Orionnnn/hub-stats huggingface dataset:
LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in
  Vision-Language Models
Authors: name: Shangqing Tu, name: Yucheng Wang, name: Daniel Zhang-Li, name: Yushi Bai, name: Jifan Yu, name: Yuhao Wu, name: Lei Hou, name: Huiqin Liu, name: Zhiyuan Liu, name: Bin Xu, name: Juanzi Li
Published: 2025-02-20 18:47:36

Existing Large Vision-Language Models (LVLMs) can process inputs with context
lengths up to 128k visual and text tokens, yet they struggle to generate
coherent outputs beyond 1,000 words. We find that the primary limitation is the
absence of long output examples during supervised fine-tuning (SFT). To tackle
this issue, we introduce LongWriter-V-22k, a SFT dataset comprising 22,158
examples, each with multiple input images, an instruction, and corresponding
outputs ranging from 0 to 10,000 words. Moreover, to achieve long outputs that
maintain high-fidelity to the input images, we employ Direct Preference
Optimization (DPO) to the SFT model. Given the high cost of collecting human
feedback for lengthy outputs (e.g., 3,000 words), we propose IterDPO, which
breaks long outputs into segments and uses iterative corrections to form
preference pairs with the original outputs. Additionally, we develop
MMLongBench-Write, a benchmark featuring six tasks to evaluate the
long-generation capabilities of VLMs. Our 7B parameter model, trained with
LongWriter-V-22k and IterDPO, achieves impressive performance on this
benchmark, outperforming larger proprietary models like GPT-4o. Code and data:
https://github.com/THU-KEG/LongWriter-V
Scaling Text-Rich Image Understanding via Code-Guided Synthetic
  Multimodal Data Generation
Authors: name: Yue Yang, name: Ajay Patel, name: Matt Deitke, name: Tanmay Gupta, name: Luca Weihs, name: Andrew Head, name: Mark Yatskar, name: Chris Callison-Burch, name: Ranjay Krishna, name: Aniruddha Kembhavi, name: Christopher Clark
Published: 2025-02-20 18:55:30

Reasoning about images with rich text, such as charts and documents, is a
critical application of vision-language models (VLMs). However, VLMs often
struggle in these domains due to the scarcity of diverse text-rich
vision-language data. To address this challenge, we present CoSyn, a framework
that leverages the coding capabilities of text-only large language models
(LLMs) to automatically create synthetic text-rich multimodal data. Given input
text describing a target domain (e.g., "nutrition fact labels"), CoSyn prompts
an LLM to generate code (Python, HTML, LaTeX, etc.) for rendering synthetic
images. With the underlying code as textual representations of the synthetic
images, CoSyn can generate high-quality instruction-tuning data, again relying
on a text-only LLM. Using CoSyn, we constructed a dataset comprising 400K
images and 2.7M rows of vision-language instruction-tuning data. Comprehensive
experiments on seven benchmarks demonstrate that models trained on our
synthetic data achieve state-of-the-art performance among competitive
open-source models, including Llama 3.2, and surpass proprietary models such as
GPT-4V and Gemini 1.5 Flash. Furthermore, CoSyn can produce synthetic pointing
data, enabling VLMs to ground information within input images, showcasing its
potential for developing multimodal agents capable of acting in real-world
environments.
SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic
  Understanding, Localization, and Dense Features
Authors: name: Michael Tschannen, name: Alexey Gritsenko, name: Xiao Wang, name: Muhammad Ferjad Naeem, name: Ibrahim Alabdulmohsin, name: Nikhil Parthasarathy, name: Talfan Evans, name: Lucas Beyer, name: Ye Xia, name: Basil Mustafa, name: Olivier Hénaff, name: Jeremiah Harmsen, name: Andreas Steiner, name: Xiaohua Zhai
Published: 2025-02-20 18:08:29

We introduce SigLIP 2, a family of new multilingual vision-language encoders
that build on the success of the original SigLIP. In this second iteration, we
extend the original image-text training objective with several prior,
independently developed techniques into a unified recipe -- this includes
captioning-based pretraining, self-supervised losses (self-distillation, masked
prediction) and online data curation. With these changes, SigLIP 2 models
outperform their SigLIP counterparts at all model scales in core capabilities,
including zero-shot classification, image-text retrieval, and transfer
performance when extracting visual representations for Vision-Language Models
(VLMs). Furthermore, the new training recipe leads to significant improvements
on localization and dense prediction tasks. We also train variants which
support multiple resolutions and preserve the input's native aspect ratio.
Finally, we train on a more diverse data-mixture that includes de-biasing
techniques, leading to much better multilingual understanding and improved
fairness. To allow users to trade off inference cost with performance, we
release model checkpoints at four sizes: ViT-B (86M), L (303M), So400m (400M),
and g (1B).
RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformers
Authors: name: Ke Cao, name: Jing Wang, name: Ao Ma, name: Jiasong Feng, name: Zhanjie Zhang, name: Xuanhua He, name: Shanyuan Liu, name: Bo Cheng, name: Dawei Leng, name: Yuhui Yin, name: Jie Zhang
Published: 2025-02-20 09:10:05

The Diffusion Transformer plays a pivotal role in advancing text-to-image and
text-to-video generation, owing primarily to its inherent scalability. However,
existing controlled diffusion transformer methods incur significant parameter
and computational overheads and suffer from inefficient resource allocation due
to their failure to account for the varying relevance of control information
across different transformer layers. To address this, we propose the
Relevance-Guided Efficient Controllable Generation framework, RelaCtrl,
enabling efficient and resource-optimized integration of control signals into
the Diffusion Transformer. First, we evaluate the relevance of each layer in
the Diffusion Transformer to the control information by assessing the
"ControlNet Relevance Score"-i.e., the impact of skipping each control layer on
both the quality of generation and the control effectiveness during inference.
Based on the strength of the relevance, we then tailor the positioning,
parameter scale, and modeling capacity of the control layers to reduce
unnecessary parameters and redundant computations. Additionally, to further
improve efficiency, we replace the self-attention and FFN in the commonly used
copy block with the carefully designed Two-Dimensional Shuffle Mixer (TDSM),
enabling efficient implementation of both the token mixer and channel mixer.
Both qualitative and quantitative experimental results demonstrate that our
approach achieves superior performance with only 15% of the parameters and
computational complexity compared to PixArt-delta. More examples are available
at https://relactrl.github.io/RelaCtrl/.
Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement
  Learning
Authors: name: Tian Xie, name: Zitian Gao, name: Qingnan Ren, name: Haoming Luo, name: Yuqian Hong, name: Bryan Dai, name: Joey Zhou, name: Kai Qiu, name: Zhirong Wu, name: Chong Luo
Published: 2025-02-20 17:49:26

Inspired by the success of DeepSeek-R1, we explore the potential of
rule-based reinforcement learning (RL) in large reasoning models. To analyze
reasoning dynamics, we use synthetic logic puzzles as training data due to
their controllable complexity and straightforward answer verification. We make
some key technical contributions that lead to effective and stable RL training:
a system prompt that emphasizes the thinking and answering process, a stringent
format reward function that penalizes outputs for taking shortcuts, and a
straightforward training recipe that achieves stable convergence. Our 7B model
develops advanced reasoning skills-such as reflection, verification, and
summarization-that are absent from the logic corpus. Remarkably, after training
on just 5K logic problems, it demonstrates generalization abilities to the
challenging math benchmarks AIME and AMC.
AlphaMaze: Enhancing Large Language Models' Spatial Intelligence via
  GRPO
Authors: name: Alan Dao, name: Dinh Bach Vu
Published: 2025-02-20 16:05:18

Large Language Models (LLMs) have demonstrated impressive capabilities in
language processing, yet they often struggle with tasks requiring genuine
visual spatial reasoning. In this paper, we introduce a novel two-stage
training framework designed to equip standard LLMs with visual reasoning
abilities for maze navigation. First, we leverage Supervised Fine Tuning (SFT)
on a curated dataset of tokenized maze representations to teach the model to
predict step-by-step movement commands. Next, we apply Group Relative Policy
Optimization (GRPO)-a technique used in DeepSeekR1-with a carefully crafted
reward function to refine the model's sequential decision-making and encourage
emergent chain-of-thought behaviors. Experimental results on synthetically
generated mazes show that while a baseline model fails to navigate the maze,
the SFT-trained model achieves 86% accuracy, and further GRPO fine-tuning
boosts accuracy to 93%. Qualitative analyses reveal that GRPO fosters more
robust and self-corrective reasoning, highlighting the potential of our
approach to bridge the gap between language models and visual spatial tasks.
These findings offer promising implications for applications in robotics,
autonomous navigation, and other domains that require integrated visual and
sequential reasoning.
MLGym: A New Framework and Benchmark for Advancing AI Research Agents
Authors: name: Deepak Nathani, name: Lovish Madaan, name: Nicholas Roberts, name: Nikolay Bashlykov, name: Ajay Menon, name: Vincent Moens, name: Amar Budhiraja, name: Despoina Magka, name: Vladislav Vorotilov, name: Gaurav Chaurasia, name: Dieuwke Hupkes, name: Ricardo Silveira Cabral, name: Tatiana Shavrina, name: Jakob Foerster, name: Yoram Bachrach, name: William Yang Wang, name: Roberta Raileanu
Published: 2025-02-20 12:28:23

We introduce Meta MLGym and MLGym-Bench, a new framework and benchmark for
evaluating and developing LLM agents on AI research tasks. This is the first
Gym environment for machine learning (ML) tasks, enabling research on
reinforcement learning (RL) algorithms for training such agents. MLGym-bench
consists of 13 diverse and open-ended AI research tasks from diverse domains
such as computer vision, natural language processing, reinforcement learning,
and game theory. Solving these tasks requires real-world AI research skills
such as generating new ideas and hypotheses, creating and processing data,
implementing ML methods, training models, running experiments, analyzing the
results, and iterating through this process to improve on a given task. We
evaluate a number of frontier large language models (LLMs) on our benchmarks
such as Claude-3.5-Sonnet, Llama-3.1 405B, GPT-4o, o1-preview, and Gemini-1.5
Pro. Our MLGym framework makes it easy to add new tasks, integrate and evaluate
models or agents, generate synthetic data at scale, as well as develop new
learning algorithms for training agents on AI research tasks. We find that
current frontier models can improve on the given baselines, usually by finding
better hyperparameters, but do not generate novel hypotheses, algorithms,
architectures, or substantial improvements. We open-source our framework and
benchmark to facilitate future research in advancing the AI research
capabilities of LLM agents.
S*: Test Time Scaling for Code Generation
Authors: name: Dacheng Li, name: Shiyi Cao, name: Chengkun Cao, name: Xiuyu Li, name: Shangyin Tan, name: Kurt Keutzer, name: Jiarong Xing, name: Joseph E. Gonzalez, name: Ion Stoica
Published: 2025-02-20 09:18:53

Increasing test-time compute for LLMs shows promise across domains but
remains underexplored in code generation, despite extensive study in math. In
this paper, we propose S*, the first hybrid test-time scaling framework that
substantially improves the coverage and selection accuracy of generated code.
S* extends the existing parallel scaling paradigm with sequential scaling to
push performance boundaries. It further leverages a novel selection mechanism
that adaptively generates distinguishing inputs for pairwise comparison,
combined with execution-grounded information to robustly identify correct
solutions. We evaluate across 12 Large Language Models and Large Reasoning
Model and show: (1) S* consistently improves performance across model families
and sizes, enabling a 3B model to outperform GPT-4o-mini; (2) S* enables
non-reasoning models to surpass reasoning models - GPT-4o-mini with S*
outperforms o1-preview by 3.7% on LiveCodeBench; (3) S* further boosts
state-of-the-art reasoning models - DeepSeek-R1-Distill-Qwen-32B with S*
achieves 85.7% on LiveCodeBench, approaching o1 (high) at 88.5%. Code will be
available under https://github.com/NovaSky-AI/SkyThought.
Which of These Best Describes Multiple Choice Evaluation with LLMs? A)
  Forced B) Flawed C) Fixable D) All of the Above
Authors: name: Nishant Balepur, name: Rachel Rudinger, name: Jordan Lee Boyd-Graber
Published: 2025-02-19 22:11:52

Multiple choice question answering (MCQA) is popular for LLM evaluation due
to its simplicity and human-like testing, but we argue for its reform. We first
reveal flaws in MCQA's format, as it struggles to: 1) test
generation/subjectivity; 2) match LLM use cases; and 3) fully test knowledge.
We instead advocate for generative formats based on human testing-where LLMs
construct and explain answers-better capturing user needs and knowledge while
remaining easy to score. We then show even when MCQA is a useful format, its
datasets suffer from: leakage; unanswerability; shortcuts; and saturation. In
each issue, we give fixes from education, like rubrics to guide MCQ writing;
scoring methods to bridle guessing; and Item Response Theory to build harder
MCQs. Lastly, we discuss LLM errors in MCQA-robustness, biases, and unfaithful
explanations-showing how our prior solutions better measure or address these
issues. While we do not need to desert MCQA, we encourage more efforts in
refining the task based on educational testing, advancing evaluations.
REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation
Authors: name: Dong-Ho Lee, name: Adyasha Maharana, name: Jay Pujara, name: Xiang Ren, name: Francesco Barbieri
Published: 2025-02-18 20:29:01

Long-term, open-domain dialogue capabilities are essential for chatbots
aiming to recall past interactions and demonstrate emotional intelligence (EI).
Yet, most existing research relies on synthetic, LLM-generated data, leaving
open questions about real-world conversational patterns. To address this gap,
we introduce REALTALK, a 21-day corpus of authentic messaging app dialogues,
providing a direct benchmark against genuine human interactions.
  We first conduct a dataset analysis, focusing on EI attributes and persona
consistency to understand the unique challenges posed by real-world dialogues.
By comparing with LLM-generated conversations, we highlight key differences,
including diverse emotional expressions and variations in persona stability
that synthetic dialogues often fail to capture.
  Building on these insights, we introduce two benchmark tasks: (1) persona
simulation where a model continues a conversation on behalf of a specific user
given prior dialogue context; and (2) memory probing where a model answers
targeted questions requiring long-term memory of past interactions.
  Our findings reveal that models struggle to simulate a user solely from
dialogue history, while fine-tuning on specific user chats improves persona
emulation. Additionally, existing models face significant challenges in
recalling and leveraging long-term context within real-world conversations.
From Tools to Teammates: Evaluating LLMs in Multi-Session Coding
  Interactions
Authors: name: Nathanaël Carraz Rakotonirina, name: Mohammed Hamdy, name: Jon Ander Campos, name: Lucas Weber, name: Alberto Testoni, name: Marzieh Fadaee, name: Sandro Pezzelle, name: Marco Del Tredici
Published: 2025-02-19 14:58:04

Large Language Models (LLMs) are increasingly used in working environments
for a wide range of tasks, excelling at solving individual problems in
isolation. However, are they also able to effectively collaborate over
long-term interactions? To investigate this, we introduce MemoryCode, a
synthetic multi-session dataset designed to test LLMs' ability to track and
execute simple coding instructions amid irrelevant information, simulating a
realistic setting. While all the models we tested handle isolated instructions
well, even the performance of state-of-the-art models like GPT-4o deteriorates
when instructions are spread across sessions. Our analysis suggests this is due
to their failure to retrieve and integrate information over long instruction
chains. Our results highlight a fundamental limitation of current LLMs,
restricting their ability to collaborate effectively in long interactions.
Judging the Judges: A Collection of LLM-Generated Relevance Judgements
Authors: name: Hossein A. Rahmani, name: Clemencia Siro, name: Mohammad Aliannejadi, name: Nick Craswell, name: Charles L. A. Clarke, name: Guglielmo Faggioli, name: Bhaskar Mitra, name: Paul Thomas, name: Emine Yilmaz
Published: 2025-02-19 17:40:32

Using Large Language Models (LLMs) for relevance assessments offers promising
opportunities to improve Information Retrieval (IR), Natural Language
Processing (NLP), and related fields. Indeed, LLMs hold the promise of allowing
IR experimenters to build evaluation collections with a fraction of the manual
human labor currently required. This could help with fresh topics on which
there is still limited knowledge and could mitigate the challenges of
evaluating ranking systems in low-resource scenarios, where it is challenging
to find human annotators. Given the fast-paced recent developments in the
domain, many questions concerning LLMs as assessors are yet to be answered.
Among the aspects that require further investigation, we can list the impact of
various components in a relevance judgment generation pipeline, such as the
prompt used or the LLM chosen.
  This paper benchmarks and reports on the results of a large-scale automatic
relevance judgment evaluation, the LLMJudge challenge at SIGIR 2024, where
different relevance assessment approaches were proposed. In detail, we release
and benchmark 42 LLM-generated labels of the TREC 2023 Deep Learning track
relevance judgments produced by eight international teams who participated in
the challenge. Given their diverse nature, these automatically generated
relevance judgments can help the community not only investigate systematic
biases caused by LLMs but also explore the effectiveness of ensemble models,
analyze the trade-offs between different models and human assessors, and
advance methodologies for improving automated evaluation techniques. The
released resource is available at the following link:
https://llm4eval.github.io/LLMJudge-benchmark/
AIDE: AI-Driven Exploration in the Space of Code
Authors: name: Zhengyao Jiang, name: Dominik Schmidt, name: Dhruv Srikanth, name: Dixing Xu, name: Ian Kaplan, name: Deniss Jacenko, name: Yuxiang Wu
Published: 2025-02-18 18:57:21

Machine learning, the foundation of modern artificial intelligence, has
driven innovations that have fundamentally transformed the world. Yet, behind
advancements lies a complex and often tedious process requiring labor and
compute intensive iteration and experimentation. Engineers and scientists
developing machine learning models spend much of their time on trial-and-error
tasks instead of conceptualizing innovative solutions or research hypotheses.
To address this challenge, we introduce AI-Driven Exploration (AIDE), a machine
learning engineering agent powered by large language models (LLMs). AIDE frames
machine learning engineering as a code optimization problem, and formulates
trial-and-error as a tree search in the space of potential solutions. By
strategically reusing and refining promising solutions, AIDE effectively trades
computational resources for enhanced performance, achieving state-of-the-art
results on multiple machine learning engineering benchmarks, including our
Kaggle evaluations, OpenAI MLE-Bench and METRs RE-Bench.
MVL-SIB: A Massively Multilingual Vision-Language Benchmark for
  Cross-Modal Topical Matching
Authors: name: Fabian David Schmidt, name: Florian Schneider, name: Chris Biemann, name: Goran Glavaš
Published: 2025-02-18 13:40:05

Existing multilingual vision-language (VL) benchmarks often only cover a
handful of languages. Consequently, evaluations of large vision-language models
(LVLMs) predominantly target high-resource languages, underscoring the need for
evaluation data for low-resource languages. To address this limitation, we
introduce MVL-SIB, a massively multilingual vision-language benchmark that
evaluates both cross-modal and text-only topical matching across 205 languages
-- over 100 more than the most multilingual existing VL benchmarks encompass.
We then benchmark a range of of open-weight LVLMs together with GPT-4o(-mini)
on MVL-SIB. Our results reveal that LVLMs struggle in cross-modal topic
matching in lower-resource languages, performing no better than chance on
languages like N'Koo. Our analysis further reveals that VL support in LVLMs
declines disproportionately relative to textual support for lower-resource
languages, as evidenced by comparison of cross-modal and text-only topical
matching performance. We further observe that open-weight LVLMs do not benefit
from representing a topic with more than one image, suggesting that these
models are not yet fully effective at handling multi-image tasks. By
correlating performance on MVL-SIB with other multilingual VL benchmarks, we
highlight that MVL-SIB serves as a comprehensive probe of multilingual VL
understanding in LVLMs.
Reducing Hallucinations in Language Model-based SPARQL Query Generation
  Using Post-Generation Memory Retrieval
Authors: name: Aditya Sharma, name: Luis Lara, name: Amal Zouaq, name: Christopher J. Pal
Published: 2025-02-19 02:08:13

The ability to generate SPARQL queries from natural language questions is
crucial for ensuring efficient and accurate retrieval of structured data from
knowledge graphs (KG). While large language models (LLMs) have been widely
adopted for SPARQL query generation, they are often susceptible to
hallucinations and out-of-distribution errors when producing KG elements like
Uniform Resource Identifiers (URIs) based on internal parametric knowledge.
This often results in content that appears plausible but is factually
incorrect, posing significant challenges for their use in real-world
information retrieval (IR) applications. This has led to increased research
aimed at detecting and mitigating such errors. In this paper, we introduce PGMR
(Post-Generation Memory Retrieval), a modular framework that incorporates a
non-parametric memory module to retrieve KG elements and enhance LLM-based
SPARQL query generation. Our experimental results indicate that PGMR
consistently delivers strong performance across diverse datasets, data
distributions, and LLMs. Notably, PGMR significantly mitigates URI
hallucinations, nearly eliminating the problem in several scenarios.