Orionnnn/hub-stats huggingface dataset:
StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction
  Following
Authors: name: Jinnan Li, name: Jinzhe Li, name: Yue Wang, name: Yi Chang, name: Yuan Wu
Published: 2025-02-20 12:22:18

Multi-turn instruction following capability constitutes a core competency of
large language models (LLMs) in real-world applications. Existing evaluation
benchmarks predominantly focus on fine-grained constraint satisfaction and
domain-specific capability assessment, yet overlook the crucial structural
dependency between dialogue turns that distinguishes multi-turn from
single-turn interactions. This structural dependency not only reflects user
intent but also establishes a second dimension for instruction following
evaluation beyond constraint satisfaction. To address this gap, we propose
StructFlowBench, a multi-turn instruction following benchmark with structural
flow modeling. The benchmark innovatively defines a structural flow framework
comprising six fundamental inter-turn relationships, which not only introduces
novel structural constraints for model evaluation but also serves as generation
parameters for creating customized dialogue flows tailored to specific
scenarios. Adopting established LLM-based automatic evaluation methodologies,
we conduct systematic evaluations of 13 leading open-source and closed-source
LLMs. Experimental results reveal significant deficiencies in current models'
comprehension of multi-turn dialogue structures. The code is available at
https://github.com/MLGroupJLU/StructFlowBench.
UPCORE: Utility-Preserving Coreset Selection for Balanced Unlearning
Authors: name: Vaidehi Patil, name: Elias Stengel-Eskin, name: Mohit Bansal
Published: 2025-02-20 22:51:10

User specifications or legal frameworks often require information to be
removed from pretrained models, including large language models (LLMs). This
requires deleting or "forgetting" a set of data points from an already-trained
model, which typically degrades its performance on other data points. Thus, a
balance must be struck between removing information and keeping the model's
other abilities intact, with a failure to balance this trade-off leading to
poor deletion or an unusable model. To this end, we propose UPCORE
(Utility-Preserving Coreset Selection), a method-agnostic data selection
framework for mitigating collateral damage during unlearning. Finding that the
model damage is correlated with the variance of the model's representations on
the forget set, we selectively prune the forget set to remove outliers, thereby
minimizing model degradation after unlearning. We evaluate UPCORE across three
standard unlearning methods consistently achieving a superior balance between
the competing objectives of deletion efficacy and model preservation. To better
evaluate this trade-off, we introduce a new metric, measuring the
area-under-the-curve (AUC) across standard metrics. We find that UPCORE
improves both standard metrics and AUC, benefitting from positive transfer
between the coreset and pruned points while reducing negative transfer from the
forget set to points outside of it.
PhotoDoodle: Learning Artistic Image Editing from Few-Shot Pairwise Data
Authors: name: Shijie Huang, name: Yiren Song, name: Yuxuan Zhang, name: Hailong Guo, name: Xueyin Wang, name: Mike Zheng Shou, name: Jiaming Liu
Published: 2025-02-20 09:35:38

We introduce PhotoDoodle, a novel image editing framework designed to
facilitate photo doodling by enabling artists to overlay decorative elements
onto photographs. Photo doodling is challenging because the inserted elements
must appear seamlessly integrated with the background, requiring realistic
blending, perspective alignment, and contextual coherence. Additionally, the
background must be preserved without distortion, and the artist's unique style
must be captured efficiently from limited training data. These requirements are
not addressed by previous methods that primarily focus on global style transfer
or regional inpainting. The proposed method, PhotoDoodle, employs a two-stage
training strategy. Initially, we train a general-purpose image editing model,
OmniEditor, using large-scale data. Subsequently, we fine-tune this model with
EditLoRA using a small, artist-curated dataset of before-and-after image pairs
to capture distinct editing styles and techniques. To enhance consistency in
the generated results, we introduce a positional encoding reuse mechanism.
Additionally, we release a PhotoDoodle dataset featuring six high-quality
styles. Extensive experiments demonstrate the advanced performance and
robustness of our method in customized image editing, opening new possibilities
for artistic creation.
One-step Diffusion Models with f-Divergence Distribution Matching
Authors: name: Yilun Xu, name: Weili Nie, name: Arash Vahdat
Published: 2025-02-21 18:59:20

Sampling from diffusion models involves a slow iterative process that hinders
their practical deployment, especially for interactive applications. To
accelerate generation speed, recent approaches distill a multi-step diffusion
model into a single-step student generator via variational score distillation,
which matches the distribution of samples generated by the student to the
teacher's distribution. However, these approaches use the reverse
Kullback-Leibler (KL) divergence for distribution matching which is known to be
mode seeking. In this paper, we generalize the distribution matching approach
using a novel f-divergence minimization framework, termed f-distill, that
covers different divergences with different trade-offs in terms of mode
coverage and training variance. We derive the gradient of the f-divergence
between the teacher and student distributions and show that it is expressed as
the product of their score differences and a weighting function determined by
their density ratio. This weighting function naturally emphasizes samples with
higher density in the teacher distribution, when using a less mode-seeking
divergence. We observe that the popular variational score distillation approach
using the reverse-KL divergence is a special case within our framework.
Empirically, we demonstrate that alternative f-divergences, such as
forward-KL and Jensen-Shannon divergences, outperform the current best
variational score distillation methods across image generation tasks. In
particular, when using Jensen-Shannon divergence, f-distill achieves current
state-of-the-art one-step generation performance on ImageNet64 and zero-shot
text-to-image generation on MS-COCO. Project page:
https://research.nvidia.com/labs/genair/f-distill
SIFT: Grounding LLM Reasoning in Contexts via Stickers
Authors: name: Zihao Zeng, name: Xuyao Huang, name: Boxiu Li, name: Zhijie Deng
Published: 2025-02-19 17:38:46

This paper identifies the misinterpretation of the context can be a
significant issue during the reasoning process of large language models,
spanning from smaller models like Llama3.2-3B-Instruct to cutting-edge ones
like DeepSeek-R1. For example, in the phrase "10 dollars per kilo," LLMs might
not recognize that "per" means "for each," leading to calculation errors. We
introduce a novel, post-training approach called **Stick to the Facts (SIFT)**
to tackle this. SIFT leverages increasing inference-time compute to ground LLM
reasoning in contexts. At the core of SIFT lies the *Sticker*, which is
generated by the model itself to explicitly emphasize the key information
within the context. Given the curated Sticker, SIFT generates two predictions
-- one from the original query and one from the query augmented with the
Sticker. If they differ, the Sticker is sequentially refined via *forward*
optimization (to better align the extracted facts with the query) and *inverse*
generation (to conform with the model's inherent tendencies) for more faithful
reasoning outcomes. Studies across diverse models (from 3B to 100B+) and
benchmarks (e.g., GSM8K, MATH-500) reveal consistent performance improvements.
Notably, SIFT improves the pass@1 accuracy of DeepSeek-R1 on AIME2024 from
78.33% to **85.67**%, establishing a new state-of-the-art in the open-source
community. The code is available at https://github.com/zhijie-group/SIFT.
Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema
  Adherence
Authors: name: Bhavik Agarwal, name: Ishan Joshi, name: Viktoria Rojkova
Published: 2025-02-18 16:44:55

In this paper, we address the challenge of enforcing strict schema adherence
in large language model (LLM) generation by leveraging LLM reasoning
capabilities. Building on the DeepSeek R1 reinforcement learning framework, our
approach trains structured reasoning skills of a 1.5B parameter model through a
novel pipeline that combines synthetic reasoning dataset construction with
custom reward functions under Group Relative Policy Optimization (GRPO).
Specifically, we first perform R1 reinforcement learning on a 20K sample
unstructured-to-structured dataset, mirroring the original DeepSeek R1 methods,
to establish core reasoning abilities. Subsequently, we performed supervised
fine-tuning on a separate 10K reasoning sample dataset, focusing on refining
schema adherence for downstream tasks. Despite the relatively modest training
scope, requiring approximately 20 hours on an 8xH100 GPU cluster for GRPO
training and 3 hours on 1xA100 for SFT, our model demonstrates robust
performance in enforcing schema consistency. We compare our ThinkJSON approach
against the original DeepSeek R1 (671B), distilled versions of DeepSeek R1
(Qwen-1.5B and Qwen-7B), and Gemini 2.0 Flash (70B), showcasing its
effectiveness in real-world applications. Our results underscore the practical
utility of a resource-efficient framework for schema-constrained text
generation.
Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular
  Language Model
Authors: name: Dongki Kim, name: Wonbin Lee, name: Sung Ju Hwang
Published: 2025-02-19 05:49:10

Understanding molecules is key to understanding organisms and driving
advances in drug discovery, requiring interdisciplinary knowledge across
chemistry and biology. Although large molecular language models have achieved
notable success in interpreting molecular structures, their instruction
datasets are limited to the specific knowledge from task-oriented datasets and
do not fully cover the fundamental characteristics of molecules, hindering
their abilities as general-purpose molecular assistants. To address this issue,
we propose Mol-LLaMA, a large molecular language model that grasps the general
knowledge centered on molecules via multi-modal instruction tuning. To this
end, we design key data types that encompass the fundamental features of
molecules, incorporating essential knowledge from molecular structures. In
addition, to improve understanding of molecular features, we introduce a module
that integrates complementary information from different molecular encoders,
leveraging the distinct advantages of different molecular representations. Our
experimental results demonstrate that Mol-LLaMA is capable of comprehending the
general features of molecules and generating relevant responses to users'
queries with detailed explanations, implying its potential as a general-purpose
assistant for molecular analysis.
InterFeedback: Unveiling Interactive Intelligence of Large Multimodal
  Models via Human Feedback
Authors: name: Henry Hengyuan Zhao, name: Wenqi Pei, name: Yifei Tao, name: Haiyang Mei, name: Mike Zheng Shou
Published: 2025-02-20 20:27:06

Existing benchmarks do not test Large Multimodal Models (LMMs) on their
interactive intelligence with human users which is vital for developing
general-purpose AI assistants. We design InterFeedback, an interactive
framework, which can be applied to any LMM and dataset to assess this ability
autonomously. On top of this, we introduce InterFeedback-Bench which evaluates
interactive intelligence using two representative datasets, MMMU-Pro and
MathVerse, to test 10 different open-source LMMs. Additionally, we present
InterFeedback-Human, a newly collected dataset of 120 cases designed for
manually testing interactive performance in leading models such as OpenAI-o1
and Claude-3.5-Sonnet. Our evaluation results show that even state-of-the-art
LMM (like OpenAI-o1) can correct their results through human feedback less than
50%. Our findings point to the need for methods that can enhance the LMMs'
capability to interpret and benefit from feedback.
The Relationship Between Reasoning and Performance in Large Language
  Models -- o3 (mini) Thinks Harder, Not Longer
Authors: name: Marthe Ballon, name: Andres Algaba, name: Vincent Ginis
Published: 2025-02-21 17:59:13

Large language models have demonstrated remarkable progress in mathematical
reasoning, leveraging chain-of-thought and test-time compute scaling. However,
many open questions remain regarding the interplay between reasoning token
usage and accuracy gains. In particular, when comparing models across
generations, it is unclear whether improved performance results from longer
reasoning chains or more efficient reasoning. We systematically analyze
chain-of-thought length across o1-mini and o3-mini variants on the Omni-MATH
benchmark, finding that o3-mini (m) achieves superior accuracy without
requiring longer reasoning chains than o1-mini. Moreover, we show that accuracy
generally declines as reasoning chains grow across all models and compute
settings, even when controlling for difficulty of the questions. This accuracy
drop is significantly smaller in more proficient models, suggesting that new
generations of reasoning models use test-time compute more effectively.
Finally, we highlight that while o3-mini (h) achieves a marginal accuracy gain
over o3-mini (m), it does so by allocating substantially more reasoning tokens
across all problems, even the ones that o3-mini (m) can already solve. These
findings provide new insights into the relationship between model capability
and reasoning length, with implications for efficiency, scaling, and evaluation
methodologies.
SurveyX: Academic Survey Automation via Large Language Models
Authors: name: Xun Liang, name: Jiawei Yang, name: Yezhaohui Wang, name: Chen Tang, name: Zifan Zheng, name: Simin Niu, name: Shichao Song, name: Hanyu Wang, name: Bo Tang, name: Feiyu Xiong, name: Keming Mao, name: Zhiyu li
Published: 2025-02-20 17:59:45

Large Language Models (LLMs) have demonstrated exceptional comprehension
capabilities and a vast knowledge base, suggesting that LLMs can serve as
efficient tools for automated survey generation. However, recent research
related to automated survey generation remains constrained by some critical
limitations like finite context window, lack of in-depth content discussion,
and absence of systematic evaluation frameworks. Inspired by human writing
processes, we propose SurveyX, an efficient and organized system for automated
survey generation that decomposes the survey composing process into two phases:
the Preparation and Generation phases. By innovatively introducing online
reference retrieval, a pre-processing method called AttributeTree, and a
re-polishing process, SurveyX significantly enhances the efficacy of survey
composition. Experimental evaluation results show that SurveyX outperforms
existing automated survey generation systems in content quality (0.259
improvement) and citation quality (1.76 enhancement), approaching human expert
performance across multiple evaluation dimensions. Examples of surveys
generated by SurveyX are available on www.surveyx.cn
Generating Skyline Datasets for Data Science Models
Authors: name: Mengying Wang, name: Hanchao Ma, name: Yiyang Bian, name: Yangxin Fan, name: Yinghui Wu
Published: 2025-02-16 20:33:59

Preparing high-quality datasets required by various data-driven AI and
machine learning models has become a cornerstone task in data-driven analysis.
Conventional data discovery methods typically integrate datasets towards a
single pre-defined quality measure that may lead to bias for downstream tasks.
This paper introduces MODis, a framework that discovers datasets by optimizing
multiple user-defined, model-performance measures. Given a set of data sources
and a model, MODis selects and integrates data sources into a skyline dataset,
over which the model is expected to have the desired performance in all the
performance measures. We formulate MODis as a multi-goal finite state
transducer, and derive three feasible algorithms to generate skyline datasets.
Our first algorithm adopts a "reduce-from-universal" strategy, that starts with
a universal schema and iteratively prunes unpromising data. Our second
algorithm further reduces the cost with a bi-directional strategy that
interleaves data augmentation and reduction. We also introduce a
diversification algorithm to mitigate the bias in skyline datasets. We
experimentally verify the efficiency and effectiveness of our skyline data
discovery algorithms, and showcase their applications in optimizing data
science pipelines.
Symmetrical Visual Contrastive Optimization: Aligning Vision-Language
  Models with Minimal Contrastive Images
Authors: name: Shengguang Wu, name: Fan-Yun Sun, name: Kaiyue Wen, name: Nick Haber
Published: 2025-02-19 18:05:42

Recent studies have shown that Large Vision-Language Models (VLMs) tend to
neglect image content and over-rely on language-model priors, resulting in
errors in visually grounded tasks and hallucinations. We hypothesize that this
issue arises because existing VLMs are not explicitly trained to generate texts
that are accurately grounded in fine-grained image details. To enhance visual
feedback during VLM training, we propose S-VCO (Symmetrical Visual Contrastive
Optimization), a novel finetuning objective that steers the model toward
capturing important visual details and aligning them with corresponding text
tokens. To further facilitate this detailed alignment, we introduce MVC, a
paired image-text dataset built by automatically filtering and augmenting
visual counterfactual data to challenge the model with hard contrastive cases
involving Minimal Visual Contrasts. Experiments show that our method
consistently improves VLM performance across diverse benchmarks covering
various abilities and domains, achieving up to a 22% reduction in
hallucinations, and significant gains in vision-centric and general tasks.
Notably, these improvements become increasingly pronounced in benchmarks with
higher visual dependency. In short, S-VCO offers a significant enhancement of
VLM's visually-dependent task performance while retaining or even improving the
model's general abilities. We opensource our code at https://s-vco.github.io/
Generating π-Functional Molecules Using STGG+ with Active Learning
Authors: name: Alexia Jolicoeur-Martineau, name: Yan Zhang, name: Boris Knyazev, name: Aristide Baratin, name: Cheng-Hao Liu
Published: 2025-02-20 18:52:42

Generating novel molecules with out-of-distribution properties is a major
challenge in molecular discovery. While supervised learning methods generate
high-quality molecules similar to those in a dataset, they struggle to
generalize to out-of-distribution properties. Reinforcement learning can
explore new chemical spaces but often conducts 'reward-hacking' and generates
non-synthesizable molecules. In this work, we address this problem by
integrating a state-of-the-art supervised learning method, STGG+, in an active
learning loop. Our approach iteratively generates, evaluates, and fine-tunes
STGG+ to continuously expand its knowledge. We denote this approach STGG+AL. We
apply STGG+AL to the design of organic pi-functional materials, specifically
two challenging tasks: 1) generating highly absorptive molecules characterized
by high oscillator strength and 2) designing absorptive molecules with
reasonable oscillator strength in the near-infrared (NIR) range. The generated
molecules are validated and rationalized in-silico with time-dependent density
functional theory. Our results demonstrate that our method is highly effective
in generating novel molecules with high oscillator strength, contrary to
existing methods such as reinforcement learning (RL) methods. We open-source
our active-learning code along with our Conjugated-xTB dataset containing 2.9
million pi-conjugated molecules and the function for approximating the
oscillator strength and absorption wavelength (based on sTDA-xTB).
How to Get Your LLM to Generate Challenging Problems for Evaluation
Authors: name: Arkil Patel, name: Siva Reddy, name: Dzmitry Bahdanau
Published: 2025-02-20 16:09:55

The pace of evolution of Large Language Models (LLMs) necessitates new
approaches for rigorous and comprehensive evaluation. Traditional human
annotation is increasingly impracticable due to the complexities and costs
involved in generating high-quality, challenging problems. In this work, we
introduce CHASE, a unified framework to synthetically generate challenging
problems using LLMs without human involvement. For a given task, our approach
builds a hard problem in a bottom-up manner from simpler components. Moreover,
our framework decomposes the generation process into independently verifiable
sub-tasks, thereby ensuring a high level of quality and correctness. We
implement CHASE to create evaluation benchmarks across three diverse domains:
(1) document-based question answering, (2) repository-level code completion,
and (3) math reasoning. The performance of state-of-the-art LLMs on these
synthetic benchmarks lies in the range of 40-60% accuracy, thereby
demonstrating the effectiveness of our framework at generating challenging
problems. We publicly release our benchmarks and code.
Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision
  Language Models
Authors: name: Michihiro Yasunaga, name: Luke Zettlemoyer, name: Marjan Ghazvininejad
Published: 2025-02-20 01:48:13

Reward models play an essential role in training vision-language models
(VLMs) by assessing output quality to enable aligning with human preferences.
Despite their importance, the research community lacks comprehensive open
benchmarks for evaluating multimodal reward models in VLMs. To address this
gap, we introduce Multimodal RewardBench, an expert-annotated benchmark
covering six domains: general correctness, preference, knowledge, reasoning,
safety, and visual question-answering. Our dataset comprises 5,211 annotated
(prompt, chosen response, rejected response) triplets collected from various
VLMs. In evaluating a range of VLM judges, we find that even the top-performing
models, Gemini 1.5 Pro and Claude 3.5 Sonnet, achieve only 72% overall
accuracy. Notably, most models struggle in the reasoning and safety domains.
These findings suggest that Multimodal RewardBench offers a challenging testbed
for advancing reward model development across multiple domains. We release the
benchmark at https://github.com/facebookresearch/multimodal_rewardbench.