Orionnnn/hub-stats huggingface dataset:
ScienceAgentBench: Toward Rigorous Assessment of Language Agents for
  Data-Driven Scientific Discovery
Authors: name: Ziru Chen, name: Shijie Chen, name: Yuting Ning, name: Qianheng Zhang, name: Boshi Wang, name: Botao Yu, name: Yifei Li, name: Zeyi Liao, name: Chen Wei, name: Zitong Lu, name: Vishal Dey, name: Mingyi Xue, name: Frazier N. Baker, name: Benjamin Burns, name: Daniel Adu-Ampratwum, name: Xuhui Huang, name: Xia Ning, name: Song Gao, name: Yu Su, name: Huan Sun
Published: 2024-10-07 14:33:50

The advancements of language language models (LLMs) have piqued growing
interest in developing LLM-based language agents to automate scientific
discovery end-to-end, which has sparked both excitement and skepticism about
the true capabilities of such agents. In this work, we argue that for an agent
to fully automate scientific discovery, it must be able to complete all
essential tasks in the workflow. Thus, we call for rigorous assessment of
agents on individual tasks in a scientific workflow before making bold claims
on end-to-end automation. To this end, we present ScienceAgentBench, a new
benchmark for evaluating language agents for data-driven scientific discovery.
To ensure the scientific authenticity and real-world relevance of our
benchmark, we extract 102 tasks from 44 peer-reviewed publications in four
disciplines and engage nine subject matter experts to validate them. We unify
the target output for every task to a self-contained Python program file and
employ an array of evaluation metrics to examine the generated programs,
execution results, and costs. Each task goes through multiple rounds of manual
validation by annotators and subject matter experts to ensure its annotation
quality and scientific plausibility. We also propose two effective strategies
to mitigate data contamination concerns. Using our benchmark, we evaluate five
open-weight and proprietary LLMs, each with three frameworks: direct prompting,
OpenHands, and self-debug. Given three attempts for each task, the
best-performing agent can only solve 32.4% of the tasks independently and 34.3%
with expert-provided knowledge. These results underscore the limited capacities
of current language agents in generating code for data-driven discovery, let
alone end-to-end automation for scientific research.
SePPO: Semi-Policy Preference Optimization for Diffusion Alignment
Authors: name: Daoan Zhang, name: Guangchen Lan, name: Dong-Jun Han, name: Wenlin Yao, name: Xiaoman Pan, name: Hongming Zhang, name: Mingxiao Li, name: Pengcheng Chen, name: Yu Dong, name: Christopher Brinton, name: Jiebo Luo
Published: 2024-10-07 17:56:53

Reinforcement learning from human feedback (RLHF) methods are emerging as a
way to fine-tune diffusion models (DMs) for visual generation. However,
commonly used on-policy strategies are limited by the generalization capability
of the reward model, while off-policy approaches require large amounts of
difficult-to-obtain paired human-annotated data, particularly in visual
generation tasks. To address the limitations of both on- and off-policy RLHF,
we propose a preference optimization method that aligns DMs with preferences
without relying on reward models or paired human-annotated data. Specifically,
we introduce a Semi-Policy Preference Optimization (SePPO) method. SePPO
leverages previous checkpoints as reference models while using them to generate
on-policy reference samples, which replace "losing images" in preference pairs.
This approach allows us to optimize using only off-policy "winning images."
Furthermore, we design a strategy for reference model selection that expands
the exploration in the policy space. Notably, we do not simply treat reference
samples as negative examples for learning. Instead, we design an anchor-based
criterion to assess whether the reference samples are likely to be winning or
losing images, allowing the model to selectively learn from the generated
reference samples. This approach mitigates performance degradation caused by
the uncertainty in reference sample quality. We validate SePPO across both
text-to-image and text-to-video benchmarks. SePPO surpasses all previous
approaches on the text-to-image benchmarks and also demonstrates outstanding
performance on the text-to-video benchmarks. Code will be released in
https://github.com/DwanZhang-AI/SePPO.
Navigating the Digital World as Humans Do: Universal Visual Grounding
  for GUI Agents
Authors: name: Boyu Gou, name: Ruohan Wang, name: Boyuan Zheng, name: Yanan Xie, name: Cheng Chang, name: Yiheng Shu, name: Huan Sun, name: Yu Su
Published: 2024-10-07 17:47:50

Multimodal large language models (MLLMs) are transforming the capabilities of
graphical user interface (GUI) agents, facilitating their transition from
controlled simulations to complex, real-world applications across various
platforms. However, the effectiveness of these agents hinges on the robustness
of their grounding capability. Current GUI agents predominantly utilize
text-based representations such as HTML or accessibility trees, which, despite
their utility, often introduce noise, incompleteness, and increased
computational overhead. In this paper, we advocate a human-like embodiment for
GUI agents that perceive the environment entirely visually and directly take
pixel-level operations on the GUI. The key is visual grounding models that can
accurately map diverse referring expressions of GUI elements to their
coordinates on the GUI across different platforms. We show that a simple
recipe, which includes web-based synthetic data and slight adaptation of the
LLaVA architecture, is surprisingly effective for training such visual
grounding models. We collect the largest dataset for GUI visual grounding so
far, containing 10M GUI elements and their referring expressions over 1.3M
screenshots, and use it to train UGround, a strong universal visual grounding
model for GUI agents. Empirical results on six benchmarks spanning three
categories (grounding, offline agent, and online agent) show that 1) UGround
substantially outperforms existing visual grounding models for GUI agents, by
up to 20% absolute, and 2) agents with UGround outperform state-of-the-art
agents, despite the fact that existing agents use additional text-based input
while ours only uses visual perception. These results provide strong support
for the feasibility and promises of GUI agents that navigate the digital world
as humans do.
What Matters for Model Merging at Scale?
Authors: name: Prateek Yadav, name: Tu Vu, name: Jonathan Lai, name: Alexandra Chronopoulou, name: Manaal Faruqui, name: Mohit Bansal, name: Tsendsuren Munkhdalai
Published: 2024-10-04 17:17:19

Model merging aims to combine multiple expert models into a more capable
single model, offering benefits such as reduced storage and serving costs,
improved generalization, and support for decentralized model development.
Despite its promise, previous studies have primarily focused on merging a few
small models. This leaves many unanswered questions about the effect of scaling
model size and how it interplays with other key factors -- like the base model
quality and number of expert models -- , to affect the merged model's
performance. This work systematically evaluates the utility of model merging at
scale, examining the impact of these different factors. We experiment with
merging fully fine-tuned models using 4 popular merging methods -- Averaging,
Task~Arithmetic, Dare, and TIES -- across model sizes ranging from 1B-64B
parameters and merging up to 8 different expert models. We evaluate the merged
models on both held-in tasks, i.e., the expert's training tasks, and zero-shot
generalization to unseen held-out tasks. Our experiments provide several new
insights about model merging at scale and the interplay between different
factors. First, we find that merging is more effective when experts are created
from strong base models, i.e., models with good zero-shot performance. Second,
larger models facilitate easier merging. Third merging consistently improves
generalization capabilities. Notably, when merging 8 large expert models, the
merged models often generalize better compared to the multitask trained models.
Fourth, we can better merge more expert models when working with larger models.
Fifth, different merging methods behave very similarly at larger scales.
Overall, our findings shed light on some interesting properties of model
merging while also highlighting some limitations. We hope that this study will
serve as a reference point on large-scale merging for upcoming research.
SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image
  Classification
Authors: name: Benjamin Feuer, name: Jiawei Xu, name: Niv Cohen, name: Patrick Yubeaton, name: Govind Mittal, name: Chinmay Hegde
Published: 2024-10-07 14:14:38

Data curation is the problem of how to collect and organize samples into a
dataset that supports efficient learning. Despite the centrality of the task,
little work has been devoted towards a large-scale, systematic comparison of
various curation methods. In this work, we take steps towards a formal
evaluation of data curation strategies and introduce SELECT, the first
large-scale benchmark of curation strategies for image classification.
  In order to generate baseline methods for the SELECT benchmark, we create a
new dataset, ImageNet++, which constitutes the largest superset of ImageNet-1K
to date. Our dataset extends ImageNet with 5 new training-data shifts, each
approximately the size of ImageNet-1K itself, and each assembled using a
distinct curation strategy. We evaluate our data curation baselines in two
ways: (i) using each training-data shift to train identical image
classification models from scratch (ii) using the data itself to fit a
pretrained self-supervised representation.
  Our findings show interesting trends, particularly pertaining to recent
methods for data curation such as synthetic data generation and lookup based on
CLIP embeddings. We show that although these strategies are highly competitive
for certain tasks, the curation strategy used to assemble the original
ImageNet-1K dataset remains the gold standard. We anticipate that our benchmark
can illuminate the path for new methods to further reduce the gap. We release
our checkpoints, code, documentation, and a link to our dataset at
https://github.com/jimmyxu123/SELECT.
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in
  Large Language Models
Authors: name: Iman Mirzadeh, name: Keivan Alizadeh, name: Hooman Shahrokhi, name: Oncel Tuzel, name: Samy Bengio, name: Mehrdad Farajtabar
Published: 2024-10-07 17:36:37

Recent advancements in Large Language Models (LLMs) have sparked interest in
their formal reasoning capabilities, particularly in mathematics. The GSM8K
benchmark is widely used to assess the mathematical reasoning of models on
grade-school-level questions. While the performance of LLMs on GSM8K has
significantly improved in recent years, it remains unclear whether their
mathematical reasoning capabilities have genuinely advanced, raising questions
about the reliability of the reported metrics. To address these concerns, we
conduct a large-scale study on several SOTA open and closed models. To overcome
the limitations of existing evaluations, we introduce GSM-Symbolic, an improved
benchmark created from symbolic templates that allow for the generation of a
diverse set of questions. GSM-Symbolic enables more controllable evaluations,
providing key insights and more reliable metrics for measuring the reasoning
capabilities of models.Our findings reveal that LLMs exhibit noticeable
variance when responding to different instantiations of the same question.
Specifically, the performance of all models declines when only the numerical
values in the question are altered in the GSM-Symbolic benchmark. Furthermore,
we investigate the fragility of mathematical reasoning in these models and show
that their performance significantly deteriorates as the number of clauses in a
question increases. We hypothesize that this decline is because current LLMs
cannot perform genuine logical reasoning; they replicate reasoning steps from
their training data. Adding a single clause that seems relevant to the question
causes significant performance drops (up to 65%) across all state-of-the-art
models, even though the clause doesn't contribute to the reasoning chain needed
for the final answer. Overall, our work offers a more nuanced understanding of
LLMs' capabilities and limitations in mathematical reasoning.
Named Clinical Entity Recognition Benchmark
Authors: name: Wadood M Abdul, name: Marco AF Pimentel, name: Muhammad Umar Salman, name: Tathagata Raha, name: Clément Christophe, name: Praveen K Kanithi, name: Nasir Hayat, name: Ronnie Rajan, name: Shadab Khan
Published: 2024-10-07 14:00:18

This technical report introduces a Named Clinical Entity Recognition
Benchmark for evaluating language models in healthcare, addressing the crucial
natural language processing (NLP) task of extracting structured information
from clinical narratives to support applications like automated coding,
clinical trial cohort identification, and clinical decision support.
  The leaderboard provides a standardized platform for assessing diverse
language models, including encoder and decoder architectures, on their ability
to identify and classify clinical entities across multiple medical domains. A
curated collection of openly available clinical datasets is utilized,
encompassing entities such as diseases, symptoms, medications, procedures, and
laboratory measurements. Importantly, these entities are standardized according
to the Observational Medical Outcomes Partnership (OMOP) Common Data Model,
ensuring consistency and interoperability across different healthcare systems
and datasets, and a comprehensive evaluation of model performance. Performance
of models is primarily assessed using the F1-score, and it is complemented by
various assessment modes to provide comprehensive insights into model
performance. The report also includes a brief analysis of models evaluated to
date, highlighting observed trends and limitations.
  By establishing this benchmarking framework, the leaderboard aims to promote
transparency, facilitate comparative analyses, and drive innovation in clinical
entity recognition tasks, addressing the need for robust evaluation methods in
healthcare NLP.
LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level
  Mathematical Reasoning
Authors: name: Di Zhang, name: Jianbo Wu, name: Jingdi Lei, name: Tong Che, name: Jiatong Li, name: Tong Xie, name: Xiaoshui Huang, name: Shufei Zhang, name: Marco Pavone, name: Yuqiang Li, name: Wanli Ouyang, name: Dongzhan Zhou
Published: 2024-10-03 18:12:29

This paper presents an advanced mathematical problem-solving framework,
LLaMA-Berry, for enhancing the mathematical reasoning ability of Large Language
Models (LLMs). The framework combines Monte Carlo Tree Search (MCTS) with
iterative Self-Refine to optimize the reasoning path and utilizes a pairwise
reward model to evaluate different paths globally. By leveraging the
self-critic and rewriting capabilities of LLMs, Self-Refine applied to MCTS
(SR-MCTS) overcomes the inefficiencies and limitations of conventional
step-wise and greedy search algorithms by fostering a more efficient
exploration of solution spaces. Pairwise Preference Reward Model~(PPRM),
inspired by Reinforcement Learning from Human Feedback (RLHF), is then used to
model pairwise preferences between solutions, utilizing an Enhanced Borda Count
(EBC) method to synthesize these preferences into a global ranking score to
find better answers. This approach addresses the challenges of scoring
variability and non-independent distributions in mathematical reasoning tasks.
The framework has been tested on general and advanced benchmarks, showing
superior performance in terms of search efficiency and problem-solving
capability compared to existing methods like ToT and rStar, particularly in
complex Olympiad-level benchmarks, including GPQA, AIME24 and AMC23.
Autonomous Character-Scene Interaction Synthesis from Text Instruction
Authors: name: Nan Jiang, name: Zimo He, name: Zi Wang, name: Hongjie Li, name: Yixin Chen, name: Siyuan Huang, name: Yixin Zhu
Published: 2024-10-04 06:58:45

Synthesizing human motions in 3D environments, particularly those with
complex activities such as locomotion, hand-reaching, and human-object
interaction, presents substantial demands for user-defined waypoints and stage
transitions. These requirements pose challenges for current models, leading to
a notable gap in automating the animation of characters from simple human
inputs. This paper addresses this challenge by introducing a comprehensive
framework for synthesizing multi-stage scene-aware interaction motions directly
from a single text instruction and goal location. Our approach employs an
auto-regressive diffusion model to synthesize the next motion segment, along
with an autonomous scheduler predicting the transition for each action stage.
To ensure that the synthesized motions are seamlessly integrated within the
environment, we propose a scene representation that considers the local
perception both at the start and the goal location. We further enhance the
coherence of the generated motion by integrating frame embeddings with language
input. Additionally, to support model training, we present a comprehensive
motion-captured dataset comprising 16 hours of motion sequences in 120 indoor
scenes covering 40 types of motions, each annotated with precise language
descriptions. Experimental results demonstrate the efficacy of our method in
generating high-quality, multi-stage motions closely aligned with environmental
and textual conditions.
LLMs Know More Than They Show: On the Intrinsic Representation of LLM
  Hallucinations
Authors: name: Hadas Orgad, name: Michael Toker, name: Zorik Gekhman, name: Roi Reichart, name: Idan Szpektor, name: Hadas Kotek, name: Yonatan Belinkov
Published: 2024-10-03 17:31:31

Large language models (LLMs) often produce errors, including factual
inaccuracies, biases, and reasoning failures, collectively referred to as
"hallucinations". Recent studies have demonstrated that LLMs' internal states
encode information regarding the truthfulness of their outputs, and that this
information can be utilized to detect errors. In this work, we show that the
internal representations of LLMs encode much more information about
truthfulness than previously recognized. We first discover that the
truthfulness information is concentrated in specific tokens, and leveraging
this property significantly enhances error detection performance. Yet, we show
that such error detectors fail to generalize across datasets, implying that --
contrary to prior claims -- truthfulness encoding is not universal but rather
multifaceted. Next, we show that internal representations can also be used for
predicting the types of errors the model is likely to make, facilitating the
development of tailored mitigation strategies. Lastly, we reveal a discrepancy
between LLMs' internal encoding and external behavior: they may encode the
correct answer, yet consistently generate an incorrect one. Taken together,
these insights deepen our understanding of LLM errors from the model's internal
perspective, which can guide future research on enhancing error analysis and
mitigation.
MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning
  in LLMs
Authors: name: Lei Wang, name: Shan Dong, name: Yuhui Xu, name: Hanze Dong, name: Yalu Wang, name: Amrita Saha, name: Ee-Peng Lim, name: Caiming Xiong, name: Doyen Sahoo
Published: 2024-10-07 02:30:07

Recent large language models (LLMs) have demonstrated versatile capabilities
in long-context scenarios. Although some recent benchmarks have been developed
to evaluate the long-context capabilities of LLMs, there is a lack of
benchmarks evaluating the mathematical reasoning abilities of LLMs over long
contexts, which is crucial for LLMs' application in real-world scenarios. In
this paper, we introduce MathHay, an automated benchmark designed to assess the
long-context mathematical reasoning capabilities of LLMs. Unlike previous
benchmarks like Needle in a Haystack, which focus primarily on information
retrieval within long texts, MathHay demands models with both
information-seeking and complex mathematical reasoning abilities. We conduct
extensive experiments on MathHay to assess the long-context mathematical
reasoning abilities of eight top-performing LLMs. Even the best-performing
model, Gemini-1.5-Pro-002, still struggles with mathematical reasoning over
long contexts, achieving only 51.26% accuracy at 128K tokens. This highlights
the significant room for improvement on the MathHay benchmark.
OmniBooth: Learning Latent Control for Image Synthesis with Multi-modal
  Instruction
Authors: name: Leheng Li, name: Weichao Qiu, name: Xu Yan, name: Jing He, name: Kaiqiang Zhou, name: Yingjie Cai, name: Qing Lian, name: Bingbing Liu, name: Ying-Cong Chen
Published: 2024-10-07 11:26:13

We present OmniBooth, an image generation framework that enables spatial
control with instance-level multi-modal customization. For all instances, the
multimodal instruction can be described through text prompts or image
references. Given a set of user-defined masks and associated text or image
guidance, our objective is to generate an image, where multiple objects are
positioned at specified coordinates and their attributes are precisely aligned
with the corresponding guidance. This approach significantly expands the scope
of text-to-image generation, and elevates it to a more versatile and practical
dimension in controllability. In this paper, our core contribution lies in the
proposed latent control signals, a high-dimensional spatial feature that
provides a unified representation to integrate the spatial, textual, and image
conditions seamlessly. The text condition extends ControlNet to provide
instance-level open-vocabulary generation. The image condition further enables
fine-grained control with personalized identity. In practice, our method
empowers users with more flexibility in controllable generation, as users can
choose multi-modal conditions from text or images as needed. Furthermore,
thorough experiments demonstrate our enhanced performance in image synthesis
fidelity and alignment across different tasks and datasets. Project page:
https://len-li.github.io/omnibooth-web/
MonST3R: A Simple Approach for Estimating Geometry in the Presence of
  Motion
Authors: name: Junyi Zhang, name: Charles Herrmann, name: Junhwa Hur, name: Varun Jampani, name: Trevor Darrell, name: Forrester Cole, name: Deqing Sun, name: Ming-Hsuan Yang
Published: 2024-10-04 18:00:07

Estimating geometry from dynamic scenes, where objects move and deform over
time, remains a core challenge in computer vision. Current approaches often
rely on multi-stage pipelines or global optimizations that decompose the
problem into subtasks, like depth and flow, leading to complex systems prone to
errors. In this paper, we present Motion DUSt3R (MonST3R), a novel
geometry-first approach that directly estimates per-timestep geometry from
dynamic scenes. Our key insight is that by simply estimating a pointmap for
each timestep, we can effectively adapt DUST3R's representation, previously
only used for static scenes, to dynamic scenes. However, this approach presents
a significant challenge: the scarcity of suitable training data, namely
dynamic, posed videos with depth labels. Despite this, we show that by posing
the problem as a fine-tuning task, identifying several suitable datasets, and
strategically training the model on this limited data, we can surprisingly
enable the model to handle dynamics, even without an explicit motion
representation. Based on this, we introduce new optimizations for several
downstream video-specific tasks and demonstrate strong performance on video
depth and camera pose estimation, outperforming prior work in terms of
robustness and efficiency. Moreover, MonST3R shows promising results for
primarily feed-forward 4D reconstruction.