Orionnnn/hub-stats huggingface dataset:
Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with
  100+ NLP Researchers
Authors: name: Chenglei Si, name: Diyi Yang, name: Tatsunori Hashimoto
Published: 2024-09-06 08:25:03

Recent advancements in large language models (LLMs) have sparked optimism
about their potential to accelerate scientific discovery, with a growing number
of works proposing research agents that autonomously generate and validate new
ideas. Despite this, no evaluations have shown that LLM systems can take the
very first step of producing novel, expert-level ideas, let alone perform the
entire research process. We address this by establishing an experimental design
that evaluates research idea generation while controlling for confounders and
performs the first head-to-head comparison between expert NLP researchers and
an LLM ideation agent. By recruiting over 100 NLP researchers to write novel
ideas and blind reviews of both LLM and human ideas, we obtain the first
statistically significant conclusion on current LLM capabilities for research
ideation: we find LLM-generated ideas are judged as more novel (p < 0.05) than
human expert ideas while being judged slightly weaker on feasibility. Studying
our agent baselines closely, we identify open problems in building and
evaluating research agents, including failures of LLM self-evaluation and their
lack of diversity in generation. Finally, we acknowledge that human judgements
of novelty can be difficult, even by experts, and propose an end-to-end study
design which recruits researchers to execute these ideas into full projects,
enabling us to study whether these novelty and feasibility judgements result in
meaningful differences in research outcome.
DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with
  Diffusion Priors
Authors: name: Thomas Hanwen Zhu, name: Ruining Li, name: Tomas Jakab
Published: 2024-09-12 17:59:49

We present DreamHOI, a novel method for zero-shot synthesis of human-object
interactions (HOIs), enabling a 3D human model to realistically interact with
any given object based on a textual description. This task is complicated by
the varying categories and geometries of real-world objects and the scarcity of
datasets encompassing diverse HOIs. To circumvent the need for extensive data,
we leverage text-to-image diffusion models trained on billions of image-caption
pairs. We optimize the articulation of a skinned human mesh using Score
Distillation Sampling (SDS) gradients obtained from these models, which predict
image-space edits. However, directly backpropagating image-space gradients into
complex articulation parameters is ineffective due to the local nature of such
gradients. To overcome this, we introduce a dual implicit-explicit
representation of a skinned mesh, combining (implicit) neural radiance fields
(NeRFs) with (explicit) skeleton-driven mesh articulation. During optimization,
we transition between implicit and explicit forms, grounding the NeRF
generation while refining the mesh articulation. We validate our approach
through extensive experiments, demonstrating its effectiveness in generating
realistic HOIs.
Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
Authors: name: Rogerio Bonatti, name: Dan Zhao, name: Francesco Bonacci, name: Dillon Dupont, name: Sara Abdali, name: Yinheng Li, name: Justin Wagle, name: Kazuhito Koishida, name: Arthur Bucker, name: Lawrence Jang, name: Zack Hui
Published: 2024-09-12 17:56:43

Large language models (LLMs) show remarkable potential to act as computer
agents, enhancing human productivity and software accessibility in multi-modal
tasks that require planning and reasoning. However, measuring agent performance
in realistic environments remains a challenge since: (i) most benchmarks are
limited to specific modalities or domains (e.g. text-only, web navigation, Q&A,
coding) and (ii) full benchmark evaluations are slow (on order of magnitude of
days) given the multi-step sequential nature of tasks. To address these
challenges, we introduce the Windows Agent Arena: a reproducible, general
environment focusing exclusively on the Windows operating system (OS) where
agents can operate freely within a real Windows OS and use the same wide range
of applications, tools, and web browsers available to human users when solving
tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse
Windows tasks across representative domains that require agent abilities in
planning, screen understanding, and tool usage. Our benchmark is scalable and
can be seamlessly parallelized in Azure for a full benchmark evaluation in as
little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we
also introduce a new multi-modal agent, Navi. Our agent achieves a success rate
of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted
human. Navi also demonstrates strong performance on another popular web-based
benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis
of Navi's performance, and provide insights into the opportunities for future
research in agent development and data generation using Windows Agent Arena.
  Webpage: https://microsoft.github.io/WindowsAgentArena
  Code: https://github.com/microsoft/WindowsAgentArena
Source2Synth: Synthetic Data Generation and Curation Grounded in Real
  Data Sources
Authors: name: Alisia Lupidi, name: Carlos Gemmell, name: Nicola Cancedda, name: Jane Dwivedi-Yu, name: Jason Weston, name: Jakob Foerster, name: Roberta Raileanu, name: Maria Lomeli
Published: 2024-09-12 17:39:08

Large Language Models still struggle in challenging scenarios that leverage
structured data, complex reasoning, or tool usage. In this paper, we propose
Source2Synth: a new method that can be used for teaching LLMs new skills
without relying on costly human annotations. Source2Synth takes as input a
custom data source and produces synthetic data points with intermediate
reasoning steps grounded in real-world sources. Source2Synth improves the
dataset quality by discarding low-quality generations based on their
answerability. We demonstrate the generality of this approach by applying it to
two challenging domains: we test reasoning abilities in multi-hop question
answering (MHQA), and tool usage in tabular question answering (TQA). Our
method improves performance by 25.51% for TQA on WikiSQL and 22.57% for MHQA on
HotPotQA compared to the fine-tuned baselines.
FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimally
Authors: name: Qiuhong Shen, name: Xingyi Yang, name: Xinchao Wang
Published: 2024-09-12 17:58:13

This study addresses the challenge of accurately segmenting 3D Gaussian
Splatting from 2D masks. Conventional methods often rely on iterative gradient
descent to assign each Gaussian a unique label, leading to lengthy optimization
and sub-optimal solutions. Instead, we propose a straightforward yet globally
optimal solver for 3D-GS segmentation. The core insight of our method is that,
with a reconstructed 3D-GS scene, the rendering of the 2D masks is essentially
a linear function with respect to the labels of each Gaussian. As such, the
optimal label assignment can be solved via linear programming in closed form.
This solution capitalizes on the alpha blending characteristic of the splatting
process for single step optimization. By incorporating the background bias in
our objective function, our method shows superior robustness in 3D segmentation
against noises. Remarkably, our optimization completes within 30 seconds, about
50times faster than the best existing methods. Extensive experiments
demonstrate the efficiency and robustness of our method in segmenting various
scenes, and its superior performance in downstream tasks such as object removal
and inpainting. Demos and code will be available at
https://github.com/florinshen/FlashSplat.
TextBoost: Towards One-Shot Personalization of Text-to-Image Models via
  Fine-tuning Text Encoder
Authors: name: NaHyeon Park, name: Kunhee Kim, name: Hyunjung Shim
Published: 2024-09-12 17:47:51

Recent breakthroughs in text-to-image models have opened up promising
research avenues in personalized image generation, enabling users to create
diverse images of a specific subject using natural language prompts. However,
existing methods often suffer from performance degradation when given only a
single reference image. They tend to overfit the input, producing highly
similar outputs regardless of the text prompt. This paper addresses the
challenge of one-shot personalization by mitigating overfitting, enabling the
creation of controllable images through text prompts. Specifically, we propose
a selective fine-tuning strategy that focuses on the text encoder. Furthermore,
we introduce three key techniques to enhance personalization performance: (1)
augmentation tokens to encourage feature disentanglement and alleviate
overfitting, (2) a knowledge-preservation loss to reduce language drift and
promote generalizability across diverse prompts, and (3) SNR-weighted sampling
for efficient training. Extensive experiments demonstrate that our approach
efficiently generates high-quality, diverse images using only a single
reference image while significantly reducing memory and storage requirements.
SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research
  Repositories
Authors: name: Ben Bogin, name: Kejuan Yang, name: Shashank Gupta, name: Kyle Richardson, name: Erin Bransom, name: Peter Clark, name: Ashish Sabharwal, name: Tushar Khot
Published: 2024-09-11 17:37:48

Given that Large Language Models (LLMs) have made significant progress in
writing code, can they now be used to autonomously reproduce results from
research repositories? Such a capability would be a boon to the research
community, helping researchers validate, understand, and extend prior work. To
advance towards this goal, we introduce SUPER, the first benchmark designed to
evaluate the capability of LLMs in setting up and executing tasks from research
repositories. SUPERaims to capture the realistic challenges faced by
researchers working with Machine Learning (ML) and Natural Language Processing
(NLP) research repositories. Our benchmark comprises three distinct problem
sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems
derived from the expert set that focus on specific challenges (e.g.,
configuring a trainer), and 602 automatically generated problems for
larger-scale development. We introduce various evaluation measures to assess
both task success and progress, utilizing gold solutions when available or
approximations otherwise. We show that state-of-the-art approaches struggle to
solve these problems with the best model (GPT-4o) solving only 16.3% of the
end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of
this task, and suggests that SUPER can serve as a valuable resource for the
community to make and measure progress.
Can Large Language Models Unlock Novel Scientific Research Ideas?
Authors: name: Sandeep Kumar, name: Tirthankar Ghosal, name: Vinayak Goyal, name: Asif Ekbal
Published: 2024-09-10 03:26:42

"An idea is nothing more nor less than a new combination of old elements"
(Young, J.W.). The widespread adoption of Large Language Models (LLMs) and
publicly available ChatGPT have marked a significant turning point in the
integration of Artificial Intelligence (AI) into people's everyday lives. This
study explores the capability of LLMs in generating novel research ideas based
on information from research papers. We conduct a thorough examination of 4
LLMs in five domains (e.g., Chemistry, Computer, Economics, Medical, and
Physics). We found that the future research ideas generated by Claude-2 and
GPT-4 are more aligned with the author's perspective than GPT-3.5 and Gemini.
We also found that Claude-2 generates more diverse future research ideas than
GPT-4, GPT-3.5, and Gemini 1.0. We further performed a human evaluation of the
novelty, relevancy, and feasibility of the generated future research ideas.
This investigation offers insights into the evolving role of LLMs in idea
generation, highlighting both its capability and limitations. Our work
contributes to the ongoing efforts in evaluating and utilizing language models
for generating future research ideas. We make our datasets and codes publicly
available.
PingPong: A Benchmark for Role-Playing Language Models with User
  Emulation and Multi-Model Evaluation
Authors: name: Ilya Gusev
Published: 2024-09-10 19:00:44

We introduce a novel benchmark for evaluating the role-playing capabilities
of language models. Our approach leverages language models themselves to
emulate users in dynamic, multi-turn conversations and to assess the resulting
dialogues. The framework consists of three main components: a player model
assuming a specific character role, an interrogator model simulating user
behavior, and a judge model evaluating conversation quality. We conducted
experiments comparing automated evaluations with human annotations to validate
our approach, demonstrating strong correlations across multiple criteria. This
work provides a foundation for a robust and dynamic evaluation of model
capabilities in interactive scenarios.
ProteinBench: A Holistic Evaluation of Protein Foundation Models
Authors: name: Fei Ye, name: Zaixiang Zheng, name: Dongyu Xue, name: Yuning Shen, name: Lihao Wang, name: Yiming Ma, name: Yan Wang, name: Xinyou Wang, name: Xiangxin Zhou, name: Quanquan Gu
Published: 2024-09-10 06:52:33

Recent years have witnessed a surge in the development of protein foundation
models, significantly improving performance in protein prediction and
generative tasks ranging from 3D structure prediction and protein design to
conformational dynamics. However, the capabilities and limitations associated
with these models remain poorly understood due to the absence of a unified
evaluation framework. To fill this gap, we introduce ProteinBench, a holistic
evaluation framework designed to enhance the transparency of protein foundation
models. Our approach consists of three key components: (i) A taxonomic
classification of tasks that broadly encompass the main challenges in the
protein domain, based on the relationships between different protein
modalities; (ii) A multi-metric evaluation approach that assesses performance
across four key dimensions: quality, novelty, diversity, and robustness; and
(iii) In-depth analyses from various user objectives, providing a holistic view
of model performance. Our comprehensive evaluation of protein foundation models
reveals several key findings that shed light on their current capabilities and
limitations. To promote transparency and facilitate further research, we
release the evaluation dataset, code, and a public leaderboard publicly for
further analysis and a general modular toolkit. We intend for ProteinBench to
be a living benchmark for establishing a standardized, in-depth evaluation
framework for protein foundation models, driving their development and
application while fostering collaboration within the field.
MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical
  Applications
Authors: name: Praveen K Kanithi, name: Clément Christophe, name: Marco AF Pimentel, name: Tathagata Raha, name: Nada Saadi, name: Hamza Javed, name: Svetlana Maslenkova, name: Nasir Hayat, name: Ronnie Rajan, name: Shadab Khan
Published: 2024-09-11 14:44:51

The rapid development of Large Language Models (LLMs) for healthcare
applications has spurred calls for holistic evaluation beyond frequently-cited
benchmarks like USMLE, to better reflect real-world performance. While
real-world assessments are valuable indicators of utility, they often lag
behind the pace of LLM evolution, likely rendering findings obsolete upon
deployment. This temporal disconnect necessitates a comprehensive upfront
evaluation that can guide model selection for specific clinical applications.
We introduce MEDIC, a framework assessing LLMs across five critical dimensions
of clinical competence: medical reasoning, ethics and bias, data and language
understanding, in-context learning, and clinical safety. MEDIC features a novel
cross-examination framework quantifying LLM performance across areas like
coverage and hallucination detection, without requiring reference outputs. We
apply MEDIC to evaluate LLMs on medical question-answering, safety,
summarization, note generation, and other tasks. Our results show performance
disparities across model sizes, baseline vs medically finetuned models, and
have implications on model selection for applications requiring specific model
strengths, such as low hallucination or lower cost of inference. MEDIC's
multifaceted evaluation reveals these performance trade-offs, bridging the gap
between theoretical capabilities and practical implementation in healthcare
settings, ensuring that the most promising models are identified and adapted
for diverse healthcare applications.
VMAS: Video-to-Music Generation via Semantic Alignment in Web Music
  Videos
Authors: name: Yan-Bo Lin, name: Yu Tian, name: Linjie Yang, name: Gedas Bertasius, name: Heng Wang
Published: 2024-09-11 17:56:48

We present a framework for learning to generate background music from video
inputs. Unlike existing works that rely on symbolic musical annotations, which
are limited in quantity and diversity, our method leverages large-scale web
videos accompanied by background music. This enables our model to learn to
generate realistic and diverse music. To accomplish this goal, we develop a
generative video-music Transformer with a novel semantic video-music alignment
scheme. Our model uses a joint autoregressive and contrastive learning
objective, which encourages the generation of music aligned with high-level
video content. We also introduce a novel video-beat alignment scheme to match
the generated music beats with the low-level motions in the video. Lastly, to
capture fine-grained visual cues in a video needed for realistic background
music generation, we introduce a new temporal video encoder architecture,
allowing us to efficiently process videos consisting of many densely sampled
frames. We train our framework on our newly curated DISCO-MV dataset,
consisting of 2.2M video-music samples, which is orders of magnitude larger
than any prior datasets used for video music generation. Our method outperforms
existing approaches on the DISCO-MV and MusicCaps datasets according to various
music generation evaluation metrics, including human evaluation. Results are
available at https://genjib.github.io/project_page/VMAs/index.html
Generative Hierarchical Materials Search
Authors: name: Sherry Yang, name: Simon Batzner, name: Ruiqi Gao, name: Muratahan Aykol, name: Alexander L. Gaunt, name: Brendan McMorrow, name: Danilo J. Rezende, name: Dale Schuurmans, name: Igor Mordatch, name: Ekin D. Cubuk
Published: 2024-09-10 17:51:28

Generative models trained at scale can now produce text, video, and more
recently, scientific data such as crystal structures. In applications of
generative approaches to materials science, and in particular to crystal
structures, the guidance from the domain expert in the form of high-level
instructions can be essential for an automated system to output candidate
crystals that are viable for downstream research. In this work, we formulate
end-to-end language-to-structure generation as a multi-objective optimization
problem, and propose Generative Hierarchical Materials Search (GenMS) for
controllable generation of crystal structures. GenMS consists of (1) a language
model that takes high-level natural language as input and generates
intermediate textual information about a crystal (e.g., chemical formulae), and
(2) a diffusion model that takes intermediate information as input and
generates low-level continuous value crystal structures. GenMS additionally
uses a graph neural network to predict properties (e.g., formation energy) from
the generated crystal structures. During inference, GenMS leverages all three
components to conduct a forward tree search over the space of possible
structures. Experiments show that GenMS outperforms other alternatives of
directly using language models to generate structures both in satisfying user
request and in generating low-energy structures. We confirm that GenMS is able
to generate common crystal structures such as double perovskites, or spinels,
solely from natural language input, and hence can form the foundation for more
complex structure generation in near future.
Instant Facial Gaussians Translator for Relightable and Interactable
  Facial Rendering
Authors: name: Dafei Qin, name: Hongyang Lin, name: Qixuan Zhang, name: Kaichun Qiao, name: Longwen Zhang, name: Zijun Zhao, name: Jun Saito, name: Jingyi Yu, name: Lan Xu, name: Taku Komura
Published: 2024-09-11 17:40:21

We propose GauFace, a novel Gaussian Splatting representation, tailored for
efficient animation and rendering of physically-based facial assets. Leveraging
strong geometric priors and constrained optimization, GauFace ensures a neat
and structured Gaussian representation, delivering high fidelity and real-time
facial interaction of 30fps@1440p on a Snapdragon 8 Gen 2 mobile platform.
  Then, we introduce TransGS, a diffusion transformer that instantly translates
physically-based facial assets into the corresponding GauFace representations.
Specifically, we adopt a patch-based pipeline to handle the vast number of
Gaussians effectively. We also introduce a novel pixel-aligned sampling scheme
with UV positional encoding to ensure the throughput and rendering quality of
GauFace assets generated by our TransGS. Once trained, TransGS can instantly
translate facial assets with lighting conditions to GauFace representation,
With the rich conditioning modalities, it also enables editing and animation
capabilities reminiscent of traditional CG pipelines.
  We conduct extensive evaluations and user studies, compared to traditional
offline and online renderers, as well as recent neural rendering methods, which
demonstrate the superior performance of our approach for facial asset
rendering. We also showcase diverse immersive applications of facial assets
using our TransGS approach and GauFace representation, across various platforms
like PCs, phones and even VR headsets.
MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View
  Synthesis
Authors: name: Hanyu Jiang, name: Jian Xue, name: Xing Lan, name: Guohong Hu, name: Ke Lu
Published: 2024-09-11 09:25:37

This paper introduces MVLLaVA, an intelligent agent designed for novel view
synthesis tasks. MVLLaVA integrates multiple multi-view diffusion models with a
large multimodal model, LLaVA, enabling it to handle a wide range of tasks
efficiently. MVLLaVA represents a versatile and unified platform that adapts to
diverse input types, including a single image, a descriptive caption, or a
specific change in viewing azimuth, guided by language instructions for
viewpoint generation. We carefully craft task-specific instruction templates,
which are subsequently used to fine-tune LLaVA. As a result, MVLLaVA acquires
the capability to generate novel view images based on user instructions,
demonstrating its flexibility across diverse tasks. Experiments are conducted
to validate the effectiveness of MVLLaVA, demonstrating its robust performance
and versatility in tackling diverse novel view synthesis challenges.