Orionnnn/hub-stats huggingface dataset:
Agents: An Open-source Framework for Autonomous Language Agents
Authors: name: Wangchunshu Zhou, name: Yuchen Eleanor Jiang, name: Long Li, name: Jialong Wu, name: Tiannan Wang, name: Shi Qiu, name: Jintian Zhang, name: Jing Chen, name: Ruipu Wu, name: Shuai Wang, name: Shiding Zhu, name: Jiyu Chen, name: Wentao Zhang, name: Ningyu Zhang, name: Huajun Chen, name: Peng Cui, name: Mrinmaya Sachan
Published: 2023-09-14 17:18:25

Recent advances on large language models (LLMs) enable researchers and
developers to build autonomous language agents that can automatically solve
various tasks and interact with environments, humans, and other agents using
natural language interfaces. We consider language agents as a promising
direction towards artificial general intelligence and release Agents, an
open-source library with the goal of opening up these advances to a wider
non-specialist audience. Agents is carefully engineered to support important
features including planning, memory, tool usage, multi-agent communication, and
fine-grained symbolic control. Agents is user-friendly as it enables
non-specialists to build, customize, test, tune, and deploy state-of-the-art
autonomous language agents without much coding. The library is also
research-friendly as its modularized design makes it easily extensible for
researchers. Agents is available at https://github.com/aiwaves-cn/agents.
Are Large Language Model-based Evaluators the Solution to Scaling Up
  Multilingual Evaluation?
Authors: name: Rishav Hada, name: Varun Gumma, name: Adrian de Wynter, name: Harshita Diddee, name: Mohamed Ahmed, name: Monojit Choudhury, name: Kalika Bali, name: Sunayana Sitaram
Published: 2023-09-14 06:41:58

Large Language Models (LLMs) have demonstrated impressive performance on
Natural Language Processing (NLP) tasks, such as Question Answering,
Summarization, and Classification. The use of LLMs as evaluators, that can rank
or score the output of other models (usually LLMs) has become increasingly
popular, due to the limitations of current evaluation techniques including the
lack of appropriate benchmarks, metrics, cost, and access to human annotators.
While LLMs are capable of handling approximately 100 languages, the majority of
languages beyond the top 20 lack systematic evaluation across various tasks,
metrics, and benchmarks. This creates an urgent need to scale up multilingual
evaluation to ensure a precise understanding of LLM performance across diverse
languages. LLM-based evaluators seem like the perfect solution to this problem,
as they do not require human annotators, human-created references, or
benchmarks and can theoretically be used to evaluate any language covered by
the LLM. In this paper, we investigate whether LLM-based evaluators can help
scale up multilingual evaluation. Specifically, we calibrate LLM-based
evaluation against 20k human judgments of five metrics across three
text-generation tasks in eight languages. Our findings indicate that LLM-based
evaluators may exhibit bias towards higher scores and should be used with
caution and should always be calibrated with a dataset of native speaker
judgments, particularly in low-resource and non-Latin script languages.
Clinical Text Summarization: Adapting Large Language Models Can
  Outperform Human Experts
Authors: name: Dave Van Veen, name: Cara Van Uden, name: Louis Blankemeier, name: Jean-Benoit Delbrouck, name: Asad Aali, name: Christian Bluethgen, name: Anuj Pareek, name: Malgorzata Polacin, name: William Collins, name: Neera Ahuja, name: Curtis P. Langlotz, name: Jason Hom, name: Sergios Gatidis, name: John Pauly, name: Akshay S. Chaudhari
Published: 2023-09-14 05:15:01

Sifting through vast textual data and summarizing key information imposes a
substantial burden on how clinicians allocate their time. Although large
language models (LLMs) have shown immense promise in natural language
processing (NLP) tasks, their efficacy across diverse clinical summarization
tasks has not yet been rigorously examined. In this work, we employ domain
adaptation methods on eight LLMs, spanning six datasets and four distinct
summarization tasks: radiology reports, patient questions, progress notes, and
doctor-patient dialogue. Our thorough quantitative assessment reveals
trade-offs between models and adaptation methods in addition to instances where
recent advances in LLMs may not lead to improved results. Further, in a
clinical reader study with six physicians, we depict that summaries from the
best adapted LLM are preferable to human summaries in terms of completeness and
correctness. Our ensuing qualitative analysis delineates mutual challenges
faced by both LLMs and human experts. Lastly, we correlate traditional
quantitative NLP metrics with reader study scores to enhance our understanding
of how these metrics align with physician preferences. Our research marks the
first evidence of LLMs outperforming human experts in clinical text
summarization across multiple tasks. This implies that integrating LLMs into
clinical workflows could alleviate documentation burden, empowering clinicians
to focus more on personalized patient care and other irreplaceable human
aspects of medicine.
Text-Guided Generation and Editing of Compositional 3D Avatars
Authors: name: Hao Zhang, name: Yao Feng, name: Peter Kulits, name: Yandong Wen, name: Justus Thies, name: Michael J. Black
Published: 2023-09-13 17:59:56

Our goal is to create a realistic 3D facial avatar with hair and accessories
using only a text description. While this challenge has attracted significant
recent interest, existing methods either lack realism, produce unrealistic
shapes, or do not support editing, such as modifications to the hairstyle. We
argue that existing methods are limited because they employ a monolithic
modeling approach, using a single representation for the head, face, hair, and
accessories. Our observation is that the hair and face, for example, have very
different structural qualities that benefit from different representations.
Building on this insight, we generate avatars with a compositional model, in
which the head, face, and upper body are represented with traditional 3D
meshes, and the hair, clothing, and accessories with neural radiance fields
(NeRF). The model-based mesh representation provides a strong geometric prior
for the face region, improving realism while enabling editing of the person's
appearance. By using NeRFs to represent the remaining components, our method is
able to model and synthesize parts with complex geometry and appearance, such
as curly hair and fluffy scarves. Our novel system synthesizes these
high-quality compositional avatars from text descriptions. The experimental
results demonstrate that our method, Text-guided generation and Editing of
Compositional Avatars (TECA), produces avatars that are more realistic than
those of recent methods while being editable because of their compositional
nature. For example, our TECA enables the seamless transfer of compositional
features like hairstyles, scarves, and other accessories between avatars. This
capability supports applications such as virtual try-on.
Tree-Structured Shading Decomposition
Authors: name: Chen Geng, name: Hong-Xing Yu, name: Sharon Zhang, name: Maneesh Agrawala, name: Jiajun Wu
Published: 2023-09-13 17:57:55

We study inferring a tree-structured representation from a single image for
object shading. Prior work typically uses the parametric or measured
representation to model shading, which is neither interpretable nor easily
editable. We propose using the shade tree representation, which combines basic
shading nodes and compositing methods to factorize object surface shading. The
shade tree representation enables novice users who are unfamiliar with the
physical shading process to edit object shading in an efficient and intuitive
manner. A main challenge in inferring the shade tree is that the inference
problem involves both the discrete tree structure and the continuous parameters
of the tree nodes. We propose a hybrid approach to address this issue. We
introduce an auto-regressive inference model to generate a rough estimation of
the tree structure and node parameters, and then we fine-tune the inferred
shade tree through an optimization algorithm. We show experiments on synthetic
images, captured reflectance, real images, and non-realistic vector drawings,
allowing downstream applications such as material editing, vectorized shading,
and relighting. Project website: https://chen-geng.com/inv-shade-trees
Dynamic NeRFs for Soccer Scenes
Authors: name: Sacha Lewin, name: Maxime Vandegar, name: Thomas Hoyoux, name: Olivier Barnich, name: Gilles Louppe
Published: 2023-09-13 08:50:00

The long-standing problem of novel view synthesis has many applications,
notably in sports broadcasting. Photorealistic novel view synthesis of soccer
actions, in particular, is of enormous interest to the broadcast industry. Yet
only a few industrial solutions have been proposed, and even fewer that achieve
near-broadcast quality of the synthetic replays. Except for their setup of
multiple static cameras around the playfield, the best proprietary systems
disclose close to no information about their inner workings. Leveraging
multiple static cameras for such a task indeed presents a challenge rarely
tackled in the literature, for a lack of public datasets: the reconstruction of
a large-scale, mostly static environment, with small, fast-moving elements.
Recently, the emergence of neural radiance fields has induced stunning progress
in many novel view synthesis applications, leveraging deep learning principles
to produce photorealistic results in the most challenging settings. In this
work, we investigate the feasibility of basing a solution to the task on
dynamic NeRFs, i.e., neural models purposed to reconstruct general dynamic
content. We compose synthetic soccer environments and conduct multiple
experiments using them, identifying key components that help reconstruct soccer
scenes with dynamic NeRFs. We show that, although this approach cannot fully
meet the quality requirements for the target application, it suggests promising
avenues toward a cost-efficient, automatic solution. We also make our work
dataset and code publicly available, with the goal to encourage further efforts
from the research community on the task of novel view synthesis for dynamic
soccer scenes. For code, data, and video results, please see
https://soccernerfs.isach.be.
DreamStyler: Paint by Style Inversion with Text-to-Image Diffusion
  Models
Authors: name: Namhyuk Ahn, name: Junsoo Lee, name: Chunggi Lee, name: Kunhee Kim, name: Daesik Kim, name: Seung-Hun Nam, name: Kibeom Hong
Published: 2023-09-13 13:13:29

Recent progresses in large-scale text-to-image models have yielded remarkable
accomplishments, finding various applications in art domain. However,
expressing unique characteristics of an artwork (e.g. brushwork, colortone, or
composition) with text prompts alone may encounter limitations due to the
inherent constraints of verbal description. To this end, we introduce
DreamStyler, a novel framework designed for artistic image synthesis,
proficient in both text-to-image synthesis and style transfer. DreamStyler
optimizes a multi-stage textual embedding with a context-aware text prompt,
resulting in prominent image quality. In addition, with content and style
guidance, DreamStyler exhibits flexibility to accommodate a range of style
references. Experimental results demonstrate its superior performance across
multiple scenarios, suggesting its promising potential in artistic product
creation.
MagiCapture: High-Resolution Multi-Concept Portrait Customization
Authors: name: Junha Hyung, name: Jaeyo Shin, name: Jaegul Choo
Published: 2023-09-13 11:37:04

Large-scale text-to-image models including Stable Diffusion are capable of
generating high-fidelity photorealistic portrait images. There is an active
research area dedicated to personalizing these models, aiming to synthesize
specific subjects or styles using provided sets of reference images. However,
despite the plausible results from these personalization methods, they tend to
produce images that often fall short of realism and are not yet on a
commercially viable level. This is particularly noticeable in portrait image
generation, where any unnatural artifact in human faces is easily discernible
due to our inherent human bias. To address this, we introduce MagiCapture, a
personalization method for integrating subject and style concepts to generate
high-resolution portrait images using just a few subject and style references.
For instance, given a handful of random selfies, our fine-tuned model can
generate high-quality portrait images in specific styles, such as passport or
profile photos. The main challenge with this task is the absence of ground
truth for the composed concepts, leading to a reduction in the quality of the
final output and an identity shift of the source subject. To address these
issues, we present a novel Attention Refocusing loss coupled with auxiliary
priors, both of which facilitate robust learning within this weakly supervised
learning setting. Our pipeline also includes additional post-processing steps
to ensure the creation of highly realistic outputs. MagiCapture outperforms
other baselines in both quantitative and qualitative evaluations and can also
be generalized to other non-human objects.
Large Language Models for Compiler Optimization
Authors: name: Chris Cummins, name: Volker Seeker, name: Dejan Grubisic, name: Mostafa Elhoushi, name: Youwei Liang, name: Baptiste Roziere, name: Jonas Gehring, name: Fabian Gloeckle, name: Kim Hazelwood, name: Gabriel Synnaeve, name: Hugh Leather
Published: 2023-09-11 22:11:46

We explore the novel application of Large Language Models to code
optimization. We present a 7B-parameter transformer model trained from scratch
to optimize LLVM assembly for code size. The model takes as input unoptimized
assembly and outputs a list of compiler options to best optimize the program.
Crucially, during training, we ask the model to predict the instruction counts
before and after optimization, and the optimized code itself. These auxiliary
learning tasks significantly improve the optimization performance of the model
and improve the model's depth of understanding.
  We evaluate on a large suite of test programs. Our approach achieves a 3.0%
improvement in reducing instruction counts over the compiler, outperforming two
state-of-the-art baselines that require thousands of compilations. Furthermore,
the model shows surprisingly strong code reasoning abilities, generating
compilable code 91% of the time and perfectly emulating the output of the
compiler 70% of the time.
Statistical Rejection Sampling Improves Preference Optimization
Authors: name: Tianqi Liu, name: Yao Zhao, name: Rishabh Joshi, name: Misha Khalman, name: Mohammad Saleh, name: Peter J. Liu, name: Jialu Liu
Published: 2023-09-13 01:07:25

Improving the alignment of language models with human preferences remains an
active research challenge. Previous approaches have primarily utilized
Reinforcement Learning from Human Feedback (RLHF) via online RL methods such as
Proximal Policy Optimization (PPO). Recently, offline methods such as Sequence
Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have
emerged as attractive alternatives, offering improvements in stability and
scalability while maintaining competitive performance. SLiC refines its loss
function using sequence pairs sampled from a supervised fine-tuned (SFT)
policy, while DPO directly optimizes language models based on preference data,
foregoing the need for a separate reward model. However, the maximum likelihood
estimator (MLE) of the target optimal policy requires labeled preference pairs
sampled from that policy. DPO's lack of a reward model constrains its ability
to sample preference pairs from the optimal policy, and SLiC is restricted to
sampling preference pairs only from the SFT policy. To address these
limitations, we introduce a novel approach called Statistical Rejection
Sampling Optimization (RSO) that aims to source preference data from the target
optimal policy using rejection sampling, enabling a more accurate estimation of
the optimal policy. We also propose a unified framework that enhances the loss
functions used in both SLiC and DPO from a preference modeling standpoint.
Through extensive experiments across three diverse tasks, we demonstrate that
RSO consistently outperforms both SLiC and DPO on evaluations from both Large
Language Model (LLM) and human raters.
Learning Disentangled Avatars with Hybrid 3D Representations
Authors: name: Yao Feng, name: Weiyang Liu, name: Timo Bolkart, name: Jinlong Yang, name: Marc Pollefeys, name: Michael J. Black
Published: 2023-09-12 17:59:36

Tremendous efforts have been made to learn animatable and photorealistic
human avatars. Towards this end, both explicit and implicit 3D representations
are heavily studied for a holistic modeling and capture of the whole human
(e.g., body, clothing, face and hair), but neither representation is an optimal
choice in terms of representation efficacy since different parts of the human
avatar have different modeling desiderata. For example, meshes are generally
not suitable for modeling clothing and hair. Motivated by this, we present
Disentangled Avatars~(DELTA), which models humans with hybrid explicit-implicit
3D representations. DELTA takes a monocular RGB video as input, and produces a
human avatar with separate body and clothing/hair layers. Specifically, we
demonstrate two important applications for DELTA. For the first one, we
consider the disentanglement of the human body and clothing and in the second,
we disentangle the face and hair. To do so, DELTA represents the body or face
with an explicit mesh-based parametric 3D model and the clothing or hair with
an implicit neural radiance field. To make this possible, we design an
end-to-end differentiable renderer that integrates meshes into volumetric
rendering, enabling DELTA to learn directly from monocular videos without any
3D supervision. Finally, we show that how these two applications can be easily
combined to model full-body avatars, such that the hair, face, body and
clothing can be fully disentangled yet jointly rendered. Such a disentanglement
enables hair and clothing transfer to arbitrary body shapes. We empirically
validate the effectiveness of DELTA's disentanglement by demonstrating its
promising performance on disentangled reconstruction, virtual clothing try-on
and hairstyle transfer. To facilitate future research, we also release an
open-sourced pipeline for the study of hybrid human avatar modeling.
LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot
  Learning
Authors: name: Kenneth Shaw, name: Ananye Agarwal, name: Deepak Pathak
Published: 2023-09-12 17:59:20

Dexterous manipulation has been a long-standing challenge in robotics. While
machine learning techniques have shown some promise, results have largely been
currently limited to simulation. This can be mostly attributed to the lack of
suitable hardware. In this paper, we present LEAP Hand, a low-cost dexterous
and anthropomorphic hand for machine learning research. In contrast to previous
hands, LEAP Hand has a novel kinematic structure that allows maximal dexterity
regardless of finger pose. LEAP Hand is low-cost and can be assembled in 4
hours at a cost of 2000 USD from readily available parts. It is capable of
consistently exerting large torques over long durations of time. We show that
LEAP Hand can be used to perform several manipulation tasks in the real world
-- from visual teleoperation to learning from passive video data and sim2real.
LEAP Hand significantly outperforms its closest competitor Allegro Hand in all
our experiments while being 1/8th of the cost. We release detailed assembly
instructions, the Sim2Real pipeline and a development platform with useful APIs
on our website at https://leap-hand.github.io/
InstaFlow: One Step is Enough for High-Quality Diffusion-Based
  Text-to-Image Generation
Authors: name: Xingchao Liu, name: Xiwen Zhang, name: Jianzhu Ma, name: Jian Peng, name: Qiang Liu
Published: 2023-09-12 16:42:09

Diffusion models have revolutionized text-to-image generation with its
exceptional quality and creativity. However, its multi-step sampling process is
known to be slow, often requiring tens of inference steps to obtain
satisfactory results. Previous attempts to improve its sampling speed and
reduce computational costs through distillation have been unsuccessful in
achieving a functional one-step model. In this paper, we explore a recent
method called Rectified Flow, which, thus far, has only been applied to small
datasets. The core of Rectified Flow lies in its reflow procedure, which
straightens the trajectories of probability flows, refines the coupling between
noises and images, and facilitates the distillation process with student
models. We propose a novel text-conditioned pipeline to turn Stable Diffusion
(SD) into an ultra-fast one-step model, in which we find reflow plays a
critical role in improving the assignment between noise and images. Leveraging
our new pipeline, we create, to the best of our knowledge, the first one-step
diffusion-based text-to-image generator with SD-level image quality, achieving
an FID (Frechet Inception Distance) of 23.3 on MS COCO 2017-5k, surpassing
the previous state-of-the-art technique, progressive distillation, by a
significant margin (37.2 rightarrow 23.3 in FID). By utilizing an
expanded network with 1.7B parameters, we further improve the FID to 22.4. We
call our one-step models InstaFlow. On MS COCO 2014-30k, InstaFlow
yields an FID of 13.1 in just 0.09 second, the best in leq 0.1 second
regime, outperforming the recent StyleGAN-T (13.9 in 0.1 second). Notably,
the training of InstaFlow only costs 199 A100 GPU days. Project
page:~https://github.com/gnobitab/InstaFlow.
PhotoVerse: Tuning-Free Image Customization with Text-to-Image Diffusion
  Models
Authors: name: Li Chen, name: Mengyi Zhao, name: Yiheng Liu, name: Mingxu Ding, name: Yangyang Song, name: Shizun Wang, name: Xu Wang, name: Hao Yang, name: Jing Liu, name: Kang Du, name: Min Zheng
Published: 2023-09-11 19:59:43

Personalized text-to-image generation has emerged as a powerful and
sought-after tool, empowering users to create customized images based on their
specific concepts and prompts. However, existing approaches to personalization
encounter multiple challenges, including long tuning times, large storage
requirements, the necessity for multiple input images per identity, and
limitations in preserving identity and editability. To address these obstacles,
we present PhotoVerse, an innovative methodology that incorporates a
dual-branch conditioning mechanism in both text and image domains, providing
effective control over the image generation process. Furthermore, we introduce
facial identity loss as a novel component to enhance the preservation of
identity during training. Remarkably, our proposed PhotoVerse eliminates the
need for test time tuning and relies solely on a single facial photo of the
target identity, significantly reducing the resource cost associated with image
generation. After a single training phase, our approach enables generating
high-quality images within only a few seconds. Moreover, our method can produce
diverse images that encompass various scenes and styles. The extensive
evaluation demonstrates the superior performance of our approach, which
achieves the dual objectives of preserving identity and facilitating
editability. Project page: https://photoverse2d.github.io/