Orionnnn/hub-stats huggingface dataset:
Demystifying GPT Self-Repair for Code Generation
Authors: name: Theo X. Olausson, name: Jeevana Priya Inala, name: Chenglong Wang, name: Jianfeng Gao, name: Armando Solar-Lezama
Published: 2023-06-16 15:13:17

Large Language Models (LLMs) have shown remarkable aptitude in code
generation but still struggle on challenging programming tasks. Self-repair --
in which the model debugs and fixes mistakes in its own code -- has recently
become a popular way to boost performance in these settings. However, only very
limited studies on how and when self-repair works effectively exist in the
literature, and one might wonder to what extent a model is really capable of
providing accurate feedback on why the code is wrong when that code was
generated by the same model. In this paper, we analyze GPT-3.5 and GPT-4's
ability to perform self-repair on APPS, a challenging dataset consisting of
diverse coding challenges. To do so, we first establish a new evaluation
strategy dubbed pass@t that measures the pass rate of the tasks against the
total number of tokens sampled from the model, enabling a fair comparison to
purely sampling-based approaches. With this evaluation strategy, we find that
the effectiveness of self-repair is only seen in GPT-4. We also observe that
self-repair is bottlenecked by the feedback stage; using GPT-4 to give feedback
on the programs generated by GPT-3.5 and using expert human programmers to give
feedback on the programs generated by GPT-4, we unlock significant performance
gains.
NAVI: Category-Agnostic Image Collections with High-Quality 3D Shape and
  Pose Annotations
Authors: name: Varun Jampani, name: Kevis-Kokitsi Maninis, name: Andreas Engelhardt, name: Arjun Karpur, name: Karen Truong, name: Kyle Sargent, name: Stefan Popov, name: André Araujo, name: Ricardo Martin-Brualla, name: Kaushal Patel, name: Daniel Vlasic, name: Vittorio Ferrari, name: Ameesh Makadia, name: Ce Liu, name: Yuanzhen Li, name: Howard Zhou
Published: 2023-06-15 13:11:30

Recent advances in neural reconstruction enable high-quality 3D object
reconstruction from casually captured image collections. Current techniques
mostly analyze their progress on relatively simple image collections where
Structure-from-Motion (SfM) techniques can provide ground-truth (GT) camera
poses. We note that SfM techniques tend to fail on in-the-wild image
collections such as image search results with varying backgrounds and
illuminations. To enable systematic research progress on 3D reconstruction from
casual image captures, we propose NAVI: a new dataset of category-agnostic
image collections of objects with high-quality 3D scans along with per-image
2D-3D alignments providing near-perfect GT camera parameters. These 2D-3D
alignments allow us to extract accurate derivative annotations such as dense
pixel correspondences, depth and segmentation maps. We demonstrate the use of
NAVI image collections on different problem settings and show that NAVI enables
more thorough evaluations that were not possible with existing datasets. We
believe NAVI is beneficial for systematic research progress on 3D
reconstruction and correspondence estimation. Project page:
https://navidataset.github.io
Agile Catching with Whole-Body MPC and Blackbox Policy Learning
Authors: name: Saminda Abeyruwan, name: Alex Bewley, name: Nicholas M. Boffi, name: Krzysztof Choromanski, name: David D'Ambrosio, name: Deepali Jain, name: Pannag Sanketi, name: Anish Shankar, name: Vikas Sindhwani, name: Sumeet Singh, name: Jean-Jacques Slotine, name: Stephen Tu
Published: 2023-06-14 02:13:25

We address a benchmark task in agile robotics: catching objects thrown at
high-speed. This is a challenging task that involves tracking, intercepting,
and cradling a thrown object with access only to visual observations of the
object and the proprioceptive state of the robot, all within a fraction of a
second. We present the relative merits of two fundamentally different solution
strategies: (i) Model Predictive Control using accelerated constrained
trajectory optimization, and (ii) Reinforcement Learning using zeroth-order
optimization. We provide insights into various performance trade-offs including
sample efficiency, sim-to-real transfer, robustness to distribution shifts, and
whole-body multimodality via extensive on-hardware experiments. We conclude
with proposals on fusing "classical" and "learning-based" techniques for agile
robot control. Videos of our experiments may be found at
https://sites.google.com/view/agile-catching
DreamHuman: Animatable 3D Avatars from Text
Authors: name: Nikos Kolotouros, name: Thiemo Alldieck, name: Andrei Zanfir, name: Eduard Gabriel Bazavan, name: Mihai Fieraru, name: Cristian Sminchisescu
Published: 2023-06-15 17:58:21

We present DreamHuman, a method to generate realistic animatable 3D human
avatar models solely from textual descriptions. Recent text-to-3D methods have
made considerable strides in generation, but are still lacking in important
aspects. Control and often spatial resolution remain limited, existing methods
produce fixed rather than animated 3D human models, and anthropometric
consistency for complex structures like people remains a challenge. DreamHuman
connects large text-to-image synthesis models, neural radiance fields, and
statistical human body models in a novel modeling and optimization framework.
This makes it possible to generate dynamic 3D human avatars with high-quality
textures and learned, instance-specific, surface deformations. We demonstrate
that our method is capable to generate a wide variety of animatable, realistic
3D human models from text. Our 3D models have diverse appearance, clothing,
skin tones and body shapes, and significantly outperform both generic
text-to-3D approaches and previous text-based 3D avatar generators in visual
fidelity. For more results and animations please check our website at
https://dream-human.github.io.
Neural Relighting with Subsurface Scattering by Learning the Radiance
  Transfer Gradient
Authors: name: Shizhan Zhu, name: Shunsuke Saito, name: Aljaz Bozic, name: Carlos Aliaga, name: Trevor Darrell, name: Christop Lassner
Published: 2023-06-15 17:56:04

Reconstructing and relighting objects and scenes under varying lighting
conditions is challenging: existing neural rendering methods often cannot
handle the complex interactions between materials and light. Incorporating
pre-computed radiance transfer techniques enables global illumination, but
still struggles with materials with subsurface scattering effects. We propose a
novel framework for learning the radiance transfer field via volume rendering
and utilizing various appearance cues to refine geometry end-to-end. This
framework extends relighting and reconstruction capabilities to handle a wider
range of materials in a data-driven fashion. The resulting models produce
plausible rendering results in existing and novel conditions. We will release
our code and a novel light stage dataset of objects with subsurface scattering
effects publicly available.
Diffusion Models for Zero-Shot Open-Vocabulary Segmentation
Authors: name: Laurynas Karazija, name: Iro Laina, name: Andrea Vedaldi, name: Christian Rupprecht
Published: 2023-06-15 17:51:28

The variety of objects in the real world is nearly unlimited and is thus
impossible to capture using models trained on a fixed set of categories. As a
result, in recent years, open-vocabulary methods have attracted the interest of
the community. This paper proposes a new method for zero-shot open-vocabulary
segmentation. Prior work largely relies on contrastive training using
image-text pairs, leveraging grouping mechanisms to learn image features that
are both aligned with language and well-localised. This however can introduce
ambiguity as the visual appearance of images with similar captions often
varies. Instead, we leverage the generative properties of large-scale
text-to-image diffusion models to sample a set of support images for a given
textual category. This provides a distribution of appearances for a given text
circumventing the ambiguity problem. We further propose a mechanism that
considers the contextual background of the sampled images to better localise
objects and segment the background directly. We show that our method can be
used to ground several existing pre-trained self-supervised feature extractors
in natural language and provide explainable predictions by mapping back to
regions in the support set. Our proposal is training-free, relying on
pre-trained components only, yet, shows strong performance on a range of
open-vocabulary segmentation benchmarks, obtaining a lead of more than 10% on
the Pascal VOC benchmark.
TryOnDiffusion: A Tale of Two UNets
Authors: name: Luyang Zhu, name: Dawei Yang, name: Tyler Zhu, name: Fitsum Reda, name: William Chan, name: Chitwan Saharia, name: Mohammad Norouzi, name: Ira Kemelmacher-Shlizerman
Published: 2023-06-14 06:25:58

Given two images depicting a person and a garment worn by another person, our
goal is to generate a visualization of how the garment might look on the input
person. A key challenge is to synthesize a photorealistic detail-preserving
visualization of the garment, while warping the garment to accommodate a
significant body pose and shape change across the subjects. Previous methods
either focus on garment detail preservation without effective pose and shape
variation, or allow try-on with the desired shape and pose but lack garment
details. In this paper, we propose a diffusion-based architecture that unifies
two UNets (referred to as Parallel-UNet), which allows us to preserve garment
details and warp the garment for significant pose and body change in a single
network. The key ideas behind Parallel-UNet include: 1) garment is warped
implicitly via a cross attention mechanism, 2) garment warp and person blend
happen as part of a unified process as opposed to a sequence of two separate
tasks. Experimental results indicate that TryOnDiffusion achieves
state-of-the-art performance both qualitatively and quantitatively.
DORSal: Diffusion for Object-centric Representations of Scenes
  et al.
Authors: name: Allan Jabri, name: Sjoerd van Steenkiste, name: Emiel Hoogeboom, name: Mehdi S. M. Sajjadi, name: Thomas Kipf
Published: 2023-06-13 18:32:35

Recent progress in 3D scene understanding enables scalable learning of
representations across large datasets of diverse scenes. As a consequence,
generalization to unseen scenes and objects, rendering novel views from just a
single or a handful of input images, and controllable scene generation that
supports editing, is now possible. However, training jointly on a large number
of scenes typically compromises rendering quality when compared to single-scene
optimized models such as NeRFs. In this paper, we leverage recent progress in
diffusion models to equip 3D scene representation learning models with the
ability to render high-fidelity novel views, while retaining benefits such as
object-level scene editing to a large degree. In particular, we propose DORSal,
which adapts a video diffusion architecture for 3D scene generation conditioned
on object-centric slot-based representations of scenes. On both complex
synthetic multi-object scenes and on the real-world large-scale Street View
dataset, we show that DORSal enables scalable neural rendering of 3D scenes
with object-level editing and improves upon existing approaches.
Anticipatory Music Transformer
Authors: name: John Thickstun, name: David Hall, name: Chris Donahue, name: Percy Liang
Published: 2023-06-14 16:27:53

We introduce anticipation: a method for constructing a controllable
generative model of a temporal point process (the event process) conditioned
asynchronously on realizations of a second, correlated process (the control
process). We achieve this by interleaving sequences of events and controls,
such that controls appear following stopping times in the event sequence. This
work is motivated by problems arising in the control of symbolic music
generation. We focus on infilling control tasks, whereby the controls are a
subset of the events themselves, and conditional generation completes a
sequence of events given the fixed control events. We train anticipatory
infilling models using the large and diverse Lakh MIDI music dataset. These
models match the performance of autoregressive models for prompted music
generation, with the additional capability to perform infilling control tasks,
including accompaniment. Human evaluators report that an anticipatory model
produces accompaniments with similar musicality to even music composed by
humans over a 20-second clip.
VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing
Authors: name: Paul Couairon, name: Clément Rambour, name: Jean-Emmanuel Haugeard, name: Nicolas Thome
Published: 2023-06-14 19:15:49

Recently, diffusion-based generative models have achieved remarkable success
for image generation and edition. However, their use for video editing still
faces important limitations. This paper introduces VidEdit, a novel method for
zero-shot text-based video editing ensuring strong temporal and spatial
consistency. Firstly, we propose to combine atlas-based and pre-trained
text-to-image diffusion models to provide a training-free and efficient editing
method, which by design fulfills temporal smoothness. Secondly, we leverage
off-the-shelf panoptic segmenters along with edge detectors and adapt their use
for conditioned diffusion-based atlas editing. This ensures a fine spatial
control on targeted regions while strictly preserving the structure of the
original video. Quantitative and qualitative experiments show that VidEdit
outperforms state-of-the-art methods on DAVIS dataset, regarding semantic
faithfulness, image preservation, and temporal consistency metrics. With this
framework, processing a single video only takes approximately one minute, and
it can generate multiple compatible edits based on a unique text prompt.
Project web-page at https://videdit.github.io
Large-scale Language Model Rescoring on Long-form Data
Authors: name: Tongzhou Chen, name: Cyril Allauzen, name: Yinghui Huang, name: Daniel Park, name: David Rybach, name: W. Ronny Huang, name: Rodrigo Cabrera, name: Kartik Audhkhasi, name: Bhuvana Ramabhadran, name: Pedro J. Moreno, name: Michael Riley
Published: 2023-06-13 20:54:12

In this work, we study the impact of Large-scale Language Models (LLM) on
Automated Speech Recognition (ASR) of YouTube videos, which we use as a source
for long-form ASR. We demonstrate up to 8\% relative reduction in Word Error
Eate (WER) on US English (en-us) and code-switched Indian English (en-in)
long-form ASR test sets and a reduction of up to 30\% relative on Salient Term
Error Rate (STER) over a strong first-pass baseline that uses a maximum-entropy
based language model. Improved lattice processing that results in a lattice
with a proper (non-tree) digraph topology and carrying context from the 1-best
hypothesis of the previous segment(s) results in significant wins in rescoring
with LLMs. We also find that the gains in performance from the combination of
LLMs trained on vast quantities of available data (such as C4) and conventional
neural LMs is additive and significantly outperforms a strong first-pass
baseline with a maximum entropy LM.
Language-Guided Music Recommendation for Video via Prompt Analogies
Authors: name: Daniel McKee, name: Justin Salamon, name: Josef Sivic, name: Bryan Russell
Published: 2023-06-15 17:58:01

We propose a method to recommend music for an input video while allowing a
user to guide music selection with free-form natural language. A key challenge
of this problem setting is that existing music video datasets provide the
needed (video, music) training pairs, but lack text descriptions of the music.
This work addresses this challenge with the following three contributions.
First, we propose a text-synthesis approach that relies on an analogy-based
prompting procedure to generate natural language music descriptions from a
large-scale language model (BLOOM-176B) given pre-trained music tagger outputs
and a small number of human text descriptions. Second, we use these synthesized
music descriptions to train a new trimodal model, which fuses text and video
input representations to query music samples. For training, we introduce a text
dropout regularization mechanism which we show is critical to model
performance. Our model design allows for the retrieved music audio to agree
with the two input modalities by matching visual style depicted in the video
and musical genre, mood, or instrumentation described in the natural language
query. Third, to evaluate our approach, we collect a testing dataset for our
problem by annotating a subset of 4k clips from the YT8M-MusicVideo dataset
with natural language music descriptions which we make publicly available. We
show that our approach can match or exceed the performance of prior methods on
video-to-music retrieval while significantly improving retrieval accuracy when
using text guidance.
KoLA: Carefully Benchmarking World Knowledge of Large Language Models
Authors: name: Jifan Yu, name: Xiaozhi Wang, name: Shangqing Tu, name: Shulin Cao, name: Daniel Zhang-Li, name: Xin Lv, name: Hao Peng, name: Zijun Yao, name: Xiaohan Zhang, name: Hanming Li, name: Chunyang Li, name: Zheyuan Zhang, name: Yushi Bai, name: Yantao Liu, name: Amy Xin, name: Nianyi Lin, name: Kaifeng Yun, name: Linlu Gong, name: Jianhui Chen, name: Zhili Wu, name: Yunjia Qi, name: Weikai Li, name: Yong Guan, name: Kaisheng Zeng, name: Ji Qi, name: Hailong Jin, name: Jinxin Liu, name: Yu Gu, name: Yuan Yao, name: Ning Ding, name: Lei Hou, name: Zhiyuan Liu, name: Bin Xu, name: Jie Tang, name: Juanzi Li
Published: 2023-06-15 17:20:46

The unprecedented performance of large language models (LLMs) necessitates
improvements in evaluations. Rather than merely exploring the breadth of LLM
abilities, we believe meticulous and thoughtful designs are essential to
thorough, unbiased, and applicable evaluations. Given the importance of world
knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark
(KoLA), in which we carefully design three crucial factors: (1) For ability
modeling, we mimic human cognition to form a four-level taxonomy of
knowledge-related abilities, covering 19 tasks. (2) For data, to ensure fair
comparisons, we use both Wikipedia, a corpus prevalently pre-trained by LLMs,
along with continuously collected emerging corpora, aiming to evaluate the
capacity to handle unseen data and evolving knowledge. (3) For evaluation
criteria, we adopt a contrastive system, including overall standard scores for
better numerical comparability across tasks and models and a unique
self-contrast metric for automatically evaluating knowledge hallucination. We
evaluate 21 open-source and commercial LLMs and obtain some intriguing
findings. The KoLA dataset and open-participation leaderboard are publicly
released at https://kola.xlore.cn and will be continuously updated to provide
references for developing LLMs and knowledge-related systems.
ChessGPT: Bridging Policy Learning and Language Modeling
Authors: name: Xidong Feng, name: Yicheng Luo, name: Ziyan Wang, name: Hongrui Tang, name: Mengyue Yang, name: Kun Shao, name: David Mguni, name: Yali Du, name: Jun Wang
Published: 2023-06-15 15:35:31

When solving decision-making tasks, humans typically depend on information
from two key sources: (1) Historical policy data, which provides interaction
replay from the environment, and (2) Analytical insights in natural language
form, exposing the invaluable thought process or strategic considerations.
Despite this, the majority of preceding research focuses on only one source:
they either use historical replay exclusively to directly learn policy or value
functions, or engaged in language model training utilizing mere language
corpus. In this paper, we argue that a powerful autonomous agent should cover
both sources. Thus, we propose ChessGPT, a GPT model bridging policy learning
and language modeling by integrating data from these two sources in Chess
games. Specifically, we build a large-scale game and language dataset related
to chess. Leveraging the dataset, we showcase two model examples ChessCLIP and
ChessGPT, integrating policy learning and language modeling. Finally, we
propose a full evaluation framework for evaluating language model's chess
ability. Experimental results validate our model and dataset's effectiveness.
We open source our code, model, and dataset at
https://github.com/waterhorse1/ChessGPT.
Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and
  Text Integration
Authors: name: Chenyang Lyu, name: Minghao Wu, name: Longyue Wang, name: Xinting Huang, name: Bingshuai Liu, name: Zefeng Du, name: Shuming Shi, name: Zhaopeng Tu
Published: 2023-06-15 12:45:25

Although instruction-tuned large language models (LLMs) have exhibited
remarkable capabilities across various NLP tasks, their effectiveness on other
data modalities beyond text has not been fully studied. In this work, we
propose Macaw-LLM, a novel multi-modal LLM that seamlessly integrates visual,
audio, and textual information. Macaw-LLM consists of three main components: a
modality module for encoding multi-modal data, a cognitive module for
harnessing pretrained LLMs, and an alignment module for harmonizing diverse
representations. Our novel alignment module seamlessly bridges multi-modal
features to textual features, simplifying the adaptation process from the
modality modules to the cognitive module. In addition, we construct a
large-scale multi-modal instruction dataset in terms of multi-turn dialogue,
including 69K image instances and 50K video instances. We have made our data,
code and model publicly available, which we hope can pave the way for future
research in multi-modal LLMs and expand the capabilities of LLMs to handle
diverse data modalities and address complex real-world scenarios.
LOVM: Language-Only Vision Model Selection
Authors: name: Orr Zohar, name: Shih-Cheng Huang, name: Kuan-Chieh Wang, name: Serena Yeung
Published: 2023-06-15 06:53:05

Pre-trained multi-modal vision-language models (VLMs) are becoming
increasingly popular due to their exceptional performance on downstream vision
applications, particularly in the few- and zero-shot settings. However,
selecting the best-performing VLM for some downstream applications is
non-trivial, as it is dataset and task-dependent. Meanwhile, the exhaustive
evaluation of all available VLMs on a novel application is not only time and
computationally demanding but also necessitates the collection of a labeled
dataset for evaluation. As the number of open-source VLM variants increases,
there is a need for an efficient model selection strategy that does not require
access to a curated evaluation dataset. This paper proposes a novel task and
benchmark for efficiently evaluating VLMs' zero-shot performance on downstream
applications without access to the downstream task dataset. Specifically, we
introduce a new task LOVM: Language-Only Vision Model Selection, where methods
are expected to perform both model selection and performance prediction based
solely on a text description of the desired downstream application. We then
introduced an extensive LOVM benchmark consisting of ground-truth evaluations
of 35 pre-trained VLMs and 23 datasets, where methods are expected to rank the
pre-trained VLMs and predict their zero-shot performance.