Orionnnn/hub-stats huggingface dataset:
Governance of the AI, by the AI, and for the AI
Authors: name: Andrew W. Torrance, name: Bill Tomlinson
Published: 2023-05-04 03:29:07

Over the past half century, there have been several false dawns during which
the "arrival" of world-changing artificial intelligence (AI) has been heralded.
Tempting fate, the authors believe the age of AI has, indeed, finally arrived.
Powerful image generators, such as DALL-E2 and Midjourney have suddenly allowed
anyone with access the ability easily to create rich and complex art. In a
similar vein, text generators, such as GPT3.5 (including ChatGPT) and BLOOM,
allow users to compose detailed written descriptions of many topics of
interest. And, it is even possible now for a person without extensive expertise
in writing software to use AI to generate code capable of myriad applications.
While AI will continue to evolve and improve, probably at a rapid rate, the
current state of AI is already ushering in profound changes to many different
sectors of society. Every new technology challenges the ability of humanity to
govern it wisely. However, governance is usually viewed as both possible and
necessary due to the disruption new technology often poses to social
structures, industries, the environment, and other important human concerns. In
this article, we offer an analysis of a range of interactions between AI and
governance, with the hope that wise decisions may be made that maximize
benefits and minimize costs. The article addresses two main aspects of this
relationship: the governance of AI by humanity, and the governance of humanity
by AI. The approach we have taken is itself informed by AI, as this article was
written collaboratively by the authors and ChatGPT.
Vera: A General-Purpose Plausibility Estimation Model for Commonsense
  Statements
Authors: name: Jiacheng Liu, name: Wenya Wang, name: Dianzhuo Wang, name: Noah A. Smith, name: Yejin Choi, name: Hannaneh Hajishirzi
Published: 2023-05-05 17:15:32

Despite the much discussed capabilities of today's language models, they are
still prone to silly and unexpected commonsense failures. We consider a
retrospective verification approach that reflects on the correctness of LM
outputs, and introduce Vera, a general-purpose model that estimates the
plausibility of declarative statements based on commonsense knowledge. Trained
on ~7M commonsense statements created from 19 QA datasets and two large-scale
knowledge bases, and with a combination of three training objectives, Vera is a
versatile model that effectively separates correct from incorrect statements
across diverse commonsense domains. When applied to solving commonsense
problems in the verification format, Vera substantially outperforms existing
models that can be repurposed for commonsense verification, and it further
exhibits generalization capabilities to unseen tasks and provides
well-calibrated outputs. We find that Vera excels at filtering LM-generated
commonsense knowledge and is useful in detecting erroneous commonsense
statements generated by models like ChatGPT in real-world settings.
Can Large Language Models Transform Computational Social Science?
Authors: name: Caleb Ziems, name: William Held, name: Omar Shaikh, name: Jiaao Chen, name: Zhehao Zhang, name: Diyi Yang
Published: 2023-04-12 17:33:28

Large Language Models (LLMs) like ChatGPT are capable of successfully
performing many language processing tasks zero-shot (without the need for
training data). If this capacity also applies to the coding of social phenomena
like persuasiveness and political ideology, then LLMs could effectively
transform Computational Social Science (CSS). This work provides a road map for
using LLMs as CSS tools. Towards this end, we contribute a set of prompting
best practices and an extensive evaluation pipeline to measure the zero-shot
performance of 13 language models on 24 representative CSS benchmarks. On
taxonomic labeling tasks (classification), LLMs fail to outperform the best
fine-tuned models but still achieve fair levels of agreement with humans. On
free-form coding tasks (generation), LLMs produce explanations that often
exceed the quality of crowdworkers' gold references. We conclude that today's
LLMs can radically augment the CSS research pipeline in two ways: (1) serving
as zero-shot data annotators on human annotation teams, and (2) bootstrapping
challenging creative generation tasks (e.g., explaining the hidden meaning
behind text). In summary, LLMs can significantly reduce costs and increase
efficiency of social science analysis in partnership with humans.
Can LLM Already Serve as A Database Interface? A BIg Bench for
  Large-Scale Database Grounded Text-to-SQLs
Authors: name: Jinyang Li, name: Binyuan Hui, name: Ge Qu, name: Binhua Li, name: Jiaxi Yang, name: Bowen Li, name: Bailin Wang, name: Bowen Qin, name: Rongyu Cao, name: Ruiying Geng, name: Nan Huo, name: Chenhao Ma, name: Kevin C. C. Chang, name: Fei Huang, name: Reynold Cheng, name: Yongbin Li
Published: 2023-05-04 19:02:29

Text-to-SQL parsing, which aims at converting natural language instructions
into executable SQLs, has gained increasing attention in recent years. In
particular, Codex and ChatGPT have shown impressive results in this task.
However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on
database schema with few rows of database contents leaving the gap between
academic study and real-world applications. To mitigate this gap, we present
Bird, a big benchmark for large-scale database grounded in text-to-SQL tasks,
containing 12,751 pairs of text-to-SQL data and 95 databases with a total size
of 33.4 GB, spanning 37 professional domains. Our emphasis on database values
highlights the new challenges of dirty database contents, external knowledge
between NL questions and database contents, and SQL efficiency, particularly in
the context of massive databases. To solve these problems, text-to-SQL models
must feature database value comprehension in addition to semantic parsing. The
experimental results demonstrate the significance of database values in
generating accurate text-to-SQLs for big databases. Furthermore, even the most
effective text-to-SQL models, i.e. ChatGPT, only achieves 40.08% in execution
accuracy, which is still far from the human result of 92.96%, proving that
challenges still stand. Besides, we also provide an efficiency analysis to
offer insights into generating text-to-efficient-SQLs that are beneficial to
industries. We believe that BIRD will contribute to advancing real-world
applications of text-to-SQL research. The leaderboard and source code are
available: https://bird-bench.github.io/.
Diffusion Explainer: Visual Explanation for Text-to-image Stable
  Diffusion
Authors: name: Seongmin Lee, name: Benjamin Hoover, name: Hendrik Strobelt, name: Zijie J. Wang, name: ShengYun Peng, name: Austin Wright, name: Kevin Li, name: Haekyu Park, name: Haoyang Yang, name: Duen Horng Chau
Published: 2023-05-04 16:14:43

Diffusion-based generative models' impressive ability to create convincing
images has captured global attention. However, their complex internal
structures and operations often make them difficult for non-experts to
understand. We present Diffusion Explainer, the first interactive visualization
tool that explains how Stable Diffusion transforms text prompts into images.
Diffusion Explainer tightly integrates a visual overview of Stable Diffusion's
complex components with detailed explanations of their underlying operations,
enabling users to fluidly transition between multiple levels of abstraction
through animations and interactive elements. By comparing the evolutions of
image representations guided by two related text prompts over refinement
timesteps, users can discover the impact of prompts on image generation.
Diffusion Explainer runs locally in users' web browsers without the need for
installation or specialized hardware, broadening the public's education access
to modern AI techniques. Our open-sourced tool is available at:
https://poloclub.github.io/diffusion-explainer/.
NeuralEditor: Editing Neural Radiance Fields via Manipulating Point
  Clouds
Authors: name: Jun-Kun Chen, name: Jipeng Lyu, name: Yu-Xiong Wang
Published: 2023-05-04 17:59:40

This paper proposes NeuralEditor that enables neural radiance fields (NeRFs)
natively editable for general shape editing tasks. Despite their impressive
results on novel-view synthesis, it remains a fundamental challenge for NeRFs
to edit the shape of the scene. Our key insight is to exploit the explicit
point cloud representation as the underlying structure to construct NeRFs,
inspired by the intuitive interpretation of NeRF rendering as a process that
projects or "plots" the associated 3D point cloud to a 2D image plane. To this
end, NeuralEditor introduces a novel rendering scheme based on deterministic
integration within K-D tree-guided density-adaptive voxels, which produces both
high-quality rendering results and precise point clouds through optimization.
NeuralEditor then performs shape editing via mapping associated points between
point clouds. Extensive evaluation shows that NeuralEditor achieves
state-of-the-art performance in both shape deformation and scene morphing
tasks. Notably, NeuralEditor supports both zero-shot inference and further
fine-tuning over the edited scene. Our code, benchmark, and demo video are
available at https://immortalco.github.io/NeuralEditor.
Single-Shot Implicit Morphable Faces with Consistent Texture
  Parameterization
Authors: name: Connor Z. Lin, name: Koki Nagano, name: Jan Kautz, name: Eric R. Chan, name: Umar Iqbal, name: Leonidas Guibas, name: Gordon Wetzstein, name: Sameh Khamis
Published: 2023-05-04 17:58:40

There is a growing demand for the accessible creation of high-quality 3D
avatars that are animatable and customizable. Although 3D morphable models
provide intuitive control for editing and animation, and robustness for
single-view face reconstruction, they cannot easily capture geometric and
appearance details. Methods based on neural implicit representations, such as
signed distance functions (SDF) or neural radiance fields, approach
photo-realism, but are difficult to animate and do not generalize well to
unseen data. To tackle this problem, we propose a novel method for constructing
implicit 3D morphable face models that are both generalizable and intuitive for
editing. Trained from a collection of high-quality 3D scans, our face model is
parameterized by geometry, expression, and texture latent codes with a learned
SDF and explicit UV texture parameterization. Once trained, we can reconstruct
an avatar from a single in-the-wild image by leveraging the learned prior to
project the image into the latent space of our model. Our implicit morphable
face models can be used to render an avatar from novel views, animate facial
expressions by modifying expression codes, and edit textures by directly
painting on the learned UV-texture maps. We demonstrate quantitatively and
qualitatively that our method improves upon photo-realism, geometry, and
expression accuracy compared to state-of-the-art methods.
TUVF: Learning Generalizable Texture UV Radiance Fields
Authors: name: An-Chieh Cheng, name: Xueting Li, name: Sifei Liu, name: Xiaolong Wang
Published: 2023-05-04 17:58:05

Textures are a vital aspect of creating visually appealing and realistic 3D
models. In this paper, we study the problem of generating high-fidelity texture
given shapes of 3D assets, which has been relatively less explored compared
with generic 3D shape modeling. Our goal is to facilitate a controllable
texture generation process, such that one texture code can correspond to a
particular appearance style independent of any input shapes from a category. We
introduce Texture UV Radiance Fields (TUVF) that generate textures in a
learnable UV sphere space rather than directly on the 3D shape. This allows the
texture to be disentangled from the underlying shape and transferable to other
shapes that share the same UV space, i.e., from the same category. We integrate
the UV sphere space with the radiance field, which provides a more efficient
and accurate representation of textures than traditional texture maps. We
perform our experiments on real-world object datasets where we achieve not only
realistic synthesis but also substantial improvements over state-of-the-arts on
texture controlling and editing. Project Page: https://www.anjiecheng.me/TUVF
NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads
Authors: name: Tobias Kirschstein, name: Shenhan Qian, name: Simon Giebenhain, name: Tim Walter, name: Matthias Nießner
Published: 2023-05-04 17:52:18

We focus on reconstructing high-fidelity radiance fields of human heads,
capturing their animations over time, and synthesizing re-renderings from novel
viewpoints at arbitrary time steps. To this end, we propose a new multi-view
capture setup composed of 16 calibrated machine vision cameras that record
time-synchronized images at 7.1 MP resolution and 73 frames per second. With
our setup, we collect a new dataset of over 4700 high-resolution,
high-framerate sequences of more than 220 human heads, from which we introduce
a new human head reconstruction benchmark. The recorded sequences cover a wide
range of facial dynamics, including head motions, natural expressions,
emotions, and spoken language. In order to reconstruct high-fidelity human
heads, we propose Dynamic Neural Radiance Fields using Hash Ensembles
(NeRSemble). We represent scene dynamics by combining a deformation field and
an ensemble of 3D multi-resolution hash encodings. The deformation field allows
for precise modeling of simple scene movements, while the ensemble of hash
encodings helps to represent complex dynamics. As a result, we obtain radiance
field representations of human heads that capture motion over time and
facilitate re-rendering of arbitrary novel viewpoints. In a series of
experiments, we explore the design choices of our method and demonstrate that
our approach outperforms state-of-the-art dynamic radiance field approaches by
a significant margin.
Masked Trajectory Models for Prediction, Representation, and Control
Authors: name: Philipp Wu, name: Arjun Majumdar, name: Kevin Stone, name: Yixin Lin, name: Igor Mordatch, name: Pieter Abbeel, name: Aravind Rajeswaran
Published: 2023-05-04 16:12:19

We introduce Masked Trajectory Models (MTM) as a generic abstraction for
sequential decision making. MTM takes a trajectory, such as a state-action
sequence, and aims to reconstruct the trajectory conditioned on random subsets
of the same trajectory. By training with a highly randomized masking pattern,
MTM learns versatile networks that can take on different roles or capabilities,
by simply choosing appropriate masks at inference time. For example, the same
MTM network can be used as a forward dynamics model, inverse dynamics model, or
even an offline RL agent. Through extensive experiments in several continuous
control tasks, we show that the same MTM network -- i.e. same weights -- can
match or outperform specialized networks trained for the aforementioned
capabilities. Additionally, we find that state representations learned by MTM
can significantly accelerate the learning speed of traditional RL algorithms.
Finally, in offline RL benchmarks, we find that MTM is competitive with
specialized offline RL algorithms, despite MTM being a generic self-supervised
learning method without any explicit RL components. Code is available at
https://github.com/facebookresearch/mtm
Real-Time Neural Appearance Models
Authors: name: Tizian Zeltner, name: Fabrice Rousselle, name: Andrea Weidlich, name: Petrik Clarberg, name: Jan Novák, name: Benedikt Bitterli, name: Alex Evans, name: Tomáš Davidovič, name: Simon Kallweit, name: Aaron Lefohn
Published: 2023-05-04 09:48:39

We present a complete system for real-time rendering of scenes with complex
appearance previously reserved for offline use. This is achieved with a
combination of algorithmic and system level innovations.
  Our appearance model utilizes learned hierarchical textures that are
interpreted using neural decoders, which produce reflectance values and
importance-sampled directions. To best utilize the modeling capacity of the
decoders, we equip the decoders with two graphics priors. The first prior --
transformation of directions into learned shading frames -- facilitates
accurate reconstruction of mesoscale effects. The second prior -- a microfacet
sampling distribution -- allows the neural decoder to perform importance
sampling efficiently. The resulting appearance model supports anisotropic
sampling and level-of-detail rendering, and allows baking deeply layered
material graphs into a compact unified neural representation.
  By exposing hardware accelerated tensor operations to ray tracing shaders, we
show that it is possible to inline and execute the neural decoders efficiently
inside a real-time path tracer. We analyze scalability with increasing number
of neural materials and propose to improve performance using code optimized for
coherent and divergent execution. Our neural material shaders can be over an
order of magnitude faster than non-neural layered materials. This opens up the
door for using film-quality visuals in real-time applications such as games and
live previews.
Tracking through Containers and Occluders in the Wild
Authors: name: Basile Van Hoorick, name: Pavel Tokmakov, name: Simon Stent, name: Jie Li, name: Carl Vondrick
Published: 2023-05-04 17:59:58

Tracking objects with persistence in cluttered and dynamic environments
remains a difficult challenge for computer vision systems. In this paper, we
introduce TCOW, a new benchmark and model for visual tracking
through heavy occlusion and containment. We set up a task where the goal is to,
given a video sequence, segment both the projected extent of the target object,
as well as the surrounding container or occluder whenever one exists. To study
this task, we create a mixture of synthetic and annotated real datasets to
support both supervised learning and structured evaluation of model performance
under various forms of task variation, such as moving or nested containment. We
evaluate two recent transformer-based video models and find that while they can
be surprisingly capable of tracking targets under certain settings of task
variation, there remains a considerable performance gap before we can claim a
tracking model to have acquired a true notion of object permanence.
BranchNorm: Robustly Scaling Extremely Deep Transformers
Authors: name: Yijin Liu, name: Xianfeng Zeng, name: Fandong Meng, name: Jie Zhou
Published: 2023-05-04 12:46:12

Recently, DeepNorm scales Transformers into extremely deep (i.e., 1000
layers) and reveals the promising potential of deep scaling. To stabilize the
training of deep models, DeepNorm (Wang et al., 2022) attempts to constrain the
model update to a constant value. Although applying such a constraint can
benefit the early stage of model training, it may lead to undertrained models
during the whole training procedure. In this paper, we propose BranchNorm,
which dynamically rescales the non-residual branch of Transformer in accordance
with the training period. BranchNorm not only theoretically stabilizes the
training with smooth gradient norms at the early stage, but also encourages
better convergence in the subsequent training stage. Experiment results on
multiple translation tasks demonstrate that BranchNorm achieves a better
trade-off between training stability and converge performance.
Learning Language-Specific Layers for Multilingual Machine Translation
Authors: name: Telmo Pessoa Pires, name: Robin M. Schmidt, name: Yi-Hsiu Liao, name: Stephan Peitz
Published: 2023-05-04 09:18:05

Multilingual Machine Translation promises to improve translation quality
between non-English languages. This is advantageous for several reasons, namely
lower latency (no need to translate twice), and reduced error cascades (e.g.,
avoiding losing gender and formality information when translating through
English). On the downside, adding more languages reduces model capacity per
language, which is usually countered by increasing the overall model size,
making training harder and inference slower. In this work, we introduce
Language-Specific Transformer Layers (LSLs), which allow us to increase model
capacity, while keeping the amount of computation and the number of parameters
used in the forward pass constant. The key idea is to have some layers of the
encoder be source or target language-specific, while keeping the remaining
layers shared. We study the best way to place these layers using a neural
architecture search inspired approach, and achieve an improvement of 1.3 chrF
(1.5 spBLEU) points over not using LSLs on a separate decoder architecture, and
1.9 chrF (2.2 spBLEU) on a shared decoder one.
Plan, Eliminate, and Track -- Language Models are Good Teachers for
  Embodied Agents
Authors: name: Yue Wu, name: So Yeon Min, name: Yonatan Bisk, name: Ruslan Salakhutdinov, name: Amos Azaria, name: Yuanzhi Li, name: Tom Mitchell, name: Shrimai Prabhumoye
Published: 2023-05-03 20:11:22

Pre-trained large language models (LLMs) capture procedural knowledge about
the world. Recent work has leveraged LLM's ability to generate abstract plans
to simplify challenging control tasks, either by action scoring, or action
modeling (fine-tuning). However, the transformer architecture inherits several
constraints that make it difficult for the LLM to directly serve as the agent:
e.g. limited input lengths, fine-tuning inefficiency, bias from pre-training,
and incompatibility with non-text environments. To maintain compatibility with
a low-level trainable actor, we propose to instead use the knowledge in LLMs to
simplify the control problem, rather than solving it. We propose the Plan,
Eliminate, and Track (PET) framework. The Plan module translates a task
description into a list of high-level sub-tasks. The Eliminate module masks out
irrelevant objects and receptacles from the observation for the current
sub-task. Finally, the Track module determines whether the agent has
accomplished each sub-task. On the AlfWorld instruction following benchmark,
the PET framework leads to a significant 15% improvement over SOTA for
generalization to human goal specifications.
Automated Code generation for Information Technology Tasks in YAML
  through Large Language Models
Authors: name: Saurabh Pujar, name: Luca Buratti, name: Xiaojie Guo, name: Nicolas Dupuis, name: Burn Lewis, name: Sahil Suneja, name: Atin Sood, name: Ganesh Nalawade, name: Matt Jones, name: Alessandro Morari, name: Ruchir Puri
Published: 2023-05-02 21:01:01

The recent improvement in code generation capabilities due to the use of
large language models has mainly benefited general purpose programming
languages. Domain specific languages, such as the ones used for IT Automation,
have received far less attention, despite involving many active developers and
being an essential component of modern cloud platforms. This work focuses on
the generation of Ansible-YAML, a widely used markup language for IT
Automation. We present Ansible Wisdom, a natural-language to Ansible-YAML code
generation tool, aimed at improving IT automation productivity. Ansible Wisdom
is a transformer-based model, extended by training with a new dataset
containing Ansible-YAML. We also develop two novel performance metrics for YAML
and Ansible to capture the specific characteristics of this domain. Results
show that Ansible Wisdom can accurately generate Ansible script from natural
language prompts with performance comparable or better than existing state of
the art code generation models.