Orionnnn/hub-stats huggingface dataset:
Relightify: Relightable 3D Faces from a Single Image via Diffusion
  Models
Authors: name: Foivos Paraperas Papantoniou, name: Alexandros Lattas, name: Stylianos Moschoglou, name: Stefanos Zafeiriou
Published: 2023-05-10 11:57:49

Following the remarkable success of diffusion models on image generation,
recent works have also demonstrated their impressive ability to address a
number of inverse problems in an unsupervised way, by properly constraining the
sampling process based on a conditioning input. Motivated by this, in this
paper, we present the first approach to use diffusion models as a prior for
highly accurate 3D facial BRDF reconstruction from a single image. We start by
leveraging a high-quality UV dataset of facial reflectance (diffuse and
specular albedo and normals), which we render under varying illumination
settings to simulate natural RGB textures and, then, train an unconditional
diffusion model on concatenated pairs of rendered textures and reflectance
components. At test time, we fit a 3D morphable model to the given image and
unwrap the face in a partial UV texture. By sampling from the diffusion model,
while retaining the observed texture part intact, the model inpaints not only
the self-occluded areas but also the unknown reflectance components, in a
single sequence of denoising steps. In contrast to existing methods, we
directly acquire the observed texture from the input image, thus, resulting in
more faithful and consistent reflectance estimation. Through a series of
qualitative and quantitative comparisons, we demonstrate superior performance
in both texture completion as well as reflectance reconstruction tasks.
To Compress or Not to Compress- Self-Supervised Learning and Information
  Theory: A Review
Authors: name: Ravid Shwartz-Ziv, name: Yann LeCun
Published: 2023-04-19 00:33:59

Deep neural networks have demonstrated remarkable performance in supervised
learning tasks but require large amounts of labeled data. Self-supervised
learning offers an alternative paradigm, enabling the model to learn from data
without explicit labels. Information theory has been instrumental in
understanding and optimizing deep neural networks. Specifically, the
information bottleneck principle has been applied to optimize the trade-off
between compression and relevant information preservation in supervised
settings. However, the optimal information objective in self-supervised
learning remains unclear. In this paper, we review various approaches to
self-supervised learning from an information-theoretic standpoint and present a
unified framework that formalizes the self-supervised information-theoretic
learning problem. We integrate existing research into a coherent framework,
examine recent self-supervised methods, and identify research opportunities and
challenges. Moreover, we discuss empirical measurement of information-theoretic
quantities and their estimators. This paper offers a comprehensive review of
the intersection between information theory, self-supervised learning, and deep
neural networks.
WikiWeb2M: A Page-Level Multimodal Wikipedia Dataset
Authors: name: Andrea Burns, name: Krishna Srinivasan, name: Joshua Ainslie, name: Geoff Brown, name: Bryan A. Plummer, name: Kate Saenko, name: Jianmo Ni, name: Mandy Guo
Published: 2023-05-09 13:20:59

Webpages have been a rich resource for language and vision-language tasks.
Yet only pieces of webpages are kept: image-caption pairs, long text articles,
or raw HTML, never all in one place. Webpage tasks have resultingly received
little attention and structured image-text data underused. To study multimodal
webpage understanding, we introduce the Wikipedia Webpage 2M (WikiWeb2M) suite;
the first to retain the full set of images, text, and structure data available
in a page. WikiWeb2M can be used for tasks like page description generation,
section summarization, and contextual image captioning.
NerfAcc: Efficient Sampling Accelerates NeRFs
Authors: name: Ruilong Li, name: Hang Gao, name: Matthew Tancik, name: Angjoo Kanazawa
Published: 2023-05-08 18:02:11

Optimizing and rendering Neural Radiance Fields is computationally expensive
due to the vast number of samples required by volume rendering. Recent works
have included alternative sampling approaches to help accelerate their methods,
however, they are often not the focus of the work. In this paper, we
investigate and compare multiple sampling approaches and demonstrate that
improved sampling is generally applicable across NeRF variants under an unified
concept of transmittance estimator. To facilitate future experiments, we
develop NerfAcc, a Python toolbox that provides flexible APIs for incorporating
advanced sampling methods into NeRF related methods. We demonstrate its
flexibility by showing that it can reduce the training time of several recent
NeRF methods by 1.5x to 20x with minimal modifications to the existing
codebase. Additionally, highly customized NeRFs, such as Instant-NGP, can be
implemented in native PyTorch using NerfAcc.
AudioSlots: A slot-centric generative model for audio separation
Authors: name: Pradyumna Reddy, name: Scott Wisdom, name: Klaus Greff, name: John R. Hershey, name: Thomas Kipf
Published: 2023-05-09 16:28:07

In a range of recent works, object-centric architectures have been shown to
be suitable for unsupervised scene decomposition in the vision domain. Inspired
by these methods we present AudioSlots, a slot-centric generative model for
blind source separation in the audio domain. AudioSlots is built using
permutation-equivariant encoder and decoder networks. The encoder network based
on the Transformer architecture learns to map a mixed audio spectrogram to an
unordered set of independent source embeddings. The spatial broadcast decoder
network learns to generate the source spectrograms from the source embeddings.
We train the model in an end-to-end manner using a permutation invariant loss
function. Our results on Libri2Mix speech separation constitute a proof of
concept that this approach shows promise. We discuss the results and
limitations of our approach in detail, and further outline potential ways to
overcome the limitations and directions for future work.
Recommender Systems with Generative Retrieval
Authors: name: Shashank Rajput, name: Nikhil Mehta, name: Anima Singh, name: Raghunandan H. Keshavan, name: Trung Vu, name: Lukasz Heldt, name: Lichan Hong, name: Yi Tay, name: Vinh Q. Tran, name: Jonah Samost, name: Maciej Kula, name: Ed H. Chi, name: Maheswaran Sathiamoorthy
Published: 2023-05-08 21:48:17

Modern recommender systems leverage large-scale retrieval models consisting
of two stages: training a dual-encoder model to embed queries and candidates in
the same space, followed by an Approximate Nearest Neighbor (ANN) search to
select top candidates given a query's embedding. In this paper, we propose a
new single-stage paradigm: a generative retrieval model which autoregressively
decodes the identifiers for the target candidates in one phase. To do this,
instead of assigning randomly generated atomic IDs to each item, we generate
Semantic IDs: a semantically meaningful tuple of codewords for each item that
serves as its unique identifier. We use a hierarchical method called RQ-VAE to
generate these codewords. Once we have the Semantic IDs for all the items, a
Transformer based sequence-to-sequence model is trained to predict the Semantic
ID of the next item. Since this model predicts the tuple of codewords
identifying the next item directly in an autoregressive manner, it can be
considered a generative retrieval model. We show that our recommender system
trained in this new paradigm improves the results achieved by current SOTA
models on the Amazon dataset. Moreover, we demonstrate that the
sequence-to-sequence model coupled with hierarchical Semantic IDs offers better
generalization and hence improves retrieval of cold-start items for
recommendations.
TidyBot: Personalized Robot Assistance with Large Language Models
Authors: name: Jimmy Wu, name: Rika Antonova, name: Adam Kan, name: Marion Lepert, name: Andy Zeng, name: Shuran Song, name: Jeannette Bohg, name: Szymon Rusinkiewicz, name: Thomas Funkhouser
Published: 2023-05-09 17:52:59

For a robot to personalize physical assistance effectively, it must learn
user preferences that can be generally reapplied to future scenarios. In this
work, we investigate personalization of household cleanup with robots that can
tidy up rooms by picking up objects and putting them away. A key challenge is
determining the proper place to put each object, as people's preferences can
vary greatly depending on personal taste or cultural background. For instance,
one person may prefer storing shirts in the drawer, while another may prefer
them on the shelf. We aim to build systems that can learn such preferences from
just a handful of examples via prior interactions with a particular person. We
show that robots can combine language-based planning and perception with the
few-shot summarization capabilities of large language models (LLMs) to infer
generalized user preferences that are broadly applicable to future
interactions. This approach enables fast adaptation and achieves 91.2% accuracy
on unseen objects in our benchmark dataset. We also demonstrate our approach on
a real-world mobile manipulator called TidyBot, which successfully puts away
85.0% of objects in real-world test scenarios.
Code Execution with Pre-trained Language Models
Authors: name: Chenxiao Liu, name: Shuai Lu, name: Weizhu Chen, name: Daxin Jiang, name: Alexey Svyatkovskiy, name: Shengyu Fu, name: Neel Sundaresan, name: Nan Duan
Published: 2023-05-08 10:00:05

Code execution is a fundamental aspect of programming language semantics that
reflects the exact behavior of the code. However, most pre-trained models for
code intelligence ignore the execution trace and only rely on source code and
syntactic structures. In this paper, we investigate how well pre-trained models
can understand and perform code execution. We develop a mutation-based data
augmentation technique to create a large-scale and realistic Python dataset and
task for code execution, which challenges existing models such as Codex. We
then present CodeExecutor, a Transformer model that leverages code execution
pre-training and curriculum learning to enhance its semantic comprehension. We
evaluate CodeExecutor on code execution and show its promising performance and
limitations. We also demonstrate its potential benefits for code intelligence
tasks such as zero-shot code-to-code search and text-to-code generation. Our
analysis provides insights into the learning and generalization abilities of
pre-trained models for code execution.
Large Language Model Programs
Authors: name: Imanol Schlag, name: Sainbayar Sukhbaatar, name: Asli Celikyilmaz, name: Wen-tau Yih, name: Jason Weston, name: Jürgen Schmidhuber, name: Xian Li
Published: 2023-05-09 11:55:36

In recent years, large pre-trained language models (LLMs) have demonstrated
the ability to follow instructions and perform novel tasks from a few examples.
The possibility to parameterise an LLM through such in-context examples widens
their capability at a much lower cost than finetuning. We extend this line of
reasoning and present a method which further expands the capabilities of an LLM
by embedding it within an algorithm or program. To demonstrate the benefits of
this approach, we present an illustrative example of evidence-supported
question-answering. We obtain a 6.4\% improvement over the chain of thought
baseline through a more algorithmic approach without any finetuning.
Furthermore, we highlight recent work from this perspective and discuss the
advantages and disadvantages in comparison to the standard approaches.
InternChat: Solving Vision-Centric Tasks by Interacting with Chatbots
  Beyond Language
Authors: name: Zhaoyang Liu, name: Yinan He, name: Wenhai Wang, name: Weiyun Wang, name: Yi Wang, name: Shoufa Chen, name: Qinglong Zhang, name: Yang Yang, name: Qingyun Li, name: Jiashuo Yu, name: Kunchang Li, name: Zhe Chen, name: Xue Yang, name: Xizhou Zhu, name: Yali Wang, name: Limin Wang, name: Ping Luo, name: Jifeng Dai, name: Yu Qiao
Published: 2023-05-09 17:58:34

We present an interactive visual framework named InternChat, or iChat for
short. The framework integrates chatbots that have planning and reasoning
capabilities, such as ChatGPT, with non-verbal instructions like pointing
movements that enable users to directly manipulate images or videos on the
screen. Pointing (including gestures, cursors, etc.) movements can provide more
flexibility and precision in performing vision-centric tasks that require
fine-grained control, editing, and generation of visual content. The name
InternChat stands for interaction, nonverbal, and chatbots. Different from
existing interactive systems that rely on pure language, by incorporating
pointing instructions, the proposed iChat significantly improves the efficiency
of communication between users and chatbots, as well as the accuracy of
chatbots in vision-centric tasks, especially in complicated visual scenarios
where the number of objects is greater than 2. Additionally, in iChat, an
auxiliary control mechanism is used to improve the control capability of LLM,
and a large vision-language model termed Husky is fine-tuned for high-quality
multi-modal dialogue (impressing ChatGPT-3.5-turbo with 93.89% GPT-4 Quality).
We hope this work can spark new ideas and directions for future interactive
visual systems. Welcome to watch the code at
https://github.com/OpenGVLab/InternChat.
Towards Building the Federated GPT: Federated Instruction Tuning
Authors: name: Jianyi Zhang, name: Saeed Vahidian, name: Martin Kuo, name: Chunyuan Li, name: Ruiyi Zhang, name: Guoyin Wang, name: Yiran Chen
Published: 2023-05-09 17:42:34

While ``instruction-tuned" generative large language models (LLMs) have
demonstrated an impressive ability to generalize to new tasks, the training
phases heavily rely on large amounts of diverse and high-quality instruction
data (such as ChatGPT and GPT-4). Unfortunately, acquiring high-quality data,
especially when it comes to human-written data, can pose significant challenges
both in terms of cost and accessibility. Moreover, concerns related to privacy
can further limit access to such data, making the process of obtaining it a
complex and nuanced undertaking. Consequently, this hinders the generality of
the tuned models and may restrict their effectiveness in certain contexts. To
tackle this issue, our study introduces a new approach called Federated
Instruction Tuning (FedIT), which leverages federated learning (FL) as the
learning framework for the instruction tuning of LLMs. This marks the first
exploration of FL-based instruction tuning for LLMs. This is especially
important since text data is predominantly generated by end users. Therefore,
it is imperative to design and adapt FL approaches to effectively leverage
these users' diverse instructions stored on local devices, while preserving
privacy and ensuring data security. In the current paper, by conducting widely
used GPT-4 auto-evaluation, we demonstrate that by exploiting the heterogeneous
and diverse sets of instructions on the client's end with the proposed
framework FedIT, we improved the performance of LLMs compared to centralized
training with only limited local instructions. Further, in this paper, we
developed a Github repository named Shepherd. This repository offers a
foundational framework for exploring federated fine-tuning of LLMs using
heterogeneous instructions across diverse categories.
FrugalGPT: How to Use Large Language Models While Reducing Cost and
  Improving Performance
Authors: name: Lingjiao Chen, name: Matei Zaharia, name: James Zou
Published: 2023-05-09 05:11:02

There is a rapidly growing number of large language models (LLMs) that users
can query for a fee. We review the cost associated with querying popular LLM
APIs, e.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have
heterogeneous pricing structures, with fees that can differ by two orders of
magnitude. In particular, using LLMs on large collections of queries and text
can be expensive. Motivated by this, we outline and discuss three types of
strategies that users can exploit to reduce the inference cost associated with
using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As
an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM
cascade which learns which combinations of LLMs to use for different queries in
order to reduce cost and improve accuracy. Our experiments show that FrugalGPT
can match the performance of the best individual LLM (e.g. GPT-4) with up to
98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost.
The ideas and findings presented here lay a foundation for using LLMs
sustainably and efficiently.
SUR-adapter: Enhancing Text-to-Image Pre-trained Diffusion Models with
  Large Language Models
Authors: name: Shanshan Zhong, name: Zhongzhan Huang, name: Wushao Wen, name: Jinghui Qin, name: Liang Lin
Published: 2023-05-09 05:48:38

Diffusion models, which have emerged to become popular text-to-image
generation models, can produce high-quality and content-rich images guided by
textual prompts. However, there are limitations to semantic understanding and
commonsense reasoning in existing models when the input prompts are concise
narrative, resulting in low-quality image generation. To improve the capacities
for narrative prompts, we propose a simple-yet-effective parameter-efficient
fine-tuning approach called the Semantic Understanding and Reasoning adapter
(SUR-adapter) for pre-trained diffusion models. To reach this goal, we first
collect and annotate a new dataset SURD which consists of more than 57,000
semantically corrected multi-modal samples. Each sample contains a simple
narrative prompt, a complex keyword-based prompt, and a high-quality image.
Then, we align the semantic representation of narrative prompts to the complex
prompts and transfer knowledge of large language models (LLMs) to our
SUR-adapter via knowledge distillation so that it can acquire the powerful
semantic understanding and reasoning capabilities to build a high-quality
textual semantic representation for text-to-image generation. We conduct
experiments by integrating multiple LLMs and popular pre-trained diffusion
models to show the effectiveness of our approach in enabling diffusion models
to understand and reason concise natural language without image quality
degradation. Our approach can make text-to-image diffusion models easier to use
with better user experience, which demonstrates our approach has the potential
for further advancing the development of user-friendly text-to-image generation
models by bridging the semantic gap between simple narrative prompts and
complex keyword-based prompts.
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning
  by Large Language Models
Authors: name: Lei Wang, name: Wanyu Xu, name: Yihuai Lan, name: Zhiqiang Hu, name: Yunshi Lan, name: Roy Ka-Wei Lee, name: Ee-Peng Lim
Published: 2023-05-06 16:34:37

Large language models (LLMs) have recently been shown to deliver impressive
performance in various NLP tasks. To tackle multi-step reasoning tasks,
few-shot chain-of-thought (CoT) prompting includes a few manually crafted
step-by-step reasoning demonstrations which enable LLMs to explicitly generate
reasoning steps and improve their reasoning task accuracy. To eliminate the
manual effort, Zero-shot-CoT concatenates the target problem statement with
"Let's think step by step" as an input prompt to LLMs. Despite the success of
Zero-shot-CoT, it still suffers from three pitfalls: calculation errors,
missing-step errors, and semantic misunderstanding errors. To address the
missing-step errors, we propose Plan-and-Solve (PS) Prompting. It consists of
two components: first, devising a plan to divide the entire task into smaller
subtasks, and then carrying out the subtasks according to the plan. To address
the calculation errors and improve the quality of generated reasoning steps, we
extend PS prompting with more detailed instructions and derive PS+ prompting.
We evaluate our proposed prompting strategy on ten datasets across three
reasoning problems. The experimental results over GPT-3 show that our proposed
zero-shot prompting consistently outperforms Zero-shot-CoT across all datasets
by a large margin, is comparable to or exceeds Zero-shot-Program-of-Thought
Prompting, and has comparable performance with 8-shot CoT prompting on the math
reasoning problem. The code can be found at
https://github.com/AGI-Edgerunners/Plan-and-Solve-Prompting.
AvatarReX: Real-time Expressive Full-body Avatars
Authors: name: Zerong Zheng, name: Xiaochen Zhao, name: Hongwen Zhang, name: Boning Liu, name: Yebin Liu
Published: 2023-05-08 15:43:00

We present AvatarReX, a new method for learning NeRF-based full-body avatars
from video data. The learnt avatar not only provides expressive control of the
body, hands and the face together, but also supports real-time animation and
rendering. To this end, we propose a compositional avatar representation, where
the body, hands and the face are separately modeled in a way that the
structural prior from parametric mesh templates is properly utilized without
compromising representation flexibility. Furthermore, we disentangle the
geometry and appearance for each part. With these technical designs, we propose
a dedicated deferred rendering pipeline, which can be executed in real-time
framerate to synthesize high-quality free-view images. The disentanglement of
geometry and appearance also allows us to design a two-pass training strategy
that combines volume rendering and surface rendering for network training. In
this way, patch-level supervision can be applied to force the network to learn
sharp appearance details on the basis of geometry estimation. Overall, our
method enables automatic construction of expressive full-body avatars with
real-time rendering capability, and can generate photo-realistic images with
dynamic details for novel body motions and facial expressions.
Multi-Space Neural Radiance Fields
Authors: name: Ze-Xin Yin, name: Jiaxiong Qiu, name: Ming-Ming Cheng, name: Bo Ren
Published: 2023-05-07 13:11:07

Existing Neural Radiance Fields (NeRF) methods suffer from the existence of
reflective objects, often resulting in blurry or distorted rendering. Instead
of calculating a single radiance field, we propose a multi-space neural
radiance field (MS-NeRF) that represents the scene using a group of feature
fields in parallel sub-spaces, which leads to a better understanding of the
neural network toward the existence of reflective and refractive objects. Our
multi-space scheme works as an enhancement to existing NeRF methods, with only
small computational overheads needed for training and inferring the extra-space
outputs. We demonstrate the superiority and compatibility of our approach using
three representative NeRF-based models, i.e., NeRF, Mip-NeRF, and Mip-NeRF 360.
Comparisons are performed on a novelly constructed dataset consisting of 25
synthetic scenes and 7 real captured scenes with complex reflection and
refraction, all having 360-degree viewpoints. Extensive experiments show that
our approach significantly outperforms the existing single-space NeRF methods
for rendering high-quality scenes concerned with complex light paths through
mirror-like objects. Our code and dataset will be publicly available at
https://zx-yin.github.io/msnerf.