Orionnnn/hub-stats huggingface dataset:
Stable Score Distillation for High-Quality 3D Generation
Authors: name: Boshi Tang, name: Jianan Wang, name: Zhiyong Wu, name: Lei Zhang
Published: 2023-12-14 19:18:38

Score Distillation Sampling (SDS) has exhibited remarkable performance in
conditional 3D content generation. However, a comprehensive understanding of
the SDS formulation is still lacking, hindering the development of 3D
generation. In this work, we present an interpretation of SDS as a combination
of three functional components: mode-disengaging, mode-seeking and
variance-reducing terms, and analyze the properties of each. We show that
problems such as over-smoothness and color-saturation result from the intrinsic
deficiency of the supervision terms and reveal that the variance-reducing term
introduced by SDS is sub-optimal. Additionally, we shed light on the adoption
of large Classifier-Free Guidance (CFG) scale for 3D generation. Based on the
analysis, we propose a simple yet effective approach named Stable Score
Distillation (SSD) which strategically orchestrates each term for high-quality
3D generation. Extensive experiments validate the efficacy of our approach,
demonstrating its ability to generate high-fidelity 3D content without
succumbing to issues such as over-smoothness and over-saturation, even under
low CFG conditions with the most challenging NeRF representation.
Challenges with unsupervised LLM knowledge discovery
Authors: name: Sebastian Farquhar, name: Vikrant Varma, name: Zachary Kenton, name: Johannes Gasteiger, name: Vladimir Mikulik, name: Rohin Shah
Published: 2023-12-15 18:49:43

We show that existing unsupervised methods on large language model (LLM)
activations do not discover knowledge -- instead they seem to discover whatever
feature of the activations is most prominent. The idea behind unsupervised
knowledge elicitation is that knowledge satisfies a consistency structure,
which can be used to discover knowledge. We first prove theoretically that
arbitrary features (not just knowledge) satisfy the consistency structure of a
particular leading unsupervised knowledge-elicitation method,
contrast-consistent search (Burns et al. - arXiv:2212.03827). We then present a
series of experiments showing settings in which unsupervised methods result in
classifiers that do not predict knowledge, but instead predict a different
prominent feature. We conclude that existing unsupervised methods for
discovering latent knowledge are insufficient, and we contribute sanity checks
to apply to evaluating future knowledge elicitation methods. Conceptually, we
hypothesise that the identification issues explored here, e.g. distinguishing a
model's knowledge from that of a simulated character's, will persist for future
unsupervised methods.
Faithful Persona-based Conversational Dataset Generation with Large
  Language Models
Authors: name: Pegah Jandaghi, name: XiangHai Sheng, name: Xinyi Bai, name: Jay Pujara, name: Hakim Sidahmed
Published: 2023-12-15 18:23:50

High-quality conversational datasets are essential for developing AI models
that can communicate with users. One way to foster deeper interactions between
a chatbot and its user is through personas, aspects of the user's character
that provide insights into their personality, motivations, and behaviors.
Training Natural Language Processing (NLP) models on a diverse and
comprehensive persona-based dataset can lead to conversational models that
create a deeper connection with the user, and maintain their engagement. In
this paper, we leverage the power of Large Language Models (LLMs) to create a
large, high-quality conversational dataset from a seed dataset. We propose a
Generator-Critic architecture framework to expand the initial dataset, while
improving the quality of its conversations. The Generator is an LLM prompted to
output conversations. The Critic consists of a mixture of expert LLMs that
control the quality of the generated conversations. These experts select the
best generated conversations, which we then use to improve the Generator. We
release Synthetic-Persona-Chat, consisting of 20k conversations seeded from
Persona-Chat. We evaluate the quality of Synthetic-Persona-Chat and our
generation framework on different dimensions through extensive experiments, and
observe that the losing rate of Synthetic-Persona-Chat against Persona-Chat
during Turing test decreases from 17.2% to 8.8% over three iterations.
ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent
Authors: name: Renat Aksitov, name: Sobhan Miryoosefi, name: Zonglin Li, name: Daliang Li, name: Sheila Babayan, name: Kavya Kopparapu, name: Zachary Fisher, name: Ruiqi Guo, name: Sushant Prakash, name: Pranesh Srinivasan, name: Manzil Zaheer, name: Felix Yu, name: Sanjiv Kumar
Published: 2023-12-15 18:20:15

Answering complex natural language questions often necessitates multi-step
reasoning and integrating external information. Several systems have combined
knowledge retrieval with a large language model (LLM) to answer such questions.
These systems, however, suffer from various failure cases, and we cannot
directly train them end-to-end to fix such failures, as interaction with
external knowledge is non-differentiable. To address these deficiencies, we
define a ReAct-style LLM agent with the ability to reason and act upon external
knowledge. We further refine the agent through a ReST-like method that
iteratively trains on previous trajectories, employing growing-batch
reinforcement learning with AI feedback for continuous self-improvement and
self-distillation. Starting from a prompted large model and after just two
iterations of the algorithm, we can produce a fine-tuned small model that
achieves comparable performance on challenging compositional question-answering
benchmarks with two orders of magnitude fewer parameters.
Self-Evaluation Improves Selective Generation in Large Language Models
Authors: name: Jie Ren, name: Yao Zhao, name: Tu Vu, name: Peter J. Liu, name: Balaji Lakshminarayanan
Published: 2023-12-14 19:09:22

Safe deployment of large language models (LLMs) may benefit from a reliable
method for assessing their generated content to determine when to abstain or to
selectively generate. While likelihood-based metrics such as perplexity are
widely employed, recent research has demonstrated the limitations of using
sequence-level probability estimates given by LLMs as reliable indicators of
generation quality. Conversely, LLMs have demonstrated strong calibration at
the token level, particularly when it comes to choosing correct answers in
multiple-choice questions or evaluating true/false statements. In this work, we
reformulate open-ended generation tasks into token-level prediction tasks, and
leverage LLMs' superior calibration at the token level. We instruct an LLM to
self-evaluate its answers, employing either a multi-way comparison or a
point-wise evaluation approach, with the option to include a ``None of the
above'' option to express the model's uncertainty explicitly. We benchmark a
range of scoring methods based on self-evaluation and evaluate their
performance in selective generation using TruthfulQA and TL;DR. Through
experiments with PaLM-2 and GPT-3, we demonstrate that self-evaluation based
scores not only improve accuracy, but also correlate better with the overall
quality of generated content.
Weight subcloning: direct initialization of transformers using larger
  pretrained ones
Authors: name: Mohammad Samragh, name: Mehrdad Farajtabar, name: Sachin Mehta, name: Raviteja Vemulapalli, name: Fartash Faghri, name: Devang Naik, name: Oncel Tuzel, name: Mohammad Rastegari
Published: 2023-12-14 19:08:56

Training large transformer models from scratch for a target task requires
lots of data and is computationally demanding. The usual practice of transfer
learning overcomes this challenge by initializing the model with weights of a
pretrained model of the same size and specification to increase the convergence
and training speed. However, what if no pretrained model of the required size
is available? In this paper, we introduce a simple yet effective technique to
transfer the knowledge of a pretrained model to smaller variants. Our approach
called weight subcloning expedites the training of scaled-down transformers by
initializing their weights from larger pretrained models.
  Weight subcloning involves an operation on the pretrained model to obtain the
equivalent initialized scaled-down model. It consists of two key steps: first,
we introduce neuron importance ranking to decrease the embedding dimension per
layer in the pretrained model. Then, we remove blocks from the transformer
model to match the number of layers in the scaled-down network. The result is a
network ready to undergo training, which gains significant improvements in
training speed compared to random initialization. For instance, we achieve 4x
faster training for vision transformers in image classification and language
models designed for next token prediction.
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak
  Supervision
Authors: name: Collin Burns, name: Pavel Izmailov, name: Jan Hendrik Kirchner, name: Bowen Baker, name: Leo Gao, name: Leopold Aschenbrenner, name: Yining Chen, name: Adrien Ecoffet, name: Manas Joglekar, name: Jan Leike, name: Ilya Sutskever, name: Jeff Wu
Published: 2023-12-14 23:07:33

Widely used alignment techniques, such as reinforcement learning from human
feedback (RLHF), rely on the ability of humans to supervise model behavior -
for example, to evaluate whether a model faithfully followed instructions or
generated safe outputs. However, future superhuman models will behave in
complex ways too difficult for humans to reliably evaluate; humans will only be
able to weakly supervise superhuman models. We study an analogy to this
problem: can weak model supervision elicit the full capabilities of a much
stronger model? We test this using a range of pretrained language models in the
GPT-4 family on natural language processing (NLP), chess, and reward modeling
tasks. We find that when we naively finetune strong pretrained models on labels
generated by a weak model, they consistently perform better than their weak
supervisors, a phenomenon we call weak-to-strong generalization. However, we
are still far from recovering the full capabilities of strong models with naive
finetuning alone, suggesting that techniques like RLHF may scale poorly to
superhuman models without further work. We find that simple methods can often
significantly improve weak-to-strong generalization: for example, when
finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence
loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our
results suggest that it is feasible to make empirical progress today on a
fundamental challenge of aligning superhuman models.
SHAP-EDITOR: Instruction-guided Latent 3D Editing in Seconds
Authors: name: Minghao Chen, name: Junyu Xie, name: Iro Laina, name: Andrea Vedaldi
Published: 2023-12-14 18:59:06

We propose a novel feed-forward 3D editing framework called Shap-Editor.
Prior research on editing 3D objects primarily concentrated on editing
individual objects by leveraging off-the-shelf 2D image editing networks. This
is achieved via a process called distillation, which transfers knowledge from
the 2D network to 3D assets. Distillation necessitates at least tens of minutes
per asset to attain satisfactory editing results, and is thus not very
practical. In contrast, we ask whether 3D editing can be carried out directly
by a feed-forward network, eschewing test-time optimisation. In particular, we
hypothesise that editing can be greatly simplified by first encoding 3D objects
in a suitable latent space. We validate this hypothesis by building upon the
latent space of Shap-E. We demonstrate that direct 3D editing in this space is
possible and efficient by building a feed-forward editor network that only
requires approximately one second per edit. Our experiments show that
Shap-Editor generalises well to both in-distribution and out-of-distribution 3D
assets with different prompts, exhibiting comparable performance with methods
that carry out test-time optimisation for each edited instance.
StemGen: A music generation model that listens
Authors: name: Julian D. Parker, name: Janne Spijkervet, name: Katerina Kosta, name: Furkan Yesiler, name: Boris Kuznetsov, name: Ju-Chiang Wang, name: Matt Avent, name: Jitong Chen, name: Duc Le
Published: 2023-12-14 08:09:20

End-to-end generation of musical audio using deep learning techniques has
seen an explosion of activity recently. However, most models concentrate on
generating fully mixed music in response to abstract conditioning information.
In this work, we present an alternative paradigm for producing music generation
models that can listen and respond to musical context. We describe how such a
model can be constructed using a non-autoregressive, transformer-based model
architecture and present a number of novel architectural and sampling
improvements. We train the described architecture on both an open-source and a
proprietary dataset. We evaluate the produced models using standard quality
metrics and a new approach based on music information retrieval descriptors.
The resulting model reaches the audio quality of state-of-the-art
text-conditioned models, as well as exhibiting strong musical coherence with
its context.
Mosaic-SDF for 3D Generative Models
Authors: name: Lior Yariv, name: Omri Puny, name: Natalia Neverova, name: Oran Gafni, name: Yaron Lipman
Published: 2023-12-14 18:52:52

Current diffusion or flow-based generative models for 3D shapes divide to
two: distilling pre-trained 2D image diffusion models, and training directly on
3D shapes. When training a diffusion or flow models on 3D shapes a crucial
design choice is the shape representation. An effective shape representation
needs to adhere three design principles: it should allow an efficient
conversion of large 3D datasets to the representation form; it should provide a
good tradeoff of approximation power versus number of parameters; and it should
have a simple tensorial form that is compatible with existing powerful neural
architectures. While standard 3D shape representations such as volumetric grids
and point clouds do not adhere to all these principles simultaneously, we
advocate in this paper a new representation that does. We introduce Mosaic-SDF
(M-SDF): a simple 3D shape representation that approximates the Signed Distance
Function (SDF) of a given shape by using a set of local grids spread near the
shape's boundary. The M-SDF representation is fast to compute for each shape
individually making it readily parallelizable; it is parameter efficient as it
only covers the space around the shape's boundary; and it has a simple matrix
form, compatible with Transformer-based architectures. We demonstrate the
efficacy of the M-SDF representation by using it to train a 3D generative flow
model including class-conditioned generation with the 3D Warehouse dataset, and
text-to-3D generation using a dataset of about 600k caption-shape pairs.
UniDream: Unifying Diffusion Priors for Relightable Text-to-3D
  Generation
Authors: name: Zexiang Liu, name: Yangguang Li, name: Youtian Lin, name: Xin Yu, name: Sida Peng, name: Yan-Pei Cao, name: Xiaojuan Qi, name: Xiaoshui Huang, name: Ding Liang, name: Wanli Ouyang
Published: 2023-12-14 09:07:37

Recent advancements in text-to-3D generation technology have significantly
advanced the conversion of textual descriptions into imaginative
well-geometrical and finely textured 3D objects. Despite these developments, a
prevalent limitation arises from the use of RGB data in diffusion or
reconstruction models, which often results in models with inherent lighting and
shadows effects that detract from their realism, thereby limiting their
usability in applications that demand accurate relighting capabilities. To
bridge this gap, we present UniDream, a text-to-3D generation framework by
incorporating unified diffusion priors. Our approach consists of three main
components: (1) a dual-phase training process to get albedo-normal aligned
multi-view diffusion and reconstruction models, (2) a progressive generation
procedure for geometry and albedo-textures based on Score Distillation Sample
(SDS) using the trained reconstruction and diffusion models, and (3) an
innovative application of SDS for finalizing PBR generation while keeping a
fixed albedo based on Stable Diffusion model. Extensive evaluations demonstrate
that UniDream surpasses existing methods in generating 3D objects with clearer
albedo textures, smoother surfaces, enhanced realism, and superior relighting
capabilities.
LIME: Localized Image Editing via Attention Regularization in Diffusion
  Models
Authors: name: Enis Simsar, name: Alessio Tonioni, name: Yongqin Xian, name: Thomas Hofmann, name: Federico Tombari
Published: 2023-12-14 18:59:59

Diffusion models (DMs) have gained prominence due to their ability to
generate high-quality, varied images, with recent advancements in text-to-image
generation. The research focus is now shifting towards the controllability of
DMs. A significant challenge within this domain is localized editing, where
specific areas of an image are modified without affecting the rest of the
content. This paper introduces LIME for localized image editing in diffusion
models that do not require user-specified regions of interest (RoI) or
additional text input. Our method employs features from pre-trained methods and
a simple clustering technique to obtain precise semantic segmentation maps.
Then, by leveraging cross-attention maps, it refines these segments for
localized edits. Finally, we propose a novel cross-attention regularization
technique that penalizes unrelated cross-attention scores in the RoI during the
denoising steps, ensuring localized edits. Our approach, without re-training
and fine-tuning, consistently improves the performance of existing methods in
various editing benchmarks.
FineControlNet: Fine-level Text Control for Image Generation with
  Spatially Aligned Text Control Injection
Authors: name: Hongsuk Choi, name: Isaac Kasahara, name: Selim Engin, name: Moritz Graule, name: Nikhil Chavan-Dafle, name: Volkan Isler
Published: 2023-12-14 18:59:43

Recently introduced ControlNet has the ability to steer the text-driven image
generation process with geometric input such as human 2D pose, or edge
features. While ControlNet provides control over the geometric form of the
instances in the generated image, it lacks the capability to dictate the visual
appearance of each instance. We present FineControlNet to provide fine control
over each instance's appearance while maintaining the precise pose control
capability. Specifically, we develop and demonstrate FineControlNet with
geometric control via human pose images and appearance control via
instance-level text prompts. The spatial alignment of instance-specific text
prompts and 2D poses in latent space enables the fine control capabilities of
FineControlNet. We evaluate the performance of FineControlNet with rigorous
comparison against state-of-the-art pose-conditioned text-to-image diffusion
models. FineControlNet achieves superior performance in generating images that
follow the user-provided instance-specific text prompts and poses compared with
existing methods. Project webpage:
https://samsunglabs.github.io/FineControlNet-project-page
VideoLCM: Video Latent Consistency Model
Authors: name: Xiang Wang, name: Shiwei Zhang, name: Han Zhang, name: Yu Liu, name: Yingya Zhang, name: Changxin Gao, name: Nong Sang
Published: 2023-12-14 16:45:36

Consistency models have demonstrated powerful capability in efficient image
generation and allowed synthesis within a few sampling steps, alleviating the
high computational cost in diffusion models. However, the consistency model in
the more challenging and resource-consuming video generation is still less
explored. In this report, we present the VideoLCM framework to fill this gap,
which leverages the concept of consistency models from image generation to
efficiently synthesize videos with minimal steps while maintaining high
quality. VideoLCM builds upon existing latent video diffusion models and
incorporates consistency distillation techniques for training the latent
consistency model. Experimental results reveal the effectiveness of our
VideoLCM in terms of computational efficiency, fidelity and temporal
consistency. Notably, VideoLCM achieves high-fidelity and smooth video
synthesis with only four sampling steps, showcasing the potential for real-time
synthesis. We hope that VideoLCM can serve as a simple yet effective baseline
for subsequent research. The source code and models will be publicly available.
SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained
  Geometry and Appearance
Authors: name: Yuanyou Xu, name: Zongxin Yang, name: Yi Yang
Published: 2023-12-13 14:48:35

Powered by large-scale text-to-image generation models, text-to-3D avatar
generation has made promising progress. However, most methods fail to produce
photorealistic results, limited by imprecise geometry and low-quality
appearance. Towards more practical avatar generation, we present SEEAvatar, a
method for generating photorealistic 3D avatars from text with SElf-Evolving
constraints for decoupled geometry and appearance. For geometry, we propose to
constrain the optimized avatar in a decent global shape with a template avatar.
The template avatar is initialized with human prior and can be updated by the
optimized avatar periodically as an evolving template, which enables more
flexible shape generation. Besides, the geometry is also constrained by the
static human prior in local parts like face and hands to maintain the delicate
structures. For appearance generation, we use diffusion model enhanced by
prompt engineering to guide a physically based rendering pipeline to generate
realistic textures. The lightness constraint is applied on the albedo texture
to suppress incorrect lighting effect. Experiments show that our method
outperforms previous methods on both global and local geometry and appearance
quality by a large margin. Since our method can produce high-quality meshes and
textures, such assets can be directly applied in classic graphics pipeline for
realistic rendering under any lighting condition. Project page at:
https://seeavatar3d.github.io.
VL-GPT: A Generative Pre-trained Transformer for Vision and Language
  Understanding and Generation
Authors: name: Jinguo Zhu, name: Xiaohan Ding, name: Yixiao Ge, name: Yuying Ge, name: Sijie Zhao, name: Hengshuang Zhao, name: Xiaohua Wang, name: Ying Shan
Published: 2023-12-14 18:59:43

In this work, we introduce Vision-Language Generative Pre-trained Transformer
(VL-GPT), a transformer model proficient at concurrently perceiving and
generating visual and linguistic data. VL-GPT achieves a unified pre-training
approach for both image and text modalities by employing a straightforward
auto-regressive objective, thereby enabling the model to process image and text
as seamlessly as a language model processes text. To accomplish this, we
initially propose a novel image tokenizer-detokenizer framework for visual
data, specifically designed to transform raw images into a sequence of
continuous embeddings and reconstruct them accordingly. In combination with the
existing text tokenizer and detokenizer, this framework allows for the encoding
of interleaved image-text data into a multimodal sequence, which can
subsequently be fed into the transformer model. Consequently, VL-GPT can
perform large-scale pre-training on multimodal corpora utilizing a unified
auto-regressive objective (i.e., next-token prediction). Upon completion of
pre-training, VL-GPT exhibits remarkable zero-shot and few-shot performance
across a diverse range of vision and language understanding and generation
tasks, including image captioning, visual question answering, text-to-image
generation, and more. Additionally, the pre-trained model retrains in-context
learning capabilities when provided with multimodal prompts. We further conduct
instruction tuning on our VL-GPT, highlighting its exceptional potential for
multimodal assistance. The source code and model weights shall be released.