Orionnnn/hub-stats huggingface dataset:
Thinking Like an Annotator: Generation of Dataset Labeling Instructions
Authors: name: Nadine Chang, name: Francesco Ferroni, name: Michael J. Tarr, name: Martial Hebert, name: Deva Ramanan
Published: 2023-06-24 18:32:48

Large-scale datasets are essential to modern day deep learning. Advocates
argue that understanding these methods requires dataset transparency (e.g.
"dataset curation, motivation, composition, collection process, etc...").
However, almost no one has suggested the release of the detailed definitions
and visual category examples provided to annotators - information critical to
understanding the structure of the annotations present in each dataset. These
labels are at the heart of public datasets, yet few datasets include the
instructions that were used to generate them. We introduce a new task, Labeling
Instruction Generation, to address missing publicly available labeling
instructions. In Labeling Instruction Generation, we take a reasonably
annotated dataset and: 1) generate a set of examples that are visually
representative of each category in the dataset; 2) provide a text label that
corresponds to each of the examples. We introduce a framework that requires no
model training to solve this task and includes a newly created rapid retrieval
system that leverages a large, pre-trained vision and language model. This
framework acts as a proxy to human annotators that can help to both generate a
final labeling instruction set and evaluate its quality. Our framework
generates multiple diverse visual and text representations of dataset
categories. The optimized instruction set outperforms our strongest baseline
across 5 folds by 7.06 mAP for NuImages and 12.9 mAP for COCO.
Beyond Scale: the Diversity Coefficient as a Data Quality Metric
  Demonstrates LLMs are Pre-trained on Formally Diverse Data
Authors: name: Alycia Lee, name: Brando Miranda, name: Sanmi Koyejo
Published: 2023-06-24 02:25:56

Current trends to pre-train capable Large Language Models (LLMs) mostly focus
on scaling of model and dataset size. However, the quality of pre-training data
is an important factor for training powerful LLMs, yet it is a nebulous concept
that has not been fully characterized. Therefore, we use the recently proposed
Task2Vec diversity coefficient to ground and understand formal aspects of data
quality, to go beyond scale alone. Specifically, we measure the diversity
coefficient of publicly available pre-training datasets to demonstrate that
their formal diversity is high when compared to theoretical lower and upper
bounds. In addition, to build confidence in the diversity coefficient, we
conduct interpretability experiments and find that the coefficient aligns with
intuitive properties of diversity, e.g., it increases as the number of latent
concepts increases. We conclude the diversity coefficient is reliable, show
it's high for publicly available LLM datasets, and conjecture it can be used to
build useful diverse datasets for LLMs.
System-Level Natural Language Feedback
Authors: name: Weizhe Yuan, name: Kyunghyun Cho, name: Jason Weston
Published: 2023-06-23 16:21:40

Natural language (NL) feedback contains rich information about the user
experience. Existing studies focus on an instance-level approach, where
feedback is used to refine specific examples, disregarding its system-wide
application. This paper proposes a general framework for unlocking the
system-level use of NL feedback. We show how to use feedback to formalize
system-level design decisions in a human-in-the-loop-process -- in order to
produce better models. In particular this is done through: (i) metric design
for tasks; and (ii) language model prompt design for refining model responses.
We conduct two case studies of this approach for improving search query
generation and dialog response generation, demonstrating the effectiveness of
the use of system-level feedback. We show the combination of system-level
feedback and instance-level feedback brings further gains, and that human
written instance-level feedback results in more grounded refinements than
GPT-3.5 written ones, underlying the importance of human feedback for building
systems.
Long-range Language Modeling with Self-retrieval
Authors: name: Ohad Rubin, name: Jonathan Berant
Published: 2023-06-23 10:18:02

Retrieval-augmented language models (LMs) have received much attention
recently. However, typically the retriever is not trained jointly as a native
component of the LM, but added to an already-pretrained LM, which limits the
ability of the LM and the retriever to adapt to one another. In this work, we
propose the Retrieval-Pretrained Transformer (RPT), an architecture and
training procedure for jointly training a retrieval-augmented LM from scratch
for the task of modeling long texts. Given a recently generated text chunk in a
long document, the LM computes query representations, which are then used to
retrieve earlier chunks in the document, located potentially tens of thousands
of tokens before. Information from retrieved chunks is fused into the LM
representations to predict the next target chunk. We train the retriever
component with a semantic objective, where the goal is to retrieve chunks that
increase the probability of the next chunk, according to a reference LM. We
evaluate RPT on four long-range language modeling tasks, spanning books, code,
and mathematical writing, and demonstrate that RPT improves retrieval quality
and subsequently perplexity across the board compared to strong baselines.
OpenMask3D: Open-Vocabulary 3D Instance Segmentation
Authors: name: Ayça Takmaz, name: Elisabetta Fedele, name: Robert W. Sumner, name: Marc Pollefeys, name: Federico Tombari, name: Francis Engelmann
Published: 2023-06-23 17:36:44

We introduce the task of open-vocabulary 3D instance segmentation.
Traditional approaches for 3D instance segmentation largely rely on existing 3D
annotated datasets, which are restricted to a closed-set of object categories.
This is an important limitation for real-life applications where one might need
to perform tasks guided by novel, open-vocabulary queries related to objects
from a wide variety. Recently, open-vocabulary 3D scene understanding methods
have emerged to address this problem by learning queryable features per each
point in the scene. While such a representation can be directly employed to
perform semantic segmentation, existing methods have limitations in their
ability to identify object instances. In this work, we address this limitation,
and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D
instance segmentation. Guided by predicted class-agnostic 3D instance masks,
our model aggregates per-mask features via multi-view fusion of CLIP-based
image embeddings. We conduct experiments and ablation studies on the ScanNet200
dataset to evaluate the performance of OpenMask3D, and provide insights about
the open-vocabulary 3D instance segmentation task. We show that our approach
outperforms other open-vocabulary counterparts, particularly on the long-tail
distribution. Furthermore, OpenMask3D goes beyond the limitations of
close-vocabulary approaches, and enables the segmentation of object instances
based on free-form queries describing object properties such as semantics,
geometry, affordances, and material properties.
Scaling MLPs: A Tale of Inductive Bias
Authors: name: Gregor Bachmann, name: Sotiris Anagnostidis, name: Thomas Hofmann
Published: 2023-06-23 15:55:44

In this work we revisit the most fundamental building block in deep learning,
the multi-layer perceptron (MLP), and study the limits of its performance on
vision tasks. Empirical insights into MLPs are important for multiple reasons.
(1) Given the recent narrative "less inductive bias is better", popularized due
to transformers eclipsing convolutional models, it is natural to explore the
limits of this hypothesis. To that end, MLPs offer an ideal test bed, being
completely free of any inductive bias. (2) MLPs have almost exclusively been
the main protagonist in the deep learning theory literature due to their
mathematical simplicity, serving as a proxy to explain empirical phenomena
observed for more complex architectures. Surprisingly, experimental datapoints
for MLPs are very difficult to find in the literature, especially when coupled
with large pre-training protocols. This discrepancy between practice and theory
is worrying: Do MLPs reflect the empirical advances exhibited by practical
models? Or do theorists need to rethink the role of MLPs as a proxy? We provide
insights into both these aspects. We show that the performance of MLPs
drastically improves with scale (93% on CIFAR10, 79% on CIFAR100, 69% on
TinyImageNet), highlighting that lack of inductive bias can indeed be
compensated. We observe that MLPs mimic the behaviour of their modern
counterparts faithfully, with some components in the learning setting however
surprisingly exhibiting stronger or unexpected behaviours. Due to their
inherent computational efficiency, large pre-training experiments become more
accessible for academic researchers. All of our experiments were run on a
single GPU.
DreamEditor: Text-Driven 3D Scene Editing with Neural Fields
Authors: name: Jingyu Zhuang, name: Chen Wang, name: Lingjie Liu, name: Liang Lin, name: Guanbin Li
Published: 2023-06-23 11:53:43

Neural fields have achieved impressive advancements in view synthesis and
scene reconstruction. However, editing these neural fields remains challenging
due to the implicit encoding of geometry and texture information. In this
paper, we propose DreamEditor, a novel framework that enables users to perform
controlled editing of neural fields using text prompts. By representing scenes
as mesh-based neural fields, DreamEditor allows localized editing within
specific regions. DreamEditor utilizes the text encoder of a pretrained
text-to-Image diffusion model to automatically identify the regions to be
edited based on the semantics of the text prompts. Subsequently, DreamEditor
optimizes the editing region and aligns its geometry and texture with the text
prompts through score distillation sampling [29]. Extensive experiments have
demonstrated that DreamEditor can accurately edit neural fields of real-world
scenes according to the given text prompts while ensuring consistency in
irrelevant areas. DreamEditor generates highly realistic textures and geometry,
significantly surpassing previous works in both quantitative and qualitative
evaluations.
Bring Your Own Data! Self-Supervised Evaluation for Large Language
  Models
Authors: name: Neel Jain, name: Khalid Saifullah, name: Yuxin Wen, name: John Kirchenbauer, name: Manli Shu, name: Aniruddha Saha, name: Micah Goldblum, name: Jonas Geiping, name: Tom Goldstein
Published: 2023-06-23 17:59:09

With the rise of Large Language Models (LLMs) and their ubiquitous deployment
in diverse domains, measuring language model behavior on realistic data is
imperative. For example, a company deploying a client-facing chatbot must
ensure that the model will not respond to client requests with profanity.
Current evaluations approach this problem using small, domain-specific datasets
with human-curated labels. These evaluation sets are often sampled from a
narrow and simplified distribution, and data sources can unknowingly be leaked
into the training set which can lead to misleading evaluations. To bypass these
drawbacks, we propose a framework for self-supervised evaluation of LLMs by
analyzing their sensitivity or invariance to transformations on the input text.
Self-supervised evaluation can directly monitor LLM behavior on datasets
collected in the wild or streamed during live model deployment. We demonstrate
self-supervised evaluation strategies for measuring closed-book knowledge,
toxicity, and long-range context dependence, in addition to sensitivity to
grammatical structure and tokenization errors. When comparisons to similar
human-labeled benchmarks are available, we find strong correlations between
self-supervised and human-supervised evaluations. The self-supervised paradigm
complements current evaluation strategies that rely on labeled data.
On-Policy Distillation of Language Models: Learning from Self-Generated
  Mistakes
Authors: name: Rishabh Agarwal, name: Nino Vieillard, name: Yongchao Zhou, name: Piotr Stanczyk, name: Sabela Ramos, name: Matthieu Geist, name: Olivier Bachem
Published: 2023-06-23 17:56:26

Knowledge distillation (KD) is widely used for compressing a teacher model to
reduce its inference cost and memory footprint, by training a smaller student
model. However, current KD methods for auto-regressive sequence models suffer
from distribution mismatch between output sequences seen during training and
those generated by the student during inference. To address this issue, we
introduce Generalized Knowledge Distillation (GKD). Instead of solely relying
on a fixed set of output sequences, GKD trains the student on its
self-generated output sequences by leveraging feedback from the teacher on such
sequences. Unlike supervised KD approaches, GKD also offers the flexibility to
employ alternative loss functions between the student and teacher, which can be
useful when the student lacks the expressivity to mimic the teacher's
distribution. Furthermore, GKD facilitates the seamless integration of
distillation with RL fine-tuning (RLHF). We demonstrate the efficacy of GKD for
distilling auto-regressive language models on summarization, translation, and
arithmetic reasoning tasks, and task-agnostic distillation for
instruction-tuning.
CLIP2Protect: Protecting Facial Privacy using Text-Guided Makeup via
  Adversarial Latent Search
Authors: name: Fahad Shamshad, name: Muzammal Naseer, name: Karthik Nandakumar
Published: 2023-06-16 17:58:15

The success of deep learning based face recognition systems has given rise to
serious privacy concerns due to their ability to enable unauthorized tracking
of users in the digital world. Existing methods for enhancing privacy fail to
generate naturalistic images that can protect facial privacy without
compromising user experience. We propose a novel two-step approach for facial
privacy protection that relies on finding adversarial latent codes in the
low-dimensional manifold of a pretrained generative model. The first step
inverts the given face image into the latent space and finetunes the generative
model to achieve an accurate reconstruction of the given image from its latent
code. This step produces a good initialization, aiding the generation of
high-quality faces that resemble the given identity. Subsequently, user-defined
makeup text prompts and identity-preserving regularization are used to guide
the search for adversarial codes in the latent space. Extensive experiments
demonstrate that faces generated by our approach have stronger black-box
transferability with an absolute gain of 12.06% over the state-of-the-art
facial privacy protection approach under the face verification task. Finally,
we demonstrate the effectiveness of the proposed approach for commercial face
recognition systems. Our code is available at
https://github.com/fahadshamshad/Clip2Protect.
Blended-NeRF: Zero-Shot Object Generation and Blending in Existing
  Neural Radiance Fields
Authors: name: Ori Gordon, name: Omri Avrahami, name: Dani Lischinski
Published: 2023-06-22 09:34:55

Editing a local region or a specific object in a 3D scene represented by a
NeRF is challenging, mainly due to the implicit nature of the scene
representation. Consistently blending a new realistic object into the scene
adds an additional level of difficulty. We present Blended-NeRF, a robust and
flexible framework for editing a specific region of interest in an existing
NeRF scene, based on text prompts or image patches, along with a 3D ROI box.
Our method leverages a pretrained language-image model to steer the synthesis
towards a user-provided text prompt or image patch, along with a 3D MLP model
initialized on an existing NeRF scene to generate the object and blend it into
a specified region in the original scene. We allow local editing by localizing
a 3D ROI box in the input scene, and seamlessly blend the content synthesized
inside the ROI with the existing scene using a novel volumetric blending
technique. To obtain natural looking and view-consistent results, we leverage
existing and new geometric priors and 3D augmentations for improving the visual
fidelity of the final result.
  We test our framework both qualitatively and quantitatively on a variety of
real 3D scenes and text prompts, demonstrating realistic multi-view consistent
results with much flexibility and diversity compared to the baselines. Finally,
we show the applicability of our framework for several 3D editing applications,
including adding new objects to a scene, removing/replacing/altering existing
objects, and texture conversion.
Continuous Layout Editing of Single Images with Diffusion Models
Authors: name: Zhiyuan Zhang, name: Zhitong Huang, name: Jing Liao
Published: 2023-06-22 17:51:05

Recent advancements in large-scale text-to-image diffusion models have
enabled many applications in image editing. However, none of these methods have
been able to edit the layout of single existing images. To address this gap, we
propose the first framework for layout editing of a single image while
preserving its visual properties, thus allowing for continuous editing on a
single image. Our approach is achieved through two key modules. First, to
preserve the characteristics of multiple objects within an image, we
disentangle the concepts of different objects and embed them into separate
textual tokens using a novel method called masked textual inversion. Next, we
propose a training-free optimization method to perform layout control for a
pre-trained diffusion model, which allows us to regenerate images with learned
concepts and align them with user-specified layouts. As the first framework to
edit the layout of existing images, we demonstrate that our method is effective
and outperforms other baselines that were modified to support this task. Our
code will be freely available for public use upon acceptance.
Quantizable Transformers: Removing Outliers by Helping Attention Heads
  Do Nothing
Authors: name: Yelysei Bondarenko, name: Markus Nagel, name: Tijmen Blankevoort
Published: 2023-06-22 14:39:04

Transformer models have been widely adopted in various domains over the last
years, and especially large language models have advanced the field of AI
significantly. Due to their size, the capability of these networks has
increased tremendously, but this has come at the cost of a significant increase
in necessary compute. Quantization is one of the most effective ways to reduce
the computational time and memory consumption of neural networks. Many studies
have shown, however, that modern transformer models tend to learn strong
outliers in their activations, making them difficult to quantize. To retain
acceptable performance, the existence of these outliers requires activations to
be in higher bitwidth or the use of different numeric formats, extra
fine-tuning, or other workarounds. We show that strong outliers are related to
very specific behavior of attention heads that try to learn a "no-op" or just a
partial update of the residual. To achieve the exact zeros needed in the
attention matrix for a no-update, the input to the softmax is pushed to be
larger and larger during training, causing outliers in other parts of the
network. Based on these observations, we propose two simple (independent)
modifications to the attention mechanism - clipped softmax and gated attention.
We empirically show that models pre-trained using our methods learn
significantly smaller outliers while maintaining and sometimes even improving
the floating-point task performance. This enables us to quantize transformers
to full INT8 quantization of the activations without any additional effort. We
demonstrate the effectiveness of our methods on both language models (BERT,
OPT) and vision transformers.
AudioPaLM: A Large Language Model That Can Speak and Listen
Authors: name: Paul K. Rubenstein, name: Chulayuth Asawaroengchai, name: Duc Dung Nguyen, name: Ankur Bapna, name: Zalán Borsos, name: Félix de Chaumont Quitry, name: Peter Chen, name: Dalia El Badawy, name: Wei Han, name: Eugene Kharitonov, name: Hannah Muckenhirn, name: Dirk Padfield, name: James Qin, name: Danny Rozenberg, name: Tara Sainath, name: Johan Schalkwyk, name: Matt Sharifi, name: Michelle Tadmor Ramanovich, name: Marco Tagliasacchi, name: Alexandru Tudor, name: Mihajlo Velimirović, name: Damien Vincent, name: Jiahui Yu, name: Yongqiang Wang, name: Vicky Zayats, name: Neil Zeghidour, name: Yu Zhang, name: Zhishuai Zhang, name: Lukas Zilka, name: Christian Frank
Published: 2023-06-22 14:37:54

We introduce AudioPaLM, a large language model for speech understanding and
generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2
[Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified
multimodal architecture that can process and generate text and speech with
applications including speech recognition and speech-to-speech translation.
AudioPaLM inherits the capability to preserve paralinguistic information such
as speaker identity and intonation from AudioLM and the linguistic knowledge
present only in text large language models such as PaLM-2. We demonstrate that
initializing AudioPaLM with the weights of a text-only large language model
improves speech processing, successfully leveraging the larger quantity of text
training data used in pretraining to assist with the speech tasks. The
resulting model significantly outperforms existing systems for speech
translation tasks and has the ability to perform zero-shot speech-to-text
translation for many languages for which input/target language combinations
were not seen in training. AudioPaLM also demonstrates features of audio
language models, such as transferring a voice across languages based on a short
spoken prompt. We release examples of our method at
https://google-research.github.io/seanet/audiopalm/examples
From Word Models to World Models: Translating from Natural Language to
  the Probabilistic Language of Thought
Authors: name: Lionel Wong, name: Gabriel Grand, name: Alexander K. Lew, name: Noah D. Goodman, name: Vikash K. Mansinghka, name: Jacob Andreas, name: Joshua B. Tenenbaum
Published: 2023-06-22 05:14:00

How does language inform our downstream thinking? In particular, how do
humans make meaning from language -- and how can we leverage a theory of
linguistic meaning to build machines that think in more human-like ways? In
this paper, we propose rational meaning construction, a computational
framework for language-informed thinking that combines neural models of
language with probabilistic models for rational inference. We frame linguistic
meaning as a context-sensitive mapping from natural language into a
probabilistic language of thought (PLoT) -- a general-purpose symbolic
substrate for probabilistic, generative world modeling. Our architecture
integrates two powerful computational tools that have not previously come
together: we model thinking with probabilistic programs, an expressive
representation for flexible commonsense reasoning; and we model meaning
construction with large language models (LLMs), which support
broad-coverage translation from natural language utterances to code expressions
in a probabilistic programming language. We illustrate our framework in action
through examples covering four core domains from cognitive science:
probabilistic reasoning, logical and relational reasoning, visual and physical
reasoning, and social reasoning about agents and their plans. In each, we show
that LLMs can generate context-sensitive translations that capture
pragmatically-appropriate linguistic meanings, while Bayesian inference with
the generated programs supports coherent and robust commonsense reasoning. We
extend our framework to integrate cognitively-motivated symbolic modules to
provide a unified commonsense thinking interface from language. Finally, we
explore how language can drive the construction of world models themselves.