Orionnnn/hub-stats huggingface dataset:
Retrieval-Enhanced Contrastive Vision-Text Models
Authors: name: Ahmet Iscen, name: Mathilde Caron, name: Alireza Fathi, name: Cordelia Schmid
Published: 2023-06-12 15:52:02

Contrastive image-text models such as CLIP form the building blocks of many
state-of-the-art systems. While they excel at recognizing common generic
concepts, they still struggle on fine-grained entities which are rare, or even
absent from the pre-training dataset. Hence, a key ingredient to their success
has been the use of large-scale curated pre-training data aiming at expanding
the set of concepts that they can memorize during the pre-training stage. In
this work, we explore an alternative to encoding fine-grained knowledge
directly into the model's parameters: we instead train the model to retrieve
this knowledge from an external memory. Specifically, we propose to equip
existing vision-text models with the ability to refine their embedding with
cross-modal retrieved information from a memory at inference time, which
greatly improves their zero-shot predictions. Remarkably, we show that this can
be done with a light-weight, single-layer, fusion transformer on top of a
frozen CLIP. Our experiments validate that our retrieval-enhanced contrastive
(RECO) training improves CLIP performance substantially on several challenging
fine-grained tasks: for example +10.9 on Stanford Cars, +10.2 on CUB-2011 and
+7.3 on the recent OVEN benchmark.
FasterViT: Fast Vision Transformers with Hierarchical Attention
Authors: name: Ali Hatamizadeh, name: Greg Heinrich, name: Hongxu Yin, name: Andrew Tao, name: Jose M. Alvarez, name: Jan Kautz, name: Pavlo Molchanov
Published: 2023-06-09 18:41:37

We design a new family of hybrid CNN-ViT neural networks, named FasterViT,
with a focus on high image throughput for computer vision (CV) applications.
FasterViT combines the benefits of fast local representation learning in CNNs
and global modeling properties in ViT. Our newly introduced Hierarchical
Attention (HAT) approach decomposes global self-attention with quadratic
complexity into a multi-level attention with reduced computational costs. We
benefit from efficient window-based self-attention. Each window has access to
dedicated carrier tokens that participate in local and global representation
learning. At a high level, global self-attentions enable the efficient
cross-window communication at lower costs. FasterViT achieves a SOTA
Pareto-front in terms of accuracy \vs image throughput. We have extensively
validated its effectiveness on various CV tasks including classification,
object detection and segmentation. We also show that HAT can be used as a
plug-and-play module for existing networks and enhance them. We further
demonstrate significantly faster and more accurate performance than competitive
counterparts for images with high resolution. Code is available at
https://github.com/NVlabs/FasterViT.
Benchmarking Neural Network Training Algorithms
Authors: name: George E. Dahl, name: Frank Schneider, name: Zachary Nado, name: Naman Agarwal, name: Chandramouli Shama Sastry, name: Philipp Hennig, name: Sourabh Medapati, name: Runa Eschenhagen, name: Priya Kasimbeg, name: Daniel Suo, name: Juhan Bae, name: Justin Gilmer, name: Abel L. Peirson, name: Bilal Khan, name: Rohan Anil, name: Mike Rabbat, name: Shankar Krishnan, name: Daniel Snider, name: Ehsan Amid, name: Kongtao Chen, name: Chris J. Maddison, name: Rakshith Vasudev, name: Michal Badura, name: Ankush Garg, name: Peter Mattson
Published: 2023-06-12 15:21:02

Training algorithms, broadly construed, are an essential part of every deep
learning pipeline. Training algorithm improvements that speed up training
across a wide variety of workloads (e.g., better update rules, tuning
protocols, learning rate schedules, or data selection schemes) could save time,
save computational resources, and lead to better, more accurate, models.
Unfortunately, as a community, we are currently unable to reliably identify
training algorithm improvements, or even determine the state-of-the-art
training algorithm. In this work, using concrete experiments, we argue that
real progress in speeding up training requires new benchmarks that resolve
three basic challenges faced by empirical comparisons of training algorithms:
(1) how to decide when training is complete and precisely measure training
time, (2) how to handle the sensitivity of measurements to exact workload
details, and (3) how to fairly compare algorithms that require hyperparameter
tuning. In order to address these challenges, we introduce a new, competitive,
time-to-result benchmark using multiple workloads running on fixed hardware,
the AlgoPerf: Training Algorithms benchmark. Our benchmark includes a set of
workload variants that make it possible to detect benchmark submissions that
are more robust to workload changes than current widely-used methods. Finally,
we evaluate baseline submissions constructed using various optimizers that
represent current practice, as well as other optimizers that have recently
received attention in the literature. These baseline results collectively
demonstrate the feasibility of our benchmark, show that non-trivial gaps
between methods exist, and set a provisional state-of-the-art for future
benchmark submissions to try and surpass.
Controlling Text-to-Image Diffusion by Orthogonal Finetuning
Authors: name: Zeju Qiu, name: Weiyang Liu, name: Haiwen Feng, name: Yuxuan Xue, name: Yao Feng, name: Zhen Liu, name: Dan Zhang, name: Adrian Weller, name: Bernhard Schölkopf
Published: 2023-06-12 17:59:23

Large text-to-image diffusion models have impressive capabilities in
generating photorealistic images from text prompts. How to effectively guide or
control these powerful models to perform different downstream tasks becomes an
important open problem. To tackle this challenge, we introduce a principled
finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image
diffusion models to downstream tasks. Unlike existing methods, OFT can provably
preserve hyperspherical energy which characterizes the pairwise neuron
relationship on the unit hypersphere. We find that this property is crucial for
preserving the semantic generation ability of text-to-image diffusion models.
To improve finetuning stability, we further propose Constrained Orthogonal
Finetuning (COFT) which imposes an additional radius constraint to the
hypersphere. Specifically, we consider two important finetuning text-to-image
tasks: subject-driven generation where the goal is to generate subject-specific
images given a few images of a subject and a text prompt, and controllable
generation where the goal is to enable the model to take in additional control
signals. We empirically show that our OFT framework outperforms existing
methods in generation quality and convergence speed.
Face0: Instantaneously Conditioning a Text-to-Image Model on a Face
Authors: name: Dani Valevski, name: Danny Wasserman, name: Yossi Matias, name: Yaniv Leviathan
Published: 2023-06-11 09:52:03

We present Face0, a novel way to instantaneously condition a text-to-image
generation model on a face, in sample time, without any optimization procedures
such as fine-tuning or inversions. We augment a dataset of annotated images
with embeddings of the included faces and train an image generation model, on
the augmented dataset. Once trained, our system is practically identical at
inference time to the underlying base model, and is therefore able to generate
images, given a user-supplied face image and a prompt, in just a couple of
seconds. Our method achieves pleasing results, is remarkably simple, extremely
fast, and equips the underlying model with new capabilities, like controlling
the generated images both via text or via direct manipulation of the input face
embeddings. In addition, when using a fixed random vector instead of a face
embedding from a user supplied image, our method essentially solves the problem
of consistent character generation across images. Finally, while requiring
further research, we hope that our method, which decouples the model's textual
biases from its biases on faces, might be a step towards some mitigation of
biases in future text-to-image models.
Augmenting Language Models with Long-Term Memory
Authors: name: Weizhi Wang, name: Li Dong, name: Hao Cheng, name: Xiaodong Liu, name: Xifeng Yan, name: Jianfeng Gao, name: Furu Wei
Published: 2023-06-12 15:13:39

Existing large language models (LLMs) can only afford fix-sized inputs due to
the input length limit, preventing them from utilizing rich long-context
information from past inputs. To address this, we propose a framework, Language
Models Augmented with Long-Term Memory (LongMem), which enables LLMs to
memorize long history. We design a novel decoupled network architecture with
the original backbone LLM frozen as a memory encoder and an adaptive residual
side-network as a memory retriever and reader. Such a decoupled memory design
can easily cache and update long-term past contexts for memory retrieval
without suffering from memory staleness. Enhanced with memory-augmented
adaptation training, LongMem can thus memorize long past context and use
long-term memory for language modeling. The proposed memory retrieval module
can handle unlimited-length context in its memory bank to benefit various
downstream tasks. Typically, LongMem can enlarge the long-form memory to 65k
tokens and thus cache many-shot extra demonstration examples as long-form
memory for in-context learning. Experiments show that our method outperforms
strong long-context models on ChapterBreak, a challenging long-context modeling
benchmark, and achieves remarkable improvements on memory-augmented in-context
learning over LLMs. The results demonstrate that the proposed method is
effective in helping language models to memorize and utilize long-form
contents. Our code is open-sourced at https://aka.ms/LongMem.
Weakly supervised information extraction from inscrutable handwritten
  document images
Authors: name: Sujoy Paul, name: Gagan Madan, name: Akankshya Mishra, name: Narayan Hegde, name: Pradeep Kumar, name: Gaurav Aggarwal
Published: 2023-06-12 02:22:30

State-of-the-art information extraction methods are limited by OCR errors.
They work well for printed text in form-like documents, but unstructured,
handwritten documents still remain a challenge. Adapting existing models to
domain-specific training data is quite expensive, because of two factors, 1)
limited availability of the domain-specific documents (such as handwritten
prescriptions, lab notes, etc.), and 2) annotations become even more
challenging as one needs domain-specific knowledge to decode inscrutable
handwritten document images. In this work, we focus on the complex problem of
extracting medicine names from handwritten prescriptions using only weakly
labeled data. The data consists of images along with the list of medicine names
in it, but not their location in the image. We solve the problem by first
identifying the regions of interest, i.e., medicine lines from just weak labels
and then injecting a domain-specific medicine language model learned using only
synthetically generated data. Compared to off-the-shelf state-of-the-art
methods, our approach performs >2.5x better in medicine names extraction from
prescriptions.
High-Fidelity Audio Compression with Improved RVQGAN
Authors: name: Rithesh Kumar, name: Prem Seetharaman, name: Alejandro Luebs, name: Ishaan Kumar, name: Kundan Kumar
Published: 2023-06-11 00:13:00

Language models have been successfully used to model natural signals, such as
images, speech, and music. A key component of these models is a high quality
neural compression model that can compress high-dimensional natural signals
into lower dimensional discrete tokens. To that end, we introduce a
high-fidelity universal neural audio compression algorithm that achieves ~90x
compression of 44.1 KHz audio into tokens at just 8kbps bandwidth. We achieve
this by combining advances in high-fidelity audio generation with better vector
quantization techniques from the image domain, along with improved adversarial
and reconstruction losses. We compress all domains (speech, environment, music,
etc.) with a single universal model, making it widely applicable to generative
modeling of all audio. We compare with competing audio compression algorithms,
and find our method outperforms them significantly. We provide thorough
ablations for every design choice, as well as open-source code and trained
model weights. We hope our work can lay the foundation for the next generation
of high-fidelity audio modeling.
Aladdin: Zero-Shot Hallucination of Stylized 3D Assets from Abstract
  Scene Descriptions
Authors: name: Ian Huang, name: Vrishab Krishna, name: Omoruyi Atekha, name: Leonidas Guibas
Published: 2023-06-09 19:24:39

What constitutes the "vibe" of a particular scene? What should one find in "a
busy, dirty city street", "an idyllic countryside", or "a crime scene in an
abandoned living room"? The translation from abstract scene descriptions to
stylized scene elements cannot be done with any generality by extant systems
trained on rigid and limited indoor datasets. In this paper, we propose to
leverage the knowledge captured by foundation models to accomplish this
translation. We present a system that can serve as a tool to generate stylized
assets for 3D scenes described by a short phrase, without the need to enumerate
the objects to be found within the scene or give instructions on their
appearance. Additionally, it is robust to open-world concepts in a way that
traditional methods trained on limited data are not, affording more creative
freedom to the 3D artist. Our system demonstrates this using a foundation model
"team" composed of a large language model, a vision-language model and several
image diffusion models, which communicate using an interpretable and
user-editable intermediate representation, thus allowing for more versatile and
controllable stylized asset generation for 3D artists. We introduce novel
metrics for this task, and show through human evaluations that in 91% of the
cases, our system outputs are judged more faithful to the semantics of the
input scene description than the baseline, thus highlighting the potential of
this approach to radically accelerate the 3D content creation process for 3D
artists.
Large Language Models as Tax Attorneys: A Case Study in Legal
  Capabilities Emergence
Authors: name: John J. Nay, name: David Karamardian, name: Sarah B. Lawsky, name: Wenting Tao, name: Meghana Bhat, name: Raghav Jain, name: Aaron Travis Lee, name: Jonathan H. Choi, name: Jungo Kasai
Published: 2023-06-12 12:40:48

Better understanding of Large Language Models' (LLMs) legal analysis
abilities can contribute to improving the efficiency of legal services,
governing artificial intelligence, and leveraging LLMs to identify
inconsistencies in law. This paper explores LLM capabilities in applying tax
law. We choose this area of law because it has a structure that allows us to
set up automated validation pipelines across thousands of examples, requires
logical reasoning and maths skills, and enables us to test LLM capabilities in
a manner relevant to real-world economic lives of citizens and companies. Our
experiments demonstrate emerging legal understanding capabilities, with
improved performance in each subsequent OpenAI model release. We experiment
with retrieving and utilising the relevant legal authority to assess the impact
of providing additional legal context to LLMs. Few-shot prompting, presenting
examples of question-answer pairs, is also found to significantly enhance the
performance of the most advanced model, GPT-4. The findings indicate that LLMs,
particularly when combined with prompting enhancements and the correct legal
texts, can perform at high levels of accuracy but not yet at expert tax lawyer
levels. As LLMs continue to advance, their ability to reason about law
autonomously could have significant implications for the legal profession and
AI governance.
Realistic Saliency Guided Image Enhancement
Authors: name: S. Mahdi H. Miangoleh, name: Zoya Bylinskii, name: Eric Kee, name: Eli Shechtman, name: Yağız Aksoy
Published: 2023-06-09 17:52:34

Common editing operations performed by professional photographers include the
cleanup operations: de-emphasizing distracting elements and enhancing subjects.
These edits are challenging, requiring a delicate balance between manipulating
the viewer's attention while maintaining photo realism. While recent approaches
can boast successful examples of attention attenuation or amplification, most
of them also suffer from frequent unrealistic edits. We propose a realism loss
for saliency-guided image enhancement to maintain high realism across varying
image types, while attenuating distractors and amplifying objects of interest.
Evaluations with professional photographers confirm that we achieve the dual
objective of realism and effectiveness, and outperform the recent approaches on
their own datasets, while requiring a smaller memory footprint and runtime. We
thus offer a viable solution for automating image enhancement and photo cleanup
operations.
GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields
Authors: name: Barbara Roessle, name: Norman Müller, name: Lorenzo Porzi, name: Samuel Rota Bulò, name: Peter Kontschieder, name: Matthias Nießner
Published: 2023-06-09 17:12:35

Neural Radiance Fields (NeRF) have shown impressive novel view synthesis
results; nonetheless, even thorough recordings yield imperfections in
reconstructions, for instance due to poorly observed areas or minor lighting
changes. Our goal is to mitigate these imperfections from various sources with
a joint solution: we take advantage of the ability of generative adversarial
networks (GANs) to produce realistic images and use them to enhance realism in
3D scene reconstruction with NeRFs. To this end, we learn the patch
distribution of a scene using an adversarial discriminator, which provides
feedback to the radiance field reconstruction, thus improving realism in a
3D-consistent fashion. Thereby, rendering artifacts are repaired directly in
the underlying 3D representation by imposing multi-view path rendering
constraints. In addition, we condition a generator with multi-resolution NeRF
renderings which is adversarially trained to further improve rendering quality.
We demonstrate that our approach significantly improves rendering quality,
e.g., nearly halving LPIPS scores compared to Nerfacto while at the same time
improving PSNR by 1.4dB on the advanced indoor scenes of Tanks and Temples.
Multi-Modal Classifiers for Open-Vocabulary Object Detection
Authors: name: Prannay Kaul, name: Weidi Xie, name: Andrew Zisserman
Published: 2023-06-08 18:31:56

The goal of this paper is open-vocabulary object detection (OVOD)
x2013 building a model that can detect objects beyond the set of
categories seen at training, thus enabling the user to specify categories of
interest at inference without the need for model retraining. We adopt a
standard two-stage object detector architecture, and explore three ways for
specifying novel categories: via language descriptions, via image exemplars, or
via a combination of the two. We make three contributions: first, we prompt a
large language model (LLM) to generate informative language descriptions for
object classes, and construct powerful text-based classifiers; second, we
employ a visual aggregator on image exemplars that can ingest any number of
images as input, forming vision-based classifiers; and third, we provide a
simple method to fuse information from language descriptions and image
exemplars, yielding a multi-modal classifier. When evaluating on the
challenging LVIS open-vocabulary benchmark we demonstrate that: (i) our
text-based classifiers outperform all previous OVOD works; (ii) our
vision-based classifiers perform as well as text-based classifiers in prior
work; (iii) using multi-modal classifiers perform better than either modality
alone; and finally, (iv) our text-based and multi-modal classifiers yield
better performance than a fully-supervised detector.
Evaluating the Social Impact of Generative AI Systems in Systems and
  Society
Authors: name: Irene Solaiman, name: Zeerak Talat, name: William Agnew, name: Lama Ahmad, name: Dylan Baker, name: Su Lin Blodgett, name: Hal Daumé III, name: Jesse Dodge, name: Ellie Evans, name: Sara Hooker, name: Yacine Jernite, name: Alexandra Sasha Luccioni, name: Alberto Lusoli, name: Margaret Mitchell, name: Jessica Newman, name: Marie-Therese Png, name: Andrew Strait, name: Apostol Vassilev
Published: 2023-06-09 15:05:13

Generative AI systems across modalities, ranging from text (including code),
image, audio, and video, have broad social impacts, but there is no official
standard for means of evaluating those impacts or for which impacts should be
evaluated. In this paper, we present a guide that moves toward a standard
approach in evaluating a base generative AI system for any modality in two
overarching categories: what can be evaluated in a base system independent of
context and what can be evaluated in a societal context. Importantly, this
refers to base systems that have no predetermined application or deployment
context, including a model itself, as well as system components, such as
training data. Our framework for a base system defines seven categories of
social impact: bias, stereotypes, and representational harms; cultural values
and sensitive content; disparate performance; privacy and data protection;
financial costs; environmental costs; and data and content moderation labor
costs. Suggested methods for evaluation apply to listed generative modalities
and analyses of the limitations of existing evaluations serve as a starting
point for necessary investment in future evaluations. We offer five overarching
categories for what can be evaluated in a broader societal context, each with
its own subcategories: trustworthiness and autonomy; inequality,
marginalization, and violence; concentration of authority; labor and
creativity; and ecosystem and environment. Each subcategory includes
recommendations for mitigating harm.
BOOT: Data-free Distillation of Denoising Diffusion Models with
  Bootstrapping
Authors: name: Jiatao Gu, name: Shuangfei Zhai, name: Yizhe Zhang, name: Lingjie Liu, name: Josh Susskind
Published: 2023-06-08 20:30:55

Diffusion models have demonstrated excellent potential for generating diverse
images. However, their performance often suffers from slow generation due to
iterative denoising. Knowledge distillation has been recently proposed as a
remedy that can reduce the number of inference steps to one or a few without
significant quality degradation. However, existing distillation methods either
require significant amounts of offline computation for generating synthetic
training data from the teacher model or need to perform expensive online
learning with the help of real data. In this work, we present a novel technique
called BOOT, that overcomes these limitations with an efficient data-free
distillation algorithm. The core idea is to learn a time-conditioned model that
predicts the output of a pre-trained diffusion model teacher given any time
step. Such a model can be efficiently trained based on bootstrapping from two
consecutive sampled steps. Furthermore, our method can be easily adapted to
large-scale text-to-image diffusion models, which are challenging for
conventional methods given the fact that the training sets are often large and
difficult to access. We demonstrate the effectiveness of our approach on
several benchmark datasets in the DDIM setting, achieving comparable generation
quality while being orders of magnitude faster than the diffusion teacher. The
text-to-image results show that the proposed approach is able to handle highly
complex distributions, shedding light on more efficient generative modeling.