Orionnnn/hub-stats huggingface dataset:
Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing
  Important Tokens
Authors: name: Zhanpeng Zeng, name: Cole Hawkins, name: Mingyi Hong, name: Aston Zhang, name: Nikolaos Pappas, name: Vikas Singh, name: Shuai Zheng
Published: 2023-05-07 10:32:18

Transformer models are foundational to natural language processing (NLP) and
computer vision. Despite various recent works devoted to reducing the quadratic
cost of such models (as a function of the sequence length n), dealing with
ultra long sequences efficiently (e.g., with more than 16K tokens) remains
challenging. Applications such as answering questions based on an entire book
or summarizing a scientific article are inefficient or infeasible. In this
paper, we propose to significantly reduce the dependency of a Transformer
model's complexity on n, by compressing the input into a representation whose
size r is independent of n at each layer. Specifically, by exploiting the
fact that in many tasks, only a small subset of special tokens (we call
VIP-tokens) are most relevant to the final prediction, we propose a VIP-token
centric compression (Vcc) scheme which selectively compresses the input
sequence based on their impact on approximating the representation of these
VIP-tokens. Compared with competitive baselines, the proposed algorithm not
only is efficient (achieving more than 3times efficiency improvement
compared to baselines on 4K and 16K lengths), but also achieves competitive or
better performance on a large number of tasks. Further, we show that our
algorithm can be scaled to 128K tokens (or more) while consistently offering
accuracy improvement.
MultiModal-GPT: A Vision and Language Model for Dialogue with Humans
Authors: name: Tao Gong, name: Chengqi Lyu, name: Shilong Zhang, name: Yudong Wang, name: Miao Zheng, name: Qian Zhao, name: Kuikun Liu, name: Wenwei Zhang, name: Ping Luo, name: Kai Chen
Published: 2023-05-08 15:45:42

We present a vision and language model named MultiModal-GPT to conduct
multi-round dialogue with humans. MultiModal-GPT can follow various
instructions from humans, such as generating a detailed caption, counting the
number of interested objects, and answering general questions from users.
MultiModal-GPT is parameter-efficiently fine-tuned from OpenFlamingo, with
Low-rank Adapter (LoRA) added both in the cross-attention part and the
self-attention part of the language model. We first construct instruction
templates with vision and language data for multi-modality instruction tuning
to make the model understand and follow human instructions. We find the quality
of training data is vital for the dialogue performance, where few data
containing short answers can lead the model to respond shortly to any
instructions. To further enhance the ability to chat with humans of the
MultiModal-GPT, we utilize language-only instruction-following data to train
the MultiModal-GPT jointly. The joint training of language-only and
visual-language instructions with the same instruction template
effectively improves dialogue performance. Various demos show the ability of
continuous dialogue of MultiModal-GPT with humans. Code and demo are at
https://github.com/open-mmlab/Multimodal-GPT
Language Models Don't Always Say What They Think: Unfaithful
  Explanations in Chain-of-Thought Prompting
Authors: name: Miles Turpin, name: Julian Michael, name: Ethan Perez, name: Samuel R. Bowman
Published: 2023-05-07 22:44:25

Large Language Models (LLMs) can achieve strong performance on many tasks by
producing step-by-step reasoning before giving a final output, often referred
to as chain-of-thought reasoning (CoT). It is tempting to interpret these CoT
explanations as the LLM's process for solving a task. However, we find that CoT
explanations can systematically misrepresent the true reason for a model's
prediction. We demonstrate that CoT explanations can be heavily influenced by
adding biasing features to model inputs -- e.g., by reordering the
multiple-choice options in a few-shot prompt to make the answer always "(A)" --
which models systematically fail to mention in their explanations. When we bias
models toward incorrect answers, they frequently generate CoT explanations
supporting those answers. This causes accuracy to drop by as much as 36% on a
suite of 13 tasks from BIG-Bench Hard, when testing with GPT-3.5 from OpenAI
and Claude 1.0 from Anthropic. On a social-bias task, model explanations
justify giving answers in line with stereotypes without mentioning the
influence of these social biases. Our findings indicate that CoT explanations
can be plausible yet misleading, which risks increasing our trust in LLMs
without guaranteeing their safety. CoT is promising for explainability, but our
results highlight the need for targeted efforts to evaluate and improve
explanation faithfulness.
Pre-training Language Model as a Multi-perspective Course Learner
Authors: name: Beiduo Chen, name: Shaohan Huang, name: Zihan Zhang, name: Wu Guo, name: Zhenhua Ling, name: Haizhen Huang, name: Furu Wei, name: Weiwei Deng, name: Qi Zhang
Published: 2023-05-06 09:02:10

ELECTRA, the generator-discriminator pre-training framework, has achieved
impressive semantic construction capability among various downstream tasks.
Despite the convincing performance, ELECTRA still faces the challenges of
monotonous training and deficient interaction. Generator with only masked
language modeling (MLM) leads to biased learning and label imbalance for
discriminator, decreasing learning efficiency; no explicit feedback loop from
discriminator to generator results in the chasm between these two components,
underutilizing the course learning. In this study, a multi-perspective course
learning (MCL) method is proposed to fetch a many degrees and visual angles for
sample-efficient pre-training, and to fully leverage the relationship between
generator and discriminator. Concretely, three self-supervision courses are
designed to alleviate inherent flaws of MLM and balance the label in a
multi-perspective way. Besides, two self-correction courses are proposed to
bridge the chasm between the two encoders by creating a "correction notebook"
for secondary-supervision. Moreover, a course soups trial is conducted to solve
the "tug-of-war" dynamics problem of MCL, evolving a stronger pre-trained
model. Experimental results show that our method significantly improves
ELECTRA's average performance by 2.8% and 3.2% absolute points respectively on
GLUE and SQuAD 2.0 benchmarks, and overshadows recent advanced ELECTRA-style
models under the same settings. The pre-trained MCL model is available at
https://huggingface.co/McmanusChen/MCL-base.
Residual Prompt Tuning: Improving Prompt Tuning with Residual
  Reparameterization
Authors: name: Anastasia Razdaibiedina, name: Yuning Mao, name: Rui Hou, name: Madian Khabsa, name: Mike Lewis, name: Jimmy Ba, name: Amjad Almahairi
Published: 2023-05-06 05:35:14

Prompt tuning is one of the successful approaches for parameter-efficient
tuning of pre-trained language models. Despite being arguably the most
parameter-efficient (tuned soft prompts constitute <0.1% of total parameters),
it typically performs worse than other efficient tuning methods and is quite
sensitive to hyper-parameters. In this work, we introduce Residual Prompt
Tuning - a simple and efficient method that significantly improves the
performance and stability of prompt tuning. We propose to reparameterize soft
prompt embeddings using a shallow network with a residual connection. Our
experiments show that Residual Prompt Tuning significantly outperforms prompt
tuning on SuperGLUE benchmark. Notably, our method reaches +7 points
improvement over prompt tuning with T5-Base and allows to reduce the prompt
length by 10x without hurting performance. In addition, we show that our
approach is robust to the choice of learning rate and prompt initialization,
and is effective in few-shot settings.
X-LLM: Bootstrapping Advanced Large Language Models by Treating
  Multi-Modalities as Foreign Languages
Authors: name: Feilong Chen, name: Minglun Han, name: Haozhi Zhao, name: Qingyang Zhang, name: Jing Shi, name: Shuang Xu, name: Bo Xu
Published: 2023-05-07 02:25:42

Large language models (LLMs) have demonstrated remarkable language abilities.
GPT-4, based on advanced LLMs, exhibits extraordinary multimodal capabilities
beyond previous visual language models. We attribute this to the use of more
advanced LLMs compared with previous multimodal models. Unfortunately, the
model architecture and training strategies of GPT-4 are unknown. To endow LLMs
with multimodal capabilities, we propose X-LLM, which converts Multi-modalities
(images, speech, videos) into foreign languages using X2L interfaces and inputs
them into a large Language model (ChatGLM). Specifically, X-LLM aligns multiple
frozen single-modal encoders and a frozen LLM using X2L interfaces, where ``X''
denotes multi-modalities such as image, speech, and videos, and ``L'' denotes
languages. X-LLM's training consists of three stages: (1) Converting Multimodal
Information: The first stage trains each X2L interface to align with its
respective single-modal encoder separately to convert multimodal information
into languages. (2) Aligning X2L representations with the LLM: single-modal
encoders are aligned with the LLM through X2L interfaces independently. (3)
Integrating multiple modalities: all single-modal encoders are aligned with the
LLM through X2L interfaces to integrate multimodal capabilities into the LLM.
Our experiments show that X-LLM demonstrates impressive multimodel chat
abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen
images/instructions, and yields a 84.5\% relative score compared with GPT-4 on
a synthetic multimodal instruction-following dataset. And we also conduct
quantitative tests on using LLM for ASR and multimodal ASR, hoping to promote
the era of LLM-based speech recognition.
Controllable Light Diffusion for Portraits
Authors: name: David Futschik, name: Kelvin Ritland, name: James Vecore, name: Sean Fanello, name: Sergio Orts-Escolano, name: Brian Curless, name: Daniel Sýkora, name: Rohit Pandey
Published: 2023-05-08 14:46:28

We introduce light diffusion, a novel method to improve lighting in
portraits, softening harsh shadows and specular highlights while preserving
overall scene illumination. Inspired by professional photographers' diffusers
and scrims, our method softens lighting given only a single portrait photo.
Previous portrait relighting approaches focus on changing the entire lighting
environment, removing shadows (ignoring strong specular highlights), or
removing shading entirely. In contrast, we propose a learning based method that
allows us to control the amount of light diffusion and apply it on in-the-wild
portraits. Additionally, we design a method to synthetically generate plausible
external shadows with sub-surface scattering effects while conforming to the
shape of the subject's face. Finally, we show how our approach can increase the
robustness of higher level vision applications, such as albedo estimation,
geometry estimation and semantic segmentation.
Locally Attentional SDF Diffusion for Controllable 3D Shape Generation
Authors: name: Xin-Yang Zheng, name: Hao Pan, name: Peng-Shuai Wang, name: Xin Tong, name: Yang Liu, name: Heung-Yeung Shum
Published: 2023-05-08 05:07:23

Although the recent rapid evolution of 3D generative neural networks greatly
improves 3D shape generation, it is still not convenient for ordinary users to
create 3D shapes and control the local geometry of generated shapes. To address
these challenges, we propose a diffusion-based 3D generation framework --
locally attentional SDF diffusion, to model plausible 3D shapes, via 2D sketch
image input. Our method is built on a two-stage diffusion model. The first
stage, named occupancy-diffusion, aims to generate a low-resolution occupancy
field to approximate the shape shell. The second stage, named SDF-diffusion,
synthesizes a high-resolution signed distance field within the occupied voxels
determined by the first stage to extract fine geometry. Our model is empowered
by a novel view-aware local attention mechanism for image-conditioned shape
generation, which takes advantage of 2D image patch features to guide 3D voxel
feature learning, greatly improving local controllability and model
generalizability. Through extensive experiments in sketch-conditioned and
category-conditioned 3D shape generation tasks, we validate and demonstrate the
ability of our method to provide plausible and diverse 3D shapes, as well as
its superior controllability and generalizability over existing work. Our code
and trained models are available at
https://zhengxinyang.github.io/projects/LAS-Diffusion.html
A Variational Perspective on Solving Inverse Problems with Diffusion
  Models
Authors: name: Morteza Mardani, name: Jiaming Song, name: Jan Kautz, name: Arash Vahdat
Published: 2023-05-07 23:00:47

Diffusion models have emerged as a key pillar of foundation models in visual
domains. One of their critical applications is to universally solve different
downstream inverse tasks via a single diffusion prior without re-training for
each task. Most inverse tasks can be formulated as inferring a posterior
distribution over data (e.g., a full image) given a measurement (e.g., a masked
image). This is however challenging in diffusion models since the nonlinear and
iterative nature of the diffusion process renders the posterior intractable. To
cope with this challenge, we propose a variational approach that by design
seeks to approximate the true posterior distribution. We show that our approach
naturally leads to regularization by denoising diffusion process (RED-Diff)
where denoisers at different timesteps concurrently impose different structural
constraints over the image. To gauge the contribution of denoisers from
different timesteps, we propose a weighting mechanism based on
signal-to-noise-ratio (SNR). Our approach provides a new variational
perspective for solving inverse problems with diffusion models, allowing us to
formulate sampling as stochastic optimization, where one can simply apply
off-the-shelf solvers with lightweight iterates. Our experiments for image
restoration tasks such as inpainting and superresolution demonstrate the
strengths of our method compared with state-of-the-art sampling-based diffusion
models.
Avatar Fingerprinting for Authorized Use of Synthetic Talking-Head
  Videos
Authors: name: Ekta Prashnani, name: Koki Nagano, name: Shalini De Mello, name: David Luebke, name: Orazio Gallo
Published: 2023-05-05 17:54:34

Modern generators render talking-head videos with impressive levels of
photorealism, ushering in new user experiences such as videoconferencing under
constrained bandwidth budgets. Their safe adoption, however, requires a
mechanism to verify if the rendered video is trustworthy. For instance, for
videoconferencing we must identify cases in which a synthetic video portrait
uses the appearance of an individual without their consent. We term this task
avatar fingerprinting. We propose to tackle it by leveraging facial motion
signatures unique to each person. Specifically, we learn an embedding in which
the motion signatures of one identity are grouped together, and pushed away
from those of other identities, regardless of the appearance in the synthetic
video. Avatar fingerprinting algorithms will be critical as talking head
generators become more ubiquitous, and yet no large scale datasets exist for
this new task. Therefore, we contribute a large dataset of people delivering
scripted and improvised short monologues, accompanied by synthetic videos in
which we render videos of one person using the facial appearance of another.
Project page: https://research.nvidia.com/labs/nxp/avatar-fingerprinting/.
A Suite of Generative Tasks for Multi-Level Multimodal Webpage
  Understanding
Authors: name: Andrea Burns, name: Krishna Srinivasan, name: Joshua Ainslie, name: Geoff Brown, name: Bryan A. Plummer, name: Kate Saenko, name: Jianmo Ni, name: Mandy Guo
Published: 2023-05-05 16:38:05

Webpages have been a rich, scalable resource for vision-language and language
only tasks. Yet only pieces of webpages are kept: image-caption pairs, long
text articles, or raw HTML, never all in one place. Webpage tasks have
resultingly received little attention and structured image-text data left
underused. To study multimodal webpage understanding, we introduce the
Wikipedia Webpage suite (WikiWeb2M) of 2M pages. We verify its utility on three
generative tasks: page description generation, section summarization, and
contextual image captioning. We design a novel attention mechanism Prefix
Global, which selects the most relevant image and text content as global tokens
to attend to the rest of the webpage for context. By using page structure to
separate such tokens, it performs better than full attention with lower
computational complexity. Experiments show that the new annotations from
WikiWeb2M improve task performance compared to data from prior work. We also
include ablations on sequence length, input features, and model size.
Composite Motion Learning with Task Control
Authors: name: Pei Xu, name: Xiumin Shang, name: Victor Zordan, name: Ioannis Karamouzas
Published: 2023-05-05 05:02:41

We present a deep learning method for composite and task-driven motion
control for physically simulated characters. In contrast to existing
data-driven approaches using reinforcement learning that imitate full-body
motions, we learn decoupled motions for specific body parts from multiple
reference motions simultaneously and directly by leveraging the use of multiple
discriminators in a GAN-like setup. In this process, there is no need of any
manual work to produce composite reference motions for learning. Instead, the
control policy explores by itself how the composite motions can be combined
automatically. We further account for multiple task-specific rewards and train
a single, multi-objective control policy. To this end, we propose a novel
framework for multi-objective learning that adaptively balances the learning of
disparate motions from multiple sources and multiple goal-directed control
objectives. In addition, as composite motions are typically augmentations of
simpler behaviors, we introduce a sample-efficient method for training
composite control policies in an incremental manner, where we reuse a
pre-trained policy as the meta policy and train a cooperative policy that
adapts the meta one for new composite tasks. We show the applicability of our
approach on a variety of challenging multi-objective tasks involving both
composite motion imitation and multiple goal-directed control.
COLA: How to adapt vision-language models to Compose Objects Localized
  with Attributes?
Authors: name: Arijit Ray, name: Filip Radenovic, name: Abhimanyu Dubey, name: Bryan A. Plummer, name: Ranjay Krishna, name: Kate Saenko
Published: 2023-05-05 17:00:16

Compositional reasoning is a hallmark of human visual intelligence; yet
despite the size of large vision-language models, they struggle to represent
simple compositions by combining objects with their attributes. To measure this
lack of compositional capability, we design Cola, a text-to-image retrieval
benchmark to Compose Objects Localized with Attributes. Using Cola as a
testbed, we explore modeling designs to adapt pre-trained vision-language
models to reason compositionally about multiple attributes attached to multiple
objects. We explore 6 finetuning strategies on 2 seminal vision-language
models, using 3 finetuning datasets and 2 test benchmarks (Cola and CREPE).
Surprisingly, our optimal finetuning strategy improves a 151M parameter CLIP,
which disjointly encodes image and language during pretraining, to perform as
well as a 241M parameter FLAVA, which uses a multi-modal transformer encoder
during pretraining to attend over both vision and language modalities. This
optimal finetuning strategy is a lightweight multi-modal adapter that jointly
attends over both image and language features generated by the pretrained
model. We show this works better than common strategies such as
prompt/fine-tuning, or tuning a comparable number of unimodal layers.
AttentionViz: A Global View of Transformer Attention
Authors: name: Catherine Yeh, name: Yida Chen, name: Aoyu Wu, name: Cynthia Chen, name: Fernanda Viégas, name: Martin Wattenberg
Published: 2023-05-04 23:46:49

Transformer models are revolutionizing machine learning, but their inner
workings remain mysterious. In this work, we present a new visualization
technique designed to help researchers understand the self-attention mechanism
in transformers that allows these models to learn rich, contextual
relationships between elements of a sequence. The main idea behind our method
is to visualize a joint embedding of the query and key vectors used by
transformer models to compute attention. Unlike previous attention
visualization techniques, our approach enables the analysis of global patterns
across multiple input sequences. We create an interactive visualization tool,
AttentionViz, based on these joint query-key embeddings, and use it to study
attention mechanisms in both language and vision transformers. We demonstrate
the utility of our approach in improving model understanding and offering new
insights about query-key interactions through several application scenarios and
expert feedback.
Otter: A Multi-Modal Model with In-Context Instruction Tuning
Authors: name: Bo Li, name: Yuanhan Zhang, name: Liangyu Chen, name: Jinghao Wang, name: Jingkang Yang, name: Ziwei Liu
Published: 2023-05-05 17:59:46

Large language models (LLMs) have demonstrated significant universal
capabilities as few/zero-shot learners in various tasks due to their
pre-training on vast amounts of text data, as exemplified by GPT-3, which
boosted to InstrctGPT and ChatGPT, effectively following natural language
instructions to accomplish real-world tasks. In this paper, we propose to
introduce instruction tuning into multi-modal models, motivated by the Flamingo
model's upstream interleaved format pretraining dataset. We adopt a similar
approach to construct our MultI-Modal In-Context Instruction Tuning (MIMIC-IT)
dataset. We then introduce Otter, a multi-modal model based on OpenFlamingo
(open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and
showcasing improved instruction-following ability and in-context learning. We
also optimize OpenFlamingo's implementation for researchers, democratizing the
required training resources from 1times A100 GPU to 4times RTX-3090 GPUs,
and integrate both OpenFlamingo and Otter into Huggingface Transformers for
more researchers to incorporate the models into their customized training and
inference pipelines.