TransformerMengxue

1 Transformer

Neural Information Processing Systems (NIPS 2017)

RNN&LSTM→Long-Term Dependencies

RNNLSTM1 可并行计算（RNN只能顺序传播）

Transformer →2 避免长程依赖的问题（用position embedding，可以快速找到远处）
3 自注意力可以产生更具可解释性的模型。我们可以从模型中检查注意力分布。
各个注意头 (attention head) 可以学会执行不同的任务Tips▪ Self-attention▪ Multihead Self-attention▪ Position encoding▪ Masked Multi-Head Attention▪ Cross Attention

Vector Set as Input

One-hot Encoding

thisisa cat

Word Embedding

apple = [ 1 0 0 0 0 …… ]bag

= [ 0 1 0 0 0 …… ]cat

= [ 0 0 1 0 0 …… ]dog = [ 0 0 0 1 0 …… ]elephant = [ 0 0 0 0 1 …… ]dogrunjumprabbit

cattreeflower

Sequence-to-sequence (Seq2seq)
SpeechRecognition

機器

學

習

MachineTranslation

machine learning你嗎machine learning

機SpeechTranslation

好器

學

習

Start with Fully-connected

FCFC

FC

FC

I

saw

asaw

window

How to consider the whole sequence? →Self-attentionSelf-attention

Self-attention

Transformer

Self-attention
Dot-product𝛼1,2 = 𝒒𝟏 ∙ 𝒌𝟐

𝛼1,3 = 𝒒𝟏 ∙ 𝒌𝟑

𝛼1,4 = 𝒒𝟏 ∙ 𝒌𝟒

𝛼1,2

𝛼1,3

𝛼1,4

𝒌𝟐 key𝒌𝟑

𝒌𝟒

𝒂𝟏𝒂𝟐𝒂𝟑𝒂𝟒

𝒒𝟏 = 𝑊 𝑞 𝒂𝟏

𝒌𝟐 = 𝑊 𝑘 𝒂𝟐

𝒌𝟑 = 𝑊 𝑘 𝒂𝟑

𝒌𝟒 = 𝑊 𝑘 𝒂𝟒

𝛼 =𝒒∙𝒌attention
score
𝒒𝑊𝑞𝒌𝑊𝑘𝒒𝟏 querySelf-attention
𝒃𝟏′
𝛼1,1′
𝛼1,2′
𝛼1,3′
𝛼1,4

𝒌𝟐 𝒗𝟐

𝒌𝟑 𝒗𝟑

𝒌𝟒 𝒗𝟒

𝒂𝟏𝒂𝟐

𝒂𝟑𝒂𝟒

𝒗𝟏 = 𝑊 𝑣 𝒂𝟏𝒗𝟐 = 𝑊 𝑣 𝒂𝟐𝒗𝟑 = 𝑊 𝑣 𝒂𝟑𝒗𝟒 = 𝑊 𝑣 𝒂𝟒

𝒒𝟏 𝒌𝟏 𝒗𝟏

Self-attention
𝒃𝟐′
𝛼2,1′
𝛼2,2′
𝛼2,3′
𝛼2,4

𝒒𝟏 𝒌𝟏 𝒗𝟏𝒒𝟐 𝒌𝟐 𝒗𝟐

𝒒𝟑 𝒌𝟑 𝒗𝟑

𝒒𝟒 𝒌𝟒 𝒗𝟒

𝒂𝟏𝒂𝟐𝒂𝟑𝒂𝟒

Self-attention

Q= 𝑊𝑞IK= 𝑊𝑘IV= 𝑊𝑣

I

Parameters to be learned

A′A= 𝐾𝑇QVA′Attention Matrix

O=Multi-head Self-attention𝒃𝒊 =𝑊𝑂𝒃𝒊,𝟏𝒃𝒊,𝟐

Positional Encoding
learned from data →BERT

𝒒𝒊 𝒌𝒊 𝒗𝒊
𝒆𝒊 + 𝒂𝒊

Word embedding output: (b, N, 512) → self.pos_embedding = nn.Parameter(torch.randn(1,N,512))

hand-craftedSeq2seq Modeloutput sequence

Encoder

Decoder

Input sequence

Transformer→

Encoder𝒉𝟏𝒉𝟐

𝒉𝟑𝒉𝟒

Encoder𝒙𝟏𝒙𝟐𝒙𝟑𝒙𝟒

…

𝑥1′𝑥2′⋮𝑥𝐾′𝒃norm

𝒂+𝒃+

𝜎norm

Layer Norm

norm

residual𝑥𝑖 − 𝑚′𝑥𝑖 =

𝒂𝑥1𝑥2
⋮𝑥𝐾

https://arxiv.org/abs/1607.0645
0mean 𝑚standard
deviation 𝜎

Self-attention+
FCDecoder機max

器

Encoder( 機器學習 )

max

max

max

習學

END

max

Decoder

START

機器學

習19

DecoderMasked Multi-Head Attention

Encoder

DecoderCross attentionFC𝒗

𝛼1′𝛼2′𝛼3′Crossattention

𝒗𝟏

𝒌𝟏

𝒂𝟏𝒗𝟐

𝒌𝟐
𝒂𝟐

Encoder

𝒗𝟑

𝒌𝟑

𝒒𝒂𝟑
Self-attention (Mask)
START2ICLR 2021 oral
ViT→Visual Transformer
Motivation: 将Transformer直接应用到图像领域，做图像分类任务，不修改Transformer结构，完全不加CNN。

ModelPatch Embedding+Positional Encoding→ 分块

→ 维度转换

Transformer Encoder

Fine-tuning and high resolution
1. Pre-train ViT on large datasets

Fine-tune to (smaller) downstream tasks
2. Remove the pre-trained prediction head
Attach a D×K feed forward layer
K is the number of downstream classes

3. If high resolution, keep the patch size the same
but a larger effective sequence lengthExperiments▪ Pre-train datasets:▪ ILSVRC-2012 ImageNet dataset: 1000 classes▪ ImagetNet-21k: 21k classes▪ JFT: 18k High Resolution Images▪ Fine-tuning datasets:▪ CIFAR-10/100▪ Oxford-IIIT Pets
▪ Oxford Flowers-102▪ VTABExperiments

3 SETRCVPR 2021
Motivation:
To break the standard FCN segmentation model, which has an encoder-decoder architecture:
the encoder is for feature representation learning, while the decoder for pixel-level classificationof the feature representations yielded by the encoder.
We propose to replace the stacked convolution layers based encoder with gradually reduced

spatial resolution with a pure transformer , resulting in a new segmentation model termed
SEgmentation TRansformer (SETR).

ModelMethod
Image to sequence:
1. Image to grid patches → 分块Length L:
2. Linear Projection → 维度转换

3. Position Embedding → 位置嵌入Transformer:

Decoder designs
1. Naive：Naive upsampling

2. PUP：Progressive Upsampling
restrict upsampling to2×

3. MLA：Multi-Level feature Aggregation

Experiments▪ Datasets: Cityscapes、ADE20K、PASCAL Context

4
ICLR 2020

Embedding&Pre-Training Task in BERT→ 上游任务：Mask Language Modelling & Next Sentence Prediction→ 下游任务：句子对分类任务、单句子分类任务、问答任务、单句标注任务Embedding in VL-BERT
下游任务：
Visual Commonsense Reasoning
Visual Question Answering (VQA)
Referring Expression task

Visual Feature Embedding : concat(Appearance Feature, Geometry Embedding)Appearance Feature: For visual element → RoI feature extracted by Fast(er) R-CNN
For non-visual elment → feature extracted on the whole image
Geometry Embedding:

Segment Embedding : Three types of segment, A, B, C, which means different input format.
• For example, for input format of , A denotes Question, B denotes Answer, and C denotes Image. For
input format of , A denotes Caption, and C denotes Image.

Pre-Training Task in VL-BERT
Task#1: Masked Language Modeling with Visual Clues· Predict the masked words  unmasked words and the visual features.
· Final output feature([mask]) → classifier → word (cross-entropy loss)

Task#2: Masked RoI Classification with Linguistic Clues· The pixels laid in the masked RoI are
set as zeros before applying Fast R-CNN· Final output feature([masked RoI]) →classifier → object category classification

Dataset for pre-training : Conceptual Captions

FINE-TUNING ON DOWNSTREAM TASKS

VQAReferring Expression

5CVPR 2021
Motivation:
Tokens are all of a fixed scale ,which is unsuitable for vision applications
Higher resolution of pixels in images compared to words in passages of text.

Overall Architecture

Hierarchical feature maps → Patch Merging

Swin Transformer Block•W-MSA (Window-Multihead Self Attention)•SW-MSA (Shifted Window-Multihead Self Attention)

W-MSA and SW-MSA

Efficient batch computation for shifted configuration

Relative position bias
具体思想参考UniLMV2Architecture VariantsExperiments

Paper List :Transformer
Attention is all you need
Visual Transformer
AN IMAGE IS WORTH 16X16 WORDS:TRANSFORMERS FOR IMAGE RECOGNITION AT SCALESETRReviewRethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Swin Transformer

Hierarchical Vision Transformer using Shifted Windows
BERT
Pre-training of Deep Bidirectional Transformers for Language Understanding

VL-BERT
PRE-TRAINING OF GENERIC VISUAL LINGUISTIC REPRESENTATIONSThank you