Self-Supervised Learning韩坤洋Self-Supervised Learning
Segmentation• Task: predict logits mask (supervised)• Image -> Semantic information -> logits mask prediction• Self-supervised task, in unlabeled dataset• Image -> Semantic information• Predict logits mask, use conv layers• Semantic information -> logits mask predictionSelf-Supervised Learning• Distortion• Patches• Rotation

Paper list
• Momentum Contrast for Unsupervised Visual Representation Learning• Improved Baselines with Momentum Contrastive Learning• Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual
Representation Learning• Exploring Simple Siamese Representation Learning

Momentum Contrast for Unsupervised Visual
Representation Learning• Mechanism for building dynamic dictionaries for contrastive learning

MoCo - Dictionary Look-up• Keys in the dictionaries• Sample from data, images or patches• Represented by encoder network• Encoded ‘query’• Similar to its matching ‘key’• Dissimilar to others• LossMoCo - Momentum Contrast• Dictionary as a queue• Dictionary can be much larger than batch size
• Current enqueued, oldest dequeued• Momentum update• Hard to train key encoder, because large dict
• Update key encoder param by:

MoCo - Details• ResNet as backbone• 128-D vector as output, L2-norm• Transform• Resize and 224 x 224-pixel crop
• Color jitter, horizontal flip, grayscale conversion• Shuffling BN, BN performs bad• Intra-batch communication leaks information• Shuffle the sample order before key encoder

MoCo - Details• Dataset• ImageNet-1M, 1.28 million images, 1000 classes• Instagram-1B, 940 million images, 1500 hashtags• Train• SGD• IN-1M, 256 batch size in 8 GPUs, 53 hours• IG-1B, 1024 batch size in 64 GPUs, 6 days

MoCo - Experiment

MoCo - ExperimentImproved Baselines with Momentum
Contrastive Learning• Implement SimCLR’s design in the MoCo framework• MLP head
• Data augmentation• Cosine lr schedule

MoCo v2 - MLP head
• Replace fc head with a 2-layer MLP head
• 2048-d hidden layer• Only influences unsupervised training stage

MoCo v2 - Augmentation• Blur augmentation• Color jitter (strengthened -> random)

MoCo v2 - Comparison

Propagate Yourself: Exploring Pixel-Level
Consistency for Unsupervised VisualRepresentation Learning• The first task directly applies contrastive learning at the pixel level.

PixPro – Pixel Contrast• Compute pairs of pixels, from two views• Contrastive loss

PixPro – Pixel-to-Propagation Consistency• Spatial sensitivity• Discriminate spatially close pixels• PixContrast• Spatial smoothness• Encourages spatially close pixels to be similar

ProPix - PPM• PPM function• g(*)• L conv layers with BN and ReLU

PixPro - PPC Loss

PixPro - Experiment

PixPro - Experiment

Exploring Simple Siamese Representation
Learning
SimSiam can learn meaningful representations without following:• Negative sample pairs• Large batches• Momentum encoders

SimSiam• Encoder f• ResNet, projection MLP head•h• Prediction MLP head

• D, cosine similarity

SimSiam - Stop GradientSimSiam - Experiment

SimSiam - Experiment• If h removed

with magnitude scaled by 1/2• Using stop-grad == removing stop-grad and scaling loss by 1/2

SimSiam - Experiment

SimSiam - Experiment

SimSiam - Experiment