01

Unsupervised Visual Representation Learning by Synchronous Momentum Grouping

Reporter： Hongguang Zhu

1.supervised learning:

2. unsupervised learning:• simple and efficient• Instance-level framework• expensive and time-consuming annotating costs• Clustering-based scheme• non-ideal generalization (annotation bias)

01

Unsupervised Visual Representation Learning by Synchronous Momentum Grouping

Instance-level framework(1) contrastive learning methods:• reduce the distance between two augmented views• push apart view from negative samples

SimCLR, MOCOs...

01

Unsupervised Visual Representation Learning by Synchronous Momentum Grouping(2) asymmetric network methods• get rid of negative sample• introduce predictor network• stop-gradient operation

idea: only adopt positive samples ↔ mode collapse• encode pervious sample information in weight of predictor• serve as negative gradient during back-propagation

DINO: centering replaces the projection and predictor
Understanding Self-Supervised Learning Dynamics without Contrastive Pairs

01

Unsupervised Visual Representation Learning by Synchronous Momentum Grouping(3) feature decorrelation methods• reduce the redundancy between differnet feature dimensions

Barlow Twins

01

Unsupervised Visual Representation Learning by Synchronous Momentum Grouping

Clustering-based scheme

SwAV: Swapping Assignments between multiple Views• clustering: K-means / optimal transport• but some restriction:• asynchronous two-satge training （DeepClustering）• no gradient propagate through group features• local equipartition constrainty to aviod degeneracy （SwAV）

code of view A → code of view B• not show an advantage in performance

instance-level discriminarion：• supervisory signal:every sample as a category• False negatives / Sampling Bias
• limited resource (large batch/ memory bank)Group-level discrimination：• supervisory signal hysteresis• local equipartition decrease the validity of grouping• not show an advantage in performance

01

Unsupervised Visual Representation Learning by Synchronous Momentum Grouping

This paper: SMoG
Novel:• integrate the instance contrastive and clustering based methods• contrastive group : allow gradient propagating through groups• synchronous: instance grouping and representation learning

Achieve:• avoid the false negatives and wipes off limiting factors of clustering methods• surpasses the vanilla supervised-level performance for the first time
• multiple downstream tasks , CNN and Transformer backbones

gap : 3% -> surpass surpervise

02

Unsupervised Visual Representation Learning by Synchronous Momentum Grouping

instance contrastive learning :(1) define group assigning function:• generate instance group ��• update the groups {g}How to design this function?
(1) �� can replace the lastest �� �� to participate in contrasting.
(2) group feature �� should update synchronously with instance feature �� ����� and ��� tend to be same group feature -> fail to gather similar instances and get them closer

02

Unsupervised Visual Representation Learning by Synchronous Momentum Grouping

group assigning function:
global clustering each iteration: computing cost -> momentum grouping scheme
momentum grouping scheme:
collapse:• the scale of the groups are unbalanced• the instance collapse into few groups-> periodically grouping on the cached large feature set to relocate the groups

02

Unsupervised Visual Representation Learning by Synchronous Momentum Grouping������
(1) momentum encoder → label:��� =argmin(sim(��� , �� ))(2) ��� update the groups → new group features:���pervious:• detach group features• not contrast groups but classify into groups

grouping contrasting:• contrasting among groups instead of classifying(3) ��� calculate the loss:(��� , new group features) ↔(��� , old group feature)(4) replace group features:

03

Unsupervised Visual Representation Learning by Synchronous Momentum Grouping
Linear Evaluation:

larger Resnet: not an increasing superiority over the supervised one

03

Unsupervised Visual Representation Learning by Synchronous Momentum Grouping

04

Unsupervised Visual Representation Learning by Synchronous Momentum Grouping

Grouping quality：

Periodical clustering is important！
Grouping number is not sensitive
Lastest feature as grouping feature is not good choice.