ViCE: Improving Dense Representation Learning by Superpixelization
and Contrasting Cluster Assignment
Robin Karlsson (1), Tomoki Hayashi (1), Keisuke Fujii (1), Alexander Carballo (2), Kento Ohtani (1), Kazuya Takeda (1,3)(1) Nagoya University, (2) Gifu University, (3) TIER IV

Email: karlsson.robin@g.sp.m.is.nagoya-u.ac.jp

Full version: https://arxiv.org/abs/2111.12460v3

Self-supervised dense representation learning

Github: https://github.com/robin-karlsson0/vice

1280x720 pixels ⇒ 921,600 pixels/vectors

・Recent self-supervised classiﬁcation models are better or equal to fully

supervised models
・However, these methods are ineﬀective for learning dense representations
・Typically reduce images into a tiny feature 7x7 map that discards
precise spatial information about content
・Prohibitive computational demand for contrastive learning methods [1]on high-resolution images

Resnet50

Embedding RGB visualizations (7x7)Superpixelization

Regional decomposition by superpixels
・Decompose images into a small set of visually coherent regions

・Adjacent similar pixels represent same semantics (no information loss)・Superpixel regions conform to image content (unlike grids)
・Reduce computational complexity by O(1000)
・Improve eﬃciency of contrastive methods on high-resolution images
・Experiments show that learning from high-resolution images is beneﬁcial

ViCE2304 vectors (99.75% reduction)

ViCE: Visual Concept Embeddings

transformed by a stochastic generative process f(X|Z)
・Stochasticity: Assume a single latent Z representation maps to many potential images X
・Same visual content but diﬀerent pixel appearance
・Method: Learn an approximative inverse function f’(Z|X) and a set C

・Objective optimize the similarity of embeddings withinall views, as well as the distribution of visual concepts
・Transform views into visualconcept embedding maps

・Generate M augmentedviews

Fundamental learning signals

Embedding RGB visualizations (1280x720)∗ Superpixels onlyused during training

・View natural images X as a result of a dense representation Z of latent visual concepts C = (c1, … , cK)

Learning method

Spatial informat
ion aboutcontent is lost②⑦③

・Same content with diﬀerent pixel
appearance by data augmentation [2,3]・Semantic meaning from spatialco-occurrence and context (similar tolearning word embeddings)⑥
・Partition image into I mutually common
superpixel regions
・Simple Linear Iterative Clustering (SLIC) [4]④①

Experiments

・Arrange all vectors in a tree structure
for convenient region-wise indexing⑤
・A single mean vector is computed
to represent each region

・Evaluate representation quality through 1) unsupervised semantic

segmentation and 2) linear model performance
・Low-resolution and high-resolution performance comparisons・Score eachembedding vector
in terms ofcloseness to eachconcept vector

Results

・Image decomposition improves performance and reduce computational time・Online clustering (ViCE) > Oﬄine clustering (baseline) [3]・Learning from high-resolution images is beneﬁcial
・Superpixels improve performance and computational time compared with grids
・General vision models learn more useful embeddings also when comparing on a
narrow domain

Conclusions

・Present a new SOTA unsupervised semantic segmentation method ViCE forlearning to generate dense embedding maps for high-resolution natural images
・Decomposing images by superpixelization improves the eﬀectiveness ofclassiﬁcation-based self-supervised methods
・Superpixels perform better than conventional grid decomposition
・Hope our work will raise interest in incorporating non-uniform image
decomposition techniques to improve other self-supervised computer vision
methods including ViT-based models [5]References: [1] Caron M. et al. NeurIPS, 2020 [2] Chen T. et al. ICML 2020 [3] Cho J. et al. CVPR 2021
[4] Achanta R. et al. PAMI, 2274-2285, 2012 [5] Caron M. et al. ICCV, 2021

C K: evaluation w. K clusters, ◇ reproduced results, ⋆ greedy assignment, ∗ ViT-based models

Acknowledgements: This work was supported by supported by JST SPRING (Grant Number JPMJSP2125), JSPS KAKENHI (21H04892, 21429770), OPERA (JP-MJOP1612)
The computation was carried out through the “General Projects” program on the supercomputer “Flow” at the Information Technology Center, Nagoya University