.
. -27Natural
7 / Images
Pre-training .without
A

Hirokatsu Kataoka1, Kazushige Okayasu1,2, Asato Matsumoto1,3,Eisuke/ Yamagata4/, Ryosuke
Yamada1,2, /,/2, Yutaka Satoh1,3Nakamasa Inoue4, Akio
Nakamura:- //2

1: AIST 2: TDU 3: Univ. of Tsukuba 4: TITech

Recent vision-driven learning•

Supervised Learning

e.g. ImageNet, Places, Open Images

Pre-train

Fine-tune

ImageNet + ResNet-50
76% @ImageNet val.
[He et al. CVPR16]

gluon-cv.mxnet.io•

Self-supervised Learning (SSL)
SimCLR + ResNet-50
69%@ImageNet val.
[Chen et al. ICML20]

Jigsaw Puzzle

DeepCluster

Rotation Classify[Noroozi al. ECCV16][Caronet al. ECCV18][Gidaris et al. ICLR18]

Existing the problems of image downloading and privacy-violations.

Can we pre-train CNN without any natural images?
•Formula-driven Supervised Learning (FDSL)–– With any mathematical formulas and functions

CNN trains a natural principle
from ImageNet dataset?
Directly render and train Fractals
Fractal geometry from ImageNet dataset

To replace a human-annotated dataset in context of pre-training
without any natural images and human labels

Proposed method: FractalDB•••

FractalDB
1) to make a pre-trained CNN without any natural images2) for a concept of Formula-driven Supervised LearningProposed method: FractalDB•••

FractalDB
1) to make a pre-trained CNN without any natural images2) for a concept of Formula-driven Supervised Learning

Surprising results which are similar to theeffects of a supervised dataset

ctseffg e eNetningtrai ImaePr ilar tosimPre-trainUnique feature representationAttention by Grad-CAM

(()# Transformation probability# Affine transformation

Definition of fractal category•

Randomly searched image category
1. Image rendering with randomized af, w through IFS
2. Add category c if filling rate (> r) in the image
3. Iterate up to defined #category (C)• Parameter separation makes a different category

Fractal categories in FractalDB

Instance augmentation in category

Experimental setting•

Pre-training & Fine-tuning– Pre-training without any natural images– Fine-tuning in an ordinal way

Pre-training on Natural Image DatasetFractalDB pre-training
FractalDB

Finetunee.g. CIFAR-10/100, Places, ImageNet

Parameters on FractalDB•

After the burden of exploration study,– #Category, #instance, and patch-rendering are the mosteffective parameters in pre-training– A more difficult pre-train is slightly better in weights

Please refer to our main paper.

Results (1/5)

Underlined bold: best score, Bold: second best score

Results (1/5)

Underlined bold: best score, Bold: second best score

FractalDB pre-trained model achieved much higher rates than training from scratch

Results (1/5)

Underlined bold: best score, Bold: second best score

In the most cases, our method is better than the DeepCluster with 10k categories

Results (1/5)

Underlined bold: best score, Bold: second best score

The FractalDB pre-trained model is still better than 100k-order supervised datasets

Results (1/5)

Underlined bold: best score, Bold: second best score

Our method partially surpasses the ImageNet/Places pre-trained models

Results (2/5)

Bold: best score

DC-10k with fractal images cannot effectively pre-train to recognize natural images
This shows our method assigns an appropriate image pattern and the category

Results (3/5)

Full fine-tuning is the best
Moreover, earlier layers tend to be good feature representations

Results (4/5)

We compare Formula-driven Supervised Learning with other principles
The FractalDB pre-trained model outperforms other methods

Results (5/5)
Visualization of Conv1

FractalDB pre-trained model acquires different representations yet look at a similar area•

If we could improve the FDSL, ImageNet pre-trainedmodel may be replaced so as to protect fairness,preserve privacy, and decrease annotation labor.

Thank you for watching.