Replacing Labeled Real-image Datasetswith Auto-generated Contours
CVPR 2022

Hirokatsu Kataoka*, Ryo Hayamizu*, Ryosuke Yamada*, Kodai Nakashima*, Sora Takashima*,**,Xinyu Zhang*,**, Edgar Josafat MARTINEZ-NORIEGA*,**, Nakamasa Inoue*,**, Rio Yokota*,**•* National Institute of Advanced Industrial Science and Technology (AIST)• **Tokyo Institute of Technology

Pre-training Image

Attention Image【ImageNet-21k】

Fine-tuning@ ImageNet-1k Top-1 Acc.

81.8
【ExFractalDB-21k; Extended FractalDataBase】

82.7【RCDB-21k; RadialContourDataBase】

82.4
The performance of FDSL(formula-driven supervised learning) can match or even exceedthat of ImageNet-21k without the use of real images, human-and self-supervision during thepre-training of ViT (vision transformers).

Hypothesis 1:

Object contours are whatmatter in FDSL datasets@FractalDB[Kataoka+, ACCV20]
Image@RadialContourDB(RCDB)

Hypothesis 2:

Task difficulty mattersin FDSL pre-training@FractalDB[Kataoka+, ACCV20]@Extended FractalDB(ExFractalDB)x ∈ R2x ∈ R3

2D → 3D
Attention

ViT activatedon contours@RadialContourDB(only #vertices)
Mainly consists of contours
from the activation

Increase#params@RadialContourDB(parameter set η)Hypothesis 1:

Object contours are what matter in FDSL datasets@FractalDB [Kataoka+, ACCV20]
Image 1

Image 2

Attention 1

Attention 2

Image 3

@RadialContourDB(RCDB)

Attention 3

ViT activated on contours of fractal images

RCDB mainly consists of contours

Hypothesis 1:

Object contours are what matter in FDSL datasets

Procedure for generating radial contours

Hypothesis 2:

Increased number of parameters in FDSL pre-training@Extended FractalDB(ExFractalDB)@FractalDB[Kataoka+,ACCV20]x ∈ R2
2D → 3D@RadialContourDB(only #vertices)x ∈ R3
Increase#params@RadialContourDB(parameter set η)Verification of Hypotheses

Regarding Hypothesis 1, we confirm that image representation using object contours
tends to yield higher scores: RCDB and FractalDB give the highest accuracy.

Regarding Hypothesis 2, we confirm that more difficult tasks improved the accuracyof RCDB and FractalDB (here, ExFractalDB).

In point-rendered FractalDB, although the fractal images with 50k points trained thevisual representations, the fractal images with 10k points failed.

At the same time, the RCDB with broken contours failed to acquire a visualrepresentation. The attention and accuracy were also broken from the visualization
and result.

Comparison

n Pre-training with ExFractalDB-21k (82.7), RCDB-21k (82.4) outperformed that with ImageNet21k (81.8).
n We can match the accuracy of pre-training on ImageNet-21k with synthetic datasets in FDSL.

@Swin Transformer-Base backbone, Mask R-CNN head, 60 epochs fine-tuning

In COCO detection and segmentation, our pre-trained model achieved scores similar
to those for the model pre-trained with ImageNet-1k.

n In comparison to SSL, ExFractalDB-1k with 1.4k instances achieved a higher
average accuracy (88.6) than that of the self-supervised PASS dataset (87.8).
n PASS and FDSL both attempt to improve ethics in datasets.

In this paper, one of our major findings is that we can surpass theaccuracy of a ViT pre-trained on ImageNet-21k using our FDSL
datasets.

We believe that further improvements in contour shapes and a morecomplex classification task are possible, which leaves open thepossibility to scale up the pre-training on synthetic datasets to one
day outperform huge-scale datasets (e.g. JFT-300M/3B, IG-3.5B).

[Kataoka+, ACCV20/IJCV22]
FDSL Proposal
Spatiotemporal
Domain3D Domain
VisionTransformers

Video Perlin Noise[Kataoka+, WACV22]

Multi-viewpoint / Point Cloud[Yamada+, IROS22/CVPR22]
EnhancHypo ed bythese
FractalDB Pre-trained ViTs

[Nakashima+, AAAI22]

W hatʼs
Next??

Replacing Labeled Real-image Datasets (This work)[Kataoka+, CVPR22]