Learning Visual Representation
with Homological Labels

ICIAM2023, Waseda, Tokyo, 21 Aug. 2023
Shizuo KAJI (Inst. of Maths-for-Industry, Kyushu Univ.)
Joint with Yohsuke Watanabe (ZOZO inc.)codes and preprint
https://github.com/shizuo-kaji/PretrainCNNwithNoDataDeep Learning and Topological Data Analysis
Human is good at• Rough estimation• Panoramic view
• Discovering rules/invariancefrom a small number ofexamples• Explaining the reason

Deep Learning(DL) is good at• Precise observation• Memorising/imitating examples• Processing huge data
• Accurate operationDeep Learning(DL)

Topological Data Analysis(TDA)
Maths-based
globaltarynelempmcoBackground• DL achieves high performance but has some weakness• TDA has been proven effective in capturing datafeatures that conventional techniques have missed

Data-driven
localBiases in Deep Learning
Algorithmic biases– image models are locally mindedConvolutional Neural Networks
are shortsighted

ImageNet-trained CNNs are biased towards texture; increasing shape bias
improves accuracy and robustness, Geirhos et al. 2019

CNNs are easily deceived

Explaining and Harnessing Adversarial Examples
Goodfellow et al. 2014

CNNs are too sensitive to local information
Convolution is a local operation

They look similar locally,
but we see a clear difference if we zoom out
c.f. Manifolds are locally all Euclidean and homology distinguishes the global topology of them.

Biases in Deep Learning
Data biases-- not only labels but also images themselves are biased

Concerns with real image
Huge cost for data collection and annotation(ImageNet consists of 14M manually-labelled images)
Bias in the annotation and images(Labels reflect the bias of the labellers.
The Image distribution itself is also biased.)
Ryan Steed and Aylin Caliskan, 2021
“Image representations learned with unsupervised pretraining contain human-like biases”

Security issues(model inversion attack)Angwin, J., Larson, J., Mattu, S. &Rights and privacy issues
Kirchner, L. (2016) ”Machine Bias”(ImageNet use ”wild” images on Internet)

Topological Image Analysis
Observe locally, understand globally

Persistent Homology of an image
Characteristics of PH• It captures global topological features• Proved stability against pixel value change• Isometry invariance (translation, rotation)

Persistent Homology of an image

A real-valued function f: X -> R defined over a
topological space X (in our case, the square) CVPR
#696
CONFIDENTIAL
REVIEWan
COPY.
DO NOT DISTRIBUTE.
definesincreasesequence of subspaces
𝑋! ≔ 𝑥 ∈ 𝑋 𝑓 𝑥 < 𝑡}ks disshapes.
tinuousprovidel Data

162
163
164

Figure 1. An example of a growing sequence of spaces is165
166
167
168
169
170

Persistent Homology of an image

CVPR
PH records the birth and death thresholds of#696
696. CONFIDENTIAL
REVIEW components
COPY. DO NOT DISTRIBUTE.
connected(PH0) and holes (PH1)in the form [birth,death].
162
163
164

ysis

at seeks diss of shapes.
r continuous
ures provide
ogical Data
that appliesof the shapetent homold, and some

165
166
167
168
169
170

1. An example ofa growing
sequence of spaces
isEx. PH0 Figure= {[0,1],[0,2]}:twoconnected
components
171
defined by the sublevel sets of a function f on a rectan172
gular grid (an image),whoseatvaluesborn
t=0are represented by the173
numbers in the grid cells. The shaded regions indicate thePH1 = {[1,2]}:andfilled
174
sublevel setaX hole= {(x, borny) | f (x, aty) <t=1t}. Thedegree
0 per-at t=2.
tsistent homology is the set of intervals {[0, 1], [0, 2]}. For

175

EX. MEDICAL IMAGE SEGMENTATION
Images from Wikipedia

healthy

COPD

IPF

COPD: Chronic obstructive pulmonary disease is the third leading cause of death (WHO 2019)
IPF: Idiopathic pulmonary fibrosis is a progressive and irreversible disease1) Explainable feature (vs blackbox DL)
2) Robust and easily transferable (vs DL needs re-training)3) 3D analysis (vs conventional 2D slice-based analysis)

DoctorPersistentHomology

Deep Learning(Unet)

Number of parameters: 5 (PH) vs over 5 million (DL)Moreover, the 5 parameters have physiological interpretations
With N. Tanabe et al. (Kyoto University Hospital), 2021

EX. IMAGE CLASSIFICATIONWITH CNN+PH

The MNIST Dataset
60k(train)+10k(test) images
10 classes (0,1,…,9)28x28 black-and-white images
Accuracy of SoTA is over 99.8%
Too easy as a benchmark

Reduced MNIST
Only 10 training images(one image per class;one-shot learning)
Difficult!

REDUCED MNISTCLASSIFICATION RESULTS(RED: ORIGINAL BLUE: +HOMOLOGY)

per class accuracy

total accuracy

Adding homological information improves the performanceWith Sudo and Ahara, 2020

Teaching Topology to
Neural Networks with
Persistent Homology(1) Synthetic image generation(2) Label generation

Goal
Transfer learning based on pretrained CNNs has some problems•

Huge labelled images are necessary for pretraining (e.g., ImageNet)• Privacy and bias issues in the training dataset• The learned model is biased towards texture

Solution: Pretraining with synthetic images with a mathematical task
Training Convolutional Neural Networks without using natural images• No need for data collection• No need for manual labelling• Acquires robust image features based on topology

Topology helps to eliminate manual labour andfairness concerns in data preparation!

Transfer learning
Middle Edge Node

Attribution Based Encryption

Task A(general)

pretraining

Usually, we do not do thepretraining by ourselves but use
some off-the-shelf pretrained model

Dataset A (big)copyweights
Encoder NetworkPredictortraining
Dataset B (small)

Output inte

Neural recording

Decoder NetworkPython/C Compiler for Neural Net(Seto@Univ. of Tokyo City)

NeuralNetworkBAccelerator in Edg
Task(O’uchi@AIST)(yourtask)

Self-supervised learning
SSL is a method to train models
without manual labels.
SSL has been very successful in NLP.
A popular scheme,contrastive learning,uses the metric in therepresentation space
A. Jaiswal et al. 2021

Attribution Based Enc

Middle Edge NodePretraining with real dataPretraining

EncoderNetworktransferPredictor

Real images
with manual or self-supervised labels

Image inputThis can be problematic

Neural recordingfrom monkey visual cortex(Hayashi@AIST)

Downstream tasks

Attribution Based Enc

Middle Edge NodePretraining with artificial data
pretraining
by SSL
Encoder
Networktransfer

Predictor

Synthetic images
with mathematically computed labels

Image input

Neural recordingfrom monkey visual cortex(Hayashi@AIST)

Downstream tasks

hun oftenrall

395
1. Create an image of dimension 256×256, where the396
value of each pixel is drawn independently from397
the uniform distribution on [0, 1]. We regard the398image as a function f : {0, 1, 2, . . . , 255}2 → [0, 1].
Each channel of the image is generated by the following formula 399
2.whereFor af isfrequency
β random
drawn fromunito
400
an image parameterwith uniform
pixelthevalues,form distribution
on [1, 2],
set from [1,2].
ap401
and β is uniformly
drawn
bed
402!
!
""
F F T (f )(x, y)
ob403
g(x, y) = Re iF F T,
NFIDENTIAL
REVIEW COPY. DO NOT ((xDISTRIBUTE.
+ 1)2 + (y + 1)2 )βhe404(1)
er’s405
𝛽 controls the decay of high-frequency components
where F F T is the 2D Fourier transform and iF F T
ge.
406
is its inverse, and Re denotes the real part of ato407
atedcomplex number.
lly,
408
with
pful
409
3.
With
a
probability
ofp,
which
we
set
to0.5,
bina, therily
410
grise
by
Otsu’sthresholding
[26].
hose

Synthetic image generation

rma4. Each
Repeatthe process
three withtimesa probability
independently
tovalchannel
wasbinarizedof0.5.
Figure2.image
Exampleof synthesised
images, thecreateacolourwithRGBchannels.
theThe final image was converted to greyscale with a probability of 0.5.

411412413

Mathematical labelling of an image
We think of an image as a function
defined over the square grid.

Any mathematical invariant of the function
can be used as the label of the image
Through the regression task of the label, themodel learns the maths!

Labelling by Persistent Homology

input

binarise

Signeddistance

PH vector

Remark: although we use synthetic images here,
any image dataset can be used.

binarised Signed distance vectrised PH

Security Node for Edge & Cloud sides (Ikeda & Fujita@Univ. of T
Fully Homomorphic Based Encryption
for secured communication with remote sites

Middle Edge Nodeinput

regre

Attribution Based Encryption

InternetlossOutput interfacession

Decoder Network

Benchmark results

CIFAR100
100-category
Natural Image
classificationfrom top tobottom

ImageNet
PH-PI
FractalDB-10k

Train accuracy

Label
ScratchValidation accuracy

The performance is behind an ImageNet trained model,
but better than training from scratch

Animal20-category
animal silhouetteclassification

pretraining
IMN: ImageNet
FDB: FractalDBScratch: no pretrainingclass: class label

Noised CIFAR100

The classification isvery hard for human eyes.
How is topology robust
against noise?

Covid-19 CT classification
2-class covid vs non-covid classification (COVID-CT dataset)
Various scanning conditions and non-uniform images

Our model shows better performance than the ImageNet pretrained model.
Perhaps because ImageNet does not contain medical images.

eft)
100

CVPR 2022 Submission #696. CONFIDENTIAL
REVIEW COPY. DO NOT
723

PH vectorisation methods

PH-BCTab.
PH-HS
FDB-10kImageNet
4. In thisFDB-1kexample, thesetting of Labeland PH(PH-A,theCIFAR100ImageNet
dataset (theH-BC
FDB-1k76.6 CPH-HS
77.9 respectively)
75.3 forFDB-10k
78.1
85.0
ScratchLabelPH-PI
na-89.1 animal
dataset,respectively)
isthesame
exceptfor the
90.6
84.6
85.2
93.3
76.6
77.9
75.3
78.1
85.0

724
725702726703 702
PH-LS727704 PH-BC
703
728705 704
78.1
76.6

PH-HS
77.9
90.6

FDB-1
75.3
84.6

eri69.6
70.3 uses the78.4classlabelCIFAR100
used for the pretraining;
Label
89.1
90.6 PH-C
84.6
85.2computed 93.3
er- pretrained
729706label whileusesmethods.
the label 80.1
with
Animal80.7
91.0 the 90.1
89.1
odels
with various
705
put730707
persistence image. Pretraining with PH improves the706
els pretrained
withindicating
various methods.
etic731
performance,thebenefitof learning not only
708
Table
1.
Classification
accuracyofmodels
pretrainedwith variou
Persistence
Image (PH-PI)
Label(PHvectorisation)
dimensions
707
732
from dimension
the class labels100
but alsowith topology.
200
400 800Compar709
Persistence Landscape (PH-LS)
en733
ing withtheentryfor
50kin
Tab.
3,
wesee
our
syn708curve (PH-BC)
CIFAR100 76.8 77.7 77.4 75.0
Betti
710
ong
734
dimension
100 89.9
200 90.5
400 87.5800learnthetic
datasetslightly
bettermaterials
for709 histogram
Animaloffers 87.4
Birth-Life(PH-HS)
711
dimension1
the pretrained
735in
model
startsalittlebehindof
Label
ing CIFAR100
topology
than theCIFAR100
dataset,which
con76.8 77.7 77.4 75.0
, in terms
736712 710
CIFAR100 7
sistsof Classification
thesamenumberof 50k of
naturalimages.
Thisthetrainingaccuracy,andquickly
surpasses
Table
2. of
accuracy
PH-PI
modelspreAnimal
87.4
89.9
90.5
87.5
711
mal
737713could
be models
attributed
to terms
the design
ofboththegener-and valiAnimal
8
trainedwithdifferent dimensionsofthePHimage
vectorisationallotherinof
training
n of738714 712
ation
model
that
produces
patterns
atvarious
scales.
(thedimensionof the The
vectorof the regression
target).
Table
Classification
accuracy
ofinPH-PI
modelsdationconvergence
behaviour
PH-PI
ion
739715
Numberof synthetic
imagesusedpretraining
pretraining
In2.theaccuracy.
caseof the animal
dataset,withPHofpre713Table 2. Classification
trainedwithdimensionsof quite
thePH
vectorisation
anddoes
FractalDB-10kmodels
aresimilar,
sitting740
benot different
lead to a large
performance
gain.
This
may
716 714trained with different didataset
size
50k
200k
400k
800k
741
t is
(thedimension
ofthevector
oftheregression
target).
be dueto the smallof training images (1,600)
tween
Scratchand number
ImageNet.
717
715(the dimension of the ve742
set
76.1limited.
77.7The78.4
78.8agrees
and CIFAR100
the variety is quite
observation
7187430}.
716
with theresult in Tab.
indicatesofAnimal
88.63 that
89.9
91.0the necessity
91.6
719
dataset size 50k 200k 400k 800k
744
asa certain amount of data for learning topological fea717
dataset size 5720
CIFAR100
76.1
77.7
78.4
78.8
745
the Table
3.
Classification
accuracyofPH-PImodels
pretures.
Pretraining with natural images
718
746721
CIFAR100 7
the trained withthe synthesised
Animal88.6 dataset
89.9 of different
91.0 sizes.
91.6 The
719
747722
1, values for 400k are reproduced
Scratch Label
PH-C
from Tab.
1. PH-A
Animal8720748ges
69.6accuracy
70.3 of 75.3
Table 3. CIFAR100
Classification
PH-PI 72.4
models pre- 723 PH-A: animal dataset
Animal80.7 dataset
80.1 of86.5
83.2sizes. The 749724 721Table 3. Classification
trained withthe synthesised
different
PH-C: CIFAR100 dataset
750725
ges
722
Tab.Table
4. In
thisareexample,thesettingofLabel and PHvalues
for400k
reproduced
fromare notused
but computed
PH-PI
trainedwith thebysynthes
4. Classification
accuracyof Tab.
PH-PI1.models pre- Labels751nt,
726 723
C (PH-A,respectively)
for the CIFAR100(thevalues for 400k are repro
trained withwith the CIFAR100(PH-C) anddataset
the animal

Interpretation

What features are learned?

PH-PI

FractalDB-10k

ImageNetFirst convolution kernels of pretrained models

What the model focuses on?

IMNPH
Task: counting the number of connected components.
Visualisation: GradCAM++

IMN focuses more on edges?

What kind of mistakes the model makes?
IMN: ○ PH-PI: x

IMN: x PH-PI: ○bearsnakeshrew

girlroadotterbeaverlizardboy

mountainboywhalebottlelion

woman

rocketclockmanlizardskunk

lizardskyscraper

snail

plate

PH-PI seems to focus more on shape than texture

MATERIALS• Codes

https://github.com/shizuo-kaji/PretrainCNNwithNoData• PH computation

https://github.com/shizuo-kaji/CubicalRipser_3dim
S Kaji, T Sudo, K Ahara, Cubical Ripser: Software forcomputing persistent homology of image and volume data
• TDA Tutorial with Google Colabhttps://github.com/shizuo-kaji/TutorialTopologicalDataAnalysis
Interactive demo on various techniques of Topological Data
Analysis including Cubical RipserSUMMARY• Topology (persistent homology) provides a way to extract

image features that are not easy to obtain by conventional
method.
• CNNs can be pretrained with synthetic images, requiring nodata collection nor manual labelling• Making CNNs learn global features encoded by topology
leads to a performance gain

FUTURE WORK
• Tolerance test against adversarial attacks• Applicability for other tasks than classification• Theoretical analysis

Thank you!