Learning Visual Representation with Synthetic Images and Topologically-defined Labels
Authors: Shizuo Kaji
Yohsuke Watanabe
Venue: Submitted to ICLR 2023 (2023)
Keywords: topology
persistent homology
self-supervised learning
synthetic image

Abstract: We propose a scheme for neural networks to learn visual representation with synthetic images and mathematically-defined labels that capture topological information. To verify that the model acquires a different visual representation than with the usual supervised learning with manually-defined labels, we show that the models pretrained with our scheme can be finetuned for image classification tasks to achieve an improved convergence compared to those trained from scratch. 
Convolutional neural networks, built upon iterative local operations, are good at learning local features of the image, such as texture, whereas they tend to pay less attention to larger structures. Our method provides a simple way to encourage the model to learn global features through a specifically designed task based on topology. Furthermore, our method requires no real images nor manual labels; hence it sheds light on some of the lately concerned topics in computer vision, such as the cost and the fairness in data collection and annotation.

Under review as a conference paper at ICLR 2023
LEARNING VISUAL REPRESENTATION WITH SYN- THETIC IMAGES AND TOPOLOGICALLY-DEFINED LA- BELS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a scheme for neural networks to learn visual representation with synthetic images and mathematiiallly-defnee labels that capture topological infor- mation. To verify that the model acquires a different visual representation than with the usual supervised learning with manually- the models pretrained with scheme can be finetuned for image classification tasks to achieve an improved convergence compared to those trained from scratch. Convolutional neural networks, built upon iterative local operations, learning local features of the image, such as texture, whereas they tend to pay less attention larger structures. Our method provides a simple way to encourage the model to learn global features through a specifically designed task based on topology. Furthermore, our method requires no real images nor manual labels; hence it sheds light on some of the lately concerned topics in computer vision, such as the cost and the fairness data collection and annotation.
labels, we show that
good
INTRODUCTION
seff-supervised learning (SSL) has seen a great success both practically and scientifically by providing scheme train neural networks without human annotated labels and by showing the similarity to humans in the learning process and the learned representation. However, there are still large gaps between SSL and human learning. In visual representation, convolutional neural networks (CNNs) are famously known to be "shortsighted" by being biased to textural information (Geirhos et al. 2019). Most image features, hand-crafted or obtained in a data-driven manner, are local the sense that they are computed from patches in the image. For example, convolution, a typical and versatile form of the operation used to construct various image features, is local unless a huge kernel is used. Although local features can be aggregated by applying reduction operations such the mean, the global information obtained in this way limited. One way to capture global characteristics of model architecture. The attention mechanism has enabled to learn global
images is to introduce image features a data-driven manner as demonstrated by Vision Transformer (Dosovitskiy et al. 2021). Another way, which concern in this paper, is to device new training scheme, which works with virtually any model architecture with little modification. Topology is a study of shapes whose ultimate goal is classify shapes by their topological types. Topologists have invented various topological invariants that can discern different shapes. For example, homology is used to classify manifolds that are locally the same Euclidean space but globally different. Our idea is design task of computing invariants from an input image by a neural network so that the neural network is encouraged learn visual representation that is relevant the topology of the image. In contrast to the popular SSL schemes, which are largely based on learning the distribution of real (plausible) images, our method based on approximating the mathernatica computation of an invariant, called persistent homology, of synthesised images.
Persistent homology (PH), one of the main tools of the emerging field of Topological Data Analysis (TDA), provides efficient machinery for computing global topological features of data mathemat- ically (Adams & Moy 202I). has been proved to be practically useful for image processing such as classification (Dunaeva et al., 2016) and segmentation (Tanabe al. 2021). In recent years, applications of persistent homology have expanded to many areas in science and led to new discoveries (Giunti 2021), but in most cases, it is used just yet another feature extractor. In view

---

Under review as a conference paper at ICLR 2023
of TDA, we proceed step further and ask if we teach topology neural network so that the model learns low-level image features together with the mechanism for computing high-level topological features from low-level features. Answering this question affirmatively would lead to the acquisition of more abstract and generalisable representation by the model. More precisely, our scheme is train the model with regression of the vectorised persistent homology for synthesised images. Through this task, the model is expected to learn relevant visual representation required to approximate persistent homology. We demonstrate the validity of our scheme by experiments showing that a CNN pretrained by the proposed method can be finetuned for image classification tasks to show an improved convergence compared with one trained from scratch.
It is also worth pointing out that our scheme does not rely on real images nor labels, but uses mathematically generated images annotated with mathematically defined features. In this way, it is free from human bias which lies not only in the manual annotation but also the image themselves; photos reflect the present world and the view of the photographer. In fact, the models trained with ImageNet by SSL, even without human annotated labels, are known to be subject to bias (Steed & Caliskan 202T
2
RELATED WORK
The proposed method is built on three key ingredients, which we discuss in this section. We introduce some novel ideas to each of the three and combine them to develop our scheme.
2.1 SELF-SUPERVISED LEARNING ON IMAGES
superviue learning tasks, which we call pretext tasks, are designed to learn image features without manually-defined labels. There are three major types of pretext tasks. The first task is to tell if given two images come from the same image or not, where variations of images are generated by applying transformations in spacial and colour domains (see Jaiswal et al. (202T) for survey). The second task is to undone degradation, such as adding noise and masking, and reconstruct the original image. The third is similar to the second one but to perform pair of (approximately) invertible processes, such as weeppeessooneeppaniiine Salakhutdinov, 2006). All of the three tasks demand the model to acquire high-level representation of the images. The main objective of these methods lies in, more or less, capturing the distribution of the training data; to be good at in-painting or compression, one has to find a low-dimensional manifold which models the training data well. We propose another type of task that put more emphasis on the computation process rather than the distribution of the data. When the computation is based on a certain mathematical structure of the data, a model will be incentivised to focus on the structure through learning the task. Our pretext task is to approximate the computation of persistent homology of the image. Persistent homology can be computed mathematically from the data and used as the label for regression task. It is also notable that our pretext task does not rely on semantics or human perception but solely on mathematics. This allows us to use synthetic images almost meaningless to human eyes, and make the procedure completely free of real data.
as represented by the celebrated autoeneoder (Hinton &
2.2 LEARNING WITH SYNTHETIC IMAGES
Even though SSL saves the annotation costs, the preparation of training data is still a vexing problem. Publicly available datasets can be of low-quality, subject to bias, or violating usage rights and privacy. ImageNet, one of the most popular large-scale datasets, suffers from fairness issues ( (Yang et al. 2020), and there have been growing interest in the fairness of machine learning Mchrabi et al. (2021). learning/Mchrabi
Steed & Caliskan (2021) points out that even models trained on ImageNet with SSL without using labels learn racial, gender, and intersectional biases from the way people are stereotypically portrayed on the web. No matter how much care and attentions are paid for data collection, it is impossible to be free from all kinds of these issues as long as real images are used. Using generative adversarial networks (GANs) to generate image datasets for training is popular and successful strategy to mitigate the situation (Besnier et al. 2020). but GANs are also trained with natural images and cannot avoid above- mentiondd problems. A promising approach is to use a algorithmically synthesised images. Formula-din driven Supervised Learning introduced in Kataoka et al. (020)) considers pretraining with synthetic images generated by mathematical formula. The labels are assigned according to the
2

---

Under review as a conference paper at ICLR 2023
parameters of the image generation. Several different formuli are tested and an iterated function system, which generates fractal images, is found to be effective. wider variety of image generation methods have been tested since then (Baradad et al. 2021 Kataoka et al. 2022 To see how synthetic images are helpful in acquiring image features is interesting also in terms of cognitive science.
In this paper, we also synthetie images, which not very meaningful for human eyes, and try to learn topological features from them. One technical difference from the Formula-driven Supervised Learning how the labels are generated. Instead of fixed labels associated with the image generation model parameters, we generate labels by a mathematical formula computed directly from the images, which has advantage of being applicable encode topological features (persistent homology, in our paper) and the model is encouraged to learn image features that are relevant to approximate the topological features.
any image generation model. The labels
2.3 PERSISTENT HOMOLOGY FOR IMAGE ANALYSIS
Persistent homology has an unusual input and output when it is seen as feature extractor. PH takes a series of nested topological spaces and outputs multiset of intervals of the real numbers. We view an image function nested spaces
X + R defined over a rectangular domain X, and obtain series of
1 c Xt, C C22, C. C. XLm = xX, X, = ((x,)) E (( st).
where fm = max(f). Applying the homology functor Ha(-) with the coefficients the field F2 with two elements, we obtain the corresponding sequence of F2 vector spaces, and this sequence is by definition the persistent homology PHaa(X,)) f) of the pair (X, f). PH f) can be written as the direct sum of so-called the interval module having the form
0
F2
F2
0
11
Ha( + .... + Ha(Xx; + Ha(X.)) + .... Ha(Xi, 1) + Ha(X,,) + ... Ha(X). We denote the summand by the interval [t., ty). When d = O. it may happen that j-1 m, in which case we represent the summand by [tz, oc). See Fig. for an example of PH of image. To sum up, PH takes function f X - R and outputs multiset of intervals. The alien output as multiset can be transformed into a fixed-length vector using vectorisation techniques (Adams) al. 2017 Bubenik 2015: Chung & Lawson 2021) so that it fits in the standard machine learning pipeline.
CNNs tend to be biased towards texture when trained with a classification task of natural im- ages (Geirhos et al., 2019). In contrast, human perception relies much on the global shape of the image content. Therefore, it would be beneficial for model to learn global topological features as well as local features. This is supported by an experiment in which combining PH with other descriptors results in increased performance in object recognition (Li et al.) 2014), showing that PH provides shape feature that is complementary to conventional ones. There are previous studies to assimilate PH into deep learning. To deal with PH with neural networks, parametric representation of persistence homology with learnable parameters is introduced in Hoter et al. (2017) so that a task-optimal vectorisation obtained data-driven manner. Dedicated architectures of CNNs are designed in Som et al. (2020) for computing vectorised PH of time-series and point clouds in the form of the persistence image. The topological autoencoder (Moor et al. 2020) learns a latent space of point cloud that preserves the topological structure in terms of persistent homology. Our idea is based on the fact that one has to acquire certain high-level image representation that encodes global topological features in order to compute PH, and this makes good pretext task for training a neural network.
3 METHOD
Our proposed scheme can be viewed as a type of SSL in which desired image features are learned by solving pretext task. The pretext task does not require any natural images, but synthesised images [Sec.3.11 are used. The model learns the input- relation (regression) for the image and the corresponding image feature vector computed through persistent homology (Sec. 3.2h. The model is
3

---

Under review as a conference paper at ICLR 2023
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
2
0
0
0
1
2
0
0
0
1
2
0
1
1
1
2
0
1
1
1
2
0
1
1
1
2
0
0
1
0
0
0
0
1
0
0
0
0
1
0
0 0
0st<1
1st<2
2
Figure 1: The figure shows the sublevel sets of an image for different ranges of t. For an image, persistent homology consists of two components PHo(X. f) and (X, (X,
which respectively records the transition of islands and holes in the sublevel sets under different threshold. In this example, PHo(X. component consisting of the single pixel the bottom-left corner in the left-most figure, which is merged to the other connected component in the central figure. Since this feature exists for E [0, 1), it is represented by the interval [0, 1), and said to have the birth time 0, death time 1, and life time 1[= 0). The other component that exists for 2 0 is represented by [0, co), which has the infinite life time. Similarly, PHz (X,
{0, 1), (0, o.)), where the interval [0, 1) corresponds to the connected
{[1, 2)} whose element represents the hole surrounding the two pixels with the value 2 in the central figure. This hole disappears in the right-most figure, and so, represented by [L. 2). Persistent homology of am image records the topological features, such as islands and holes, with the threshold in which the features emerge and disappear. The distinctive idea of persistence to trace the emergence and disappearance rather than computing features each threshold separately.
Mathematical formula
Pretraining
Neural Network
Persistent homology
Synthetic Image
Label
Finetuning
Human annotation
Real Image
Figure 2: The overall structure of the proposed method.
expected to learn relevant image features which are required for approximating PH. Intuitively, the procedure described as follows. The teacher knows topology and gives problems to the student with the answer computed mathematically. The student tries to guess how the teacher's answer is computed and search for clues in the image. So it is not guaranteed that the student learns how to compute PH in the way is defined mathematically, but the student collects image features that helpful to approximate PH, and those features are necessarily global and topological, as so is PH. Similarly, any algorithmically computable vector associated with the image can be used. However, we particularly choose PH as our image feature since (1) it captures global topological characteristics of the image and is known have discriminative power (Adams et al., (017]. (2) has guaranteed stability (including robustness against the change in the pixel values and the invariance against isometric geometric transformations (Cohen-Stenis et al. 22005] Skraba Turner (20TT). ) and (3) defined for any image and is efficiently computed (Kaji et al. 2020). We elaborate on these points below.
(1) The image features obtained by PH are topological and global, whereas the features acquired by CNNs through supervised learning with ImageNet are often biased towards textures. We expect that learning abstract topological features first in pretraining and local features such texture in
4

---

Under review as a conference paper at ICLR 2023
finetuning will have complementary effects.
(2) Learning invariant features is one of the main principles in visual representation learning (Misra & Maaten 2020). PH gives the same labelling for images that are rotated and reflected, and similar labelling for images with certain types of pixel values alteration, which is guaranteed by the stability theorems. The regression of PH encourages the model learn those invariances. Furthermore, when a certain transformation of the image which changes its PH is applied, the model is asked to learn the change in the label; persistent homology is functorial. That is, the transformation in the input is systematically reflected by the transformation in the output, and the model is asked to learn this higher relation as well when trained with data augmentation such by affine transformations. cannot rely semanties of the images that is meaningful for
(3) For randomly generated images, human eyes to design the pretext task. We not have this problem with PH, which is mathematically defined. Moreover, computation of PH is cheap (Appendix A.1 so that it does not increase overall pretraining time substantially.
We evaluate our scheme by finetuning the model pretrained by scheme for image classification task with some popular datasets T dataset and the performance in terms of the convergence of the classification accuracy for the validation dataset is measured. At the beginning of finetuning, we replace the fully-connected layer with a randomly initialised one. We remark that the topological features encoded by persistent homology alone are insufficient for image classification tasks, and they are meant to complement local features such as texture. For this reason, the weights of all layers are updated during finetuning.
schema). finetune the model with the training
The experiments are conducted with our codes publicly available under the MIT license at at //xxx (the URL is removed temporarily for the double-blind review. The codes are found in supplemental materials.).
3.1 IMAGE GENERATION
We use the following frequency-based random image generation method to create dataset for pretraining. We choose this image generation model mainly because it computationally inexpensive and produces images with various frequency profiles (see also Appendix A.5). In particular, the resulting images (see Fig. contain patterns at different scales, and hence, have rich topological information.
1. Create an array h of dimension 256 x 256, where the values are drawn independently from the uniform distribution on [0, 1].
2. For a frequency parameter ,3 drawn from the uniform distribution [1, set
(FT v) ((x 1)2 1)2)$
g(z, y) Re iFFT
(I)
where FFT is the 2D discrete Fourier transform and iFFT is its inverse, and Re denotes the real part of complex number.
3. With probability of p, which we set to 0.5, binarise g by Otsu's thresholding Otsu 1979).
4. Repeat the process three times independently to create a colour image with RGB channels.
5. Convert the image into greyscale with probability 0.5.
The random parameter controls how fast the high-frequency components decay. The effect of the choices for the hyper-parameters p. g and the range of 3 does not seem to be large and we haven't 't done comprehensive search. We have only checked that setting p = 0.5 and q = 0.5 slightly 1.
better than p 0,
3.2
PH LABEL COMPUTATION
The images labels for the pretext regression task are computed using persistent homology. There various ways to utilise PH to encode topological information of images (see Turkes et al. (202)):
'Some more experiments which corroborate complement this evaluation experiment are given in Ap- pendix A. 2 and Appendix A.4
5

---

Under review as a conference paper at ICLR 2023
50
0
50
50
Figure Two examples of synthesised images and their signed distance functions. Persistent homology is computed for the sequence of the sublevel sets for the signed distance function.
Garin & Tauzin 2019) for survey). Our choice aims extracting shapes contained in the image by their contours and capturing its topological features such as connected components and holes, together with their scale. To this end, nested sequence of spaces, which the input for PH, is built from the image by the following procedure: (1) Convert the image into greyscale (2) Binarise it using Otsu's 's thresholding (Oisu 1979) (3) Compute the signed distance function (see Figure 3
min (ar' w (e) (x, y) (z', 3/
if (I,, y) is foreground
((x, v) =
(2)
min (1r' .y Joreggoond (zz, y) (z', y/)l
if (x,, y) is background.
(4) Finally, obtain the sequence of the sublevel sets X, {(x,)) E X 6(x y) S f). where X is the domain of the original image; that is, the rectangular grid of the size of the image. This procedure gives a bounded nested sequence satisfying X, C X, when s < and X-n 0, XD = X, where D is the length of the diagonal of the image. The PH (with the coefficients in the field F22) of this sequence is computed by the software called Cubical Ripser (Kaji et al. (020). An important remark that instead of considering the sublevel sets with respect to the original (greyscaled) pixel values, we consider the sublevel sets of the signed distance function. In this way, the resulting PH captures the metric-aware structure of the original image; the scale of topological features encoded as life time in persistent homology.
The degree 0 and parts of persistence homology are vectorised separately and concatenated into single vector, which serves as the label for the image. We test four different vectorisation techniques; the persistence image (Adams
al. 2017), the persistence landscape (Bubenik) 2015). the Betti number curve (Chung & Lawson 2021), and the birth-life histogram. The last one is simply the histograms of the birth time (the left end of the interval) and the life time (the length of the interval) of the persistence homology for each degree. All the vectorisation techniques have hyper-parameete for the "resolution" of the output, which are determined from the specified output dimension in our experiments In most of the experiments, the output dimension is fixed to 200. We allocate the same dimension for the degree O and I persistence homology so that we obtain a pair of 100-dimensional vectors that are concatenated to form a 200-dimena -i0-eimensional output vector. To reduce the dynamic range and suppress overflow, the square root is taken for each coordinate of the vector. Although we may elaborate the hyppe-parameter tuning for the PH vectorisation, Tab. [ shows that the choice of the vectorisation techniques does not have a large impact. Therefore, we limit ourselves to an essential set of experiments to find out the nature of the scheme as detailed in Sec. 3.3
3.3 EVALUATION RESULT
We evaluate the effect of the pretraining by looking at the convergence of the accuracy in finetuning for classification tasks. Along with the popular ImageNet- 1k (IMN-1K) (Deng et al.) 2009) and CIFAR (C100) (Krizhevsky] (009) datasets consisting of natural images, we use the animal dataset (ANM) (Bai et al. 2009). which is of relatively small size and consists of 2,000 binary images of animal contours of various size that are labelled with 20 classes. Since no texture information is present in the animal dataset, is suitable to verify the effect of topology learning, and is used in
'The only non cano-nnnical choices among the standard set of sypee--arrmeter: of the vectorisation techniques are the number of landscape functions for the persistence landscape and the sigma of the Gaussian kernel used in the persistence image, which
and 1.0 respectively.

---

Under review as a conference paper at ICLR 2023
Hofer et al. (2017) for evaluation of their method of incorporating PH into deep learning. We split the animal dataset into two sets with 1,600 training images (80 images per class) and 400 validation images.
The papee--rraeeterss for learning are fixed in a standard manner as follows. ResNet50 network architecture is used. Input images resized to 256 x 256 and then cropped randomly down to 224 x 224. Random horizontal flipping is applied. We vary the number of epochs according to the dataset; both in pretraining and finetuning, 90 epochs for the CIFARIO0 dataset, 90 epochs for synthesised dataset, 300 epochs for the animal dataset, and 45 epochs for ImageNet. The stochastic gradient descent with a momentum of 0.9 and weight decay of 10 is used as the optimiser. The initial learning rate is set to 0.1 for pretraining (0,01 for finetuning) and multiplied by 0.1 twice at the 1/3 and the 2/3 of the total training epochs. The batch size is set to 128.
Table 1 shows the classification accuracy of the validation dataset after each pretrained model is finetuned with the training dataset using its labels. The finetuning phase is exactly the same for all pretrained models. Scratch is without any pretraining. Label is pretrained with the training dataset using its labels F That is, the model is trained twice with the training dataset but with different learning rates. The four PH-based models are pretrained with the synthetic dataset with 400,000 images described in Sec. 3.2 with different PH vectorisation targets (PH-PI with the persistence image, PH-LS with the persistence landscape, PH-BC with the Betti number curve, and PH-HS with the bifth-life histogram). For comparison, a model pretrained by a popular SSL scheme, MoCo-v2 (Chen) et al. 2020), with the same synthetic dataset with 400,000 images is shown. As comparison target for pretraining with synthesised images, the results of the FractalDB-pretrainge preetrained models (FDB-Ik consists of 1,000,000 images with 1,000 classes and FDB-10k consists of 10, 0,000,000 images with 10,000 classes) that are publicly available are shown. The values in Tab. I differ slightly from the ones reported in Kataoka et al. (2020), , possibly due to minor differences in the experimental settings. We also list the accuracy of the ImageNet pretrained model (IMN) that is provided as a part of PyTorrhh's 's torchvision library (version 0. Although publicly available pretrained weights are used for FDB-1k 1k and IMN, the hours taken for pretraining with FDB-1k and IMN are measured on the same system used for pretraining other models. For FDB-10k, the time of FDB-1k is extrapolated by simply multiplied by ten.
Since there are only 400 validation images for the animal dataset, the accuracy values fluctuate. The values listed in Tabs.
[ and are the mean for the last epochs from 290 to 300. The 90 percentile falls within =0.6 of the mean. Moreover, to see the stability of the results, we have performed ten trials of pretraining and finetuning with different random seeds for the entry of PH-PI for C100 in Tab. and observed that the 90 percentile falls within +.3.3 of the mean.
The PH- Preprrerdeeedd models show better convergence than Scratch and Label, and comparable convergence to FDB-10k even though the PH models are trained with a much smaller number of 400,000 images and about 70 times less computation time. All PH-pretrained models show more or less similar convergence regardless of the used PH vectorisation techniques. Among them, PH-BC performs the worst, and this can be attributed to the fact that only PH-BC does not care about the "persistence"; PH-BC can be computed just as the sequence of the Betti numbers of the sublevel set X, one t. at a time without considering the relation among different l's, while the other three, PH-PI, PH-LS, and PH-HS, capture the relation. This suggests that not only homology but also its persistence structure helps learn visual representation.
Figure T shows the transition of the accuracy in the finetuning of the models selected from Table 1 for the CIFARI00 dataset. A similar figure for the ImageNet dataset is given in Figure 9 The ImageNet-pretrained model IMN surpasses all other models in both training and validation. The convergence behaviour of PH-PI and FDB-10k are quite similar, sitting between IMN and MoCo-v2.
To see the effect of various parameters and the nature of our method, we conduct a few more experiments. We fix the vectorisation method to the persistence image. Unless otherwise stated, the output dimension is fixed to 200 and the size of the synthetic image dataset is fixed to 200,000.
'The deteriorated accuracy of Label for Animal compared to Scratch is likely due to over-fitting We observe the validation loss converges quickly although to higher value. hhtpss: g/ thub. com /hi rokat 6/ //rractaa 10B- www.etrriiidd-eeeeeeee ReaNet- PyTerch
7

---

Under review as a conference paper at ICLR 2023
Table 1:
Classification accuracy of models pretrained with various methods. For FDBB 1 and FDB-
10k, the pretrained weight file made publicly available by the authors of Kataoka et al. (2020) is used. For IMN, the weight file that comes with the torchvision library is used. The hours taken for pretraining is measured on PC with single NVIDIA RTX 3090 and an Intel Core 19- 19-10850K.
Scratch
Label
PH-PI
PH-LS
PH-BC
PH-HS
MoCo-v2
FDB-IK
FDB- 10k
IMN
C100
69.6
70.3
78.4
78.1
76.6
77.9
71.6
75.3
78.1
85.0
ANM
80.7
91.0
90.1
89.1
90.6
85.0
84.6
85.2
93.3
IMN-1K
70.8
NA
72.9
72.0
71.4
72.8
71.0
71.8
72.4
75.3
time
Oh
3h
22h
22h
22h
21h
38h
144h
1442h
74h
from top
90
70
to bottom
70
50
PH--I
FDB-100 1DB-00k
MoCo-v2
Scratch
13
10
20
80
40
50
60
90
50 Validation accuracy
30
40
60
70
80
90
Train accuracy
Figure 4: Transition of the training and validation accuracy the finetuning of the CIFARI00 dataset.
The impact of the choice of the vectorisation dimension of PH is assessed by varying the value among {100, 200, 400, 800]. As we observe in Tab. Tab. (left,, the choice affects the performance and the optimal value, in this case, is about 200. However, guess the optimal value depends the complexity and the dimension of the synthesised images as the vectorisation dimension controls the resolution of the discretisation of PH.
The impact the size of the pretraining dataset is assessed by varying the size of the synthesised dataset among (50, 000, 200, 000, 400, 000, 800, 000). As observed in Tab. 2 (middle), the per- formance increases as the size gets larger. At the size of 50,000, which the same as that of the CIFARIO0 training dataset, the PH-PI model performs better than Label Tab. I which is pretrained with the same number of images and iterations with the same learning rates.
Table 2: (Left) Classification accuracy of PH-PI models pretrained with different dimensions of the PH vectorisation (the dimension of the vector of the regression target). (Middle) Classification accuracy of PH-PI models pretrained with the synthesised dataset of different sizes. The values for 400k are reproduced from Tab. [ (Right) Classification accuracy of PH-PI models pretrained with with the CIFARION 100 (PH-C) and the animal (PH-A) datasets. The values for Scratch are reproduced
from Tab. I
100
200
400
800
50k
200k
400k
800k
Scratch
PH-C
PH-A
C100
76.8
77.7
77.4
75.0
76.1
77.7
78.4
78.8
69.6
75.3
72,4
ANM
87.4
89.9
90.5
87.5
88.6
89.9
91.0
91.6
80.7
86.5
83.2
Our pretraining scheme also works with real images in place of synthetic ones. In the next experiment, we pretrain the model with the training split of the CIFAR 100 dataset (and the animal dataset) using the PH-regression task, and then finetune the model using the class label of the dataset. The result is shown in Tab. 2 (right). In this example, the setting Label and PH-C (PH-A, respectively) for the CIFAR 100 dataset (the animal dataset, respectively) is the same except for the label used for the pretraining; Label uses the class label while PH-C (PH-A, respectively) uses the label computed with the persistence image. Pretraining with PH improves the performance, indicating the benefit of learning not only from the class labels but also with topology- based labels. Comparing with the entry for 50k in Tab. (middle)
synthetic dataset offers slightly better quality for learning
8

---

Under review as a conference paper at ICLR 2023
than the CIFARIO0 dataset, which consists of the same number of 50k natural images. This could be attributed to the design of the image generation model that produces patterns at various scales. In the case of the animal dataset. pretraining with PH does not lead to a large performance gain. This may be due to the small number of training images (1,600) and the variety is quite limited. The observation agrees with the result in Tab. (middle) that indicates the necessity of certain amount of data for learning topological features.
LIMITATIONS
We have seen that CNNs can learn useful image features from a regression task of persistent homology. However, (vectorised) PH alone is not strong image feature for a classification task although it captures different characteristics of images than conventional techniques. We believe adding topology helpful but does not replace textural information that CNNs naturally learns from natural image classification. Our proposed scheme forces the model to focus on topology during pretraining: the PH label is computed from binarised and signed distance transformed images from which textural information is stripped off. The PH label encodes topological features with their scale. The model learns other local features only during finetuning, where topology is not paid a particular attention. Although this splitting of topology learning from conventional learning makes the analysis more controlled, it would not be the optimal strategy to learn both local and global image features if the primary goal is better performance in practical applications.
Another limitation, which is related to the explainability just as in most deep-learning schemes, is that are not really sure what features are learned in our method although we have designed it for learning global topological features of images. We have observed that both the training and the validation losses converge during pretraining the PH-based models, but the latter stays about twice higher than the former, indicating that the models learn to approximate PH but fails to learn to compute PH in a fully generalisable (that is, mathematical) manner. Nonetheless, we have observed that the visual representation obtained by the model pretrained by PH is quite different from those pretrained by FractalDB and ImageNet (see Appendix A..).. From a practical viewpoint, this means the model would have strengths and weaknesses depending on the downstream tasks. Detailed and controlled experiments including more than simple classification tasks should be performed to verify the usefulness of the learned features across different tasks.
5 CONCLUSION
We proposed a scheme for learning visual representations that were relevant to the topological features of images through regression task for mathematical function defined by persistent homology. Our method can be applied to any image dataset and virtually any neural network architecture with a small cost. Our experiment with a synthetic image dataset suggested that even images that were little meaningful for human eyes could be used for learning certain visual representation. Our main contributions are two fold: (I) We proposed a pretext task that was mathematically defined and applicable to synthetic images without labels. (II) We experimentally showed that a convolutional neural network pretrained with the pretext task acquired visual representation that was different from those obtained by the usual supervised learning.
We would like to conclude the paper with a few possible future directions to investigate. (a) Since global topological features are generally robust compared to local ones, learning them could add resistance against adversarial attacks. (b) Regressing other mathematical invariants could be used to pretrain neural networks to equip them with understanding of geometric, topological, and algebraic structures of the image. This would provide a generic method to incorporate various knowledge established by mathematicians over past centuries to deep learning research. terms of learning with synthetic images, regressing the random parameters used in the image generation models such as in (Kataoka et al. 20000 Baradad et al. (02I)) would also be interesting. (c) A theoretical question related to this work is on the relation between the neural network architecture and the learning capacity. It is known that wapeesaallpproximmattee that relate topology of the target function to be learned and the width of neural networks (Johnson] 2019). It would be interesting if we could compare the way in which a CNN learns to approximate PH and the rigorous mathematical definition of PH.
statements have some constraints
9

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
The experiments are conducted with our codes included as supplemental materials, We will release them at github under the MIT license. Robustness of the numbers in the tables and figures are discussed in Section 3.3
REFERENCES
Henry Adams and Michael Moy. Topology applied to machine learning: From global to local. Frontiers in Artificial Intelligence, 4:54, 2021. ISSN 2624-8212. doi: 203338//frr.2222116666000 263302
URL
www. http://www.ooooaeoaooorrrrrrrarr
Henry Adams, Tegan Emerson, Michael Kirby, Rachel Neville, Chris Peterson, Patrick Shipman, Sofya Chepushtanova, Eric Hanson, Francis Motta, and Lori Ziegelmeier. Persistence images: stable vector representation of persistent homology. J. Mach. Learn. Res., 18(1):218-253) January 2017. ISSN 1532-4435.
Xiang Bai, Wenyu Liu, and Zhuowen Tu. Integrating contour and skeleton for shape classification. In Inf. Conf. Comput. Vis. Worksh., pp. 360-367, 2009, doi: 10.1 109/CCCV.2099.4577677 URL http:///a...oo/.......////////// 2009. 0457779754579999
Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to by looking at noise. Adv. Neural Inform. Process. Syst., 2021.
Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. This dataset does not exist: training models from generated images. Int. Conf. Acoustics, Speech, & Sign. Process..
pp. 1-5, 2020.
Peter Bubenik. Statistical topological data analysis using persistence landscapes. J. Mach. Learn. Res., 16(1):77-102, January 2015. ISSN 1532-4435.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference Applications of Computer Vision (WACV), pp. 839-847, 2018. doi:
N0.1109/WACV.2018.0000
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.00 04297, 2020.
Yu-Min Chung and Austin Lawson. Persistence curves: A canonical framework for summarizing persistence diagrams, 2021. arXiv:1904.07766v4.
David Cohen Cohen-Steiner, Herbert Edelsbrunner, and John Harer. Stability of persistence diagrams. In Proceedings of the Twenty-Fine Annual Symposium on Computational Geometry, SCG '05, pp. 263-271, New York, NY, USA, 2005, Association for Computing Machinery. ISBN 139118 doi: 10. 00.1144//064009211041111 1064133 URL httpt:t////// /oi. org 100g/ 10e11 1145/1064092 1064133
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Inl. Conf. Learn. Represent. 2021. URL httts:://ppereviieee wttp:/wwwwwwwwweefeeeeeeeeeeeeee id=YicbF
Olga Dunaeva, Herbert Edelsbrunner, Anton Lukyanov, Michael Machin, Daria Malkova, Roman Kuvaev, and Sergey Kashin. The classification of endoscopy images with persistent homology. Pattern Recognition Letters, 83:13 22, 2016. ISSN 0167-8655. doi: http://doi.org//0.00666 URL https: http://wwwwciiiieeeeer...... com wtt..///eence//riiieeeee
Pi 0111500667865515000003
10

---

Under review as a conference paper at ICLR 2023
Adélie Garin and Guillaume Tauzin. A topologiaal "reading" lesson: Classification of MNIST using TDA. Proceedings of the IEEE International Conference on Machine Learning And Applications (ICMLA), pp. 1551-1556. IEEE, 2019. doi: 10.1 10.100/1CMLA22219000226 URL
htttps://doi. www.oo..orgo.. wwo. 1 201 9. c0256
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In Int. Conf. Learn. Represent., 2019. URL https://
openreview.co http://wwww.etttfommmmmmmmmmmm Bd 90
Barbara Giunti. TDA-Applications: Zotero database. /ttp::///wwwww/ wwwpw//wwwwooo///ooooo./o 2425412/tda app capplicaiions 2021. [Online; accessed 14-Oct-2021]..
Geoffrey Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504-07, July 2006. doi: 10.s126/sc.ence.
1127647. URL http://wwer ncbi www. htt.://.i...././tte/eeeetttte..e http://ttee/etttrereerdd--bbbbe
Christoph Hofer, Roland Kwitt, Marc Niethammer, and Andreas Uhl. Deep learning with topological signatures. In Adv. Neural Inform. Process. Syst.
2017.
URL https://proceeddnng nwwrips htt.://aa.....a.oooooooooo////// hctppaapeee...///i.e.e.. pdf
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1), 2021. ISSN 2227-
7080. doi: 10.390//ecimologgis00000000 URL https: /www//wwwwww/// mdpi.
1/2
Jesse Johnson, Deep, skinny neural networks are not universal approximators. In Int. Conf. Learn. Represent. 2019. URL https:://ppnnreviee http:/wwwwwww...e.....eeeeeeeeee http://wwnne//fouuddd///rregeeee
Shizuo Kaji, Takeki Sudo, and Kazushi Ahara. Cubical ripser: Software for computing persistent homology of image and volume data, 2020. a2Xiv:2005.12692.
Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, and Yutaka Satoh. Pre-training without natural images. In Asian. Conf. Comput. Vis., 2020.
Hirokatsu Kataoka, Ryo Hayamizu, Ryosuke Yamada, Kodai Nakashima, Sora Takashima, Xinyu Zhang, Edgar Josafat Martinez-Noriega, Nakamasa Inoue, and Rio Yokota. Replacing labeled real-image datasets with auto-generated contours. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 21232-21241, June 2022.
Alex Krizhevsky, Learning multiple layers features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Chunyuan Li, Maks Ovsjanikov, and Frederic Chazal. Persistence-ban based structural recognition. In IEÉE Conf. Comput. Vis. Pattern Recog., 2014.
Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Comput. Surv., 54(6), July 2021. ISSN 0360-0300. doi: (0.1145/3457600 URL htttttts: :/do:////do/ //doi. www.ai.oorg/..... wttrg/o..11.....1 1115....11111111510
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext- invariant representations. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 7045- 7054, 2020. URL htttt::::/pproceede ngs. m www.s//essvvveee 1tc1199//ooo m0 html
Nobuyuki Otsu. A Threshold Selection Method from Gray-level Histograms. IEEE Transactions on Systems, Man and Cybermetics, 91) 66, 1979. doi: 10.1 10977SMC.19999441000 URL httpp:/dxx. doi http:///w..ooi...r...111101/////
1979. 4310076

---

Under review as a conference paper at ICLR 2023
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran As- sociates, Inc., 2013. URL http:tt////oooceedinnggg/oo wwwngngg..euuuuuuuee neurips. wttpppppppccpp........... 45b.a44bbbbbbbbbba hww.a:a000022ee00000000----00000
14bfa 45bbb
Primoz Skraba and Katharine Turner.
Wasserstein stability for persistence diagrams, 2021.
arXiv:2006.16824.
Anirudh Som, Hongjun Choi, Karthikeyan Natesan Ramamurthy, Matthew P. Buman, and Pavan K. Turaga. Pi-net: A deep learning approach to extract topological persistence images. In IEEE Conf. Comput. Vis. Pattern Recog. Worksh. pp. 3639-3648, 2020. doi: 10.1 PV.1R009 00425. URL
10. 1109/ CVPRW5 98 2020 00425
Ryan Steed and Aylin Caliskan. Image representations learned with unsupervised pre-training contain like biases. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT "21, pp. 701-713, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 450383097. doi: 10. (0.1145/9442188.444550000 URL.https://ddt org/
10.1145/34 444445/44422
3445932
Naoya Tanabe, Shizuo Kaji, Susumu Sato, Tomoo Yokoyama, Tsuyoshi Oguma, Kiminobu Tanizawa, Tomohiro Handa, Takashi Sakajo, and Toyohiro Hirai. homological approach to a mathematical definition of pulmonary fibrosis and emphysema on computed tomography. J. Appl. Physiol., 1985: 601-612, 2021. doi: https://doi.org/1O..
00..00.0210002221 50.2021.
Renata Turkeš, Jannes Nys, Tim Verdonck, and Steven Latré. Noise robustness of persistent ho- mology on greyscale images, across filtrations and signatures. PLOS ONE, 16(9):1-26, 09 2021. doi: howp://joooall.oo.02227777... 0257215
URL https: /tt.:ttt///aa..ccc/0000111/////0
Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Fil- tering and balancing the distribution of the people subtree in the imagenet hierarchy. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT '20, pp. 547-558, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 4781450369367. doi: 10.1 10.11553351095.33557099 URL https: //w./ orgpoioorg//oo 114.:/....114111 /1t1:///01...010..111141313111.3 3375700 /
3375709
A APPENDIX
A.1 COMPUTATIONAL COST OF PH LABELLING
The worst-case complexity of the standard matrix reduction algorithm for computing persistent homology is K33 where K is the number of columns, which is linear with respect to the number of pixels in the cubical complex. The average time of computation for persistent homology and its persistence image as in the PH-PI pretraining for synthesised image described in Sec. 3.1 in different sizes is shown in Tab, B The increase rate looks almost linear with respect to the number of pixels in this particular set of images.
Table 3: Average computation time for PH and persistence image
image size
256 x 256
512 x 512
1024 x 1024
2048 x 2048
computation time in ms
22.62
89.85
436.93
2181.13
A.2 WHAT IS REALLY LEARNED?
Understanding and explaining what the model is learning is difficult problem, which forms of the central topics in machine learning research. Here, we show some experiments that indicate the difference between the models trained in our scheme and in the classical supervised manner with real
12

---

Under review as a conference paper at ICLR 2023
images and manual labels. First, observe distinction in the weights of the first few convolution layers among different models. Figure [ shows that the model learns different low-level features through scheme.
PH-PI
FractalDB-10k
ImageNet
Figure 5: Visualisation of the filters of the first convolutional layer of PH-PI, FractalDB- 10k, and ImageNet pretrained models.
To understand this phenomenon, we conducted an experiment "the other way around" see the visual representation acquired through supervised learning with real images is sufficient to approximate persistent homology. Specifically, we replaced the fully-connected layer the ImageNet pretrained model with randomly initialised one. Then, the model was finetuned for the PH-regression task as in the PH-PI pretraining while the weights of but fully-connected layers were frozen (we call this model IMG frr. The model compared with the one trained from scratch (by updating all layers), which was precisely the same manner as the PH-PI pertaining (we call this model Scratch). Figure 0 shows that the validation loss, well the training loss, converges for Scratch, and hence, the model was learning to approximate PH to some extent. However, we observe that the convergence of IMN_K fr is much worse than Scratch, indicating that the image features learned by supervised training with ImageNet is not sufficient for approximating PH. That is, not only cally-connected layers but also convolutional layers need be trained for the computation of PH. This explains what we have observed in Fig. 5 the convolutional layers of PH-PI learn features specific PH computation, which are different from those features learned with supervised image classification task with ImageNet. This behaviour sharply contrasts with the fact that the ImageNet pretrained model IMN can be finetuned for various natural image classification tasks by updating only the weights of fully-connected layers. The convolutional layers of IMN have obtained image features sufficient for general classification tasks of natural images, which are insufficient for approximating PH.
0.6
IMN_I fr
Scratch
0.4
0.4
10
20
30
10
50
60
70
80
90
10
20
30
40
50
90
Figure 6: Transition of the loss for the PH-regression task for the ImageNet pretrained model with the weights of all but fully-connected layers frozen (IMN fr) and model with randomly initialised weights (Scratch).
To investigate this direction further, performed another experiment of the PH-regression with a different set of synthesised images with simple random blobs which had virtually no texture information (Fig. (Top). We finetuned the ImageNet pretrained (IMN_fr fr) and the PH-PI pretrained
13

---

Under review as a conference paper at ICLR 2023
(PH-PI fr) models while the weights of all but fulle-connected layers were frozen. The transition of the loss values is shown in Fig. g The validation loss of PH-PI fr converges to lower value than that of IMN fr. This shows the visual representation learned by the convolutional layers of PH-PI is useful to approximate the PH of a different type of image than was used for pretraining. On the contrary, the features learned by pretraining with ImageNet PH, even for very simple images. FigureßJalso shows the transition of the loss values for IMN and PH-PI, where all layers including convolutional ones were updated during finetuning. Both models had better convergence than the ones updating only the fully-connected layers, but PH-PI had lower loss values than IMN indicating that updating all layers did not completely override the learned features during pretraining.
sufficient approximate
The visualisation of -aad-CA+++ (Chattopadhay al. 2018) (with the target of the last convolution layer) of the two models, PH-PI_f fr and IMN_fr fr, is shown in the middle and bottom rows of Fig. Although is hard to interpret, we may be able to say, PH-PI fr (Bottom) identifies large circle as single shape whereas IMN fr (Middle) has more focus edges and finds large circle collection two or three shapes.
Figure 7: (Top) randomly synthesised blob images (Middle) Grad-CAM+ visualisation of IMN [f_ and (Bottom) PH-PI_ fr
024
0.25
IMN_fr
PH-PI_fr fr
0.2
IMN
PH-PI
0.04
20
50
Train loss
Validation loss
Figure 8: Transition of loss for PH-regression task with the random blob images. The ImageNet pretrained model (IMN) and the PH-PI model (PH-PI) pretrained with the synthetic dataset described Section 3.1 are finetuned to compute the persistent homology (vectorised by persistence images) of another set of synthetic images (Fig. Top)). Corresponding models with the weights of all but fully-connected layers frozen during finetuning are named IMN fr and PH-PI fr, respectively. Note that the two sets of randomly generated images (Fig. Top) for finetuning and Fig. for pretraining PH-PI) are very different.
14

---

Under review as a conference paper at ICLR 2023
A.3 TRANSITIONS STATISTICS DURING FINETUNING
The transition of the accuracy during the finetuning of the models selected from Table for the ImageNet-Ik dataset given Fig- [ We observe similar tendency to Fig. f the CI dataset. The MuageNet-pretratie model IMN surpasses all other models in both training and validation. The convergence behaviour of PH-PI and FDB-10k are quite similar, sitting between IMN and Scratch.
from top to bottom
IMN PH-PI
FDB-10k Scratch
0
10
25
30
Train accuracy
Validation accuracy
Figure 9: Transition of the training and validation accuracy during the finetuning of the selected models with ResNet50 for the ImageNet- 1 dataset.
To see if the proposed scheme is effective for powerful CNN architecture, performed an experiment with ResNet101. The experiment configuration was the same as Table Mobleeexee except for the model architecture. Figure shows the transition of the accuracy during the finetuning of the selected models with ResNet101 for the CIFARIO0 dataset. Note that due to the large computational time required for pretraining FDB-10k, we could not include FDB-10k in the figure.
from top bottom
90
50
PH-PI
MoCo-v2
Scratch
10
Train accuracy
Validation accuracy
Transition of the training and validation accuracy during the finetuning of the selected Figure 10:
models with ResNet101 for the CIFAR100 dataset.
As an example of another downstream task than classification, we performed object detection with Faster R-CNN (Ren al. 2015) for the PASCAL VOC 2007 dataset. We used the trainval split for training and the test split for validation. We tested five ResNet50 models in Table [] Scratch, PH-PI, MoCo-v2, FDB-10K, 10k, and IMN as the backbone of Faster R-CNN. Figure [T shows the transition of mAP for the first 20 epochs of training and validation. The mAP observed in experiment was very low for SSL models. It seems that the backbone network should be pretrained a supervised manner with a large dataset as in the original paper of Faster R- R-CNN. Therefore, it may not be meaningful to do any comparison, but we see that mAp increased faster with PH-PI than the other SSL models and training from scratch.
A.4 CLASS-WISS WISE INSPECTION ON THE CIFAR 100 CLASSIFICATION
Here, investigate what kind of images are better classified with scheme. We looked the predictions for the CIFAR100 validation dataset (100 classes x 100 images) of the three models, Label, PH-PI, and IMN selected from Table T after finetuning. We computed the Fi-score for each
15

---

Under review as a conference paper at ICLR 2023
from top to bottom
0.4
0.08
0.08
IMN
0.08
0.08
PH-PI
763
FDB-10k
MoCo-v2
2+3
263
Scratch
2
10
12
16
18
12
16
Training mAP (log scale)
Validation mAP (log scale)
Figure 11: Transition of mAP during the finetuning of the selected models with Faster R-CNN with ResNe150 backbone for the PASCAL VOC 2007 test dataset.
Table Fcoscsss of the predictions of selected models from Table I for the validation dataset of C100. The list is sorted in the increasing order of the difference in the Fi Fi-score between PH-PI and IMN and we show the first three and the last three classes.
class
Label
PH-PI
IMN
(PH-PI)-(IMN)
(PH-PI)((Label)
bear
0.524
0.653
0.854
-0.201
0.129
snake
0.617
0.670
0.831
-0.161
0.053
shrew
0.545
0.573
0.725
-0.152
0.028
plate 0.710
0.777
0.772
0.006
0.068
road girl 0.472
0.906
0.927
0,916
0.011
0.020
0.653
0.623
0.030
0.181
of the 100 classes and sorted the result in terms of (Fi-score of PH-PI) (Fi-score of IMN). Having a positive value for ( (Fi-score of PH-PI) (Fi-score of IMN) means that PH-PI performed better than IMN for the class. The result is shown in Table 4 Although is not shown on the list, we note that PH-PI had improved Fi Fi-scores than Label for all 100 classes.
Figure shows sample images from the six classes that appear in Table We selected three images from each class in the following manner. For the classes of bear, snake, and shrew, with which IMN performed better then PH-PI, we selected those images where the prediction of PH-PI was incorrect and that of IMN was correct. Then, we chose three images with the lowest ranking of the true label PH-PI. For the classes of plate, road, and girl, with which PH-PI performed better than IMN, we did the same by swapping the role of PH-PI and IMN. The caption below each sample image is the prediction of the incorrect model. How the models make incorrect predictions is interesting and it seems to agree with our claim that model put more emphasis the shape than the texture. Especially the plate class provides an illustrative example; PH-PI focuses on the round shape, while IMI looks at the texture of the plate. We give few more interpretations of the result: The snake in the second row can be mistaken as a bottle if we look at its silhouette without its distinctive texture. In the first row, the shrew's pose with the forefoot resembles that of lizard. The girl in the third row wears colourful striped clothes but cannot be mistaken as a lizard by human. All images of road consist mainly of large and simple geometric shapes, which are easily characterised by persistent homology.
in the prediction
A.5 IMAGE SYNTHESIS METHOD
There are many different ways to synthesise images, such as various noise generation methods. We have not conducted thorough test on dataset creation since the focus in the design of our pretext task is not on the distribution of the data. The key is in the process of approximating a certain computation (in our case, persistent homology) based on mathematical structure. We list some reasons why we chose the particular image synthesis method. We can see from Eq. (I: (i) It is rotation invariant, except for the fact that the image is rectangular. (ii) It is scale (image size) independent. (iii) It produces various frequency profiles: The lifetime profile of the persistent homology is a very important factor of the topology of the image. With the signed distance transform, the total persistence (the sum of lifetime of all eycles) is bounded so that have either many short lifetime cycles
16

---

Under review as a conference paper at ICLR 2023
PH-PI:
bear
snake
shrew
girl
plate
otter
beaver
lizard
boy
mountain
boy
bottle
lion
woman
rocket
clock
man
lizard
skunk
lizard
skyscraper
shail
Figure 12: Sample images of the classes in Table The label of each image indicates the (incorrect) prediction by PH-PI (for the classes bear, snake, and shrew) and by IMN (for the classes girl, road, and plate).
small number of long lifetime cycles. In natural images, short lifetime cycles arise mainly from noise or texture, which are local. The denominator of Eq. ① discourages the resulting image long lifetime cycles, which represent global topology. (iv) It is computationally cheap. We emphasise that the fractal model used in Formula-driven Supervised Learning (Kataoka et al. 2020pp requires tremendous computational power and their datasets are prepared on computing cluster. Creation of our dataset with 400,000 images takes less than twenty minutes on personal computer, and generated on the fly during the first epoch of training.
from having many high-frequency components that there are
17


Reviews:
[{"review_id": "zehDsDUuKqy", "reviewer": "Reviewer_nihr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a technique to learn local and global visual representation via Persistent Homology.\nThe overall structure of the proposed method is to perform self-supervised learning (regression) on synthetic images for Persistent Homology as a pre-training, and then natural images for fine-tuning.\n\nThe synthetic image utilizes frequency-based (FFT) random generation method, where there are patterns at different scales\nThe PH label computation (using Cubical Ripser) aims at extracting shapes using contours by using signed distance function.\nThe author then utilizes four different vectorization techniques: the persistence image, the persistence landscape, the Betti number curve, and the birth-life histogram.\n\nThe algorithm is tested on ImageNet and CIFAR100. It also uses 2000 binary images of animal contours for a second (mid-step) topology learning.\nThat is, for ImageNet for example, it uses 90 epoch synthetic data, 300 epochs for animal dataset, then 45 epochs for ImageNet.\nThe first experiment is testing various pre-training (including vectorization techniques) and fine-tuning training combinations, as well as other methods, such as MoCo v2 and FractralDB-pre-trained models.\n\nThe PH-pre-trained models show better convergence than without pre-training (just fine-tuning) or fine-tuned twice, and comparable to FractalDB.\nAmong the vectorization, all are similar, with Betti count performs worst. This may be because of the lack of persistence structure learning.\nThe experiments also show that the number of PH-training images is important as there is only 1600 of the animal training images.\n", "summary_of_the_review": "As stated the problem and solution (technique) is explained well. However, the testing and analysis can be more in-depth.", "strength_and_weaknesses": "Strengths: The paper explains the intentions and technical parts quite well. It is tested reasonably thorough, comparing the various vectorization techniques, as well as with other modules.\n\nWeaknesses: it is hard to understand what is actually being learned. It would help to know what kind of images are now better classified after the pre-training and fine-tuning regiments to demonstrate the usage of structure learned in improving final task accuracy.\nThat is, would it be possible to show example testing images that are moving from incorrect to correct.\nAlso, it would be helpful to describe what types of global structures (and invariances) are eventually being learned by PH synthetic image pre-training.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WvNfLXQQ7-n", "reviewer": "Reviewer_tB82", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new scheme to pretrain the neural networks, which combines the calculated persistent homology of a sublevel set filtration of images as the topology information. The topology information aims to guide the neural networks to learn non-local topological structures. To avoid the human bias in the annotated datasets, the authors utilize synthetic images generated by specific formulas during the pre-training. The authors provide empirical evaluations and show that the topology-related pre-training scheme can improve the convergence with comparable accuracy on natural image classification tasks.", "summary_of_the_review": "Overall, this paper contributes some new ideas but the empirical evaluation is limited. Also, it's unclear to me what the model is actually learning (interpretability).", "strength_and_weaknesses": "**Strengths**:\n\n1. The idea of guiding the neural networks to learn global structures with non-local features, like persistent homology is novel and interesting.\n\n2. Pre-trained with synthetic images generated by specific formulas seems to prevent the human bias existing in the annotated datasets and ensure fairness to some extent.\n\n3. The paper is clearly structured and well written.\n\n**Weaknesses**:\n\n1. The authors only provide empirical results, without giving theoretical analysis of the proposed pre-training scheme.\n\n2. The authors mention the weights of all layers are updated during the phase of fine tuning. However, it is unclear that the neural network is still able to preserve the learned global topological features if we update all layers’ parameters during the phase of fine tuning.\n\n3. The experiments are all conducted on small datasets of natural image classification tasks, which is not sufficient to indicate that the proposed scheme can be used in other larger datasets or different downstream tasks.\n\n**Questions**:\n\n1. It may be helpful to conduct experiments on how the learned global features change between the pre-training phase and the fine-tuning phase.\n\n2. Using hand-crafted features to guide the training of neural networks seems alright. However, how to prove that your hand-crafted features actually reflect the global information of the training dataset? I would expect more theoretical analysis to support the main claim of this paper.\n\n3. It seems that the models are only tested on rather small datasets within only the natural image classification tasks. Is it possible to conduct experiments on larger datasets on other scenarios, like segmentation or tracking, to see the performance?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9Gwk_4pcY7f", "reviewer": "Reviewer_U7F9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use synthetic data to learn a better representation that can enforce the model to learn holistic features. The synthetic data is generated using persistent homology (PH). This method does not require real images. In the experiment, authors try to apply the learnt representation to downstream tasks. The experiment is done on the animal dataset, which consist of binary images. ", "summary_of_the_review": "The paper proposes an interesting way to generate synthetic data. Then use this data to train a better representation that can encode holistic information. The experiment is limited to a binary dataset, which makes it hard to judge whether the training strategy can work well on real-world data. Due to the challenge and complexity of real data, many existing models can already encode long-term context and have large receptive fields. This fact makes the advantage of the proposed method seem less significant. Based on these reasons, I think the paper is below the threshold of ICLR.", "strength_and_weaknesses": "Strength:\n- Training a better representation without using real data is an interesting direction. This paper presents an innovative way to learn holistic features.\n\nNegative:\n- The effectiveness of this method is uncertain. It is uncertain whether the feature learned using this synthetic data can work well together with real data and tasks. The authors mentioned, \"The image features obtained by PH are topological and global, whereas the features extracted through iterated convolutions by CNNs are mostly local.\"  This disadvantage might not hold for various cases. Many models can encode long-term context effectively.\n- The experiment is not persuasive enough to show whether this method can be generalized to in-the-wild real data. The binary image is an effective way to prove the concept, but it is also helpful to include results for more challenging real data to show its effectiveness.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject