SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While we can use the topological features alone, we can also concatenate them with the features computed by a deep neural network (DNN). In order to do this, we need to learn not only (a) networks in our framework but also (b) DNN. Although (a) and (b) can be learned simultaneously since the output of the resulting feature is differentiable with all parameters, the networks cannot be successfully optimized in our experiments. This would be because the loss function is not smooth with respect to the parameters in (a), which can also make the optimization of (b) unstable. To deal with this issue, we propose to learn (a) and (b) separately.
Let us describe the whole training procedure briefly here. Suppose  $X$  is a input point cloud. We write  $\Psi_{\mathrm{topo}}(X) \in \mathbb{R}^{L_1}$  for the topological feature which comes from our method, and  $\Psi_{\mathrm{DNN}}(X) \in \mathbb{R}^{L_2}$  for the feature that comes from a DNN-based method. Let  $\ell$  be a loss function for the classification task and  $n$  be a number of classes. We propose the following two-phase training procedure:
1st Phase. Let  $C_1\colon \mathbb{R}^{L_2}\to \mathbb{R}^n$  be a classifier that receives feature from  $\Psi_{\mathrm{DNN}}$ . Then,  $C_1$  and  $\Psi_{\mathrm{DNN}}$  are learned through the classification task, which is achieved by minimizing  $\sum_{k = 1}^{M}\ell (C_1(\Psi_{\mathrm{DNN}}(X_k)),y_k)$ .
2nd Phase. Let  $C_2 \colon \mathbb{R}^{L_1 + L_2} \to \mathbb{R}^n$  be a classifier. We discard the classifier  $C_1$  and fix  $\Psi_{\mathrm{DNN}}$ . Then,  $C_2$  and  $\Psi_{\mathrm{topo}}$  are learned through the classification task, which is achieved by minimizing  $\sum_{k=1}^{M} \ell(C_2([ \Psi_{\mathrm{DNN}}(X_k)^\top, \Psi_{\mathrm{topo}}(X_k)^\top ]^\top), y_k)$ .
This training procedure is observed to make the optimization stable and to help the architecture to achieve higher accuracy. We show the experimental result in §5.
4 Approximation Ability of Weight Function
In this section, we theoretically show that our architecture has a good expression power. We prove that approximation of any continuous map can be realized by our idea to concatenate two finite-dimensional feature vectors.
Consider the space  $2^{[0,1]^m}$  of subsets in  $[0,1]^m$  equipped with the Hausdorff distance. For a fixed  $N \in \mathbb{N}$ , we define the following subspace of  $2^{[0,1]^m}$ :
$$
\mathcal {X} := \left\{X \subset [ 0, 1 ] ^ {m} \mid | X | \leq N \right\}.
$$
The following theorem states that we can approximate any continuous function on  $\mathcal{X} \times [0,1]^m$  by concatenating the finite dimensional feature vectors of a point cloud and a point. The proof of the theorem is given in Appendix B.
Theorem 4.1. Let  $f\colon \mathcal{X}\times [0,1]^m\to \mathbb{R}$  be a continuous function. Then for any  $\epsilon >0$ , there exist  $K\in \mathbb{N}$  and continuous maps  $\psi_{1}\colon \mathcal{X}\to \mathbb{R}^{K}$ ,  $\psi_{2}\colon \mathbb{R}^{K + m}\to \mathbb{R}$  such that
$$
\sup _ {X \in \mathcal {X}} \int_ {[ 0, 1 ] ^ {m}} \left(f (X, x) - \psi_ {2} ([ \psi_ {1} (X) ^ {\top}, x ^ {\top} ] ^ {\top})\right) ^ {2} \mathrm {d} x <   \epsilon .
$$
To investigate the performance of our method in classification tasks, we conducted numerical experiments on two types of public real-world datasets. In §5.1, we present a simple experiment on a protein dataset to demonstrate the efficacy of our method. Next, in §5.2, we show the result of a experiment on a 3D CAD dataset. The classification accuracy was used as the evaluation metric.
The first experiment was conducted to show the validity of our method where we only used the topological feature, without DNN features. In the second experiment, on the other hand, we combine the topological features and features from DNNs using the method described in §3.2
All the code was implemented in Python 3.9.13 with PyTorch 1.10.2. The experiments were conducted on CentOS 8.1 with AMD EPYC 7713 2.0 GHz CPU and 512 GB memory. Persistent homology was calculated by Python interface of GUDHI. All of the source codes we used for experiments is publicly available. More details of the experiments can be found in Appendix C.
5.1 Protein classification
Dataset. In this section, we utilize the protein dataset in Kovacev-Nikolic et al. (2016) consisting of dense configuration data of 14 types of proteins. The authors of the paper analyzed the maltose-binding protein (MBP), whose topological structure is important for investigating its biological role. They constructed a dynamic model of 370 essential points each for these 14 types of proteins and calculated the cross-correlation matrix  $C$  for each type. They then define the distance matrix, called the dynamical distance, by  $D_{ij} = 1 - |C_{ij}|$ , and they use them to classify the proteins into two classes, "open" and "close" based on their topological structure. In this paper, we subsampled 60 points and used the distance matrices with the shape  $60 \times 60$  for each instance. We subsampled 1,000 times, half of which were from open and the rest were from closed. The off-diagonal elements of the distance matrix were perturbed by adding noise from a normal distribution with a standard deviation of 0.1.
Architectures and comparison baselines. Given a distance matrix, we computed the proposed adaptive filtration and calculate the persistent homology. Then, the 1st persistence diagram of that filtration was vectorized by PersLay. The obtained feature was input into a classifier using a linear model, and all the models were trained with cross entropy loss. For comparison, we evaluate the accuracy when we replace our learnable filtration with a fixed filtration. As a fixed filtration, we used Rips or DTM filtration (Anai et al., 2020), the latter of which is robust to outliers. The hyperparameter for DTM filtration was set to maximize the classification accuracy. Additionally, we replaced our topological feature by the output of (2), which we call DistMatrixNet, and compared the classification accuracy. Note that we used DistMatrixNet not for the computation of filtration weights but for the direct computation of point clouds' feature. Since the protein dataset is given in the format of distance matrices, we did not utilize standard neural network methods for point clouds.
Results. We present the results of the binary classification task of proteins in Table [1]. We can see that our framework overperformed the other methods in terms of the accuracy of the classification.
We describe some observations below: Firstly, the result of our method is better than those of Rips and DTM filtrations. This would be because our method learned a weighted filtration adaptively to data and tasks and could provide more informative topological features than the classical ones. Secondly, our method achieved higher accuracy compared to DistMatrixNet, which has a similar
https://gudhi.inria.fr/ https://github.com/git-westriver/FiltrationLearningForPointClouds 3The correlation matrix  $C$  can be found in https://www.researchgate.net/publication/ 301543862_corr.
Table 1: The accuracy of the binary classification task of protein structure. We compared our methods with DistMatrixNet and persistent homology with Rips/DTM filtration. We can see that our method performs better than other three methods.
DistMatrixNet	Rips	DTM	Ours
65.0 ± 12.0	79.9 ± 3.0	78.0 ± 1.6	81.9 ± 2.1
number of parameters to the proposed method. This suggests that persistent homology is essential in the proposed method rather than the expressive power of DistMatrixNet.
5.2 3D CAD data classification
Dataset. In this section, we deal with the classification task on ModelNet10 (Wu et al., 2015). This is a dataset consisting of 3D-CAD data given by collections of polygons of 10 different kinds of objects, such as beds, chairs, and desks. We choose 1,000 point clouds from this dataset so that the resulting dataset includes 100 point clouds for each class. The corresponding point cloud was created with the sampling from each polygon with probability proportional to its area. Moreover, to evaluate the performance in the case where the number of points in each point cloud is small, we subsampled 128 points from each point cloud. The number of points is relatively small compared with the previous studies, but this setting would be natural from the viewpoint of practical applications. We added noise to the coordinates of each sampled point using a normal distribution with a standard deviation of 0.1.
Architecture and comparison baselines. We utilized the two-phase training procedure described in §3.2 where we concatenated a feature computed by DNN-based method and a topological feature. As DNN-based methods, which were trained through the 1st Phase, we utilized DeepSets (Zaheer et al., 2017), PointNet (Qi et al., 2017a), and PointMLP (Ma et al., 2022). A given point cloud was input into our adaptive filtration architecture, and the resulting 1st persistence diagram was vectorized by PersLay. The concatenated features were input into a classifier based on a linear model, and the parameters are tuned with the cross entropy loss.
We compared the result of 1st Phase, which is the accuracy of a DNN alone, and that of 2nd Phase to validate the efficacy of our learnable filtration. We also compared our method with a fixed filtration, Rips or DTM, instead of our learnable filtration. Furthermore, we replaced our topological feature replaced by the output of DistMatrixNet (2) and compared the classification accuracy.
Results. The results are shown in Table 2. Below, we present some observations for the results.
Firstly, our method outperformed Rips filtration, which is one of the most common filtrations. Moreover, the accuracy of our method is almost the same as or higher than that of DTM filtration. While DTM filtration has hyperparameters that need to be tuned, our method achieved such a result by learning an adaptive filtration automatically through the training. This result implies that our method is useful in efficiently choosing filtrations.
Secondly, our method again overperformed DistMatrixNet. This indicates that the combination of persistent homology and DistMatrixNet was crucial similarly to the protein dataset.
Thirdly, and most importantly, it can be observed that the classification accuracy is better when our method was concatenated with DeepSets/PointNet compared to using DeepSets/PointNet alone. Additionally, the accuracy when we combine our method and DeepSets/PointNet is competitive when using PointMLP alone. These observations indicate that concatenating the topological features obtained by our method yields positive effects. The accuracy when we combine our method and PointMLP is not higher than that of PointMLP. This would mean that PointMLP has already captured enough information including topological features during the 1st phase, so that the information obtained by persistent homology may be redundant, potentially negatively impacting the classification.
In this paper, we tackled the problem to obtain adaptive topological features of point clouds. To this end, we utilize a weighted filtration and train a neural network that generates a weight for each point.
Table 2: Results for the classification task of 3D CAD data. The row named "1st Phase" shows the results when we used the feature calculated by DNN only, and the rows named "2nd Phase" shows the results when we concatenated them with the feature calculated by DistMatrixNet or topological feature obtained by persistent homology. Compared to using the DeepSets/PointNet alone, we can achieve higher accuracy when we concatenate the topological feature obtained by our method. Additionally, regardless which DNN method we use, using topological feature computed by our method improves accuracy better than using topological feature computed by Rips filtration and feature computed by DistMatrixNet. Moreover, the accuracy when we combine our method with DeepSets/PointNet is competitive with using PointMLP alone.
DeepSets	PointNet	PointMLP
# of parameters	813,488	3,472,500	3,524,386
1st Phase	65.7 ± 1.4	64.3 ± 4.4	68.8 ± 6.3
2nd Phase	DistMatrixNet	65.7 ± 4.8	55.7 ± 13.9	53.8 ± 7.4
Rips	67.0 ± 2.6	68.4 ± 2.4	57.8 ± 12.4
DTM	68.0 ± 2.5	68.7 ± 2.3	57.2 ± 6.8
Ours	67.5 ± 2.5	68.8 ± 2.0	60.0 ± 6.3
Since the model is desired to be invariant with respect to isometric transformation, we proposed a network architecture that satisfies such invariance. We theoretically showed a finite-dimensional approximation property for a wide class of weight functions, which supports the validity of our architecture. Our numerical experiments demonstrated the effectiveness of our method in comparison with the existing methods.
Limitations and future work There are several limitations and remained future work for this study. First, while we concentrated on determining the weight function of the weighted Rips filtration in this paper, we can also consider other kinds of filtrations. For example, we can expand the balls from each point at different speeds. Second, we did not applied our method to larger datasets as we needed to compute persistent homology repeatedly in the training, which makes the computational cost higher. In fact, it took about seven hours to train the neural networks in our method for each cross-validation. While we believe our method is meaningful since it sometimes improves classification accuracy, this is one of the largest limitations of our study. Addressing this issue is one of our future work. Third, since our method adaptively chooses a filtration, our framework is expected to work well even if a subsequent machine learning model has a simple architecture. We plan to validate the hypothesis experimentally and theoretically.
We appreciate Marc Glisse and Théo Lacombe for the helpful discussion. This work was partially supported by Grant-in-Aid for Transformative Research Areas (A) (22H05107) and JST KAKENHI JP19H01114.
S. Zhang, S. Mukherjee, and T. K. Dey. Gefl: Extended filtration learning for graph classification. In Learning on Graphs Conference, pages 16-1. PMLR, 2022.
C. Zhou, Z. Dong, and H. Lin. Learning persistent homology of 3d point clouds. Computers & Graphics, 102:269-279, 2022.