SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fangzhou Lin $^{1,2*}$  Yun Yue $^{1*}$  Ziming Zhang $^{1\dagger}$  Songlin Hou $^{1,3}$  Kazunori D Yamada $^{2}$  Vijaya B Kolachalama $^{4}$  Venkatesh Saligrama $^{4}$
$^{1}$ Worcester Polytechnic Institute, USA  $^{2}$ Tohoku University, Japan
$^{3}$ Dell Technologies, USA  $^{4}$ Boston University, USA
{flin2, yyue, zzhang15, shou}@wpi.edu, yamada@tohoku.ac.jp,
{vkola,srv}@bu.edu
A point cloud is a discrete set of data points sampled from a 3D geometric surface. Chamfer distance (CD) is a popular metric and training loss to measure the distances between point clouds, but also well known to be sensitive to outliers. We propose InfoCD, a novel contrastive Chamfer distance loss, and learn to spread the matched points to better align the distributions of point clouds. As such InfoCD leads to an improved surface similarity metric. We show that minimizing InfoCD is equivalent to maximizing a lower bound of the mutual information between the underlying geometric surfaces represented by the point clouds, leading to a regularized CD metric which is robust and computationally efficient for deep learning. We conduct comprehensive experiments for point cloud completion using InfoCD and observe significant improvements consistently over all the popular baseline networks trained with CD-based losses, leading to new state-of-the-art results on several benchmark datasets. Demo code is available at https://github.com/Zhang-VISLab/NeurIPS2023-InfoCD.
Point Cloud Completion. Point clouds, one of the most important data representations that can be easily acquired, play a key role in modern robotics and automation applications [1-3]. However, raw data of point clouds captured by existing 3D sensors are usually incomplete and sparse due to occlusion, limited sensor resolution, and light reflection [4-8], which can negatively impact the performance of downstream tasks that require high-quality representation, such as point cloud segmentation and detection. Point cloud completion [9] refers to the task of inferring the complete shape of an object or scene from incomplete raw point clouds. Recently, many (deep) learning based approaches have been introduced to point cloud completion ranging from supervised learning, self-supervised learning to unsupervised learning [10-15]. Among these methods, supervised learning with a general encoder-decoder structure is the prevailing architectural choice for many researchers, consistently achieving state-of-the-art results on mainstream benchmarks [16, 17, 8, 1, 18].
Learning with Chamfer Distance (CD). CD is a commonly employed metric in point cloud completion, for example in studies like [19, 20]. It assesses the dissimilarity in shape between two sets of point clouds by calculating the average distance from each point in one set to its nearest neighbor in the other set. Minimizing the Euclidean distances between matched points, as done in CD, is a known method that is sensitive to outliers, resulting in clumping behavior that involves
Figure 1: Illustration of comparison among CD, MI, and InfoCD with different numbers of samples.
a substantial number of many-to-one correspondences in point matching, forming small clusters visually. These observations can significantly deviate from the assumption of uniform sampling from the underlying geometric surfaces, which is commonly used to generate point clouds.
Learning with Mutual Information (MI). A more fundamental problem in measuring point cloud similarity (or distance alternatively) is: How can we measure the similarities between the underlying geometric surfaces represented by the point clouds? To address this issue, one potential method is to compute their MI by taking each point cloud as a discrete set of point samples from a random variable following a certain distribution, and then plugging both point clouds into the MI formula to measure the similarity of the two random variables. However, two limitations prevent us from using it directly as a loss function in learning: (1) It may be nontrivial to define the joint probability distribution between two point clouds since we do not have any prior knowledge, and (2) The requirement for extensive memory, necessary for handling a large volume of data points and deep learning models, could render the practical implementation of the MI (Mutual Information) loss unfeasible.
Our Approach and Contributions. Motivated by contrastive learning, we propose a novel contrastive Chamfer distance loss, namely InfoCD, to learn to spread the matched points for better distribution alignments between point clouds as well as accounting for surface similarity estimation. Similar to InfoNCE [21], our InfoCD minimization tends to maximize a lower bound of the MI between the underlying geometric surfaces represented by the point clouds. This indeed leads to a regularized CD loss that can be minimized effectively and efficiently with similar computational complexity to CD.
We further illustrate our high level ideas in Fig. 1. To plot such figures, we consider generating point clouds uniformly sampled from two 1D Gaussian distributions both with unit variance but different means, one fixed at 0 and the other varying from -10 to 10 at step-size of 0.1. We then compute CD, InfoCD (see Eq. 5 with  $\tau = 1$ ), the lower bound of MI (see Eq. 8), and MI where the joint probability distribution is computed based on the exponential of negative Euclidean distances. To better view the difference between the curves of CD and InfoCD, we also rescale the InfoCD curve by aligning its minimum and maximum values with those for CD based on linear scaling so that their value ranges are the same. With different numbers of samples, we can clearly see that: (1) InfoCD is more robust to outliers (i.e., the cases far away from 0 mean) than CD by penalizing larger deviations with some similar numbers (though smaller numbers of samples have more fluctuations); (2) The sharper purple curves around 0 indicate that InfoCD may lead to better convergence; (3) The lower bound based on InfoCD can approximate MI well with the maximum at 0, which indicates that InfoCD can be taken as a good MI estimator and lead to better point distributions with fewer visual clusters.
To summarize, we list our main contributions as follows:
- We propose InfoCD by introducing contrastive learning into the CD loss, leading to a regularized CD loss for better point distribution alignments.
- We analyze the connection between InfoCD and MI as well as learning behavior with InfoCD.
- We achieve state-of-the-art results on popular benchmark datasets for point cloud completion.
Point Cloud Completion. As the first learning-based point cloud completion network, PCN [10] extracts global features in a similar way PointNet [22] did and generates points through folding
operations as in FoldingNet [23]. In order to obtain local structures among points, Zhang et al. [24] proposed extracting multi-scale features from different layers in the feature extraction part to enhance the performance. CDN [25] uses a cascaded refinement network to bridge the local details of partial input and the global shape information together. Lyu et al. [26] proposed treating point cloud completion as a conditional generation problem in the framework of denoising diffusion probabilistic models (DDPM) [27]. Attention mechanisms such as Transformer [28], demonstrate their superiority in capturing long-range interaction as compared to CNNs' constrained receptive fields. For instance, to preserve more detailed geometry information for point cloud generation in the decoder, SA-Net [29] uses the skip-attention mechanism to merge local region information from the encoder and point features of the decoder. SnowflakeNet [17] and PointTr [16] pay extra attention to the decoder part with Transformer-like designs. PointAttN [1] was proposed solely based on Transformers.
Distance Metrics for Point Clouds. Distance in point clouds is a non-negative function that measures the dissimilarity between them. Since point clouds are inherently unordered, the shape-level distance is typically derived from statistics of pair-wise point-level distances based on a particular assignment strategy [20]. With relatively low computational cost fair design, CD and its variants are extensively used in learning-based methods for point cloud completion tasks [30, 26, 31, 32]. Earth Mover's Distance (EMD), which is another widely used metric, relies on finding the optimal mapping function from one set to the other by solving an optimization problem. In some cases, it is considered to be more reliable than CD, but it suffers from high computational overhead and is only suitable for sets with exact numbers of points [33, 34]. Recently, Wu et al. [20] propose a Density-aware Chamfer Distance (DCD) as a new metric for point cloud completion which can balance the behavior of CD and computational cost in EMD to a certain level. Lin et al. [35] proposed a HyperCD that computes CD in a hyperbolic space and achieves significantly better performance than DCD.
Contrastive Learning. Recently, learning representations from unlabeled data in contrastive way [36, 37] has been one of the most competitive research fields [21, 38-50]. Popular model structures like SimCLR [42] and Moco [45] apply the commonly used loss function InfoNCE [21] to learn a latent representation that is beneficial to downstream tasks. Several theoretical studies show that contrastive loss optimizes data representations by aligning the same image's two views (positive pairs) while pushing different images (negative pairs) away on the hypersphere [51-54]. A good survey on contrastive learning can be found in [55]. More recently, contrastive learning has found its way into point cloud applications as well. For instance, Tang et al. [56] proposed a contrastive boundary learning framework for point cloud segmentation. Yang et al. [57] proposed the mutual attention module and co-contrastive learning for point cloud object co-segmentation. Jiang et al. [58] proposed a guided point contrastive loss to enhance the feature representation and model generalization ability in semi-supervised settings for point cloud segmentation. Du et al. [59] proposed a self-contrastive learning approach for self-supervised point cloud representation learning. Wang et al. [60] proposed exploring whether maximizing the mutual information across shallow and deep layers is beneficial to improve representation learning on point clouds, leading to a new design of Maximizing Mutual Information (MMI) Module. Afham et al. [61] proposed CrossPoint, a simple cross-modal contrastive learning approach to learn transferable 3D point cloud representations with 2D images. Shao et al. [62] proposed a spatial consistency guided network (SCRnet) using contrastive learning for point cloud registration.
3 InfoCD Loss
InfoNCE Loss. In [21], the InfoNCE loss is defined as follows:
$$
\mathcal {L} _ {\text {I n f o N C E}} = - \sum_ {x} \log \left\{\frac {\exp \left\{\frac {1}{\tau} s \left(x ^ {+} , x ; \theta\right) \right\}}{\exp \left\{\frac {1}{\tau} s \left(x ^ {+} , x ; \theta\right) \right\} + \sum_ {x ^ {-}} \exp \left\{\frac {1}{\tau} s \left(x ^ {-}, x; \theta\right) \right\}} \right\}, \tag {1}
$$
where  $x, x^{+}, x^{-}$  denote the anchor, its positive and negative samples,  $s$  denotes a similarity function parametrized by  $\theta$ , and  $\tau \geq 0$  is a predefined temperature that controls the sharpness.
Proposition 1 (InfoNCE vs. MI [21]). Let  $c_{t}$  be the context at the  $t$ -th time step, and  $x_{t+k}$  be the future target. Then given a set of  $N$  random samples,  $\{x_{1},\dots,x_{N}\}$ , containing one positive sample from the distribution  $p(x_{t+k}|c_{t})$  and  $N - 1$  negative samples from the "proposal" distribution
$p(x_{t + k})$  , we have
$$
I \left(x _ {t + k}, c _ {t}\right) \geq \log (N) - \mathcal {L} _ {\text {I n f o N C E}}, \tag {2}
$$
where  $I$  denotes the mutual information (MI) operator.
Please refer to the appendix in [21] for the proof. This proposition provides us with an alternative way to measure MI approximately and implicitly. If point clouds can fit into the setting of InfoNCE, we then may better estimate the underlying surface similarities from point clouds.
Chamfer Distance Loss. In the sequel, we denote  $(x_{i},y_{i})$  as the  $i$ -th point cloud pair, with  $x_{i} = \{x_{ij}\}$  and  $y_{i} = \{y_{ik}\}$  as two sets of 3D points, and  $d(\cdot ,\cdot)$  as a certain distance metric. Then the CD loss for point clouds can be defined as follows:
$$
\mathcal {L} _ {\mathrm {C D}} \left(x _ {i}, y _ {i}\right) = \ell_ {\mathrm {C D}} \left(x _ {i}, y _ {i}\right) + \ell_ {\mathrm {C D}} \left(y _ {i}, x _ {i}\right) = \frac {1}{| y _ {i} |} \sum_ {k} \min  _ {j} d \left(x _ {i j}, y _ {i k}\right) + \frac {1}{| x _ {i} |} \sum_ {j} \min  _ {k} d \left(x _ {i j}, y _ {i k}\right), \tag {3}
$$
where  $|\cdot|$  denotes the cardinality of a set. For point cloud completion, function  $d$  usually refers to
$$
d \left(x _ {i j}, y _ {i k}\right) = \left\{ \begin{array}{l l} \| x _ {i j} - y _ {i k} \| & \text {a s L 1 - d i s t a n c e} \\ \| x _ {i j} - y _ {i k} \| ^ {2} & \text {a s L 2 - d i s t a n c e} \end{array} \right. \tag {4}
$$
where  $\| \cdot \|$  denotes the Euclidean  $\ell_2$  norm of a vector.
3.2 Our Loss Function
Given the considerations above, we propose the following formula as our InfoCD loss:
$$
\mathcal {L} _ {\text {I n f o C D}} \left(x _ {i}, y _ {i}\right) = \ell_ {\text {I n f o C D}} \left(x _ {i}, y _ {i}\right) + \ell_ {\text {I n f o C D}} \left(y _ {i}, x _ {i}\right), \text {w h e r e}
$$
$$
\begin{array}{l} \ell_ {\mathrm {I n f o C D}} (x _ {i}, y _ {i}) = - \frac {1}{| y _ {i} |} \sum_ {k} \log \left\{\frac {\exp \{- \frac {1}{\tau^ {\prime}} \min _ {j} d (x _ {i j} , y _ {i k}) \}}{\sum_ {k} \exp \{- \frac {1}{\tau} \min _ {j} d (x _ {i j} , y _ {i k}) \}} \right\} \\ \propto \frac {1}{\tau | y _ {i} |} \sum_ {k} \min  _ {j} d \left(x _ {i j}, y _ {i k}\right) + \lambda \log \left\{\sum_ {k} \exp \left\{- \frac {1}{\tau} \min  _ {j} d \left(x _ {i j}, y _ {i k}\right) \right\} \right\} (5) \\ = - \frac {1}{| y _ {i} |} \sum_ {k} \log \left\{\frac {\exp \left\{- \frac {1}{\tau} \min  _ {j} d \left(x _ {i j} , y _ {i k}\right) \right\}}{\left[ \sum_ {k} \exp \left\{- \frac {1}{\tau} \min  _ {j} d \left(x _ {i j} , y _ {i k}\right) \right\} \right] ^ {\lambda}} \right\} (6) \\ \Rightarrow \mathcal {L} _ {\text {I n f o C D}} \left(x _ {i}, y _ {i}\right) \propto \frac {1}{\tau} \mathcal {L} _ {\text {C D}} \left(x _ {i}, y _ {i}\right) + \lambda \mathcal {R} \left(x _ {i}, y _ {i}\right) (7) \\ \end{array}
$$
with  $\mathcal{R}(x_i,y_i) = \log \left\{\sum_{m,n}\exp \left\{-\frac{1}{\tau} [\min_jd(x_{ij},y_{in}) + \min_kd(x_{im},y_{ik})]\right\} \right\}$  as a regularizer and  $\lambda = \frac{\tau'}{\tau} \in (0,1]$  as a predefined constant controlling the trade-off between the loss and the regularizer. The smaller  $\mathcal{L}_{\mathrm{CD}}(x_i,y_i)$  is, the larger  $\mathcal{R}(x_i,y_i)$  is accordingly. From this perspective, we can easily see that our InfoCD loss is equivalent to a regularized CD loss.
InfoCD vs. MI. To see the connections between InfoCD and MI, we have the following lemma:
Lemma 1. Consider two point clouds  $x_{i} = \{x_{ij}\}$ ,  $y_{i} = \{y_{ik}\}$  representing two underlying geometric surfaces  $\mathcal{X}_i, \mathcal{Y}_i$ . Now we introduce a new random variable  $z_{y_{ik}}$  whose probability distribution  $p(z_{y_{ik}} | \mathcal{X}_i = \mathcal{Y}_i)$  indicates how likely a point  $y_{ik}$  can be sampled from  $\mathcal{Y}_i$  conditional on  $\mathcal{X}_i = \mathcal{Y}_i$ , and "proposal" distribution  $p(z_{y_{ik}})$  indicates the likelihood of generating an arbitrary  $y_{ik}$ . With these notations, we will have
$$
I \left(z _ {y _ {i k}}; \mathcal {X} _ {i} = \mathcal {Y} _ {i}\right) \geq \log \left(\left| y _ {i} \right|\right) - \ell_ {\text {I n f o C D}} \left(x _ {i}, y _ {i}\right). \tag {8}
$$
Proof. By following Prop. 1 and the proof in [21], we can take  $z_{y_{i_k}}$ ,  $\mathcal{X}_i = \mathcal{Y}_i$  as the replacements for  $x_{t + k}, c_t$ . Then samples from  $p(z_{y_{i_k}} | \mathcal{X}_i = \mathcal{Y}_i)$  will be "positive", and ones from  $p(z_{y_{i_k}})$  will be
"negative" in the language of contrastive learning. Now we can construct  $|y_i|$  groups, as illustrated in Fig. 2, where each group consists of the entire point cloud  $x_i$  and a point in  $y_i$ . We further can take an arbitrary group as a positive and the rest as negatives. Finally, by parametrizing  $\frac{p(z_{y_{ik}}|\mathcal{X}_i = \mathcal{Y}_i)}{p(z_{y_{ik}})} \propto \exp\left\{-\frac{1}{\tau'}\min_j d(x_{ij},y_{ik})\right\}$  for positives and similarly to negatives, we can complete our proof.
Therefore, based on this lemma, minimizing InfoCD tends to better estimate the lower bound of MI that indicates the underlying surface similarities between point clouds.
Point Spread in Learning & Testing. To better understand the behavior of different losses, we first introduce another lemma as follows:
Lemma 2. Consider an optimization problem  $\min_{\omega \in \Omega} \sum_{i} g(h(x_i; \omega))$  where  $h: \mathcal{X} \times \Omega \to \mathbb{R}, g: \mathbb{R} \to \mathbb{R}$  are both Lipschitz continuous functions,  $h$  is also smooth over  $\omega$ , and  $x_i \in \mathcal{X}$  is a data point. Then based on gradient descent, i.e.,  $\omega_{t+1} = \omega_t - \eta_t \sum_{i} \nabla g(h(x_i; \omega_t))$  with a learning rate  $\eta_t \geq 0$  at the t-th iteration and a gradient operator  $\nabla$ , it holds that given a new data point  $\tilde{x}$ ,
Figure 2: Illustration of group construction process for MI estimation.
$$
h \left(\tilde {x}; \omega_ {t}\right) - h \left(\tilde {x}; \omega_ {t + 1}\right) \approx \sum_ {i} \eta_ {t} \left. \frac {\partial g}{\partial h} \right| _ {\left(x _ {i}; \omega_ {t}\right)} \nabla h \left(x _ {i}; \omega_ {t}\right) ^ {T} \nabla h \left(\tilde {x}; \omega_ {t}\right) = \eta_ {t} \sum_ {i} \left. \frac {\partial g}{\partial h} \right| _ {\left(x _ {i}; \omega_ {t}\right)} \kappa \left(x _ {i}, \tilde {x}; \omega_ {t}\right), \tag {9}
$$
where  $\omega_0$  is the initialization of  $\omega$ ,  $\kappa$  denotes a (neural) tangent kernel function parametrized by  $\omega_t$ ,  $(\cdot)^T$  is the matrix transpose operator, and  $\left.\frac{\partial g}{\partial h}\right|_{(x_i;\omega_t)}$  is the derivative of  $g$  over  $h$  at point  $(x_i;\omega_t)$ .
Proof. Using the linear approximation of  $h$  and the assumptions, we can easily prove this lemma.
To connect this lemma with CD and InfoCD, let us first compute the gradients of  $\ell_{\mathrm{CD}}$  and  $\ell_{\mathrm{InfoCD}}$ :
$$
\nabla \ell_ {\mathbf {C D}} \left(x _ {i}, y _ {i}\right) = \sum_ {k} \frac {1}{\left| y _ {i} \right|} \nabla d \left(x _ {i k ^ {\prime}}, y _ {i k}\right), \tag {10}
$$
$$
\nabla \ell_ {\text {I n f o C D}} \left(x _ {i}, y _ {i}\right) = \frac {1}{\tau} \sum_ {k} \left[ \frac {1}{| y _ {i} |} - \tau^ {\prime} \frac {\exp \left\{- \frac {1}{\tau} d \left(x _ {i k ^ {\prime}} , y _ {i k}\right) \right\}}{\sum_ {k} \exp \left\{- \frac {1}{\tau} d \left(x _ {i k ^ {\prime}} , y _ {i k}\right) \right\}} \right] \nabla d \left(x _ {i k ^ {\prime}}, y _ {i k}\right), \tag {11}
$$
where  $k' = \arg \min_{j} d(x_{ij}, y_{ik}), \forall k$ . By viewing  $d$  as  $h$  and  $\ell_{\mathrm{CD}}$  (or  $\ell_{\mathrm{InfoCD}}$ ) as  $g$ , we can see the weight  $\frac{\partial \ell_{\mathrm{CD}}(x_{ik'}, y_{ik}; \omega_t)}{\partial d} = \frac{1}{|y_i|} > 0$  but  $\frac{\partial \ell_{\mathrm{InfoCD}}(x_{ik'}, y_{ik}; \omega_t)}{\partial d} = \frac{1}{|y_i|} - \tau' \frac{\exp\left\{-\frac{1}{\tau} d(x_{ik'}, y_{ik})\right\}}{\sum_k \exp\left\{-\frac{1}{\tau} d(x_{ik'}, y_{ik})\right\}} \in \mathbb{R}$ . To simplify the analysis and explanation, assuming that the normalized gradients,  $\frac{\nabla d(\cdot, \cdot)}{\|\nabla d(\cdot, \cdot)\|}$ , can be viewed as random samples from a high dimensional (i.e., the number of network parameters that is much larger than the number of samples in mini-batches) normal distribution, then for two different random inputs  $(x_{im'}, y_{im})$ ,  $(x_{in'}, y_{in})$ , it will be expected [63] that the corresponding gradients will be close to being orthogonal to each other, i.e.,  $\nabla d(x_{im}, y_{ik})^T \nabla d(x_{in}, y_{ik}) \approx 0$ . Now by substituting this assumption into Eq. 9, we can have
$$
\ell_ {\mathbf {C D}} \left(x _ {i}, y _ {i}; \omega_ {t}\right) - \ell_ {\mathbf {C D}} \left(x _ {i}, y _ {i}; \omega_ {t + 1}\right) \approx \eta_ {t} \sum_ {k} \frac {\partial \ell_ {\mathbf {C D}} \left(x _ {i k ^ {\prime}} , y _ {i k} ; \omega_ {t}\right)}{\partial d} \| \nabla d \left(x _ {i k ^ {\prime}}, y _ {i k}\right) \| ^ {2}, \tag {12}
$$
$$
\ell_ {\text {I n f o C D}} \left(x _ {i}, y _ {i}; \omega_ {t}\right) - \ell_ {\text {I n f o C D}} \left(x _ {i}, y _ {i}; \omega_ {t + 1}\right) \approx \frac {\eta_ {t}}{\tau} \sum_ {k} \frac {\partial \ell_ {\text {I n f o C D}} \left(x _ {i k ^ {\prime}} , y _ {i k} ; \omega_ {t}\right)}{\partial d} \| \nabla d \left(x _ {i k ^ {\prime}}, y _ {i k}\right) \| ^ {2}, \tag {13}
$$
with another assumption that the matched point pairs keep unchanged over iterations. From this perspective, we can easily see that the negative weights of  $\frac{\partial\ell_{InfoCD}(x_{ik'},y_{ik};\omega_t)}{\partial d}$  with smaller distances will push the predicted points away from the matched ground-truth points, while positive weights will tend to reduce the distances in both metrics, as illustrated in Fig. 3. In this contrastive way, the predicted points will be more likely to be spread for better alignment with the ground-truth points.
(a) Prediction  $\rightarrow$  Ground truth
(b) Ground truth  $\rightarrow$  Prediction
To demonstrate the capability of our InfoCD loss to spread the matched points at test time, we illustrate some comparison results in Fig. 4, where we can see clearly that InfoCD can significantly reduce the numbers of many-to-one matched points, leading to better point distribution alignments.
Convergence. In general, there is no guarantee that training networks with InfoCD using (stochastic) gradient descent will converge if the matched point pairs between the predictions and ground truth are frequently changed. However,
empirically we observe that such training stabilization can be efficiently reached using different networks on different datasets as well. For instance, as illustrated in Fig. 5(a)<sup>3</sup>, InfoCD with L1-distance in Eq. 4 behaves similarly in terms of convergence rate to L1-CD that has convergence guarantee, but reduces the loss more significantly.
As illustrated in Fig. 5(b), ideally CD aims to reach the status where all the predicted points will be aligned perfectly with ground truth with no errors by consistently minimizing the distances. Such a requirement may be so strict that in learning the optimization may be much easier to be stuck at the suboptimal solutions, e.g., forming clusters around some points. In contrast, InfoCD aims to align the point distributions with sufficiently small errors, as illustrated in Fig. 5(c) with the dotted lines indicating a Voronoi diagram for the ground-truth point cloud. This is much less strict than CD, and thus will be more likely to locate better solutions. In fact, the movements of predicted points in Fig. 3(b) provide an effective way towards learning such an alignment in Fig. 5(c) by avoiding bad suboptimal solutions.
Choice of  $d$  in InfoCD. In particular, we utilize L1-distance in Eq. 4 as the choice for  $d$  in InfoCD, because it is unbiased to the distance, i.e.,  $\nabla d(x_{ij},y_{ik}) = \nabla \| x_{ij} - y_{ik}\|$ . In practice, we observe that using  $d(x_{ij},y_{ik}) = \| x_{ij} - y_{ik}\|^{p},p > 0,p\neq 1$ , the performance of InfoCD is very unstable for different networks