# KD-PatchMatch: A Self-Supervised Training Learning-Based PatchMatch
## 3 Method
### Probabilistic Depth Sampling  
With \(N\) input images of size \(W\times H\), including a reference image \(I_{0}\) and its neighboring source images \(\{l_{i}\}_{i=1}^{N-1}\), in the coarse-to-fine optimization, features are extracted hierarchically at different resolution levels by using a Feature Pyramid Network (FPN) [31]. At Stage \(k\), the resolution of feature maps is \(\frac{W}{2^{k}}\times\frac{H}{2^{k}}\). Figure 1: The framework of KD-PatchMatch, including PatchMatch-based MVS framework and self-supervised training pipeline based on knowledge distillation: (**a**) This figure illustrates how the depth candidates are sampled during the depth initialization phase, The curves and shadows represent the estimated depth probability distributions and depth sampling range during the depth initialization phase. (**b**) The pipeline for self-supervised training based on knowledge distillation is shown in this image, along with the creation of pseudo-probabilistic knowledge and student training. **Single-View Depth Probability. ** At Stage 3 of PatchMatch, the depth initialization is originally performed in a random way and uses the same set of depth candidates for all pixels. In order to find the correct depth, the network has to evaluate a large number of depth candidates, resulting in a large number of computational resources being wasted. To this end, following [PERSON] et al., 2022 ([PERSON] et al., 2022), we propose a depth sampling method based on D-Net ([PERSON] et al., 2022), which allows the network to perform better while considering fewer depth candidates. Instead of uniform sampling in image space, we sample per-pixel depth candidates for input images by the single-view depth probability distribution. For each image of size \(W\times H\), our network estimates a single-view depth probability distribution map at the resolution \(\frac{H}{\mathcal{B}}\times\frac{W}{\mathcal{B}}\) of the input image \(I_{0}\). The depth distribution of each pixel \(p_{0}\) in the reference image \(I_{0}\), \(P_{p_{0}}(d\mid I_{0})\) is parameterized as a Gaussian distribution,  
\[P_{p_{0}}(d\mid I_{0})=\frac{1}{\sigma_{p_{0}}(I_{0})\sqrt{2\pi}}e^{-\frac{1}{2 }\left(\frac{d-\mu_{p_{0}}(I_{0})}{\sigma_{p_{0}}(I_{0})}\right)^{2}} \tag{1}\]  
where \(d\) is the depth value of pixel \(p_{0}\), and \(\mu_{p_{0}}\) and \(\sigma_{p_{0}}\) are the mean and the variance of depth values of pixel \(p_{0}\). Efficient-Net B5 ([PERSON] et al., 2022) is used as a lightweight convolutional network encoder-decoder. Our single-view network is pre-trained, with the weights fixed while the rest of the pipeline is being trained. The training loss is defined as negative log-likelihood (\(NLL\)) of the ground-truth depth,  
\[L_{p_{0}}\left(d_{p_{0}}^{gt}\mid I_{0}\right)=\frac{1}{2}\log\sigma_{p_{0}}^{2 }(I_{0})+\frac{\left(d_{p_{0}}^{gt}-\mu_{p_{0}}(I_{0})\right)^{2}}{2\sigma_{p_ {0}}^{2}(I_{0})} \tag{2}\]  
Equation (2) is an \(L2\) loss with a decaying learning rate; \(d_{p_{0}}^{gt}\) is the ground truth of the depth value of pixel \(p_{0}\). When reducing the error \(((d^{gt}-\mu)^{2}\) is difficult, the \(\sigma^{2}\) estimated by network learning will be high.This usually happens at points near and far from the object boundary ([PERSON] et al., 2020). Conversely, when the estimated \(\sigma^{2}\) is low, the correct depth may be near the estimated \(\mu\). **Probabilistic Depth Sampling. ** We use the single-view depth probability distribution estimated for the reference image \(I_{0}\) for the pixel-by-pixel depth hypothesis sampling. First, we need to define the search space \(\left[\mu_{p_{0}}-\beta\sigma_{p_{0}},\mu_{p_{0}}+\beta\sigma_{p_{0}}\right]\) for each pixel \(p_{0}\), where \(\beta\) is the hyperparameter, and we will assign a value to \(\beta\) in the experimental part. Then, we divide the search space into \(D_{f}\) intervals, ensuring that each interval has the same probability mass. Finally, the midpoint of each interval is chosen as the depth candidate.