SlowGuess/NeurIPS_2023_OCR huggingface dataset:
-  $u$ : This is the parameter for the Bernoulli distribution. In our Prob. DMT module, for every structure, we have a choice to either retain the structure as obtained from DMT, or, generate a sample skeleton using the perturb-and-walk algorithm. We model this choice using the Bernoulli distribution. Essentially, in some runs we would like the original DMT structures to also interact with the others. Thus a low value of  $u$  works best. We found  $u = 0.3$  to give the best performance, that is, for every structure there is a  $30\%$  chance that it's DMT form is used and a  $70\%$  chance that a sample variant is used. We find that  $0.15 \leq u \leq 0.3$  have comparable performance.
-  $\gamma$ : This hyperparameter is used in the weighted combination of distance  $Q_{d}$  and likelihood  $f_{n}$  to obtain  $Q(c')$ , which is used to determine the next pixel location. It maintains a tradeoff between the distance regularizer  $Q_{d}$  and the perturbed likelihood  $f_{n}$ . The higher the value of  $\gamma$ , the greater the distance regularizer, and consequently the generated path will become closer to that of a straight line. This is not desirable, as a straight line would lose the original composition of the structure. Additionally, because of the perturbation in the likelihood, we do not want the path to go astray. To ensure path completeness, we require  $\gamma$  to be non-zero. Through experiments, we obtain the best performance when  $\gamma = 0.2$ .
-  $\alpha, \beta$ : These are prior hyperparameters of the Inverse Gamma (IG) distribution. We perturb the likelihood using a Gaussian model. As the variance of the Gaussian model is unknown, we use Bayesian probability theory to sample the variance from the IG distribution (its conjugate prior). And so,  $\alpha$  is the shape parameter and  $\beta$  is the scale parameter of this IG distribution. Ideally we would like a small perturbation of the likelihood and not a strong one. This is because a strong perturbation would corrupt wholly and we would not be able to sample a reasonable skeleton. At the same time, the perturbation should not be too small, otherwise we will not obtain a significant variant. The mean of the IG distribution is  $\frac{\beta}{\alpha - 1}$  (when  $\alpha > 1, \beta > 0$ ), which on average is the value of the sampled variance for the Gaussian distribution. We achieve the best performance when  $\alpha = 2.0$  and  $\beta = 0.01$ . The resulting sampled variance for the Gaussian model thus generates reasonable perturbation.
C Computational complexity
We report the inference time for 5 runs on a  $256 \times 256$  input image patch as follows: Prob.-UNet: 0.196 sec; PHiSeg: 1.811 sec; Hu et al.: 5.485 sec; Ours: 7.433 sec.
The module which takes the most time is the DMT / Prob.DMT computation. Presently, this is the most optimized version as we have implemented it as an external module in  $\mathbf{C} + +$ . We will work towards porting the code to run on GPU to bring down the runtime even more.
Following [10], the computational complexity of DMT is  $O(n \log n)$ , where  $n$  is the number of pixels in the image. Since Prob. DMT additionally computes structure variants, the complexity is  $O(n \log n + m)$  where  $m$  is approximately the number of foreground pixels, and typically  $m << n$  for curvilinear structure datasets. The linear term is added as we traverse each foreground pixel only once when generating the sample skeleton.
D Inference procedure
We illustrate the inference procedure in Fig. 12. There are two outcomes of our framework  $M_{\phi}$ , namely, the structure-wise uncertainty heatmap as well as an improved discrete segmentation map that can be used instead of the one obtained by  $F_{\theta}$ .
For each structure  $e$ , we obtain  $\hat{p}(e)$  (the regression output) and  $\hat{\delta}_e^2$  (the uncertainty) from  $M_{\phi}$ . We take  $T$  runs of  $M_{\phi}$  and then for each structure  $e$ , we compute the mean across  $T$  runs as  $\bar{\delta}_e^2 = \frac{1}{T}\sum_{t=1}^{T}(\hat{\delta}_e^2)_t$ , and,  $\bar{p}(e) = \frac{1}{T}\sum_{t=1}^{T}\hat{p}(e)_t$ . Next, we consider only those structures  $e$  for which  $\bar{p}(e) \geq 0.5$ , i.e.,  $e$  has a minimum probability of  $50\%$  of being a true positive. This is the threshold step in Fig. 12. We do this so as to consider only the true positive, false positive, and false negative
Figure 12: Inference procedure. Stars  $(\star)$  denote the final outcomes of our framework  $M_{\phi}$ .
Figure 13: Zoomed-in views of Fig. 12.
structures in the final outcomes. We use these structures to create a skeletal discrete segmentation map (see Fig. 12(c)) which has the same spatial resolution as Fig. 12(a). As we want to recover the thickness of each structure, we overlay the two maps to get the final discrete segmentation map (see Fig. 12(e)).
The uncertainty heatmap that we obtain from  $M_{\phi}$  is also skeletal (see Fig. 12(d)). We recover the structure thickness to get the final uncertainty heatmap (see Fig. 12(f)). We use shortest distance to do this. Shortest distance is used to assign uncertainty values from Fig. 12(d) to the pixels in Fig. 12(e). The shortest distance uses paths only along the foreground pixels and not along the background ones. This ensures that pixels within a structure are not assigned uncertainty values from other nearby structures. We provide a zoomed-in view in Fig. 13.
We note that generating the Morse complex is computationally heavy, however, it needs to be computed only once across the  $T$  runs. As described in Sec. 3.1, the sampled structures are between  $(c_s, c_m)$ , and so the Morse complex is generated only in the first run.
In this paper, we validate our results on four segmentation datasets: DRIVE [65], ROSE [40], ROADS [44], and PARSE 2022 Grand Challenge [39, 70].
DRIVE [65] The DRIVE dataset is a 2D retinal vessel dataset with 40 images. Each image has a resolution of  $584 \times 565$ . We use the dataset's predetermined split of 20 training images and 20 test images. For training, we keep aside four randomly-chosen samples as validation, and train on the remaining 16 samples.
ROSE [40] The ROSE dataset is a 2D retinal OCTA (Optical Coherence Tomography Angiography) segmentation dataset. We use ROSE-1 (SVC) in this work. It has a predetermined split of 30 train and 9 test samples, with each sample having a resolution of  $304 \times 304$ . For training, we keep aside four randomly-chosen samples as validation, and train on the remaining 26 samples.
ROADS [44] The ROADS dataset is a large, non-medical dataset containing 1171 aerial images (1108 train, 14 validation, and 49 test), each of  $1500 \times 1500$  resolution. It is a challenging dataset due to obstruction from nearby trees, shadows, varying texture/color of roads, road class imbalance, and so on.
PARSE [39, 70] The PARSE dataset is a 3D CT dataset containing pulmonary artery segmentations. The dataset contains 100 volumes and their sizes vary from  $512 \times 512 \times 228$  to  $512 \times 512 \times 376$ . As there is no predetermined train/test split, we use 4-fold cross-validation and report the average performance.
We use the PyTorch framework, a single NVIDIA Tesla V100-SXM2 GPU (32G Memory), and a Dual Intel Xeon Silver 4216 CPU@2.1Ghz (16 cores) for all the experiments. The training hyperparameters for our method as well as the baselines are as tabulated in Tab. 4. Note that although PARSE is a 3D dataset, all the segmentation networks (backbones)  $F_{\theta}$  are 2D, that is, the networks are trained on 2D slices of the dataset. This was done to maintain a fair comparison across all baselines, as methods such as Prob.-UNet and PHiSeg only had 2D implementations available.
We use the publicly available codes for the baselines:
- UNet [57]: https://github.com/johschmidt42/PyTorch-2D-3D-UNet-Tutorial
- Prob.-UNet [32]: https://github.com/stefanknegt/Probabilistic-Unet-Pytorch
- PHiSeg [4]: https://github.com/gigantenbein/UNet-Zoo
Hu et al. [28]: https://github.com/HuXiaoling/Structural_Uncertainty
- DeepVesselNet [68]: https://github.com/dhavalshah18/deepvesselnet
- CS²-Net [48]: https://github.com/iMED-Lab/CS-Net
F.2 Our method
Our code is available at https://github.com/Saumya-Gupta-26/struct-uncertainty. For reproducibility, we provide the architecture details as follows. The 'Joint reasoning' module described in Sec. 3.2 of our framework is a Graph Neural Network (GNN) [59], specifically a Graph Convolution Network (GCN) [31]. As per Fig. 6, the input feature vector for each graph node is constructed by passing  $[x^c, f^c, m]$  through the following architecture:  $C(3,24) \to ReLU \to D(0.2) \to C(3,32) \to ReLU \to D(0.2) \to MaxPool \to Concat(pers)$ , where,  $C(a,b)$  denotes a convolution layer having kernel size  $a$  and number of output channels  $b$ ;  $D(p)$  denotes a Dropout layer with probability  $p$ ; MaxPool denotes the adaptive maxpool layer<sup>4</sup> returning a  $1 \times 1$  output for each channel; and pers is a scalar value denoting the persistence of the structure. Furthermore, the bounding box size of  $[x^c, f^c, m]$  is  $32 \times 32$  centered at each structure for 2D (for 3D, it is  $32 \times 32 \times 32$ ).
The aforementioned layers generate the input feature vector for each graph node. They are then passed through the GNN which contains the following layers:  $GCN(32) \to ReLU \to D(0.2) \to$
<sup>4</sup>https://pytorch.org/docs/stable/generated/torch(nn.AdaptiveMaxPool2d.html
Table 4: Training configuration
Dataset	Model	Patch Size; Batch Size	Learning rate (LR); Optimizer
DRIVE	UNet	128 × 128; 8	1e-3 with LR scheduler6; Adam7 with weight decay 3e-5
DeepVesselNet	128 × 128; 8	1e-3 with LR scheduler; Adam with weight decay 3e-5
CS2-Net	128 × 128; 8	1e-3 with LR scheduler; Adam with weight decay 3e-5
Prob.-UNet	128 × 128; 8	1e-3; Adam with weight decay 0
PhiSeg	128 × 128; 8	1e-4 with LR scheduler; Adam with weight decay 1e-5
Hu et al.	128 × 128; 8	1e-3; Adam with weight decay 0
Ours	128 × 128; 8	1e-3; Adam with weight decay 0
ROSE,ROADS	UNet	128 × 128; 6	1e-3 with LR scheduler; Adam with weight decay 3e-5
DeepVesselNet	128 × 128; 6	1e-3 with LR scheduler; Adam with weight decay 3e-5
CS2-Net	128 × 128; 6	1e-3 with LR scheduler; Adam with weight decay 3e-5
Prob.-UNet	128 × 128; 6	1e-3; Adam with weight decay 0
PhiSeg	128 × 128; 6	1e-4 with LR scheduler; Adam with weight decay 1e-5
Hu et al.	128 × 128; 6	1e-3; Adam with weight decay 0
Ours	128 × 128; 6	1e-3; Adam with weight decay 0
PARSE	UNet	128 × 128; 8	1e-3 with LR scheduler; Adam with weight decay 3e-5
DeepVesselNet	128 × 128; 8	1e-3 with LR scheduler; Adam with weight decay 3e-5
CS2-Net	128 × 128; 8	1e-3 with LR scheduler; Adam with weight decay 3e-5
Prob.-UNet	128 × 1128; 8	1e-3; Adam with weight decay 0
PhiSeg	128 × 128; 8	1e-4 with LR scheduler; Adam with weight decay 1e-5
Hu et al.	128 × 128; 8	1e-3; Adam with weight decay 0
Ours	128 × 1128; 8	1e-3; Adam with weight decay 0
$GCN(64) \to ReLU \to D(0.2) \to GCN(32) \to ReLU \to D(0.2)$ , where  $GCN(a)$  denotes a GCNConv layer having  $a$  number of output channels. The output from this sequence of layers is then fed to two separate  $GCN(1)$  layers to output the regression likelihood  $\hat{p}(e)$  and the uncertainty  $\hat{\delta}_e^2$ . As per GNN fashion, the weights of the layers are shared across all the nodes.
G Evaluation metrics
We use both segmentation and uncertainty metrics to evaluate our method. We describe the metrics in detail below.
G.1 Uncertainty metrics
We use Reliability diagrams [8] and Expected calibration error (ECE) [50] to evaluate the quality of uncertainty. As both the metrics were originally designed for classification, we adapt from the classification task to semantic segmentation by treating each pixel as an independent sample. For both metrics, we first divide the probability interval  $[0,1]$  into  $N$  equal-sized probability intervals (each of size  $\frac{1}{N}$ ). We use  $N = 20$  bins in this work. We then calculate the accuracy and confidence of each bin.
Reliability diagrams (RD) [8] Reliability diagrams are a visual representation of model calibration by plotting the expected accuracy as a function of confidence (confidence  $= 1 - \text{uncertainty}$ ). Perfect calibration corresponds to an identity function in the RD, i.e., the model is not over/under-confident. Consider the set of pixels/structures whose predicted probabilities fall into the bin  $B_{i}$ . The accuracy and confidence are given by:
$$
a c c (B _ {i}) = \frac {1}{| B _ {i} |} \sum_ {\forall x \in B _ {i}} \mathbf {1} (\hat {Y} (x) = Y (x))
$$
$$
\operatorname {c o n f} \left(B _ {i}\right) = \frac {1}{\left| B _ {i} \right|} \sum_ {\forall x \in B _ {i}} \hat {P} (x)
$$
$^{5}$ https://pytorch-geometric.readthedocs.io/en/latest/generated/torch/geometric.nn. conv.GCNConv.html#torch_geometric.nn.conv.GCNConv
$^{6}$ https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler. ReduceLROnPlateau.html
<sup>7</sup>https://pytorch.org/docs/stable/generated/torch.optim.Adam.html
Table 5: Comparison on topological metrics Betti Number error and Betti Matching error for different datasets. All methods use UNet as the backbone.  $\beta_0^{\mathrm{err}}$ ,  $\beta_1^{\mathrm{err}}$ , and  $\beta_2^{\mathrm{err}}$  denote Betti Number error in 0-dim, 1-dim, and 2-dim respectively.  $\mu_0^{\mathrm{err}}$  and  $\mu_1^{\mathrm{err}}$  denote Betti Matching error in 0-dim and 1-dim respectively. The statistically significant better performances are highlighted in bold
Dataset	Method	β0err↓	β1err↓	β2err↓	μ0err↓	μ1err↓
DRIVE	UNet	166.3154 ± 12.1065	9.3149 ± 4.2062	-	205.8312 ± 12.4496	28.9587 ± 5.3766
Prob.-UNet	146.6373 ± 11.0831	7.8197 ± 3.1980	-	191.2790 ± 11.2324	24.7826 ± 4.9684
PHiSeg	145.3777 ± 12.4873	7.1542 ± 3.6436	-	190.0528 ± 10.3376	24.0893 ± 4.1123
Hu et al.	140.8317 ± 11.5502	6.3083 ± 2.2372	-	188.6573 ± 10.2403	23.7263 ± 4.3402
Ours	127.4041 ± 10.7344	4.6172 ± 2.6586	-	161.4536 ± 9.7017	20.6835 ± 3.4121
ROSE	UNet	231.5081 ± 15.5573	9.8826 ± 1.6486	-	243.2775 ± 16.9274	14.8922 ± 1.6793
Prob.-UNet	229.7987 ± 15.4307	9.0396 ± 1.6315	-	240.3295 ± 16.8371	14.0771 ± 2.0375
PHiSeg	220.0644 ± 14.6356	7.8644 ± 2.0692	-	228.4348 ± 17.8907	11.0377 ± 1.9442
Hu et al.	219.7530 ± 15.8446	7.6981 ± 1.5677	-	226.2989 ± 16.1992	10.6838 ± 1.8091
Ours	203.5791 ± 13.6467	5.0553 ± 1.4734	-	210.1763 ± 15.1485	8.6489 ± 1.5646
ROADS	UNet	75.6666 ± 8.1079	25.5777 ± 7.4432	-	78.2291 ± 9.5055	30.5104 ± 6.6921
Prob.-UNet	70.3564 ± 7.5929	24.3852 ± 7.0812	-	72.8129 ± 9.1638	29.8830 ± 6.1977
PHiSeg	68.7237 ± 7.9177	24.1772 ± 6.5982	-	70.2788 ± 8.2474	28.9467 ± 5.9164
Hu et al.	61.5167 ± 6.1625	23.5863 ± 5.3985	-	62.4951 ± 7.7601	26.2681 ± 5.8736
Ours	45.6735 ± 5.9286	17.2653 ± 4.6162	-	47.1429 ± 6.7905	23.1837 ± 5.4451
Parse	UNet	673.7016 ± 23.9541	79.5825 ± 10.9693	18.4316 ± 2.9432	-	-
Prob.-UNet	620.1903 ± 22.0012	51.4995 ± 8.4096	16.7046 ± 2.2419	-	-
PHiSeg	587.2137 ± 22.6801	45.9331 ± 8.7251	15.8529 ± 3.0218	-	-
Hu et al.	555.9788 ± 23.5735	40.0707 ± 8.2376	13.9498 ± 2.2883	-	-
Ours	520.4991 ± 22.4327	33.0532 ± 7.8453	10.3831 ± 2.1035	-	-
where,  $Y$  is the discrete segmentation ground truth (GT), and  $\hat{Y}$  is the discrete segmentation map outputted by the model. In our method,  $\hat{Y}$  is as shown in Fig. 12(c). Additionally,  $\hat{P}$  is the pixelwise probability (likelihood) outputted by the model, whereas in our case, it is the structure-wise uncertainty  $\bar{\delta}^2$  (Fig.12(d)). For our method and Hu et al., the  $x \in B_i$  denotes structures, while in the other methods, it denotes pixels.
Expected calibration error (ECE) [50] RDs are only a visual cue, and so we also use ECE: a scalar to summarize the calibration performance. RDs do not take into account the number of pixels/structures in each bin. Thus, to account for such variations of the number of samples in a bin, we use ECE. It is given by:
$$
E C E = \sum_ {i = 1} ^ {N} \frac {\left| B _ {i} \right|}{n} \left| a c c (B _ {i}) - c o n f (B _ {i}) \right|
$$
where  $n = \sum_{i}^{N}|B_{i}|$  is the total number of pixels/structures. The difference between acc and conf for a given bin represents the calibration gap. When there is perfect calibration, ECE is zero.
The definition of  $acc$  and  $conf$  remains the same as defined for RDs.
G.2 Segmentation metrics
DICE [73] DICE score is a popular metric which measures the area/volumetric overlap between the predicted and ground truth discrete masks. It overcomes the class imbalance problem in the pixel-wise accuracy metric by considering only the foreground classes for measuring the overlap. The higher the DICE, the better the segmentation.
clDice [62] clDice is derived from DICE, however, clDice uses the skeleton of the predictions. This makes it sensitive to the performance of thin structures like vessels which is important in curvilinear segmentation. The higher the value, the better the segmentation.
ARI [54] The Rand index computes similarity between two clusterings. This raw score is "adjusted for chance" to get ARI (Adjusted Rand Index) [2]. The ARI takes into account the fact that some agreement between the two clusterings can occur by chance, and it adjusts the Rand index to account for this possibility. The higher the value, the better the segmentation.
VOI [42] The VOI metric is defined as the sum of the conditional entropies between two segmentations. A lower VOI value indicates better segmentation.
Table 6: Additional ablation study results of our method on the DRIVE dataset using UNet as the backbone. The best results (as reported in Tab. 1 of the main paper) are in bold. In the table, Feature vector denotes the length of the input feature vector to the GCN, while Bounding box denotes the size of the crop/bounding box centered on each structure. These hyperparameters are discussed in detail in Appendix B. The  $+1$  denotes the concatenation of the scalar persistence value
Feature vector	ECE (%)↓	clDice↑
16 + 1	4.9621 ± 0.0048	0.7906 ± 0.0394
32 + 1	4.1633 ± 0.0043	0.7974 ± 0.0372
64 + 1	4.1667 ± 0.0045	0.7972 ± 0.0367
Bounding box	ECE (%)↓	clDice↑
16 × 16	5.1085 ± 0.0047	0.7842 ± 0.0416
32 × 32	4.1633 ± 0.0043	0.7974 ± 0.0372
64 × 64	4.1689 ± 0.0042	0.7921 ± 0.0363
Betti Number error [27] This metric computes the difference between the Betti numbers of the prediction and the ground truth. We separately compute this metric for 0-dimension and 1-dimension structures.
Betti Matching error [66] This metric is similar to Betti Number, however, in Betti Matching, the spatial location of the 0-dimension and 1-dimension components is also taken into account. Hence this metric is stricter compared to Betti Number error.
H Quantitative results on topological metrics
In Tab. 5, we provide results on two topologically important metrics, namely, Betti Number error [27] and Betti Matching error [66]. Our method consistently improves the segmentation result in terms of topology. This is consistent with our results in Tab. 1 of the main paper where our method outperforms the baselines on other topology-based metrics like clDice [62], ARI [2] and VOI [42]. Note that for the 3D PARSE dataset, we were unable to provide Betti Matching error results as its official implementation handles only 2D inputs.
I Ablation study
We check the effect of the different hyperparameters in our work by conducting experiments on the DRIVE dataset using UNet as the backbone. Our main hyperparameters are  $u, \alpha, \beta, \gamma$ , with  $u$  used in the Bernoulli distribution,  $\gamma$  in the path-generation algorithm, and  $(\alpha, \beta)^8$  as prior hyperparameters of the Inverse Gamma distribution. We test different values and report the ECE (the lower the better) in Fig. 14. For all the experiments, we set  $u = 0.3$ . We achieve the best ECE when  $\gamma = 0.2, \alpha = 2.0, \beta = 0.01$ , however, a reasonable range always yields improvement (notice how non-zero  $\gamma$  results in a sharp improvement). This demonstrates the robustness of our proposed method.
We also provide additional ablation studies in Tab. 6. We now include ablation studies on the dimensionality of the input feature vector, and size of the crops/bounding boxes, and report this in Tab. 6. We obtain the best results when the input feature vector size is 32 and the bounding box is  $32 \times 32$ . For lower values (16 and  $16 \times 16$ ), the performance reduces, while for higher values (64 and  $64 \times 64$ ) we did not observe any statistically significant improvement. Thus to maintain the tradeoff between complexity and performance, we respectively use 32 and  $32 \times 32$  for these hyperparameters.
Figure 14: Effect of hyperparameters.
$^{8}\alpha$  is the shape parameter;  $\beta$  is the scale parameter.
J Broader impact
In this work, we aim to capture structure-wise uncertainty of a given network, where a structure is defined to be a coherent set of pixels a user can intuitively understand, e.g., small vessels/branches, short stretches of road etc. Fine-scale structures such as vessels, neurons, and membranes often consist of interconnected branches or structures that form a cohesive entity. Thus structure-wise uncertainty maps can highlight uncertain instances or branches as a whole, providing a more accurate indication of regions where the segmentation may be inaccurate or uncertain. This is beneficial for proofreading or error-correction tasks as they can direct the focus of human annotators to uncertain structures that require further attention. This can save time and effort compared to pixel-wise uncertainty maps that highlight numerous pixels as uncertain, many of which do not require correction. Thus structure-wise uncertainty can provide more interpretable estimates and is a desirable approach for improving segmentation accuracy and supporting downstream analysis tasks. This can go a long way as the benefit of proofreading is twofold: it improves segmentation quality, and it also helps expand the body of labeled data that can be further used to train automatic segmentation methods. Our work is thus a useful tool in streamlining the process of scalable annotation. At the present stage, we do not foresee any potential negative societal impacts.
K Limitations
Our method currently fits in the context of curvilinear segmentation. In general, large object segmentation could also benefit from structure-wise uncertainty (structures in this case would be smaller patches/volumes). Discrete Morse theory can be used in this setting, however, we would need to make use of topological features other than the stable manifold. In its present form, our proposed solution is currently not applicable in a setting beyond curvilinear segmentation.