SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Benchmark Evaluation
Dataset (#train/#test) (vol size) (voxel resolution)	Method Reference	#Param	Dice(%)↑	HD95(mm)↓	ASD(mm)↓	NSD@1.5mm↑
TotalSeg-Femur-DRR	SwinUNET [21]	62.2M	93.64	3.32	0.94	0.85
AttentionUnet [39]	1.5M	93.66	3.12	0.93	0.85
2DConcat [4]	1.2M	93.05	3.73	1.02	0.83
UNet [29]	1.2M	93.36	3.32	0.95	0.84
MultiScale2DConcat [58]	3.5M	92.79	3.64	1.11	0.81
786/138	UNETR [22]	96.2M	92.39	3.96	1.16	0.80
128x128x128	TL-Embedding [18]	6.6M	90.43	4.28	1.41	0.73
1.0	1DConcat [11]	40.6M	89.66	4.86	1.57	0.67
TotalSeg-Rib-DRR	SwinUNETR	62.2M	54.05	6.17	0.90	0.49
AttentionUnet	1.5M	53.34	4.15	0.93	0.52
2DConcat	1.2M	50.90	4.56	1.10	0.49
UNet	1.2M	49.10	4.86	1.35	0.48
MultiScale2DConcat	3.5M	49.23	7.29	1.12	0.47
408/73	UNETR	96.2M	40.32	7.10	1.62	0.39
128x128x128	TL-Embedding	6.6M	29.15	7.67	2.22	0.31
2.5	1DConcat	40.6M	27.22	16.98	2.60	0.28
TotalSeg-Hip-DRR	SwinUNETR	62.2M	85.78	2.55	0.81	0.54
AttentionUnet	1.5M	85.03	2.60	0.78	0.52
2DConcat	1.2M	84.75	2.69	0.81	0.49
UNet	1.2M	84.45	3.16	0.88	0.48
MultiScale2DConcat	3.5M	84.48	2.98	0.84	0.50
320/56	UNETR	96.2M	82.27	3.35	0.97	0.44
128x128x128	TL-Embedding	6.6M	79.33	3.87	1.12	0.38
2.25	1DConcat	40.6M	78.85	3.67	1.11	0.37
VerSe19-Spine-DRR	SwinUNETR	62.2M	83.59	2.56	0.80	0.86
AttentionUnet	1.5M	83.66	2.43	0.74	0.85
2DConcat	1.2M	83.62	2.35	0.72	0.85
UNet	1.2M	82.17	2.61	0.83	0.82
MultiScale2DConcat	3.5M	81.85	2.74	0.78	0.82
1451/271	UNETR	96.2M	81.84	2.68	0.81	0.83
64x64x64	TL-Embedding	6.6M	79.20	3.00	0.99	0.76
1.5	1DConcat	40.6M	80.92	2.76	0.83	0.80
Aggregate	SwinUNETR	62.2M	79.27	3.65	0.86	0.68
AttentionUnet	1.5M	78.92	3.07	0.84	0.69
TwoDPermuteConcat	1.2M	78.08	3.33	0.91	0.67
UNet	1.2M	77.27	3.49	1.00	0.66
MultiScale2DPermuteConcat	3.5M	77.09	4.16	0.96	0.65
UNETR	96.2M	74.20	4.27	1.14	0.62
TLPredictor	6.6M	69.53	4.70	1.43	0.54
OneDConcat	40.6M	69.16	7.07	1.53	0.53
model performance of clinically relevant subgroups. In fact, good reconstruction performance on these minority subgroups, such as pathological cohorts, may be more valuable.
Figure 3: Disaggregated metrics on VerSe19-Spine-DRR: Reconstruction performance in clinically relevant subgroups that shows differing results for different subgroups.
Reporting of Clinically Relevant Metrics Figure 4 presents quantitative results for mean DSC vs. clinical parameter estimation error. One can see that the improvement in the mean DSC, averaged over the whole test set, does not always mean improvement in clinical metric estimation. For example, i) first row: in the fourth and fifth columns, OneDConcat has worse DSC by greater than five against UNet, but almost the same  $PT_{R}$  and better  $IS_{L}$ . Similar results can be seen on the third from the right column when comparing OneDConcat against MultiScale2DPermuteConcat or SwinUNETR, ii) second row: Similar observation can be seen with DSC difference of greater than two for other pair of methods in column 3, 4, and 5, iii) third row: The third column has NSA not much correlated to DSC when DSC is higher than 90, and even +ve correlation between DSC vs. FDA estimation error in the third column from the right.
Figure 4: DSC vs Clinical Metrics: Top (hip), Middle (vertebra), Bottom (femur); The x-axis denotes the mean DSC obtained by each architecture whereas the y-axis denotes corresponding clinical metric estimation error. Improvement in mean DSC across the whole dataset does not always result in reduced clinical parameter estimation error, as we see in the figure for many parameters across architectures.
This effect is similar for other Image-based metrics such as ASD, HD95 and NSD.
Figure 5: Qualitative examples where clinical metric and DSC diverge
In Figure 5, using sample results from our experiments, we present qualitative illustrations for three anatomies where DSC and clinical metrics are not consistent.
In the first row, the reconstruction with higher DSC is less anatomically plausible with higher Vertebra Canal Length (VCL) estimation error. This error can cause an overestimation of the safe zone for pedicle screw insertion resulting in incorrect surgical planning and outcome[30]. Similarly, in the second row, the Neck-Shaft Angle (NSA) is reconstructed well in the left example despite poor DSC compared to the one on the right. This signifies the need for measuring clinically relevant metrics, as NSA is an important characteristic of the proximal femur and useful in the diagnosis of various pathologies such as acetabular impingement, risk of hip fracture, hip dysplasia, and osteoarthritis [34, 42, 48]. Additionally, clinical metrics may be useful in quantifying reconstruction error in a localized region where DSC measures overall reconstruction accuracy. Case-in-point, in the last row of Figure 5, the Anterior Superior Illiac Spine (ASIS) is well reconstructed on the left with a large reconstruction error in the posterior ilium region whereas, on the right, the larger reconstruction error of the right ASIS,  $8.7\mathrm{mm}$  compared to  $2.62\mathrm{mm}$ , is compensated for by good reconstruction in other regions. These landmarks define points where muscles and ligaments attach, which may be used in personalized biomechanical evaluation[45] as well as for defining anatomical reference frame[57] useful for surgical planning, statistical modelling[53] etc.
Anatomical Landmarks and Metrics
Figure 6: Reduction in performance due to Domain Shift: We find that shifts in the population cohort, as well as X-ray-specific shifts, hinder clinical translation.
Domain Shift Results Robustness to Domain shift is a challenge for clinical adoption. Figure 6 shows performance gaps of models on various groups of test images that have specific types of domain shift compared to the training set. Kits19 subset of CTPelvic1k comes from different population cohorts and/or scanner-type. Clinic subset of CTPelvic1K contains images with hip fracture; RSNACervical contains images of the fractured cervical vertebra; Clinic-metal contains images with foreign implants such as cement, bone implants, screws, and rods. We observe that there is a substantial decrease in performance for all these new subsets. It is interesting to observe that while OneDConcat and TL-Embedding have lower DSC, their decrement in the score with domain shift is less severe than other methods in cohort shift and fractured bones.
Misalignment Results
Figure 7: Reduction in performance due to misalignment: Misalignment due to perturbation of the relative angle of LAT view w.r.t AP view results in reduced performance for femur, rib and hip anatomies while vertebra-specific models are quite robust since such perturbed views occur naturally in the dataset.
Ideally, AP and LAT views are perfectly orthogonal. However, in practical clinical settings, misalignment errors can often result in slightly non-orthogonal AP and LAT views. We simulate such scenarios by generating misaligned DRR pairs of AP and Lateral views by generating LAT views at five different angles in the 91 to 100 degrees range. When evaluating the robustness of all the models to accurately reconstruct 3D when such misaligned data are presented during inference, we see (in Figure 7) that the models have degraded performance, which increases with the increase in misalignment. The reduction in DSC in the rib is most prominent with up to 30, while the performance is stable in the vertebra with delta DSC limited to 3. In the femur and hip delta DSC is moderate in the range of six and ten, respectively. Surprisingly, MultiScaleConcat[9, 58] architecture seems considerably more robust to such shift than others. The spine is a naturally curved structure with relatively bigger variation in the curvature in all three axes across the population. Thus, a more robust performance of the models in vertebra reconstruction may be because the models might already have learned to see the vertebra shape from diverse angles due to intrinsic natural variation across the population.
5 Discussion, Limitations, and Future Work
We introduced a novel open-source benchmarking platform for Biplanar X-rays to 3D bone reconstruction that – has a careful collection of six publicly available but scattered datasets on four anatomies; experimental designs to study versatility and robustness against clinically important subgroups and domain shift due to misalignment errors; provides a set of preprocessing and utility tools, and reference implementations of several reconstruction models with no public source code, reference implementation of several clinical metrics for three anatomies. We included only encoder-decoder architecture in this work, as they are the most widely used ones and there is a tradeoff in going for
exhaustive set of methods vs. performing large number of insightful experiments to build useful insights with limited resource and time budget. Nevertheless, our framework is not limited to the specific architecture, and we hope this work allows the community to compare newly proposed or other yet unimplemented models in a common set of clinically relevant benchmarking tasks across multiple anatomies. Using this benchmark, we evaluated task-specific as well as off-the-shelf architectures. Our results show that while existing reconstruction methods perform fairly well on aggregated data when using image-based evaluation metrics such as DSC. However, a closer look into disaggregated data show that the clinically important minority subgroups within datasets such as fractured bones or bones with implant have substantially reduced performance compared to commonly reported average scores. When analyzing the relationship between common image-based metrics to clinical metrics qualitatively and quantitatively, we find that improvement in metrics like DSC do not always lead to improved performance on clinical metrics.
When exploring several papers in the literature to implement clinical parameter estimation methods, we found that these methods were validated only on a small cohort with less diversity than the datasets on which we report our results. Despite challenges in automatically extracting clinical metrics on a large, diverse cohort, including model-predicted shapes that are not always anatomically plausible, the methods we implemented did reasonably well with only a handful of failure cases when visually going through the test dataset. Nevertheless, building more robust clinical parameter estimation methods might be an interesting avenue for the community to work on.
We use digitally reconstructed radiographs (DRRs) from publicly available CT scans only, but the real target clinical application would take during inference real X-rays pair. Hence, more accurate evaluations require several pairs of people's AP-LAT X-ray and CT scans, which are not readily available and are rarely used in the literature. The community's concerted efforts may help bring such datasets to public access. One of the very important clinical applications of 2D-3D reconstruction models is in resource-constrained settings such as large parts of LMICs in rural and community hospitals where CT Scans are unavailable or most people cannot afford the costly CT scans. However, the very use of the reconstruction would then be intended for diagnosing or planning surgery for abnormal bones. Thus, future work should focus on building robust methods that can reliably detect for various demographic subgroups and subgroups with specific abnormalities. Finally, those who are interested in prospective trials and evaluating the models effect on clinical decision making will likely need various clinical parameters estimated in this work. Thus, our work provides an important first step in consolidating the community's effort, and offering the ML researchers a useful platform and direction towards building 2D-3D reconstruction models that could be translated to the real world.
We would like to thank Dr. Bhaskar Raj Pant for providing the initial impetus for this project and convincing the team about the need for X-ray to 3D Reconstruction in trauma surgery for rural healthcare. This work was made possible via the institutional support by NAAMII Nepal. Additionally, We would like to thank our colleagues at NAAMII Nepal, especially TOGAI Team members - Kanchan, Prasiddha, Rabin, Safal and Manish, for being our sounding board to bounce ideas off and providing valuable technical support.
[1] D. F. Almeida, P. Astudillo, and D. Vandermeulen. Three-dimensional image volumes from two-dimensional digitally reconstructed radiographs: A deep learning approach in lower limb ct scans. Medical Physics, 48(5):2448-2457, 2021. 2
[2] M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menze, O. Ronneberger, R. M. Summers, et al. The medical segmentation decathlon. Nature communications, 13 (1):4128, 2022. 4
[3] B. Aubert, C. Vazquez, T. Cresson, S. Parent, and J. A. de Guise. Toward automated 3d spine reconstruction from biplanar radiographs using cnn for statistical spine model fitting. IEEE Transactions on Medical Imaging, 38(12):2796-2806, 2019. 2
[4] A. Bayat, A. Sekuboyina, J. C. Paetzold, C. Payer, D. Stern, M. Urschler, J. S. Kirschke, and B. H. Menze. Inferring the 3d standing spine posture from 2d radiographs. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 775-784. Springer, 2020. 2, 4, 5, 6, 7