The Center for Computational Sciences, University of Tsukuba MCRP-2021 report

3D Volume Estimation from X-Ray Images Using Implicit NeuralRepresentation
Chun Xie
Center for Computational Sciences, University of Tsukuba

１．Project Purpose
Reconstruction of 3D Structures from few x-ray images is a non-trivial task. A
common technique used in orthopedic procedures involve intra-operative
fluoroscopy, that is taking the x-ray images of the target during the operation andassessing the progress. This requires thinking and localizing in 3D mentally fromthe x-ray images. Our aim is to relax the mental burden on surgeons posed by
ambiguous depth information in the image, through 3D reconstruction andregistration of the image with a pre-operatively obtained 3D model from the CTscan. The initial plan was to incorporate implicit neural representations forgenerating the 3D structure however we find that learning such a representationof the same volume but with different orientation for each image is redundant andinefficient. Therefore, we opted for a scene coordinate based approach where ascene coordinate map containing 3D point locations of the surface points for eachcorresponding image pixel is regressed using a U-Net architecture. This allows fora partial point cloud reconstruction from the image as well as registration with apre-obtained 3D model (Fig. 1).

Rendering(alignment)

Registra�on

Input Image

U-Net

Scene Coordinate Map

3D ModelOverlay

Fig. 1 Overview of our proposed method. Scene coordinate regression (red box) and registration (blue box).

２．Results
We evaluated the registration accuracy for the reconstructed scene coordinate
maps. With a simulated dataset of 2700 DRR images which were taken in different
orientations, The rotation error was 3.96 +/- 2.26 deg and translation error was
23.07 +/- 19.29 mm. Example overlays of 3D model aligned to the X-Ray are shownin Fig. 2. Since the registration result is close to the ground truth pose, intensitybased optimization method could be applied for further refinement.

The Center for Computational Sciences, University of Tsukuba MCRP-2021 report

Fig. 2. Examples of registration results. Case with low aligned error (top) and high alignment error (bottom)

３．Roles of the MCRP and its significanceThe computing resource is crucial in developing a deep learning system. MCRP
provided the essential resource for training our proposed model. The main
contribution however was during the generation of simulated x-ray images fromthe CT-scan data. It has been shown in previous researches that training a model
on simulated images using domain randomization produces as good result as thosetrained on real x-ray images. The medical community usually has limited amountof data available for public use. Therefore, generating a simulated dataset is ofgreat importance to the field and can be accelerated using resources from MCRP.
４．Future planAs mentioned above, we would like to create and share a dataset for x-ray image
reconstruction and registration tasks. Currently, there very few public datasets
available in the community and those that are available contain few hundred images
for each patient. To create a common benchmark dataset, we plan to generaterealistic simulation images in the order of thousands for each patient anatomy andtrain large models for reconstruction as well as registration tasks.
５．Publications and conference presentations
Presentations
Pragyan Shrestha, Xie Chun, Hidehiko Shishido, Yuichi Yoshii, Itaru Kitahara, "X-Ray to CT Registration Using Scene CoordinateRegression Network", International Forums on Medical Imaging in Asia (IFMIA2023), 2 pages, (2023/1)

Supercomputer

UseCygnus

Yes

Oakforest-PACSNo*in units of node-hour product

Allocated resources*
InitialAdditionalresources

resources

4050

0