Limited capture range, and the requirement to provide high quality initialization for optimization-based 2D 3D image registration methods, can significantly degrade the performance of 3D image reconstruction and motion compensation pipelines. Challenging clinical imaging scenarios, which contain significant subject motion such as fet al in-utero imaging, complicate the 3D image and volume reconstruction process. In this paper we present a learning based image registration method capable of predicting 3D rigid transformations of arbitrarily oriented 2D image slices, with respect to a learned canonical atlas co-ordinate system. Only image slice intensity information is used to perform registration and canonical alignment, no spatial transform initialization is required. To find image transformations we utilize a Convolutional Neural Network (CNN) architecture to learn the regression function capable of mapping 2D image slices to a 3D canonical atlas space. We extensively evaluate the effectiveness of our approach quantitatively on simulated Magnetic Resonance Imaging (MRI), fet al brain imagery with synthetic motion and further demonstrate qualitative results on real fet al MRI data where our method is integrated into a full reconstruction and motion compensation pipeline. Our learning based registration achieves an average spatial prediction error of 7 mm on simulated data and produces qualitatively improved reconstructions for heavily moving fetuses with gestational ages of approximately 20 weeks. Our model provides a general and computationally efficient solution to the 2D 3D registration initialization problem and is suitable for real-time scenarios.
@cite_15 trained CNNs to regress the transformation parameters for 2D to 3D registration. Their regression is performed directly on image slices without feature extraction. The input to the network are pairs of target 2D X-ray images with synthetic DRR source images that are generated from a Cone-Beam Computed Tomography (CBCT) volume. Each image pair is augmented with varying levels of anisotropic diffusion. Iterative updates of the CNN yield new transformation parameters to generate new DRR images until similarly to the target X-Ray converges.
@cite_15
2755211235
The 2D-3D registration is a cornerstone to align the inter-treatment X-ray images with the available volumetric images. In this paper, we propose a CNN regression based non-rigid 2D-3D registration method. An iterative refinement scheme is introduced to update the reference volumetric image and the digitally-reconstructed-radiograph (DRR) for convergence to the target X-ray image. The CNN-based regressor represents the mapping between an image pair and the in-between deformation parameters. In particular, the short residual connections in the convolution blocks and long jump connections for the multi-scale feature map fusion facilitate the information propagation in training the regressor. The proposed method has been applied to 2D-3D registration of synthetic X-ray and clinically-captured CBCT images. Experimental results demonstrate the proposed method realizes an accurate and efficient 2D-3D registration of craniofacial images. Generation of digitally reconstructed radiographs (DRRs) is computationally expensive and is typically the rate-limiting step in the execution time of intensity-based two-dimensional to three-dimensional (2D-3D) registration algorithms. We address this computational issue by extending the technique of light field rendering from the computer graphics community. The extension of light fields, which we call attenuation fields (AFs), allows most of the DRR computation to be performed in a preprocessing step; after this precomputation step, DRRs can be generated substantially faster than with conventional ray casting. We derive expressions for the physical sizes of the two planes of an AF necessary to generate DRRs for a given X-ray camera geometry and all possible object motion within a specified range. Because an AF is a ray-based data structure, it is substantially more memory efficient than a huge table of precomputed DRRs because it eliminates the redundancy of replicated rays. Nonetheless, an AF can require substantial memory, which we address by compressing it using vector quantization. We compare DRRs generated using AFs (AF-DRRs) to those generated using ray casting (RC-DRRs) for a typical C-arm geometry and computed tomography images of several anatomic regions. They are quantitatively very similar: the median peak signal-to-noise ratio of AF-DRRs versus RC-DRRs is greater than 43 dB in all cases. We perform intensity-based 2D-3D registration using AF-DRRs and RC-DRRs and evaluate registration accuracy using gold-standard clinical spine image data from four patients. The registration accuracy and robustness of the two methods is virtually identical whereas the execution speed using AF-DRRs is an order of magnitude faster. This paper presents work towards a system for intra-operative registration of 3D CT data to 2D X-ray radiographs. The registration procedure involves iterative comparison of Digitally Reconstructed Radiographs (DRRs) with X-ray images acquired during surgery. A new data structure called a Transgraph permits rapid generation of DRRS, and greatly speeds up the registration process. The underlying data structures are described, and the registration algorithm is evaluated for application to an existing image guided radiosurgery system. This paper aims to solve a fundamental problem in intensity-based 2D 3D registration, which concerns the limited capture range and need for very good initialization of state-of-the-art image registration methods. We propose a regression approach that learns to predict rotations and translations of arbitrary 2D image slices from 3D volumes, with respect to a learned canonical atlas co-ordinate system. To this end, we utilize Convolutional Neural Networks (CNNs) to learn the highly complex regression function that maps 2D image slices into their correct position and orientation in 3D space. Our approach is attractive in challenging imaging scenarios, where significant subject motion complicates reconstruction performance of 3D volumes from 2D slice data. We extensively evaluate the effectiveness of our approach quantitatively on simulated MRI brain data with extreme random motion. We further demonstrate qualitative results on fet al MRI where our method is integrated into a full reconstruction and motion compensation pipeline. With our CNN regression approach we obtain an average prediction error of 7 mm on simulated data, and convincing reconstruction quality of images of very young fetuses where previous methods fail. We further discuss applications to Computed Tomography (CT) and X-Ray projections. Our approach is a general solution to the 2D 3D initialization problem. It is computationally efficient, with prediction times per slice of a few milliseconds, making it suitable for real-time scenarios.