ACCV'07 Workshop on Multi-dimensional and Multi-view Image Processing, Tokyo, Nov., 2007

MM-O-09

Patient-specific femoral shape estimation using a parametric model and two 2D
fluoroscopic images
Yumi IwashitaÝ , Ryo KurazumeÝ , Kahori NakamuraÝ , Toshiyuki OkadaÝÝ ,Yoshinobu SatoÝÝ , Nobuhiko SuganoÝÝ, Tsuyoshi KoyamaÝÝ, and Tsutomu HasegawaÝ
ÝGraduate Faculty of Infomation Sciencen and Electrical Engineering
Kyushu University
744 Motooka, Nishi-ku, Fukuoka, JapanÝÝ

Graduate School of Medicine
Osaka University
2-2 Yamadaoka, Suita-shi, 565-0871, Japan

Abstract

However, these apparatuses are generally large and very expensive, and thus, they are usually installed in large medical institutions rather than small local clinics. Since anappointment is also required before examination, these systems are not suitable for urgent fracture diagnosis in emergency treatment.
However, X-ray has been widely used as traditional medical diagnosis. Recently, digital fluoroscopy has been developed and widely used in many hospitals. The cost of thisfluoroscopic inspection system is much lower than that ofCT or MRI systems and the system can be dealt with moreconveniently. Furthermore, the risk of radiation exposure isalso lower than that of the CT inspection system.
In the present paper, a technique by which to estimate
the patient-specific 3D shape of a femur from only two fluoroscopic images is proposed. The proposed technique utilizes a parametric femoral model constructed by statisticalanalysis of 3D femoral shapes created from CT images of
56 patients. The position/attitude and shape parameters of
the parametric model are then estimated from two 2D fluoroscopic images using a distance map constructed by theLevel Set Method. Experiments using in vivo images for
hip prosthesis patients, are successfully carried out, and it isverified that the proposed system has practical applications.

In medical diagnostic imaging, the X-ray CT scannerand the MRI system have been widely used to examine
3D shapes and internal structures of living organisms andbones. However, these apparatuses are generally large andvery expensive. Since an appointment is also required before examination, these systems are not suitable for urgent
fracture diagnosis in emergency treatment. However, Xray/fluoroscopy has been widely used as traditional medical diagnosis. Therefore, the realization of the reconstruction of precise 3D shapes of living organisms or bones froma few conventional 2D fluoroscopic images might be very
useful in practice, in terms of cost, labor, and radiation exposure. The present paper proposes a method by which toestimate a patient-specific 3D shape of a femur from only
two fluoroscopic images using a parametric femoral model.
First, we develop a parametric femoral model by the statistical analysis of 3D femoral shapes created from CT imagesof 56 patients. Then, the position and shape parametersof the parametric model are estimated from two 2D fluoroscopic images using a distance map constructed by theLevel Set Method. Experiments using in vivo images for
hip prosthesis patients are successfully carried out, and it isverified that the proposed system has practical applications.

2 Related works
1. Introduction

The 2D/3D registration problem is well established inimage processing, especially for texture mapping in Computer Graphics or Augmented Reality. For a rigid object, 1) feature-based techniques [1],[2],[3], 2) imagebased techniques using 3D texture, reflectance, brightness,and shading [4],[5],[6], and 3) silhouette-based techniques

In medical diagnostic imaging, the X-ray Computed Tomography (CT) scanner and the Magnetic Resonance Imaging (MRI) system have been widely used to examine the 3D
shape or internal structure of living organisms and bones.

59

ACCV'07 Workshop on Multi-dimensional and Multi-view Image Processing, Tokyo, Nov., 2007

MM-O-091
93%[7],[8],[9],[10], have been proposed. In particular, in surgical navigation systems, Digitally Reconstructed Radiographs (DRRs) [11],[12] are widely used in 2D/3D registration for the fluoroscopy-guided surgery.
In 2D/3D registration of a non-rigid object such as soft
tissues in medical imaging, similarity measure [13],[14],mutual imformation [15], affine [16],[17], geometric hashing [18], and displacement-field-based transformation [19]have been proposed and tested. In addition, the 3D shapeestimation of a parameterized object, such as the shape reconstruction of mathematical plaster models with unknown
parameters using a laser range finder [20], or the comparison of multiple cross-section images of a 3D model and a
3D parametric model [21], has also been studied. However,these studies assumed the use of a sufficient number of images or a precise 3D shape taken by a laser range finder,and only a few studies have examined 3D non-rigid shapereconstruction from only a few 2D images [22],[23],[24].

Contribution

0.8
0.6
0.4
0.2
0

0

10
20
30
40
Number of parameters

50

Figure 1. Contribution of parametric modelare determined according to the displacement vector
fields. Each of the femur surface models is represented
by 1500 3D positions, which is regarded as a 1500 ¢ 3
dimensional shape vector. Given shape vectors, theaverage shape vector is given by their average. PCA isapplied to a set of the shape vectors subtracted by theaverage shape vector to obtain the eigenvectors whose
coefficients correspond to the shape parameters.

3 3D parametric femoral model
We utilize the statistical shape model of the femur proposed by Okada [25],[26]. In this technique, a number of
3D femoral shapes created from CT images are analyzed
statistically, and the parametric femoral model [27],[28],which consists of the average 3D shape and several shapeparameters, is created. With this parametric femoral model,
a general 3D shape of the femur is expressed by the average
shape and several shape parameters.
The concrete procedure for creating a parametric 3D
femoral model is as follows:

The parametric femoral model used in the following
experiments was created using CT images of 56 patients.
By applying PCA to 56 samples of 3D femoral shapes,
we extracted the most significant 50 principal components(         ), standard deviation (         ), andcorresponding principal vectors (         ). With theobtained parametric femoral model, the general 3D shapeof a femur is expressed as

¼   ¡  ¡     ¡  ¡    ¡ ¡ ¡ (1)where  is the surface point of the average shape and  ¼
is the surface point of the general shape. Therefore, thegeneral 3D shape of a femur is expressed by the parametricfemoral model with

1. Surface models of femurs are created from CT imagesby manual segmentation and Marching Cubes.
2. Local coordinate axes of the surface models are determined by applying the principal component analysis(PCA) to the set of 3D positions of the node points ineach surface model. The Z axis is determined as theaxis corresponding to the largest eigenvalue, which istoward the longitudial direction of the femoral shaft.

¯ average 3D shape and several principal vectors (predetermined)¯ several (up to 50) shape parameters (estimated)
Figure 1 shows the contribution ratio of the shape parameters for the statistical femoral model.

The region where the length from the top of thefemoral head is less than 35 % of total length of thefemur is extracted as a proximal femur and the centerof gravity is defined as the origin of the local coordinate system.

4. Reconstruction of 3D femoral shape fromtwo 2D fluoroscopic images

3. One of the femoral model is selected as the reference
model and displacement vector fields to all other models described by the thin plate spline are calculated using the non-rigid registration algorithm[29].

In this section, we introduce the 2D/3D registration algorithm and the estimation procedure of the optimum shapeparameters using two fluoroscopic images.
This 2D/3D registration algorithm utilizes the contour
lines of the silhouette of the 2D image and the projected
contour lines of the 3D model. The optimum position of the

4. 1500 surface points on the reference model are selected,
and their corresponding surface points on other models

60

ACCV'07 Workshop on Multi-dimensional and Multi-view Image Processing, Tokyo, Nov., 2007

MM-O-09fi

3D model is determined such that the contour lines coincide with each other on the 2D image plane. In commonlyused approaches such as the ICP algorithm, the error metricis usually defined as the sum of the distances between the
points on the 2D contour lines and their nearest points on
the projected contour lines of the 3D model. However, thenearest point search is a laborious task and is time consuming even for the kd tree-based algorithm [30].
In the present approach, the 2D distance map [7] is utilized. First, the 2D distance map from the contour lines iscreated on the 2D image using the Fast Marching Method[31],[32] or raster scan algorithms [33]. Once the 2D distance map is created, the error metric is obtained directlyfrom the 2D distance map as the value at the points on the
projected contour lines of the 3D model. Using the courseto-fine strategy called “Distance Band” [7], a 2D distancemap can be constructed quite rapidly using the Fast Marching Method.
After creating the 2D distance map, the parametricfemoral model is placed at an arbitrary position and the2D projection image of the 3D model is calculated. Then,contour lines of the projected image and corresponding 3D
patches of the 3D model are extracted. Finally, the force
which is calculated from the 2D distance map at the projected contour points is applyed directly to the corresponding 3D patch, and the optimum position and the shape parameters are estimated as shown in Fig.3.

Object

COG

ri
M

fi

F

Image planefiProjected
contour pointyx(s, t)z
Steepest descentdirectionv
Focal point

(a)

(b)

Figure 3. Calculation of the total force andmoment around the COG
90 mm
50 mm

ABS
150 mmABS
20 mm
Stainless steel ballNon-coplanar marker for
external parameter calibration

Stainless steel disk

Coplanar marker forinternal parameter calibration

Figure 4. Non-coplanar and coplanar markers+20 pixels

Fig.4, are used for the calibration of external and internalparameters, respectively. These markers are constructed ofacrylonitrile butadiene styrene (ABS), which indicated thehighest transmission of X-rays in preliminary experimentation. The non-coplanar marker contains nine small stainless steel spheres and the coplanar marker contains 16 small
stainless steel disks. The calibration procedure using thesemarkers is as follows:+40 pixels

Boundary

Contour line in distance map

Figure 2. 2D distance map from contour in thefemoral image

step 1. The non-coplanar marker is captured by X-ray CT
and the relative positions of the stainless steel spheres

5 In vivo experiments
We conducted in vivo experiments for hip prosthesis patients. Fluoroscopic images of four patients were taken inclinical practice using the fluoroscopic imaging apparatus(Siemens, Siremobil ISO-C), and shapes measured by the
CT scanner and estimated by parameter estimation werecompared.

    

Figure 5. Example fluoroscopic image of noncoplanar marker

First, we measured the internal and external parameters.
A non-coplanar marker and a coplanar marker, shown in61

ACCV'07 Workshop on Multi-dimensional and Multi-view Image Processing, Tokyo, Nov., 2007

MM-O-09

Step 2 and the projection image of stainless steel disks
by Tsai’s method.

X-ray apparatus  First, we manually extracted the contour of the femur(Fig.8) in the fluoroscopic images (Fig.7). Image size andresolution are 512  512 pixels and 0.454 mm/pixel. We
then estimated the position and 10 shape parameters fromthe silhouette of the femur. The precise 3D shapes of patients’ femurs were measured precisely by CT scanner.
The average errors and standard deviations of the estimated femoral shapes are shown in Fig.9. In these figures,“0” in the horizontal axis shows the case in which only the
position is estimated without parameter estimation.
The experimental results show that the average error between the estimated shape and the actual shape is approximately 0.8 mm to 1.1 mm for the in vivo experiments. One
example of average, actual, and estimated shapes for Case
4 is shown in Fig.10, and the distribution of average error isshown in Fig.11. In Fig.11, dark regions indicate less error,
and the brightness of each point is proportional to its average error. From this figure, we verified that the errors in thefemoral head and lesser trochanter are reduced.

Coplanar marker

Figure 6. X-ray photography of coplanarmarkerFigure 7. Two fluoroscopic femoral images of
hip prosthesis patients

1.6
case1

Average error [mm]1.4

are measured.
step 2. A fluoroscopic image of the non-coplanar marker iscaptured, as shown in Fig.5, and the internal parameters of the fluoroscopic imaging apparatus are estimated by Tsai’s method.

case2

1.2
1.0
case3

0.8

case4

0.6
0.4
0.2
0

step 3. The coplanar marker is placed under the patient’s
hip and two fluoroscopic images of the femur are captured from two directions, as shown in Figs.6,7.

0

1

2

3

1.2

step 4. The external parameters of two fluoroscopic images
are estimated using the internal parameters obtained in

4 5 6 7 8 9
Number of parameters

case1

10 11 12case2

S.D. [mm]

1.0
0.8
case3

0.6

case4

0.4
0.2
0

0

1

2

3

4 5 6 7 8
Number of parameters

9

10 11 12Figure 9. Average error and standard deviation for the number of estimated shape parameters for the femurs of four patients

Figure 8. Extracted contours of the femur influoroscopic images

62

ACCV'07 Workshop on Multi-dimensional and Multi-view Image Processing, Tokyo, Nov., 2007

0

Initial shape Estimated shape Actual shape
Figure 10. Actual and estimated femoral
model with and without shape parameter estimation for Case 4

5
Number of parameters

MM-O-0910

Figure 11. Error distribution indicated bybrightness (higher brightness indicateslarger error

6 Conclusions[3] L. Liu and I. Stamos. Automatic 3d to 2d registration for thephotorealistic rendering of urban scenes. In IEEE International Conference on Robotics & Automation, 2005.

We proposed a method by which to estimate the 3D
shape of the in vivo femur from only two fluoroscopic images using a parametric femoral model. Although the precise 3D shape of the femur is usually measured using a CT
scanner or an MRI system, the proposed method enablesa precise 3D shape to be estimated using only two fluoroscopic images taken by an inexpensive fluoroscopic inspection apparatus. Thus, the cost of the inspection system canbe dramatically reduced and the 3D image-based medical
diagnosis becomes available even in small clinics.
In vivo experiments revealed the average error between
the estimated shape and the actual shape to be 0.8 mm to 1.1mm, and it was verified that the 3D shape can be estimatedusing two 2D fluoroscopic images taken from different view
points with the same accuracy as in the case of 3D shapes
being compared directly.

[4] R. Kurazume, K. Noshino, Z. Zhang, and K. Ikeuchi. Simultaneous 2d images and 3d geometric model registration
for texture mapping utilizing reflectance attribute. In Proc.
of Fifth Asian Conference on Computer Vision (ACCV), pp.
99—106, 2002.
[5] M. D. Elstrom and P. W. Smith. Stereo-based registration ofmulti-sensor imagery for enhanced visualization of remoteenvironments. In Proc. of the 1999 IEEE International Conference on Robotics and Automation, pp. 1948–1953, 1999.
[6] K. Umeda, G. Godin, and M. Rioux. Registration of range
and color images using gradient constraints and range intensity images. In Proc. of 17th International Conference on
Pattern Recognition, pp. 12–15, 2004.
[7] Y. Iwashita, R. Kurazume, K. Hara, and T. Hasegawa, Fast
Alignment of 3D Geometrical Models and 2D Color Images
using 2D Distance Maps, in Proc. The 5th International
Conference on 3-D Digital Imaging and Modeling (3DIM),
pp.164-171, 2005.

References[1] I. Stamos and P. K. Allen. Integration of range and image
sensing for photorealistic 3d modeling. In Proc. of the 2000
IEEE International Conference on Robotics and Automation, pp. 1435–1440, 2000.

[8] Q. Delamarre and O. Faugeras. 3d articulated models andmulti-view tracking with silhouettes. In Proc. of the International Conference on Computer Vision, Vol. 2, pp. 716–721,
1999.

[2] I. Stamos and P. K. Allen. Automatic registration of 2-dwith 3-d imagery in urban environments. In Proc. of theInternational Conference on Computer Vision, pp. 731–737,
2001.

[9] K. Matsushita and T. Kaneko. Efficient and handy texture
mapping on 3d surfaces. In Comput. Graphics Forum 18,
pp. 349–358, 1999.

63

ACCV'07 Workshop on Multi-dimensional and Multi-view Image Processing, Tokyo, Nov., 2007[10] P. J. Neugebauer and K. Klein. Texturing 3d models of
real world objects from multiple unregistered photographic
views. In Computer Graphics Forum 18, pp. 245–256, 1999.

MM-O-09

bony anatomy in image guided orthopaedic surgery. In
Lecture Notes in Computer Science, 3217 (Proc. 7th International Conference on Medical Image Computing andComputer Assisted Intervention, Part II (MICCAI 2004), StMalo, France), pp. 397–404, 2004.

[11] G. Penny, J. Weese, J. Little, P. Desmedt, D. Hill, and D.
Hawkes, A Comparison of Similarity Measures for Use in
2-D/3-D Medical Image Registration, IEEE Trans. Medical
Imaging, Vol.17, No.4, pp.586-595, 1998.

[22] D. Terzopoulos and D. Metaxas, Dynamic 3D Models with Local and Global Deformations: Deformable Superquadrics, IEEE Trans. on Pattern Analysis and MachineIntelligence, Vol 13, No.7, pp.703-714, 1991.

[12] L. Zollei, E. Grimson, A. Norbash, and W. Wells, 2D-3D
rigid registration of x-ray fluoroscopy and CT images using
mutual information and sparsely sampled histogram estimators, In Proc. of Computer Vision and Pattern Recognition,
pp.696–703, 2001.

[23] K. Nakamura, R. Kurazume, T. Okada, Y. Sato, N. Sugano,and T. Hasegawa, 3D reconstruction of a femoral shape using a parametric model and two 2D radiographs In Proc. of
Meeting on Image Recognition and Understanding, pp.7883, 2006.

[13] C. V. Stewart, C. L. Tsai, and A. Perera. A view-based approach to registration: Theory and application to vascular
image regisrtaion. In International Conference on Information Processing in Medical Imaging (IPMI), pp. 475–486,
2003.

[24] G. Zheng, M. Ballester, M. Styner, and L. Nolte, Reconstruction of Patient-specific 3D Bone Surface from 2D Calibrated Fluoroscopic Images and Point Distribution Model In
Proc. of Medical Image Computing and Computer-Assisted
Intervention(MICCAI’06), pp.25-32, 2006.

[14] A. Guimond, A. Roche,M. Ayache, and J. Meunier. Threedimensional multimodal brain warping using the demons algorithm and adaptive intensity corrections. IEEE Transaction on Medical Imaging, Vol. 20, No. 1, pp. 58–69, 2001.

[25] Keita Yokota and Toshiyuki Okada and Masahiko Nakamoto
and Yoshinobu Sato and Masatoshi Hori and J. Masumotoand Hironobu Nakamura and Shinichi Tamura, ”Construction of conditional statistical atlases of the liver based on
spatial normalization using surrounding structures”, International Journal of Computer Assisted Radiology andSurgery, pp. 39-40, 2006,[15] F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and
P. Suetens. Multimodality image registration by maximization of mutual infomation. IEEE Transaction on Medical
Imaging, Vol. 16, No. 2, pp. 187–198, 1997.
[16] C. V. Stewart, C. L. Tsai, and A. Perera. Rigid and affine
registration of smooth surfaces using differential properties.
Proc. of Third European Conference on Computer Vision(ECCV’94), pp. 397–406, 1994.

[26] Toshiyuki Okada and Ryuji Shimada and Yoshinobu Satoand Masatoshi Hori and Keita Yokota and MasahikoNakamoto and Yen-Wei Chen and Hironobu Nakamura andShinichi Tamura, ”Automated segmentation of the liverfrom 3D CT images using probabilistic atlas and multi-levelstatistical shape model”, Proceedings of the 10th International Conference on Medical Image Computing and Computer Assisted Intervention (Proc. MICCAI 2007), in press,
2007[17] C. R. Meyer, J. L. Boes, B. Kim, P. H. Bland, K. R. Zasadny, P. V. Kison, K. Korak, K. A. Fery, and R. L. Wahl.
Demonstration of accuracy and clinical versatility of mutual
information for automatic multimodality image fusion using
affine and thin plate spline warped geometric deformations.
Medical Image Analysis, Vol. 1, No. 3, pp.195–206, 1997.

[27] TF Cootes TF, CJ Cooper, CJ Taylor, and J Graham. Active
shape models — their training and application,. Computer
Vision and Image Understanding, Vol. 61, No. 1, pp. 38–59,
1995.

[18] A. Gueziec, X. Pennec, and N. Ayache. Medical image registration using geometric hashing. IEEE Computational Science and Engineering, special issue on Geometric Hashing,Vol. 4, No. 4, pp. 29–41, 1997.

[28] T. Okada, M. Nakamoto, Y. Sato, N. Sugano, H. Yoshikawa,S. Tamura, T. Asaka, Y.-W. Chen, Effects of surface correspondence methods in statistical shape modelling of theproximal femur on approximation accuracy In Proc. of the20th International Congress and Exhibition, Computer Assisted Radiology and Surgery CARS 2006, 016, 2006.

[19] P. R. Andresen and M. Nielsen. Non-rigid registration bygeometry constrained diffusion. Medical Image Computing and Computer-Assisted Intervention(MICCAI’ 99), pp.
533–543, 1999.
[20] T. Masuda, Y. Hirota, K. Ikeuchi, and K. Nishino. Simultaneous determination of registration and deformation parameters among 3d range images. In Fifth International Conference on 3-D Digital Imaging and Modeling, pp. 369–376,
2005.

[29] Haili Chui and Anand Rangarajan, ”A new point matchingalgorithm for non-rigid registration”, Computer Vision andImage Understanding, Vol 89, No. 2-3, pp 114-141, 2003.

[21] CSK Chan, DC Barratt, PJ Edwards, GP Penney, M Slomczykowski, TJ Charter, and DJ Hawkes. Cadaver validation of the use of ultrasound for 3d model instantiation of[30] M. Greenspan, M. Yurick, Approximate k-d tree search for
efficient ICP, in Proc. of 3-D Digital Imaging and Modeling(3DIM) 2003 pp.442–448, 2003.

64

ACCV'07 Workshop on Multi-dimensional and Multi-view Image Processing, Tokyo, Nov., 2007[31] J. Sethian. Level Set Methods and Fast Marching Methods,
second edition. Cambridge University Press, UK, 1999.
[32] J. Sethian. A fast marching level set method for monotonically advancing fronts. In Proc. of the National Academy of
Science, Vol. 93, pp. 1591–1595, 1996.
[33] S. W. Shih and Y. T. Wu, Fast Euclidean distance transformation in two scans using a 3 3 neighborhood, Computer
Vision and Image Understanding, Vol.93, No.2, pp.195–
205, 2004.
[34] R. Y. Tsai, An Efficient and Accurate Camera Calibration
Technique for 3D Machine Vision, in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, pp.
364-374, 1986.

65

MM-O-09