5th MICCAI Workshop on
Computational Methods
and Clinical Applicationsin Musculoskeletal Imaging

Proceedings

Editors
Ben Glocker
Jianhua Yao
Tomaž Vrtovec
Alejandro F. Frangi
Guoyan Zheng

MICCAI-MSKI2017 • Sep 10, 2017 • Quebec City, Canada
https://mski2017.wordpress.com

5th MICCAI Workshop on
Computational Methods
and Clinical Applicationsin Musculoskeletal Imaging

Proceedings

Editors
Ben Glocker
Jianhua Yao
Tomaž Vrtovec
Alejandro F. Frangi
Guoyan Zheng

MICCAI-MSKI2017 • Sep 10, 2017 • Quebec City, Canada
https://mski2017.wordpress.com

ii
5th MICCAI Workshop on Computational Methods and ClinicalApplications in Musculoskeletal Imaging – MICCAI-MSKI2017
c 2017

The papers included in this proceedings book were part of the technical workshop cited on the
cover. The papers were selected and reviewed by the editors and the scientific review committee.
Some workshop presentations may not be available for publication. The papers published in thisproceedings book reflect the work and thoughts of the corresponding authors and are publishedherein as submitted with minor editorial revisions. Neither the authors, the editors, nor theworkshop organizers can accept any legal responsibility for any errors or omissions that may bemade. Please use the following format to cite materials from this proceedings book:<Authors>, <Paper Title>, In: B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proceedings of the 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. <Page Numbers>, 2017.
All articles published herein are protected by copyright, which covers the exclusive rights to
reproduce and distribute the articles, as well all translation rights. No materials published herein
may be reproduced or stored electronically without first obtaining written permission from theauthors.

Table of Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

v

Workshop Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

vii

Localization of Bone Surfaces from Ultrasound Data Using
Local Phase Information and Signal Transmission Maps
Ilker Hacihaliloglu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1

Shape-Aware Deep Convolutional Neural Network for
Vertebrae Segmentation
S. M. Masudur Rahman Al Arif, Karen Knapp, Greg Slabaugh . . . . . . . . . .

12

Automated Characterization of Body Composition and
Frailty with Clinically Acquired CT
Peijun Hu, Yuankai Huo, Dexing Kong, J. Jeffrey Carr,
Richard G. Abramson, Katherine G. Hartley, Bennett A. Landman . . . . . . .

25

OpenRib Clinical Application
Catalina Tobon-Gomez, Tyler Stroud, John Cameron, Dave Elcock,
Andrew Murray, Daniel Wyeth, Chris Conway, Steven Reynolds,
Pedro Augusto Gondim Teixeira, Alain Blum, Costas Plakas . . . . . . . . . . . .

35

3D Cobb Angle Measurements from Scoliotic Mesh Models
with Varying Face-Vertex Density
Uroš Petković, Robert Korez, Stefan Parent, Samuel Kadoury,
Tomaž Vrtovec . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

46

Automatic Localization of the Lumbar Vertebral
Landmarks in CT Images with Context Features
Dimitrios Damopoulos, Ben Glocker, Guoyan Zheng . . . . . . . . . . . . . . . . . . .

56

Joint Multimodal Segmentation of Clinical CT and MR
from Hip Arthroplasty Patients
Marta B.M. Ranzini, Michael Ebner, M. Jorge Cardoso,
Anastasia Fotiadou, Tom Vercauteren, Johann Henckel,
Alister Hart, Sébastien Ourselin, Marc Modat . . . . . . . . . . . . . . . . . . . . . . . .

69

Reconstruction of 3D Muscle Fiber Structure Using High
Resolution Cryosectioned Volume
Yoshito Otake, Kohei Miyamoto, Axel Ollivier, Futoshi Yokota,
Norio Fukuda, Lauren J. O’Donnell, Carl-Fredrik Westin, Masaki Takao,
Nobuhiko Sugano, Beom Sun Chung, Jin Seo Park, Yoshinobu Sato . . . . . . .

iii

82

iv

Segmentation of Pathological Spines in CT Images Using a
Two-Way CNN and a Collision-Based Model
Robert Korez, Boštjan Likar, Franjo Pernuš, Tomaž Vrtovec . . . . . . . . . . . . .

91

Attention-Driven Deep Learning for Pathological Spine
Segmentation
Anjany Sekuboyina, Jan Kukačka, Jan S. Kirschke, Bjoern H. Menze,
Alexander Valentinitsch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

Automatic Full Femur Segmentation from Computed
Tomography Datasets Using an Atlas-Based Approach
Bryce A. Besler, Andrew S. Michalski, Nils D. Forkert, Steven K. Boyd . . . . 116

Classification of Osteoporotic Vertebral Fractures Using
Shape and Appearance Modelling
Paul A. Bromiley, Eleni P. Kariki, Judith E. Adams, Timothy F. Cootes . . . 128

DSMS-FCN: A Deeply Supervised Multi-Scale Fully
Convolutional Network for Automatic Segmentation ofIntervertebral Disc in 3D MR Images
Guodong Zeng, Guoyan Zheng . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

Preface

The musculoskeletal system consists of the skeleton, muscles, cartilage,
ligaments, joints, and other connective tissue that supports and binds tissues
and organs together, and provides form, support, protection, stability andmovement to the body. Specific subsystems like the spine provide both a vital
central axis for the musculoskeletal system and a flexible protective shellsurrounding the most important neural pathway in the body, the spinal cord.
The musculoskeletal system is involved in various disease processes associatedwith aging and degeneration of bones and joints, like osteoporosis andosteoarthritis. Osteoporosis is a condition where bones become brittle andfragile from loss of tissue due to hormonal changes, or deficiency in calcium orvitamin D. Osteoporosis leads to an increased bone fracture risk, which isfurther exacerbated in the elderly due to the lost muscular strength and frailty.
Osteoarthritis, or degenerative arthritis, is caused by inflammation and theeventual loss of cartilage in the joints, which wears down with time. These arejust a few relevant examples of the conditions associated to the musculoskeletal
system, not to mention therapeutic procedures in orthopedic surgery, and therelated medical implant and devices where imaging plays a crucial role in theplanning, guidance and monitoring phases. Giving the increasing volume ofmultimodal imaging examinations associated to musculoskeletal diseases andthe complexity of their assessment, there is a pressing need for advanced
computational methods that support the diagnosis, therapy planning andinterventional guidance, with a number of related challenges in bothmethodology and clinical applications.
The goal of the workshop series on “Computational Methods and ClinicalApplications in Musculoskeletal Imaging” is to bring together clinicians,
researchers and industrial vendors in the field of musculoskeletal imaging, forreviewing the state-of-the-art techniques, sharing the novel and emerging
analysis and visualization techniques, and discussing the clinical challenges andopen problems in this field. Topics of interest include all major aspects ofmusculoskeletal imaging, for example: clinical applications of musculoskeletal
computational imaging; computer-aided detection and diagnosis of conditionsof the bones, muscles and joints; image-guided musculoskeletal surgery andinterventions; image-based assessment and monitoring of surgical andpharmacological treatment; segmentation, registration, detection, localizationand visualization of the musculoskeletal anatomy; statistical and geometricalmodeling of the musculoskeletal shape and appearance; image-based
microstructural characterization of musculoskeletal tissue; novel techniques formusculoskeletal imaging.

v

vi

MSKI2017, the 5th Workshop on Computational Methods and ClinicalApplications in Musculoskeletal Imaging, is a satellite event of MICCAI 2017,the 20th International Conference on Medical Image Computing andComputer-Assisted Intervention, held from September 10 to 14, 2017 in Quebec
City, QC, Canada. The workshop is a continuation of the former Workshop on
Computational Methods and Clinical Applications for Spine Imaging, CSI,
which was after four successful consecutive editions at MICCAI 2013, 2014,2015 and 2016 opened up to a wider community by broadening the scope fromspine to musculoskeletal imaging, therefore recognizing the relevant progress
made in spine imaging and the emerging needs in imaging of other bones,
joints and muscles of the musculoskeletal system. We received several highquality submissions addressing many of the above mentioned issues. All papers
underwent a double-blind review, with each paper being reviewed by threemembers of the scientific review committee. We finally accepted 13 papers,
which appear in these proceedings in the chronological order of submission.
We would like to thank everyone who contributed to this workshop: theauthors for their contributions, the members of the program and scientificreview committee for their review work, promotion of the workshop andgeneral support, and the MICCAI Society for the opportunity to exchange
research ideas and build the community within the premier conference in thefield of medical imaging.

Ben Glocker
Jianhua Yao
Tomaž Vrtovec
Alejandro F. Frangi
Guoyan Zheng

Workshop Organization

Workshop Organizing Committee
Ben Glocker
Jianhua Yao
Tomaž Vrtovec
Alejandro F. Frangi
Guoyan Zheng

Imperial College London, UK
National Institutes of Health, USA
University of Ljubljana, Slovenia
The University of Sheffield, UK
University of Bern, Switzerland

Program & Review Committee
Ulas Bagci
University of Central Florida, USA
Paul A. Bromiley
The University of Manchester, UK
Weidong Cai
The University of Sydney, Australia
Yunliang Cai
Amazon & Worcester Polytechnic Institute, USA
Ananda S. Chowdhury
Jadavpur University, India
Daniel Forsberg
Sectra & Linköping University, Sweden
Huiguang He
The Chinese Academy of Sciences, China
Bulat Ibragimov
Stanford University School of Medicine, USA
Jianfei Liu
National Institutes of Health, USA
Cristian Lorenz
Philips Research, Germany
Simon Pezold
University of Basel, Switzerland
Greg Slabaugh
City, University of London, UK
Darko Štern
Ludwig Boltzmann Inst. for Clinical Forensic Imaging, Austria
Sovira Tan
National Institutes of Health, USA
Tamas Ungi
Queen’s University, Canada
Yiqiang Zhan
Siemens Healthcare, USA

vii

viii

Proceedings Editors
Ben Glocker

Imperial College London, UK
b.glocker@imperial.ac.uk

Jianhua Yao

National Institutes of Health, USAjyao@cc.nih.gov

Tomaž Vrtovec

University of Ljubljana, Slovenia
tomaz.vrtovec@fe.uni-lj.si

Alejandro F. Frangi

The University of Sheffield, UK
a.frangi@sheffield.ac.uk

Guoyan Zheng

University of Bern, Switzerland
guoyan.zheng@istb.unibe.chLocalization of Bone Surfaces from UltrasoundData Using Local Phase Information and
Signal Transmission Maps
Ilker Hacihaliloglu1,2(

)

1

2

Department of Biomedical Engineering, Rutgers University, USA
Department of Radiology, Rutgers Robert Wood Johnson Medical School, USA
ilker.hac@soe.rutgers.edu

Abstract. Low signal-to-noise ratio, imaging artifacts and bone boundaries appearing several millimeters in thickness have hampered the success of ultrasound (US) guided computer assisted orthopedic surgery
procedures. In this paper we propose a robust and accurate bone localization method. The proposed approach is based on the enhancement ofbone surfaces using the combination of three different local image phasefeatures. The extracted local phase image features are used as an input
to an L1 norm-based contextual regularization method for the enhancement of bone shadow regions. During the final stage the enhanced bonefeatures and shadow region information is combined into a dynamic programming solution for the localization of the bone surface data. Qualitative and quantitative validation was performed on 150 in vivo US scansobtained from seven subjects by scanning femur, knee, distal radius andvertebrae bones. Validation against expert segmentation achieved a meansurface localization error of 0.26 mm a 67% improvement over state of
the art.
Keywords: Ultrasound, bone segmentation, orthopedics, local phase,signal transmission.

1

Introduction

In order to decrease the total amount of radiation exposure, caused by intraoperative fluoroscopy, and provide real-time three-dimensional (3D) guidance,
ultrasound (US) has been incorporated as an alternative imaging modality into
various computer assisted orthopedic surgery (CAOS) procedure [1]. Nevertheless, due to the continuing challenges faced during the extraction of relevant
anatomical information from US data, most of the proposed US-based CAOS
guidance systems have not succeeded in clinical settings. Ultrasound images
typically contain significant speckle and imaging artifacts, which do not correspond to any specific anatomy, complicating image interpretation and automatic
processing. Furthermore, orientation of the US transducer with respect to theimaged anatomy and the elevational beam width strongly influence bone surface
response profile and corresponding bone boundaries appear several millimeters
B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proc. 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. 1−11, 2017.

2

Hacihaliloglu

Fig. 1. Bone imaging appearance in ultrasound (US). (a),(b) High intensity soft tissue
interfaces above the bone surface with similar intensity profile as the bone surfaces andreverberation artifacts inside the shadow region. (c) Separate low intensity spine bone
surfaces. (d) Low intensity bone surface obtained due to non-optimal orientation of the
US transducer.

in thickness. In order to overcome some of these challenges bone segmentationor enhancement methods have been proposed by various groups.
The previously proposed image-based segmentation or enhancement methods
can be classified into three groups: (i) methods using image intensity/gradient
information [2, 3], (ii) methods based on local phase image features [4, 5], and(iii) hybrid approaches which combine the strengths of intensity and phase-based
methods [6–8]. Intensity-based approaches are not robust to low contrast bone
responses and high intensity soft tissue interfaces (Fig. 1). One of the distinct
features in bone US data is the shadow region. A large transition in acoustic
impedance between the tissue and the bone causes most of the acoustic signal
to be reflected back creating a low intensity region extending from the boneboundary to the bottom of the image. Incorporating this information into theirframework improved the accuracy and robustness of the proposed intensityand phase-based methods [3, 6–8]. In [6], the percentage of overlapping surfaces
between the manual segmented and automatic method was 62.5%. The hybrid
approach proposed in [7] integrated machine learning into their framework. The
method was validated on 35 US scans obtained from a single subject achieving
an accuracy score of 86% for 1 mm tolerance with 0.59 mm localization error forthis tolerance. The computation time for the proposed method was 2 min. In [8],computed tomography (CT) derived bone surfaces were registered to US derived
bone surfaces. The reported average surface fit error for the in vivo pelvis data
was close to 0.5 mm.
Although previously reported results provided promising outcomes, acquisition of high quality US data in clinical settings continues to be and ongoingchallenge in US-based CAOS procedures effecting the accuracy and robustness ofthe segmentation methods. In this work, we propose a bone localization method
which is accurate and robust to different US imaging artifacts. Local phase-based
image features are utilized to enhance the bone surface response profile and suppress the soft tissue interfaces and imaging artifacts. The enhanced images areused as an input to an L1 norm-based contextual regularization method whichLocalization of Bone Surfaces from Ultrasound Data

3

Fig. 2. Flowchart of the proposed bone localization method.

emphasizes uncertainty in the shadow regions. The enhanced bone response andshadow region images are incorporated into a dynamic programming solution for
localizing the bone surfaces. Qualitative and quantitative validation results onscans collected from seven volunteers are presented. The proposed method is also
compared against previously developed intensity-based [3] and phase-based [9]methods.

2

Methods

The flowchart of the proposed method is provided in Fig. 2 and is based on
our previous experience where local phase image features are used for boneenhancement and/or segmentation.
2.1

Enhancement of Bone Surface Response

Bone surface response profile in US is highly affected by the orientation of thebeam with respect to the imaged bone boundary and the 3D anatomy of theimaged surface. If the US beam is perfectly aligned and the attenuation fromsoft tissue interface is low the bone response profile appears as a dominant ridgeedge along the scanline direction. However, while imaging complex shape bone
surfaces, such as spine, or if the attenuation from soft tissue interface is large thebone response profile can be dominated by different edge profiles. The first stepin our framework involves the enhancement of the low intensity bone surfaces
by constructing a local phase enhancement metric, similar to [9], as:P Prs b[erm (x, y) − ors (x, y)] − Tr cp
U SE(x, y) = P.
(1)P
e2rs (x, y) − o2rs (x, y) + rsHere o(x, y) and e(x, y) represent the even and odd symmetric filter response
and are obtained by filtering the B-mode US image, U S(x, y), in the frequency
domain using Log-Gabor filter [10]. Since the first step in the proposed framework
is to provide an initial general ultrasound enhancement, in this new metric we are
not using the absolute response values of the even and odd filter responses which
was done previously for enhancement of bone interfaces [9]. Filter orientations

4

Hacihaliloglu

Fig. 3. Local phase image bone features. Top row shows the extracted local
phase image features where the enhanced U SE(x, y) image was used to extract
LP T (x, y), LP E(x, y), LwP A(x, y), and LP (x, y) image features. Bottom row showsthe extracted local phase image features where the B-mode U S(x, y) image was used
to extract LP T (x, y), LP E(x, y), LwP A(x, y), and LP (x, y) image features. Red arrows
point to bone surfaces and soft tissue interfaces where the improvement was achieved.
Distance map is shown on the far right.

and scale are represented with r and s respectively.  is a small constant included
to avoid division by zero. Tr is a noise dependent threshold calculated as a
specified number of standard deviations above the mean of the local energy
distribution because of noise [11]. The standard deviation and mean of the local
energy is calculated for each orientation separately using the response of thesmallest scale filter [10].
Fig. 3 shows that U SE(x, y) results in the enhancement of low intensity bone
surfaces and soft tissue interfaces. Hacihaliloglu et al. [12] recently proposed atensor-based feature descriptor, called local phase tensor (LP T (x, y)), for theenhancement of bone features while suppressing high intensity soft tissue interfaces. The second step in the bone enhancement framework is to calculate
the LP T (x, y) image. LP T (x, y) is obtained using even and odd filter responses
which are defined as:T

Teven = [H(U SDB (x, y))] [H(U SDB (x, y))] ,TTodd = −0.5 × ([∇U SDB (x, y)] ∇∇2 U SDB (x, y) +T∇∇2 U SDB (x, y) [∇U SDB (x, y)] ).

(2)

Here Teven represents symmetric features and Todd represents the asymmetricfeatures. H, ∇ and ∇2 denote the Hessian, Gradient and Laplacian operations,
respectively. U SDB (x,y) is obtained by masking the band-pass filtered U SE(x, y)
image with a distance map which improves the enhancement of bone surfaces
located deeper in the image while masking out of soft tissue interfaces close to thetransducer. Band-pass filtering was performed using a Log-Gaborfilter [12]. Thep
2
2
final LP T (x, y) image is obtained using LP T (x, y) = Teven+ Todd× cos(ϕ).
The instantaneous phase obtained from the symmetric (Teven ) and asymmetric(Todd ) features responses is represented with ϕ [12]. Investigating the obtained
LP T (x, y) image (Fig. 1) we can see that the descriptor enhances soft tissue
interfaces close the to bone surface as well. In order to provide an enhancementwith less soft tissue interfaces and more compact bone representation, local phaseLocalization of Bone Surfaces from Ultrasound Data

5energy (LP E(x, y)) and local weighted mean phase angle (LwP A(x, y)) image
features are extracted using monogenic signal theory where the monogenic signal
image (U SM (x,y)) is formed by combining the bandpass filtered LP T (x, y) image(LP TB (x, y)) with the Riesz filtered components as:
U SM (x, y) = LP TB (x, y), LP TB × h1 (x, y), LP TB (x, y) × h2 (x, y) .
(3)Here h1 and h2 represent the vector valued odd filter (Riesz filter) [13]. For bandpass filtering α-scale space derivative quadrature filters (ASSD) are used which
are shown to produce produce improved edge detection results on simulated US
images [14]. The LP E(x, y) image is obtained by averaging the phase sum of the
response vectors over many scales using:qX
2 (x, y) + U S 2 (x, y) .
(4)
LP E(x, y) =U SM 1 (x, y) − U SM2
M3
sc

In the above equation sc represents the number of scales. LP E(x, y) encodes theunderlying shape of the bone boundary by accumulating the local energy of theimage along several filter responses. LwP A(x, y) is calculated using:
!
P
U
S
(x,
y)
M
1
sc
LwP A(x, y) = arctan pP
(5)
P
2
2
sc U SM 2 (x, y)
sc U SM 1 (x, y) +during the calculation of the LwP A(x, y) feature map noise compensation is not
performed and the LwP A(x, y) image preserves all the structural details of the
US image such as the soft tissue interfaces and bone surface. The final improved
local phase bone image (LP (x, y)) is obtained using: LP (x, y) = LP T (x, y) ×
LP E(x, y) × LwP A(x, y). Fig. 1 shows the obtained local phase feature images(LP T (x, y), LP E(x, y), LwP A(x, y)). One common property of the extracted
local phase image feature images is that the enhanced bone surfaces are well
localized in all of the three images while soft tissue interfaces are not. Therefore,
the combination of these three phase feature images results in the suppression ofsoft tissue interfaces while keeping the bone surfaces more compact and localized.
In Fig. 1 (bottom row) we also show the bone enhancement results obtained if
we used U S(x, y) image as an input to the tensor-based phase descriptor. Red
arrows point to the enhanced soft tissue artifacts and missing bone boundaries
since. These are the locations in the B-mode US image (U S(x, y)) where the bone
response is weaker compared to the soft tissue interfaces above the bone surface.
The obtained LP (x, y) image is used in the next section for the enhancement ofbone shadow region.
2.2

Enhancement of Shadow Region

Automatic identification of shadow regions is important since it can be used as
an additional feature to improve the robustness and accuracy of the segmentation or registration methods. The bone shadow enhancement is based on the

6Hacihaliloglu

Fig. 4. (a) Enhanced bone shadow image BSE(x, y). (b) Bone probability image obtained by masking LP (x, y) with BSE(x, y). (c) Bone localization presented as curveBL(s). The curve BL(s) is overlaid on the actual bone surface for better representation.
(d) Localized bone surface is overlaid on the B-mode ultrasound image of in vivo knee.

modification of previously proposed US confidence map (CM) approach [15].
However, instead of using the US image intensity information we use LP (x, y)
image features. We achieve this by modeling the interaction of the US signal
within the tissue using scattering and attenuation information. The model, denoted as US signal transmission map (U SA (x, y)), maximizes the visibility ofhigh intensity features inside a local region and satisfies the constraint that themean intensity of the local region is less than the echogenicity of the tissue confining the bone. The scattering and attenuation effects in the tissue are combined
as: CMLP (x, y) = U SA (x, y)BSE(x, y) + (1 − U SA (x, y))ρ. Here CMLP (x, y)
represents CM image obtained from LP (x, y) using [15], ρ is a constant value
representative of echogenicity in the tissue surrounding the bone, and BSE(x, y)is the enhanced bone shadow image which we are trying to calculate. In orderto calculate BSE(x, y), U SA (x, y) is estimated first by minimizing the following
objective function [16]:X
λ2
U SA (x, y) − CMLP (x, y) 2 +Wj ◦ (Dj ∗ U SA (x, y)) 1 .
2
j∈χ

(6)

Here ◦ represents element-wise multiplication, χ is an index set, and ∗ is convolution operator. Dj is calculated using a bank of high order differential filters [17].
The filter bank results in the enhancement of bone features in the local region
while attenuating the image noise. Wj is a weighting matrix calculated using:
Wj (x, y) = exp(− | Dj (x, y) ∗ CMLP (x, y) |2 ). In (6) the first part measuresthe dependence of U SA (x, y) on CMLP (x, y)) and the second part models thecontextual constraints of U SA (x, y). These two terms are balanced using a regularization parameter λ [16]. After estimating U SA (x, y), BSE(x, y) image is
obtained using: BSE(x, y) = [(CMLP (x, y) − ρ)/[max(U SA (x, y), )]δ ] + ρ. δ
is related to tissue attenuation coefficient (η) and  is a small constant used
to avoid division by zero [16]. Fig. 4 shows the enhanced bone shadow image
BSE(x, y) where the soft tissue interface above the bone surface is represented
with uniform intensity and the shadow region is represented with low intensity
values corresponding to a low probability value that the signal reaching backto the transducer imaging array (high intensity denoted with dark red and low
intensity with blue color coding). Investigating the BSE(x, y) image we can see

Localization of Bone Surfaces from Ultrasound Data

7

that the transition from soft tissue interface to bone shadow region is represented with a sharp intensity change clearly differentiating the two interfaces.
The enhanced bone shadow region image (BSE(x, y)) and local phase bone image (LP (x, y)) are used during the bone surface localization which is explainedin the next section.
2.3

Bone Surface Localization

The localization of the bone feature within a column s, denoted as BL(s), is
achieved by minimizing a cost function composed of two energy functions denoted as internal energy (Eint (x, y)) and external energy (Eext (x, y)). Eint (x, y)
is obtained by masking the LP (x, y) image with the BSE(x, y) image which
provides a bone probability map (Fig. 4(b)). The external energy (Eext (x, y)) isconstructed by dividing the US image into three regions denotes as bone region,boneless region and the jump region (the region between the first two regions)(Fig. 4(c)). Eext (x, y) is constructed using these three regions as [3]:d2 BL 2dBL 2
Bone region, ν|| ds || + ξ|| ds2 || + ς;
(7)
Eext (i, j) = JumpCost;
Jump region,νD12 + ξD22 ;Boneless region.
Here ν and ξ are the weights of the smoothness (the first derivative of BL(s))
and the curvature (the second derivative of BL(s)), and ς is small negative scalar
ensuring larger connected bone regions to stay connected. Bone connectivity isfurther maintained with the JumpCost constant which penalizes frequent jumps
between bone and boneless regions. As there is no bone information present
in the boneless region, first and second order derivatives are assigned constantvalues D1 and D2 . Dynamic programming optimization is used to solve:
BLmin(i, j) = Eint (i, j) + min BLmin(k, j − 1) + Eext (k, j) .
(8)kBLmin(i, j) represents the minimum cost of moving from first column to the
pixel in i th row and j th column. Row index is represented with k. The index of
the pixel k,j with its minima is stored in Indexmin(i, j) = arg mink [BLmin(k, j−1) + Eext (k, j)]. Dynamic programming provides a fast optimization of the costfunction. The final optimized bone localization if obtained by tracing back fromthe last column of the US image using:
NR + 1s = N C;
BLopt (s) =(9)Indexmin[s + 1, BLopt (s + 1)];
s = 1, . . . , (N C − 1).
BLopt is the optimized segmentation path where the energy cost function isminimized. The number of rows and columns are indicated with NR and NC of
the B-mode US image. NR and NC also indicate the last row and last column
in the US image. The final localized bone surfaces is shown in Fig. 4.

8Hacihaliloglu

2.4

Data Acquisition and Experiments

After obtaining the institutional review board (IRB) approval a total of 150
different US images, from seven healthy subjects, were collected using SonixTouch US machine (Analogic Corporation, Peabody, MA, USA). Dependingon the anatomical region of interest two different transducers were used (C52 curvilinear, L14-5 linear transducer). Depth settings and image resolutions
varied between 3 − 8 cm and 0.12 − 0.19 mm respectively. All the proposed image enhancement and localization methods were implemented using MATLAB
2014a software package and run on a 2.3 GHz Intel(R) CoreTM i5 CPU, 16 GB
RAM windows PC. The localized bone surfaces were compared to manual localization results obtained from an expert user. The quality of the localization
was evaluated by computing average Euclidean distance (AED) between thetwo surfaces. We also compare the localization results against the methods proposed in [3, 9]. For bone shadow enhancement, λ = 2 and ρ, the constant related
to tissue echogenicity, was chosen as 90% of the maximum intensity value of
CMLP (x, y). LP T (x, y) images were calculated using the filter parameter values defined in [12]. The CM (x, y) and CMLP (x, y) images were obtained using the constant values as: η = 2, β = 90, γ = 0.03. For bone surface localization
the constant values were chosen as: ν = 50, ξ = 100, JumpCost = 0.8, ς = 0.15,
D1 = D2 = 1. These values were determined empirically and kept constant during
qualitative and quantitative analysis.

3

Results

Investigating the qualitative results we can see that the surfaces localized withthe proposed method have a good alignment with the expert manual localization(Fig. 5). The combination of enhanced local phase bone features and shadow
region information provides a robust estimate even if (i) the shadow region had
intensity variations (Fig. 5; femur and spine), (ii) disconnected bone surfaces(Fig. 5; spine), (iii) low intensity bone boundary (Fig. 5; radius, spine and femur),and (iv) high intensity soft tissue interfaces (Fig. 5; femur, spine and radius).
The overall AED error for the proposed method was 0.26 mm (SD: 0.22). The
overall AED error for [9] and [3] were 0.78 mm (SD: 0.68) and 4.5 mm (SD:4.39) respectively. The maximum AED was 1.36 mm for the proposed method,
and 19.08 mm for [3], and 4.2 mm for [9] (Tab. 1). Tab. 1 also shows the 95%
confidence level calculated for the localization results obtained for all the threemethods compared. We can see that the the proposed method outperforms [3],and [9]. The average computation time was 9.4 s.

4Discussion and Conclusion

We have presented a method for accurate, robust and fully automatic localization of bone surfaces in two-dimensional US data based on enhanced local phasebone and shadow region information. The method was validated on 150 in vivo

Localization of Bone Surfaces from Ultrasound Data

9

Fig. 5. Qualitative results. First, third and fifth rows represent the B-mode ultrasoundimage of in vivo radius, spine and femur respectively. Second, fourth and sixth rowspresent the localization result. Green represents manual expert segmentation and red
is obtained using the proposed algorithm.

US data, obtained from seven volunteers, and achieved an overall AED error of
0.26 mm. We achieved a 67% improvement in terms of surface localization overstate of the art methods and 94% improvement compared to intensity-based localization methods. Although we have not directly compared our method to ma-

10Hacihaliloglu

Table 1. Comparative results of the proposed approach.
Proposed

Phase symmetry [9]Dynamic programming [3]Mean SD 95% CL

Mean

SD 95% CL

Mean

SD

95% CL

Femur
Radius
Tibia
Knee
Spine

0.32
0.44
0.22
0.19
0.34

0.25
0.26
0.24
0.16
0.17

0.64
0.81
0.56
0.42
0.53

0.74
1.08
0.68
0.83
0.73

0.50
0.88
0.42
0.88
0.65

1.43
2.93
1.58
2.5
1.95

7.60
6.97
6.69
3.29
1.77

4.27
5.73
0.41
4.23
1.61

15.42
16.00
11.20
11.96
5.25

Overall

0.26

0.22

0.63

0.78

0.68

2.21

4.50

4.39

13.79

(SD - standard deviation, CL - confidence level)chine learning-based approaches [7] our reported localization results have 54%
improved accuracy. However, the proposed shadow enhancement method andlocal phase features extracted in the proposed work can also be incorporated
into existing machine learning approaches as additional features which could results in the improvement of the localization results reported for these methods.
The specific contributions include: (1) the use of α-scale filters for extraction ofbone phase features, (2) calculation of a new bone probability map for improvedbone surface localization, and (3) combination of enhanced bone shadow features with three different image phase features for bone localization. Previously,
it was shown that by optimizing the filter parameter selection, using information derived from the collected data, improvements can be achieved in terms of
surface localization and robustness to artifacts [9]. Therefore, the filter parameter selection process should be automated. Another limitation of the proposed
method is the achieved mean computation time which was around 9.4 s. This isa large computational cost considering that any intra-operative procedure performed requires real time feedback. Future work will involve (i) improvement of
the computation speed, (ii) validation on more in vivo scans, and (iii) optimization of the filter parameters. Finally, we would like to mention that although
there were no failed cases for the proposed method a more extensive validationis required in order to fully address clinical challenges that can be faced during
the application of the method. Specifically, volunteers with high body mass index will require a special investigation which we will be performing as part ofour future work.

References
1. Tonetti, J., Carrat, L., Blendea, S., Merloz, P., Troccaz, J., Lavallée, S., Chirossel,
J.P.: Clinical results of percutaneous pelvic surgery. computer assisted surgery
using ultrasound compared to standard fluoroscopy. Comput. Aided Surg. 6(4),204–211 (2001)
2. Baka, N., Leenstra, S., van Walsum, T.: Machine learning based bone segmentation
in ultrasound. In: Yao, J., et al. (eds.) Proc. 4th MICCAI Workshop on Compu-

Localization of Bone Surfaces from Ultrasound Data11

tational Methods and Clinical Applications for Spine Imaging – CSI 2016, LNCS,
vol. 10182, pp. 16–25. Springer (2016)
3. Foroughi, P., Boctor, E., Swartz, M., Taylor, R., Fichtinger, G.: P6d-2 ultrasoundbone segmentation using dynamic programming. In: Proc. 2007 IEEE Ultrasonics
Symposium. pp. 2523–2526. IEEE (2007)
4. Anas, E., Seitel, A., Rasoulian, A., John, P., Ungi, T., Lasso, A., Darras, K.,
Wilson, D., Lessoway, V., Fichtinger, G., Rohling, R., Abolmaesumi, P.: Boneenhancement in ultrasound based on 3D local spectrum variation for percutaneous
scaphoid fracture fixation. In: Ourselin, S., et al. (eds.) Proc. 19th International
Conference on Medical Image Computing and Computer-Assisted Intervention –MICCAI 2016, LNCS, vol. 9900, pp. 465–473. Springer (2016)
5. Hacihaliloglu, I., Guy, P., Hodgson, A., Abugharbieh, R.: Automatic extractionof bone surfaces from 3D ultrasound images in orthopaedic trauma cases. Int. J.
Comput. Assist. Radiol. Surg. 10(8), 1279–1287 (2015)
6. Jia, R., Mellon, S., Hansjee, S., Monk, A., Murray, D., Noble, J.: Automatic bonesegmentation in ultrasound images using local phase features and dynamic programming. In: Proc. 13th IEEE International Symposium on Biomedical Imaging– ISBI 2016. pp. 1005–1008. IEEE (2016)
7. Ozdemir, F., Ozkan, E., Goksel, O.: Graphical modeling of ultrasound propagationin tissue for automatic bone segmentation. In: Ourselin, S., et al. (eds.) Proc. 19thInternational Conference on Medical Image Computing and Computer-Assisted
Intervention – MICCAI 2016, LNCS, vol. 9901, pp. 256–264. Springer (2016)
8. Quader, N., Hodgson, A., Abugharbieh, R.: Confidence weighted local phase features for robust bone surface segmentation in ultrasound. In: Linguraru, M., et al.
(eds.) Proc. MICCAI Workshop on Clinical Image-Based Procedures – CLIP 2014,
LNCS, vol. 8680, pp. 76–83. Springer (2014)
9. Hacihaliloglu, I., Abugharbieh, R., Hodgson, A., Rohling, R.: Automatic adaptive
parameterization in local phase feature-based bone segmentation in ultrasound.
Ultrasound Med. Biol. 37(10), 1689–1703 (2011)
10. Hacihaliloglu, I., Abugharbieh, R., Hodgson, A., Rohling, R.: Bone surface localization in ultrasound using image phase-based features. Ultrasound Med. Biol. 35(9),1475–1487 (2009)
11. Kovesi, P.: Image features from phase congruency. Videre: J. Comput. Vis. Res.
1(3), 1–26 (1999)
12. Hacihaliloglu, I., Rasoulian, A., Rohling, R., Abolmaesumi, P.: Local phase tensor
features for 3D ultrasound to statistical shape+pose spine model registration. IEEE
Trans. Med. Imaging 33(11), 2167–2179 (2014)
13. Felsberg, M., Sommer, G.: The monogenic signal. IEEE Trans. Signal Process.
49(12), 3136–3144 (2001)
14. Belaid, A., Boukerroui, D.: α scale spaces filters for phase based edge detection in
ultrasound images. In: Proc. 11th IEEE International Symposium on Biomedical
Imaging – ISBI 2014. pp. 1247–1250. IEEE (2014)
15. Karamalis, A., Wein, W., Klein, T., Navab, N.: Ultrasound confidence maps using
random walks. Med. Image Anal. 16(6), 1101–1112 (2012)16. Hacihaliloglu, I.: Enhancement of bone shadow region using local phase-based ultrasound transmission maps. Int. J. Comput. Assist. Radiol. Surg. 12(6), 951–960(2017)
17. Meng, G., Wang, Y., Duan, J., Xiang, S., Pan, C.: Efficient image dehazing withboundary constraint and contextual regularization. In: Proc. IEEE International
Conference on Computer Vision – ICCV 2013. pp. 617–624. IEEE (2013)

Shape-Aware Deep Convolutional Neural
Network for Vertebrae Segmentation
S. M. Masudur Rahman Al Arif 1(
1), Karen Knapp2 , and Greg Slabaugh1

City, University of London, UK
2
University of Exeter, UK
s.al-arif@city.ac.uk

Abstract. Shape is an important characteristic of an object, and fundamental topic in computer vision. In image segmentation, shape has
been widely used in segmentation methods, like the active shape model,
to constrain a segmentation result to a class of learned shapes. However,to date, shape has been underutilized in deep segmentation networks.
This paper addresses this gap by introducing a shape-aware term inthe segmentation loss function. A deep convolutional network has been
adapted in a novel cervical vertebrae segmentation framework and compared with traditional active shape model based methods. The proposed
framework has been trained on an augmented dataset of 26370 vertebrae and tested on 792 vertebrae collected from a total of 296 real-life
emergency room lateral cervical X-ray images. The proposed framework
achieved an average error of 1.11 pixels, signifying a 36% improvement
over the traditional methods. The introduction of the novel shape-aware
term in the loss function significantly improved the performance by further 12%, achieving an average error of only 0.99 pixel.
Keywords: Convolutional neural networks, vertebrae, segmentation,
shape-aware, X-rays.

1

Introduction

Deep learning has revolutionized the field of image classification [1–4], segmentation [5–7] and many other aspects of computer vision. Segmenting an anatomical body part in medical images is a challenging problem in the field. Although,training a deep network requires huge amount of data, which is usually not available for medical images, recent techniques using data augmentation have shownpromising results for segmentation problem on medical images [8,9]. Shape characteristics have long been used for image segmentation problems, especially in
medical images [10–13]. Medical image modalities, e.g. X-ray, dual-energy Xray absorptiometry, magnetic resonance imaging, often produce noisy capturesof anatomical body parts, where segmentation must rely on the shape information to produce reliable results. However, combining shape information ina deep segmentation network is not straightforward. In this paper, we try tosolve the problem by introducing a novel shape-aware term in the segmentation
B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proc. 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. 12−24, 2017.

Shape-Aware Deep CNN for Vertebrae Segmentation

13

loss function. To test its capability of shape preservation, we adapted the novel
shape-aware deep segmentation network in a semi-automatic cervical vertebrae
segmentation framework.
Segmenting the vertebrae correctly is a crucial part for further analysis in
an injury detection system. Previous work in vertebrae segmentation has largely
been dominated by statistical shape model (SSM) based approaches [14–22].
These methods record statistical information about the shape and/or the appearance of the vertebrae based on a training set. Then the mean shape isinitialized either manually or semi-automatically near the actual vertebra. The
model then tries to converge to the actual vertebra boundary based on a search
procedure. Recent work [19–22] utilizes random forest based machine learningmodels in order to achieve shape convergence. In contrast to these methods,
we propose a novel deep convolutional neural network based method for vertebrae segmentation. Instead of predicting the shape of a vertebra, our framework
predicts the segmentation mask of a vertebrae patch. In order to preserve thevertebra shape, a novel shape-aware loss term has been proposed. From a training set of 124 X-ray images containing 586 cervical vertebrae, 26370 vertebrae
patch-segmentation mask pairs have been generated through data augmentation for training the deep network. The trained framework has been tested ondataset of 172 images containing 792 vertebrae. An average pixel-level accuracyof 97.01%, Dice similarity coefficient 0.9438 and shape error of 0.99 pixel havebeen achieved.
The key contributions of this work are two fold. First, the introduction of
a novel shape-aware term in the loss function of a deep segmentation network
which learns to preserve the shape of the target object and significantly improved
the segmentation accuracy. Second, the application and adaptation of deep segmentation networks to achieve vertebrae segmentation in real life medical images
which outperformed the traditional SSM based methods by 35%.

2

Data

A total of 296 lateral cervical spine X-ray images were collected from Royal Devon and Exeter Hospital in association with the University of Exeter, UK. The
age of the patients varied from 17 to 96. Different radiographic systems (Philips,
Agfa, Kodak, GE) were used to produce the scans. Image resolution varied from

Fig. 1. X-ray images and manual annotations. Center: blue plus (+), vertebrae boundary curve: green).

14

Al Arif et al.

0.1 to 0.194 mm per pixel. The images include examples of vertebrae with fractures, degenerative changes and bone implants. The data is anonymized andstandard research protocols have been followed. The size, shape, orientation ofspine, image intensity, contrast, noise level all varied greatly in the dataset. For
this work, five vertebrae C3-C7 are considered. C1 and C2 have an ambiguous
appearance due to their overlap in lateral cervical radiographs, and our clinicalexperts were not able to provide ground truth segmentations for these vertebral
bodies. For this reason they are excluded in this study, similar to other cervical spine image analysis research [15, 23]. Each vertebra from the images was
manually annotated for the vertebral body boundaries and centers by an expert
radiographer. A few examples with corresponding manual annotations are shownin Fig. 1.
The images were received in two sets. The first set of 124 images are used fortraining and the rest are kept for testing. The manually clicked center points andthe vertebrae boundary curves are used extract the vertebrae image patch andcorresponding segmentation masks. Different patch size and rotation angles are
considered in order to augment the training data. After data augmentation, weended up with 26370 vertebrae training patches. All the patches were then resized
to 64 × 64 pixel patches. The corresponding vertebrae curves were converted tobinary segmentation masks of the same size. A few training vertebrae patches
and corresponding overlaid segmentation masks are shown in Fig. 2. Similarly
vertebrae patches were also collected from the test images. The orientation andscale for the test vertebrae were computed by the manually clicked center points
only, shape information was not used. Our assumption is that the center points
will be manually provided at test time, making the process semi-automatic.
Some test vertebra are shown in Fig. 3. Note the differences in intensity, texture,and contrast, coupled with the possibility of surgical implants, making for achallenging problem on real-world data.

3

Methodology

Several deep segmentation networks have achieved outstanding performance innatural images [5–7]. However, medical images have their own set of challenges toovercome. The UNet architecture have shown excellent capability of segmenting
different target objects in different medical image modalities [8,9]. Following the

Fig. 2. Training vertebrae patches and corresponding segmentation masks (blue).

Shape-Aware Deep CNN for Vertebrae Segmentation

15

Fig. 3. Examples of test vertebrae patches.

literature, for our shape-aware vertebrae segmentation problem, we have chosena modified version of the original UNet [8] deep segmentation network.
3.1

Network Architecture

The UNet architecture consists of a contracting path and an expanding pathin the network. The contracting path reduces the spatial dimension of an input
image to a smaller version and the expanding path expands spatial dimension
and results in a segmentation map at the output. In the original architecture [8],the spatial dimension output segmentation map is smaller than that of the input images due to the use of convolution layers without padding. In our version, we want to keep the spatial dimension of the input image and the output
segmentation map same. Our architecture has nine convolutional layers in thecontracting path. Each convolutional layer is followed by a batch normalization
and a rectified linear unit (ReLU) layer. Three 2 × 2 pooling layers, one each
after two consecutive convolutional layers, reduce the input size of 64 × 64 toa smaller dimension of 8 × 8 at the end of the contracting path. This data isthen forwarded through a mirrored expanding path of the network. The upsampling after every two convolution layer in the expanding path is achieved bya deconvolution layer with 2 × 2 kernel size. The network shares intermediate
information from the contracting path to the expanding path by concatenationof data. After each upsampling, the data in the expanding path is concatenatedby the corresponding data from the contracting path. This helps the network
to recover some of the information lost during max-pooling operation. Our network takes a single channel vertebra patch of spatial dimension 64 × 64 andpredicts a two channel probabilistic output for the prediction vertebrae maskof the same size. Fig. 4 details the network diagram. The number of filters ineach convolutional/deconvolutional layer can be tracked from the intermediate
data dimensions in Fig. 4. The total number of parameters in the network is
24.238.210.
3.2

Loss Function

Given a dataset of training image (x)-segmentation label (y) pairs, training a
deep segmentation network means finding a set of parameters W that minimizesa loss function, Lt . The simplest form of the loss function for segmentation

16

Al Arif et al.

(a)

(b)

Fig. 4. UNet architecture. (a) Network diagram (b) Legends.

problem is the pixel-wise log loss:
Ŵ = arg min
W

N
X

n=1



Lt {x(n) , y (n) }; W ,(1)where N is the number of training examples and {x(n) , y (n) } represents n-th
example in the training set with corresponding manual segmentation. The pixelwise segmentation loss per image can be defined as:
Lt ({x, y}; W ) = −MXXyij log P (yij = 1|xi ; W ),(2)iΩp j=1

exp(aj (xi )),P (yij = 1|xi ; W ) = PM
k=1 exp(ak (xi ))(3)

where aj (xi ) is the output of the penultimate activation layer of the network forthe pixel xi , Ωp represents the pixel space, M is the total number of segmentation class labels and P are the corresponding class probabilities. However, thisterm does not constrain the predicted masks to conform to possible vertebra
shapes. Since vertebrae shapes are known from the provided manual segmentation curves, we add a novel shape-aware term in the loss function to force the
network to learn to penalize predicted areas outside the curve.
3.3

Shape-Aware Term

For training the deep segmentation network, we introduce a novel shape-based
term, Ls . This term forces the network to produce a prediction masks similar to
the training vertebra shapes. This term can be defined as:
Ls ({x, y}; W ) = −MXXyij Ei log P (yij = 1|xi ; W ); Ei = D(Ĉ, CGT ),iΩ̂p j=1

(4)

Shape-Aware Deep CNN for Vertebrae Segmentation

(a)

(b)

(c)

17

(d)

Fig. 5. Shape-aware loss: (a) Ground truth mask. (b) Predicted mask. (c) Groundtruth shape, CGT (green) and predicted shape, Ĉ (red). (d) Refined pixel space, Ω̂p :False positive (purple) and false negative (red).

where Ĉ is the curve surrounding the predicted regions and CGT is groundtruth curve. The function, D(·), computes the average point to curve Euclidean
distance between the predicted shape, Ĉ and the ground truth shape, CGT .
Ĉ is generated by locating the boundary pixels of the predicted mask. The
redefined pixel space, Ω̂p , contains the set of pixels where the prediction mask
does not match the ground truth mask. These terms can also be explained using
the toy example shown in Fig. 5. Given a ground truth mask (Fig. 5(a)) and a
prediction mask (Fig. 5(b)), Ei is computed by measuring the average distance
between the ground truth (green) curve and prediction (red) curve (Fig. 5(c)).
Fig. 5(d) shows the redefined pixel space, Ω̂p . This term adds additional penalty
proportional to Euclidean distance between predicted and ground truth curveto the pixels that do not match the ground truth segmentation mask. In thecase when the predicted mask is a cluster of small regions, especially during thefirst few epochs in training, Ei becomes very large because of the increase in theboundary perimeters from the disjoint predictions. Thus, this term also implicitly
forces the network to learn to predict single connected prediction masks faster.
3.4

Updated Loss Function

Finally, the loss function of (1) can be extended as:Ŵ = arg min
W

N 
X

n=1





Lt {x(n) , y (n) }; W + Ls {x(n) , y (n) }; W.

(5)

The contribution of each term in the total loss can be controlled by introducing
a weight parameter in (5). However, in our case, best performance was achieved
when both terms contributed equally.

4

Experiments

We have two versions of the deep segmentation network: UNet and UNet-S,
where “-S” signifies the use of updated shape-aware loss function of (5). The
networks are trained for 30 epochs with batch size of 25 vertebrae patches. To
update the network parameters, RMSprop version of mini-batch gradient descent

18

Al Arif et al.

algorithm is used [24]. Each network took around 30 hours to complete training
in computer equipped with a NVIDIA Pascal Titan X GPU. In order to comparewith the deep segmentation network based prediction results, three active shapemodel (ASM)-based shape prediction frameworks have been implemented. A
simple maximum gradient-based image search based ASM (ASM-G) [14], a Mahalanobis distance based ASM (ASM-M) [15] and a random forest based ASM(ASM-RF) [21]. The later two have been used in cervical vertebrae segmentationin different datasets.
4.1

Inference and Metrics

At test time, 792 vertebrae from 172 test images are extracted based on the
manually clicked vertebral centers. These patches are forwarded through each of
the networks to get the prediction masks. These prediction masks are comparedwith the ground truth segmentation mask to compute number pixels detected
as true positive (TP), true negative (TN), false positive (FP) and false negative(FN). Based on these measures two metrics are computed for each set of test
vertebra patch and prediction masks: pixel-wise accuracy (pA) and Dice similarity coefficients (DSC). For the ASM based shape predictors, the predicted shapeis converted to a prediction map to measure these metrics.
2T P,
2T P + F P + F N
TP + TN]pA =× 100%.
TP + TN + FP + FN

DSC =

(6)
(7)

These metrics are well suited to capture the number of correctly segmented
pixels, but they fail to capture the differences in shape. In order to compare
the shape of the predicted mask appropriately with the ground truth vertebrae
boundary, the predicted masks of the deep segmentation networks are converted
into shapes by locating the boundary pixels. These shapes are then comparedmanually annotated vertebral boundary curves by measuring average point tocurve Euclidean distance between them, similar to (4). A final metric, called fit
failure [20], is also computed which measures the percentage of vertebrae having
an average point to ground truth curve error of greater than 2 pixels.

5

Results

Tab. 1 reports the average median, mean and standard deviation metrics over
the test dataset of 792 vertebrae for all the methods. The deep segmentation
networks clearly outperform the ASM-based methods. Even the worst versionof our framework, UNet achieves a 2.9% improvement in terms of pixel-wise
accuracy and an increase of 0.055 for Dice similarity coefficient. Among the two
version of deep networks, the use of novel loss function improves the performanceby 0.31% in terms of pixel-wise accuracy. In terms of Dice similarity coefficient,Shape-Aware Deep CNN for Vertebrae Segmentation

19

Table 1. Average quantitative metrics for mask prediction.
Pixel-wise accuracy (%)
Median Mean Std

p-value

Dice similarity coefficientMedian Mean Std

p-value

ASM-RF
ASM-M
ASM-G

95.09 90.77 8.98
95.09 93.48 4.92
95.34 93.75 4.48

0.881 0.774 0.220
0.900 0.877 0.073
0.906 0.883 0.066

UNet
UNet-S97.71 96.69 3.04
7.17 × 10−13
97.92 97.01 2.79

0.952 0.938 0.048
7.76 × 10−130.957 0.944 0.044

the improvement is in the range of 0.006. Although, subtle, the improvements
are statistically significant according to a paired t-test at a 5% significance level.
Corresponding p-values between the two versions of the network are reported
in Tab. 1. Bold fonts indicates the best performing metrics. Interestingly, among
the ASM-based methods, the simplest version, ASM-G, performs better than
the alternatives. Recent methods [15, 21], have failed to perform robustly on ourchallenging dataset of test vertebrae.
The average point to curve error for the methods are reported in Tab. 2. The
deep segmentation framework, UNet, produced a 35% improvement over the
ASM-based methods in terms of the mean values. The introduction of the novel
loss term in the training further reduced the average error by 12% achieving
the best error of 0.99 pixels. The most significant improvement can be seenin the fit failure which denotes the percentage of the test vertebrae having an
average error of higher than 2 pixels. The novel shape-aware network, UNet-S,
has achieved drop of around 37% from the ASM-RF method. The cumulative
distribution of the point to curve error is also plotted in the performance curveof Fig. 6. It can be seen adaptation deep segmentation network provides a big
improvement in area under the curve. The box plot of the quantitative metrics
are shown in Fig. 7. It can be seen that, even the worst outlier for shape-aware
network, UNet-S, have a pixel-wise accuracy higher than 70%, signifying theregularizing capability of the novel term. Most of the outliers are caused by
Table 2. Average quantitative metric for shape prediction.
Average point to curve error (pixels)
Median Mean Std
ASM-RF
ASM-M
ASM-G

1.82
1.54
1.38

2.59
1.88
1.73

UNet
UNet-S

0.77
0.78

1.11 1.29
0.99 0.67

p-value

1.85
1.05
0.99

Fit failure (%)
43.43
32.70
26.89

0.0043

8.59
6.06

20

Al Arif et al.

Fig. 6. Performance curve: cumulative distribution of point to curve errors.

bone implants, fractured vertebrae or abnormal artifacts in the images. A few
examples for qualitative assessment are shown in Fig. 8. Fig. 8(a) shows an easy
example where all the methods perform well. Examples with bone implants areshown in Fig. 8(b),(c). Fig. 8(d),(e) shows vertebrae with abrupt contrast change.
Vertebrae with fracture and osteoporosis are shown in Fig. 8(f),(g). Fig. 8(g)also shows how UNet-S has been able capture the vertebrae fractures pattern.
Fig. 8(h),(i) show vertebrae with image artefacts. A complete failure case isshown in Fig. 8(j). In all cases the shape-aware network, UNet-S, has produced
better segmentation results than its counterpart.
5.1

Analysis on Harder Cases

Although statistically significant, the difference in performance between the
UNet and UNet-S is subtle over the whole dataset of test vertebrae. This isbecause majority of the vertebrae are healthy and shape-awareness does not improve the results by a big margin. To show the shape-awareness capability of
UNet-S a selection 52 vertebrae with severe clinical conditions are chosen. The
average metrics for this subset of test vertebrae between UNet and UNet-S is

(a)

(b)

(c)

Fig. 7. Box plot of quantitative metrics: (a) pixel-level accuracy, (b) Dice similarity
coefficients, and (c) point to manual segmentation curve error.

Shape-Aware Deep CNN for Vertebrae Segmentation

21

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Fig. 8. Qualitative segmentation results: true positive (green), false positive (blue) andfalse negative (red).

reported in Table 3. An improvement of 1.2% and 0.02 have been achieved interms of pixel-wise accuracy and Dice similarity coefficient, respectively. The
difference over the whole dataset were only 0.31% and 0.006. The metric, point

22

Al Arif et al.

Table 3. Comparison of UNet and UNet-S.
Average quantitative metrics

UNet
UNet-S

Pixel-wise
accuracy (%)

Dicecoefficient

Point to curve
error (pixels)

94.01
95.21

0.91
0.93

1.61
1.24

to curve error produces the most dramatic change. The novel shape-aware network, UNet-S, reduced the error by 22.9% for this subset of vertebrae with severeclinical conditions. Fig. 9 shows a few example of these subset of images.

6

Conclusion

Deep segmentation networks have shown exciting application in different medical image modalities. The shape of an anatomical object is very important for
automated computer aided diagnosis and injury detection. Our overarching goal
is to build a computer aided system that can help the emergency department
physicians to detect injuries with better accuracy. Towards this goal, in thispaper, we proposed a robust semi automatic vertebrae segmentation method using deep convolutional neural networks that incorporate the shape information
in to achieve better segmentation accuracy. The proposed deep segmentation
method has outperformed the traditional active shape model based approaches
by a significant margin. In order to incorporate shape information with themask prediction capability of the deep neural networks, a novel shape-aware
loss function has been formulated. Inclusion of this novel term in the training
provided significant quantitative and qualitative improvements. A maximum average pixel-level segmentation accuracy of 97.01%, Dice coefficient of 0.9438 andFig. 9. Comparison of performance for vertebrae with severe clinical condition.

Shape-Aware Deep CNN for Vertebrae Segmentation

23

point to ground truth curve error of less than 1 pixel has been achieved over adiverse dataset of 792 test vertebrae collected from real life medical emergency
rooms. Currently, we are working on a fully automatic localization framework
to locate the vertebrae centers in arbitrary X-ray images. In the future, we will
be using the segmented vertebrae column to automatically determine various
clinical conditions like misalignment of the vertebral body, osteoporosis, bonedensity abnormality and type and severity of different vertebral fractures.
Acknowledgements. We gratefully acknowledge the support of NVIDIA Corporation
with the donation of the Titan X Pascal GPU used for this research.

References
1. Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: Pereira, F., et al. (eds.) Proc. Neural Information
Processing Systems – NIPS 2012, vol. 25, pp. 1097–1105. NIPS (2012)
2. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale
image recognition. arXiv p. arXiv:1409.1556 (2014)
3. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D.,
Vanhoucke, V., Rabinovich, .: Going deeper with convolutions. In: Proc. IEEE
Conference on Computer Vision and Pattern Recognition – CVPR 2015. pp. 1–9.
IEEE (2015)
4. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.
In: Proc. IEEE Conference on Computer Vision and Pattern Recognition – CVPR
2016. pp. 770–778. IEEE (2016)
5. Shelhamer, E., Long, J., Darrell, T.: Fully convolutional networks for semantic
segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 640–651 (2017)
6. Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang,C., Torr, P.: Conditional random fields as recurrent neural networks. In: Proc. IEEE
International Conference on Computer Vision – ICCV 2015. pp. 1529–1537. IEEE
(2015)
7. Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. In: Proc. IEEE International Conference on Computer Vision – ICCV 2015.
pp. 1520–1528. IEEE (2015)
8. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., et al. (eds.) Proc. 18th International Conference on Medical Image Computing and Computer-Assisted Intervention – MICCAI2015, LNCS, vol. 9351, pp. 234–241. Springer (2015)
9. BenTaieb, A., Hamarneh, G.: Topology aware fully convolutional networks forhistology gland segmentation. In: Ourselin, S., et al. (eds.) Proc. 19th International
Conference on Medical Image Computing and Computer-Assisted Intervention –MICCAI 2016, LNCS, vol. 9901, pp. 460–468. Springer (2016)
10. Yushkevich, P., Piven, J., Hazlett, H., Smith, R., Ho, S., Gee, J., Gerig, G.: Userguided 3d active contour segmentation of anatomical structures: significantly improved efficiency and reliability. Neuroimage 31(3), 1116–1128 (2006)
11. Pluempitiwiriyawej, C., Moura, J., Wu, Y.J., Ho, C.: STACS: new active contour
scheme for cardiac MR image segmentation. IEEE Trans. Med. Imaging 24(5),593–603 (2005)

24

Al Arif et al.

12. Weese, J., Wächter-Stehle, I., Zagorchev, L., Peters, J.: Shape-constrained deformable models and applications in medical imaging. In: Li, S., et al. (eds.) ShapeAnalysis in Medical Image Analysis, LNCVB, vol. 14, pp. 151–184. Springer (2014)
13. Farag, A., Shalaby, A., El Munim, H., Farag, A.: Variational shape representation
for modeling, elastic registration and segmentation. In: Li, S., et al. (eds.) ShapeAnalysis in Medical Image Analysis, LNCVB, vol. 14, pp. 95–121. Springer (2014)
14. Cootes, T., Taylor, C., Cooper, D., Graham, J.: Active shape models - their training
and application. Comput. Vis. Image Understand. 61(1), 38–59 (1995)
15. Benjelloun, M., Mahmoudi, S., Lecron, F.: A framework of vertebra segmentation
using the active shape model-based approach. Int. J. Biomed. Imaging 2011, 621905(2011)
16. Larhmam, M.A.and Mahmoudi, S., Benjelloun, M.: Semi-automatic detection of
cervical vertebrae in X-ray images using generalized Hough transform. In: Proc.
3rd IEEE International Conference on Image Processing Theory, Tools and Applications – IPTA 2012. pp. 396–401. IEEE (2012)
17. Roberts, M., Cootes, T., Adams, J.: Vertebral morphometry: semiautomatic determination of detailed shape from dual-energy X-ray absorptiometry images using
active appearance models. Invest. Radiol. 41(12), 849–859 (2006)
18. Roberts, M., Pacheco, E., Mohankumar, R., Cootes, T., Adams, J.: Detection ofvertebral fractures in DXA VFA images using statistical models of appearance anda semi-automatic segmentation. Osteoporos. Int. 21(12), 2037–2046 (2010)
19. Roberts, M., Cootes, T., Adams, J.: Automatic location of vertebrae on DXA
images using random forest regression. In: Ayache, N., et al. (eds.) Proc. 15th
International Conference on Medical Image Computing and Computer-Assisted
Intervention – MICCAI 2012, LNCS, vol. 7512, pp. 361–368. Springer (2012)20. Bromiley, P., Adams, J., Cootes, T.: Localisation of vertebrae on DXA images
using constrained local models with random forest regression voting. In: Yao, J.,et al. (eds.) Proc. 2nd MICCAI Workshop on Computational Methods and ClinicalApplications for Spine Imaging – CSI 2014, LNCVB, vol. 20, pp. 159–171. Springer(2015)
21. Al Arif, S., Gundry, M., Knapp, K., Slabaugh, G.: Improving an active shape modelwith random classification forest for segmentation of cervical vertebrae. In: Yao, J.,et al. (eds.) Proc. 4th MICCAI Workshop & Challenge on Computational Methods
and Clinical Applications for Spine Imaging – CSI 2016, LNCS, vol. 10182, pp. 3–
15. Springer (2016)22. Bromiley, P., Kariki, E., Adams, J., Cootes, T.: Fully automatic localisation ofvertebrae in CT images using random forest regression voting. In: Yao, J., et al.
(eds.) Proc. 4th MICCAI Workshop & Challenge on Computational Methods andClinical Applications for Spine Imaging – CSI 2016, LNCS, vol. 10182, pp. 51–63.
Springer (2016)23. Mahmoudi, S., Lecron, F., Manneback, P., Benjelloun, M., Mahmoudi, S.: GPUbased segmentation of cervical vertebra in X-ray images. In: Proc. IEEE International Conference on Cluster Computing Workshops and Posters – CLUSTERWORKSHOPS. pp. 1–8. IEEE (2010)
24. Ruder, S.: An overview of gradient descent optimization algorithms. arXiv p.
arXiv:1609.04747 (2016)

Automated Characterization of Body
Composition and Frailty with
Clinically Acquired CT
Peijun Hu1,2( ) , Yuankai Huo3 , Dexing Kong1 , J. Jeffrey Carr4 ,Richard G. Abramson4 , Katherine G. Hartley4 , and Bennett A. Landman2,3,4
1

School of Mathematical Sciences, Zhejiang University, China
2
Computer Science, Vanderbilt University, USA
3
Electrical Engineering, Vanderbilt University, USA
4Radiology and Radiological Sciences, Vanderbilt University, USA
peijunhu.zju@gmail.com

Abstract. Quantification of fat and muscle on clinically acquired computed tomography (CT) scans is critical for determination of body composition, a key component of health. Manual tracing has been regarded
as the gold standard method of body segmentation; however, manual
tracing is time-consuming. Many semi-automated/automated algorithms
have been proposed to avoid the manual efforts. Previous efforts largely
focused on segmenting two-dimensional cross-sectional images (e.g., at
L3/T4 vertebra locations) rather than on the whole-body volume. In thispaper, we propose a fully automated three-dimensional (3D) body composition estimation framework for segmenting the muscle and fat fromabdominal CT scans. The 3D whole body segmentations were reconstructed from a slice-wise multi-atlas label fusion (MALF) based framework. First, we used a low-dimensional atlas representation to estimate
each class for each axial slice. Second, the abdominal wall and psoas
muscle were segmented by combining MALF with active shape models
and deformable models. Third, skeletal muscle, visceral adipose tissue(VAT) and subcutaneous adipose tissue (SAT) were measured to assess
the areas of muscle and fat tissue. The proposed method was compared tomanual segmentation and demonstrated high accuracy. Then, we evaluated the approach on 40 CT scans comparing the new method to aprior atlas-based segmentation method and achieved 0.854, 0.740, 0.887
and 0.933 on Dice similarity index for the skeletal muscle, psoas muscle,
VAT and SAT, respectively. Compared with the baseline, our method
showed significantly (p < 0.001) higher accuracy on skeletal muscle, VAT
and SAT estimation.
Keywords: Skeletal muscle, psoas muscle, visceral fat, subcutaneousfat, multi-atlas.

1

Introduction

Body composition of fat and muscle mass is an important biomarker in cancer
treatment. The quantitative measurement of body composition is related to the
B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proc. 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. 25−34, 2017.

26

Hu et al.

efficacy and toxicity of chemotherapy, post therapy functional status, surgicalcomplication rates, length of hospital stay and overall survival [1]. In addition,body composition estimation will lead to a more reliable replacement of basic
measures of healthy weight, such as body mass index. Manual delineation on
computed tomography (CT) images has been regarded as the gold standard
in body composition estimation [2]. However, manual tracing on muscle andfat regions are time-consuming and cannot be easily applied to large cohorts.
Therefore, many automated or semi-automated methods have been proposed toperform the segmentation. The previous studies estimated the body compositionof muscle and fat based on the two-dimensional (2D) axial slice at the L3 lumbar
vertebra position. However, single slice based estimation is a rough approximation of the whole-body composition, which is sensitive to slice selection.
For fat segmentation, the Hounsfield unit (HU) intensity is typically used
to distinguish muscle and fat on CT images when performing the manual segmentation (e.g., [−29, 150] for muscle tissue and [−190, −30] for fat tissue [3]).
Importantly, the compartment in which adipose tissue resides relates to theclinical significance of that fat. For example, it is found that nonagenarian individuals with and without frailty syndrome presented marked differences in the
pericardial and visceral adipose tissue [4]. Therefore, to generate a meaningful measurement, efforts are required to separate fat into visceral adipose tissue(VAT) and subcutaneous adipose tissue (SAT). The VAT is the adipose tissue included in intra-abdominal cavity, while SAT is the adipose tissue bounded by the
inner abdominal wall musculature and the skin surface [4]. Muscle segmentation
presents a greater challenge as the HU for muscle overlaps with other abdominalorgans and tissues. Moreover, the variable shape and location of muscle makethe segmentation even more difficult.
Previous efforts were typically focused on segmenting either muscle or fat. For
fat segmentation, Yao et al. [5] separated the subcutaneous and visceral fat by asingle surface at the abdominal wall driven by active contour models (ACM). For
muscle segmentation, shape models were typically used. For instance, Tsutomu
et al. [6] incorporated a shape prior represented as logistic curves in higher-ordergraph cut models to segment psoas from CT images. Chung et al. [7] presented
a muscle segmentation method in which a thresholded binary image was warped
to a mean shape prior by a free deformation model. Popuri et al. [3] proposed a
FEM-based registration model to perform template-based segmentation of skeletal muscle. Although these methods achieved high accuracy, they were based on
2D cross-sectional images taken at the 3rd lumbar vertebra (L3) or the 4ththoracic vertebra (T4) locations rather than on whole three-dimensional (3D)
volumes. Zhang et al. [8] presented an atlas-based approach to segment the musculature on CT volumes using five pre-defined muscle atlas models, and thenrefined by ACM. Xu et al. [9] proposed a slice-wise method called augmented
active shape model (AASM) that integrated multi-atlas label fusion (MALF) andlevel set into the active shape model (ASM) framework. To overcome the large
intra-slice variations in the abdominal wall along the cranial-caudal direction,
this method pre-classifies slice-wise images to five exclusive classes using land-Characterization of Body Composition and Frailty with CT

27

Fig. 1. Illustration of challenges in muscle and fat segmentation. The sagittally reconstructed images show differing fields of view while the axial images demonstrated thelarge intra-slice and inter-slice variations in muscle and fat cross-sectional area on CT
images.

marks. However, for clinical data with large variations on fields of view (FOV),
such landmarks are not typically available (Fig. 1).
In this study, we propose a fully automated framework to segment the skeletal
muscle, psoas muscle, VAT and SAT from clinically acquired CT scans. Briefly,
we first used a PCA-based low-dimensional representation to estimate the atlas class for the target image. Second, the abdominal wall was segmented using
AASM under a slice-wise MALF framework. Here, the abdominal wall is characterized as an enclosed region bounded by the inner surface and outer surface.
Then, the psoas muscle was segmented using combination of MALF and a deformable model. Finally, the skeletal muscle, VAT and SAT were extracted using
the generated abdominal wall mask and pre-defined HU ranges. The main novelty of our work lies in the application of MALF on the challenging problem ofmuscle and fat quantification. Considering different anatomies of target regions,an active shape model and a deformable model are used to refine and regularize
the initial results of MALF. The whole pipeline is fully automated, without theneed of user interaction or parameter adjustment, which gives it potential to be
applied in clinic.

2

Methods

The proposed pipeline is shown in Fig. 2. It integrates a slice-wise multi-atlaslabel fusion framework with an active shape model and a deformable model.
2.1

Atlas Class Estimation via PCA

We use a PCA-based low-dimensional representation to decide the atlas class foreach testing image. Following [9], three biomarkers, i.e., xiphoid process (XP),pubic symphysis (PS) and umbilicus (UB) are manually labeled on each training volume. According to the location of these biomarkers, training axial slices

28

Hu et al.

are separated into five exclusive classes. For training image j, we assume that
we are given the image aj ∈ RD and its atlas class cj , where D is the total
number of pixels and cj ∈ {1, 2, 3, 4, 5}. Given a set of training images, the goal
is to assign a target slice image x into the most similar atlas class. Firstly, wealign all training images to the same space. Specifically, one image is randomly
selected and others are registered to it using an affine transform. Then, an average image is generated from all registered images and all images are registered
to the average image again. We denote the aligned training data
P5 set as matrix
A = (a11 , . . . , a1n1 , . . . , a51 , . . . , a5n5 ) ∈ RD×N , where N = i=1 ni is the total number of training images. Secondly, by using PCA dimension reduction, thehigh-dimensional datapoints A are transformed into a low-dimensional representation Y = (y 11 , . . . , y 1n1 , . . . , y 51 , . . . , y 5n5 ) ∈ Rd×N , whered  D. The PCAP
is performed by minimizing the cost function φ(Y) = i,j d2ij − ||y i − y j ||2 ,where dij represents the Euclidean distance between the high-dimensional points
ai and aj . Finally, the target image x is projected to the low-dimensional space
and the k-nearest neighbors in Euclidean distance are selected to estimate its
class using majority voting.
2.2

Abdominal Wall and Psoas Segmentation

Prior Probability Map Learned from MALF. For each axial slice of thetest CT volume, all the training images in the estimated class are considered
as atlases to perform slice-wise multi-atlas label fusion with respect to regionsof interest, i.e., the abdominal wall and psoas muscle. Each atlas is non-rigidly
registered to the target image with NiftyReg package [10]. Atlas labels are thenwarped to the target image and combined using the joint label fusion algorithm[11], yielding prior probability maps for the abdominal wall and psoas muscle.
Abdominal Wall Segmentation. In abdominal wall segmentation, we use
AASM [9] to search the optimal shape iteratively. In the training stage, anactive shape model and a local appearance model are trained from each atlasclass. In testing, the trained ASM, local appearance model and the probability
map generated from MALF guide landmarks along current contour move to new
positions. Specifically, firstly level set evolution is applied on the probability map
to move the current contour. Then, the zero-crossing points along the normal
direction of the zero level set are collected as landmarks. These landmarks are
updated with the active shape search and used as the initialization of level set
evolution for the next iteration. The constrained ASM parameters guarantee theregularization of the shape.
Psoas Muscle Segmentation. For psoas muscle, we adopt a 3D deformablemodel that integrates intensity statistical information, a prior probability map,and a gradient map to refine the initial surface. Given a volume image I : x ∈Ω → R defined on Ω ⊂ R3 , we denote the indicator function of psoas as u(x) =Characterization of Body Composition and Frailty with CT

29

Fig. 2. Flowchart of the proposed segmentation framework.

{0, 1}, x ∈ Ω, which is obtained by minimizing the energy functional:ZE(u) = λ1 Edata (u) + λ2 Eprior (u) +g(x)|∇u|dx.

(1)Ω

The first data term formulates
the intensity statistics inside and outside theRregion, i.e., Edata (u) = − Ω u log pin (I(x)) + (1 − u) log pout (I(x)) dx, where
pin and pout are intensity distributions of foreground and background pixels
defined by the initial psoas muscle
region given by MALF. The second prior termR
presented as Eprior (u) = − Ω u log L(x) + (1 − u) log (1 − L(x)) dx is to provide
spatial constraints, where L(x) is the psoas muscle probability map learned from
MALF. The last weighted total-variation term acts as regularization term, where1g(x) = 1+β|∇I(x)|2 , β > 0. The energy function (1) is globally optimized in asurface evolution way with the continuous max-flow algorithm [12, 13].

30

Hu et al.

2.3

Skeletal Muscle and Adipose Tissue Measurement

Using the segmented abdominal wall as a mask, skeletal muscle and fat tissue ineach axial slice are extracted based on pre-defined HU ranges according to [3].
The region that locates inside the inner wall surface with HU [−190, −30] is
extracted as VAT, and the region resides outside inner wall surface within the
body mask with HU [−190, −30] is extract as SAT. The skeletal muscle tissue
is segmented inside abdominal wall mask with HU [−29, −150].
2.4

Baseline Method

We take the method of Zhang et al. [8] as baseline, which also focused on whole
CT volume segmentation and used slice-wise algorithms. Following [8], we usedfuzzy c-means and an active contour model (ACM) to segment VAT and SAT,
and then used an atlas-based method to segment skeletal muscle. To avoid theeffect of atlas model selection on muscle segmentation, the ACM was applied torefine the results from our MALF as re-implemented.
All experiments in this paper were run on a machine with 8 cores of Intel
Xeon W3520 processors and 12 GB RAM available running Ubuntu 14.04.1 LTS(64 bit). Algorithms were implemented in MATLAB 2014b environment. The
proposed segmentation method has been made available online in open-source1 .

3Experiments ans Results

3.1

Data

Abdominal CT data on 60 patients from two clinical datasets were randomly
retrieved in de-identified form under IRB approval (40 patients from PHC and
20 from GIONC). Specifically, 20 scans from PHC were used for training, andthe remaining 20 PHC scans and 20 GIONC scans were for testing. The FOVs
of PHC scans ranged from 335 × 335 × 390 mm3 to 500 × 500 × 708 mm3 , withresolutions ranging from 0.65 × 0.65 × 2.50 mm3 to 0.98 × 0.98 × 5.00 mm3 . The
FOVs of GIONC scans were from 346 × 346 × 165 mm3 to 412 × 412 × 505 mm3 ,with resolutions from 0.65 × 0.65 × 1.50 mm3 to 0.85 × 0.85 × 5.00 mm3 . All 60
scans were manually labeled by an experienced rater to generate ground truthof skeletal muscle, psoas, VAT and SAT. Specifically, for the 40 PHC scans,essential biomarkers, i.e., xiphoid process, pubic symphysis, and umbilicus wereidentified, and the abdominal wall and psoas muscle were delineated on axial
slices spaced every 5 cm, generating 177 and 184 axial slices for training andtesting, respectively. Besides, 40 axial slices at the L3 position of the 40 testingscans were extracted and labeled. Axial slices from the same patient for training
were not used for testing.
1

https://www.nitrc.org/project/showfiles.php?group_id=385&release_id=3557

Characterization of Body Composition and Frailty with CT

31

Fig. 3. Scatter plots showing the correlation between estimated tissue area by the
proposed method and the ground truth on 184 axial slices.

3.2

Experimental Setting

For all training and testing slices, a body mask was obtained by thresholding
the image to remove the background, then selecting the largest 3D connectedcomponent to ensure that the CT table was excluded before image analysis. All
the images were centered after body extraction. A leave-one-out approach that
excluded slices from the same patient was used to optimize the parameters in
PCA dimension reduction on the labeled slices from PHC dataset. As a result, 9modes of variation in the low-dimensional space and k = 3 neighboring images
are chosen to estimate the target image class.
In the AASM-based abdominal wall segmentation, 177 labeled axial slices
from 20 patients were used as training set (also atlases) to build the active shapemodel with default parameters as in [9]. The parameters in energy function (1)of psoas refinement were empirically set as λ1 = 0.1, λ2 = 0.01, β = 0.2.
3.3

Results

Automated results were validated against the manual labels on 184 axial slices
taken at different positions in 20 scans (Fig. 3). The Spearman’s rank correlation
coefficient between estimated cross-sectional tissue area (cm2 ) and the truth was
0.91, 0.75, 0.99, and 0.99 for skeletal muscle, psoas, VAT and SAT, respectively.
Fig. 4 shows qualitative segmentation results at different position slices from one
CT. Although there are large shape variations in axial slices taken at different
positions, the automated segmentations of muscle, VAT and SAT region match
well with the manual label. Failures of the psoas segmentation occur on the slices
taken at the bottom position, where the psoas muscle region is very small anddifficult to detect (see Fig. 4, row 1, column 3).
In Fig. 5, we used the Spearman rank correlation to compare the estimatedtissue area by the proposed method with the ground truth on axial slices taken
at L3 position from 40 testing scans. Average tissue areas of three and five axial
slices centered at L3 are also measured.
In Tab. 1, we compared our method with the baseline method [8] on 40 L3 axial slices from testing scans. Both the proposed method and baseline focused onwhole CT scan segmentation and used slice-wise algorithms. As shown in Tab. 1,
our method achieved significantly (p < 0.001) higher Dice similarity index (Dice)values for the skeletal muscle, VAT and SAT using Wilcoxon signed rank test.

32

Hu et al.

Fig. 4. Qualitative results on different axial slices from one subject. Manual label(green), automated label (red) and overlap (yellow).

4

Conclusions

We presented an automated workflow to segment the skeletal muscle, psoas muscle, visceral adipose tissue and subcutaneous adipose tissue from clinically acquired abdominal CT scans. The results on 40 subjects from two separate clinicaldatasets demonstrated that the proposed framework was able to achieve 0.854,
0.740, 0.887 and 0.933 Dice similarity coefficient for the skeletal muscle, psoas
muscle, VAT and SAT, respectively. The future work can be the improvementof the proposed method by using shape prior in psoas muscle segmentation.
Acknowledgements. This research was supported by NIH 1R03EB012461, NIH 2R01
EB006136, NIH R01EB006193, NIH P30 CA068485, NIH R01 HL 098445 (PI - Carr),and AUR GE Radiology Research Academic Fellowship, and in part using the resourcesof the Advanced Computing Center for Research and Education (ACCRE) at Vanderbilt University, Nashville, TN. This project was supported in part by VISE/VICTR
Table 1. Comparison of tissue area estimation by the proposed method andZhang et al. [8] on 40 L3 axial slices.
SkeletalmuscleTissue2

Psoas
muscleVAT

SAT

Manual area (cm )142.7 ± 35.0
15.6 ± 5.9
168.4 ± 97.5
224.6 ± 126.5
Proposed Dice
0.854 ± 0.110 0.740 ± 0.259 0.887 ± 0.075 0.933 ± 0.046
Proposed area error (cm2 )17.1 ± 14.9
4.5 ± 7.1
12.0 ± 14.0
15.5 ± 14.4
Zhang et al. [8] Dice
0.758 ± 0.128−
0.828 ± 0.054 0.852 ± 0.054
Zhang et al. [8] area error (cm2 )46.8 ± 27.7−
46.6 ± 22.1
44.0 ± 21.7
Proposed vs. Zhang et al. [8] Dice p-value
3.3 × 10−5−
2.4 × 10−5
5.6 × 10−8−6−7
Proposed vs. Zhang et al. [8] area error p-value
3.8 × 10−
7.6 × 10
1.6 × 10−6

Characterization of Body Composition and Frailty with CT

33

Fig. 5. Scatter plots showing the correlation between estimated tissue area by the
proposed method and the ground truth. The first row shows tissue area estimated on
one slice at middle L3 position. The second and third row show the average tissue area
estimated at three and five slices at the L3 position, respectively.

VR3029 and the National Center for Research Resources, Grant UL1 RR024975-01,
and is now at the National Center for Advancing Translational Sciences, Grant 2 UL1TR000445-06. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. This research was supported in part by
the National Natural Science Foundation of China (Grant No. 91630311) and the Fundamental Research Funds for the Central Universities (Grant No. 2017XZZX007-02).

References
1. Yip, C., Dinkel, C., Mahajan, A., Siddique, M., Cook, G., Goh, V.: Imaging bodycomposition in cancer patients: visceral obesity, sarcopenia and sarcopenic obesity
may impact on clinical outcome. Insights Imaging 6(4), 489–497 (2015)
2. Lee, S., Gallagher, D.: Assessment methods in human body composition. Curr.
Opin. Clin. Nutr. Metab. Care 11(5), 566–572 (2008)
3. Popuri, K., Cobzas, D., Esfandiari, N., Baracos, V., Jagersand, M.: Body composition assessment in axial CT images using FEM-based automatic segmentation ofskeletal muscle. IEEE Trans. Med. Imaging 35(2), 512–520 (2016)
4. Idoate, F., Cadore, E., Casas-Herrero, A., Zambom-Ferraresi, F., Marcellán, T., deGordoa, A., Rodriguez-Mañas, L., Bastarrika, G., Marques, M., Martı́nez-Velilla,N.: Adipose tissue compartments, muscle mass, muscle fat infiltration, and coronary calcium in institutionalized frail nonagenarians. Eur. Radiol. 25(7) (2015)

34

Hu et al.

5. Yao, J., Sussman, D., Summers, R.: Fully automated adipose tissue measurement
on abdominal CT. In: Weaver, J., Molthen, R. (eds.) Proc. SPIE Medical Imaging
2011: Biomedical Applications in Molecular, Structural, and Functional Imaging,vol. 7965, p. 79651Z. SPIE (2011)
6. Inoue, T., Kitamura, Y., Li, Y., Ito, W., Ishikawa, H.: Psoas major muscle segmentation using higher-order shape prior. In: Menze, B., et al. (eds.) Proc. MICCAIInternational Workshop on Medical Computer Vision: Algorithms for Big Data –MCV 2015, LNCS, vol. 9601, pp. 116–124. Springer (2015)
7. Chung, H., Cobzas, D., Birdsell, L., Lieffers, J., Baracos, V.: Automated segmentation of muscle and adipose tissue on CT images for human body composition
analysis. In: Miga, M., Wong, K. (eds.) Proc. SPIE Medical Imaging 2009: Visualization, Image-Guided Procedures, and Modeling, vol. 7261, p. 72610K. SPIE
(2009)
8. Zhang, W., Liu, J., Yao, J., Summers, R.: Segmenting the thoracic, abdominaland pelvic musculature on CT scans combining atlas-based model and active contour model. In: Novak, C., Aylward, S. (eds.) Proc. SPIE Medical Imaging 2013:
Computer-Aided Diagnosis, vol. 8670, p. 867008. SPIE (2013)
9. Xu, Z., Conrad, B., Baucom, R., Smith, S., Poulose, B., Landman, B.: Abdomen
and spinal cord segmentation with augmented active shape models. J. Med. Imaging 3(3), 036002 (2016)
10. Modat, M., Ridgway, G., Taylor, Z., Lehmann, M., Barnes, J., Hawkes, D., Fox, N.,
Ourselin, S.: Fast free-form deformation using graphics processing units. Comput.
Methods Programs Biomed. 98(3), 278–284 (2010)
11. Wang, H., Suh, J., Das, S., Pluta, J., Craige, C., Yushkevich, P.: Multi-atlas segmentation with joint label fusion. IEEE Trans. Pattern Anal. Mach. Intell. 35(3),611–623 (2013)
12. Yuan, J., Bae, E., Tai, X.C.: A study on continuous max-flow and min-cut approaches. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition– CVPR 2010. pp. 2217–2224. IEEE (2010)
13. Yuan, J., Ukwatta, E., Tai, X., Fenster, A., Schnörr, C.: A fast global optimizationbased approach to evolving contours with generic shape prior. UCLA Tech. Reportp. CAM 12 38 (2012)

OpenRib Clinical Application
Catalina Tobon-Gomez1( ) , Tyler Stroud1 , John Cameron1 , Dave Elcock1 ,Andrew Murray1 , Daniel Wyeth1 , Chris Conway1 , Steven Reynolds1 ,Pedro Augusto Gondim Teixeira2 , Alain Blum2 , and Costas Plakas1
1

2

Toshiba Medical Visualization Systems, UK
Service d’Imagerie Guilloz, CHRU Nancy, France
ctobon-gomez@tmvse.com

Abstract. The recommended exam for assessing chest trauma is a computed tomography (CT) chest scan. Using multi-planar reconstructions
to evaluate a CT volume to assess the ribcage is a tedious and timeconsuming task. We have designed an application that provides an automatically rendered unfolded unobstructed view of the entire ribcage:
OpenRib. This paper describes the underlying algorithm which has two
main steps: ribcage segmentation and ribcage unfolding. The unfolding
technique we developed preserves the relative size and location of theribs and surrounding tissue, providing a natural anatomical reference
for the reader. It also demonstrated usefulness to identify other musculoskeletal conditions such us scoliosis, calcified cartilage, bone tumours.
To evaluate the usefulness of the application, we evaluated it on 70 representative CT chest scans. The evaluation was performed by a clinicalexpert who graded the specialized rib view on a 5 point Likert scale according to the level of diagnostic confidence. Results showed that 84%
of the studies were clinically useful (above grade 3). The algorithm isfully automatic and it runs in an average time of 24 s. The evaluation
described in this paper gives positive initial feedback on the usefulnessof the application. A multi-user clinical study is under way to evaluate:
1) sensitivity and specificity of fracture detection, and 2) reading timewith respect to conventional multi-planar reconstruction views.
Keywords: Rib fractures, clinical application, unfolded ribcage.

1

Introduction

1.1

Clinical Motivation

Injury-related emergency hospital visits reach up to 37 million in the UnitedStates [1] and 39 million in the European Union [2] each year. Fractured ribs
are the most common injury in chest trauma. Therefore, it is mandatory toevaluate all ribs in any computed tomography (CT) image of the chest, and it isof particular importance in trauma patients as 25% of injury-related deaths arecaused by chest trauma [3].
According to radiological guidelines the recommended exam for assessing
chest trauma is a CT chest scan. Diagnosing rib fractures using a CT scan is
B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proc. 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. 35−45, 2017.

36

Tobon-Gomez et al.

a convenient single diagnostic exam, but it is tedious and time-consuming forthe reader. Each rib follows a diagonal orientation covering multiple CT slices.
Conventional tools require the user to continually create and adjust obliquemulti-planar reconstructions (MPRs) due to the curvature of the ribs. Moreover,the full evaluation is performed one rib at a time from side to side.
This has motivated us to develop a clinical application that provides anautomatically rendered unfolded unobstructed view of the entire ribcage. This
specialized view can be browsed as a volume, allowing the clinician to quickly
assess the ribcage [4]. The clinical application is interactive, allowing the user
to triangulate the position of suspected fractures from the unfolded view to theconventional MPR views to confirm diagnosis. We call it OpenRib.
1.2

State of the Art

Currently there are two commercial applications with a similar purpose. On the
one hand, Siemens’ syngo.CT application traces the centreline of each rib todefine a curved planar reformation (CPR) image for each rib [5]. The displayed
image is a composite of the collection of CPR images of the ribs plus a CPRof the spine. The advantage of this approach is that non-displaced fractures areeasily visible. The disadvantage is that straightening the ribs complicates the understanding of their relative position and hides other musculoskeletal conditions(e.g., scoliosis).
On the other hand, Carestream’s Radial View application provides an unfolded maximum intensity projection of the ribcage. The advantage of this approach is that a projection gives an excellent overview of ribcage. The disadvantage is that maximum intensity projection of ribs hides non-displaced fractures.
Our solution makes improvements on the state of the art by: 1) removing
obstructing objects from the projection, 2) improve sensitivity to subtle nondisplaced fractures, and 3) preserving the relative size and location of the ribs
and surrounding tissue.

2

Data

Landmarks Classifier Training. We used a retrospective in-house database of
369 CT datasets for landmark classifier training. On these datasets, key anatomical markers were collected as ground truth. The ground-truth covers a wide rangeof anatomies, contrast and non-contrast acquisitions, from multiple scanner vendors.
Segmentation Atlas. Since the main requirement for this tool is computational
efficiency, we used a single non-contrast CT dataset as segmentation atlas. We
selected an average size individual with no fractures or other evident pathology.
On this atlas, several structures were manually segmented by a clinical expert:
ribs, spine, sternum, scapulae, clavicles, kidneys. Additionally, anatomical markers were manually collected in the thorax to aid initial alignment (see Sec. 3.1).

OpenRib Clinical Application

37

Fig. 1. Overview of the ribcage segmentation process (part 1).

Testing Database. We used a database of 70 CT studies provided by ourclinical collaborators. From this database, we used 44 datasets for algorithm
optimisation (seen datasets), leaving 26 datasets for blinded evaluation (unseen
datasets). From the seen datasets, six datasets were manually labelled for ribs
and spine to provide a quantitative reference.

3

Methods

3.1

Ribcage Segmentation

Anatomical Landmarks Detection. Using technology developed within our
research group [6, 7], we are able to accurately and efficiently detect anatomical
landmarks on novel datasets of any anatomical location (see Fig. 1). The method
localizes landmarks using a random forest classifier comprising long-range intensity features [8], histogram of oriented gradients (HOG) features [9], and atlaslocation features. The classifier was trained off-line on a large sample of data
(see Sec. 2).
Thorax Bounding Box. To bound the segmentation of the ribs, we predefine
a thorax box in a synthetics atlas. Using the landmarks detected on the novel
volume (previous step), we compute a transformation from synthetic atlas space
to novel patient space. The bounds are mapped using this transformation. Everything outside the thorax bounds is discarded from subsequent processing (seeFig. 1).

38

Tobon-Gomez et al.

Fig. 2. Overview of the ribcage segmentation process (part 2). Note in the bottom
row the areas where vessels/cables were removed (yellow arrows).

Estimate Threshold Value. Even though CT intensity values are relatively
calibrated, the values may change in the presence of contrast or pathology. To
obtain an initial mask with a raw segmentation of the bone, we vary the thresholdvalue in a case-by-case basis. To define this value, we create a sphere centredat each of the thorax skeletal landmarks and sample intensity values within it.
We then compute the 70th percentile of the sampled values. Using this value
as a lower threshold, we obtain a mask that contains everything from bones
to vessels/cables. In certain cases, it also includes lung fluid, heart chambers,
kidneys and/or liver (see Fig. 1).
Atlas Based Segmentation. The landmarks detected on the novel dataset
are used to initialize the atlas based segmentation. We use non-rigid registration
based on a demons algorithm [10] to warp the segmentation atlas to the novel
dataset. With the obtained warp field, we warp the wanted labelled masks: ribs,
spine and sternum. These three labelled masks will be used on following processing steps. Additionally, we remove from the initial raw mask any identified object
that is not a rib: clavicles, scapulae, kidneys, spine and sternum (see Fig. 1). This
outputs a clean mask that contains mainly ribs with other non-identified objects(e.g., vessels/cables).

OpenRib Clinical Application

39

Fig. 3. Overview of the ribcage unfolding process.

Rib Tracing. Due to the similarity of ribs with each other, it is a challenging
task for the registration alone. As a result, the warped ribs will be missing
sections, specially distally to the spine. To complete the ribs we run a tracing
algorithm that adds the missing sections. The tracing starts with the warped
ribs as seeds and searches for a path forwards (i.e., away from the spine) anddownwards. This naive tracing approach allows to keep computation time low.
As a disadvantage, the tracing will stop at the presence of fractures. This is
mitigated by seeding all possible sections of ribs with the wrapped rib. The pathcan follow any location along the full thickness of the potential rib (i.e., it is
not constrained to the centrelines). As a final stage, the paths are thickened bydilation and intersection with the input mask (see Fig. 2).
Vessel/Cable Removal. Despite the multiple steps above, portions of vessels
and/or cables may remain in the rib mask (specially in contrasted studies). To
remove this portions of vessel/cables, we run a vesselness filter [11] and extract
only areas with very high response. We then find connected components withvery high intensity values and remove them from the initial rib mask. We finalize
with morphological operations to polish the ribs after removal (see Fig. 2).
3.2

Ribcage Unfolding

In order to unfold the ribcage, we want to find a three-dimensional manifoldthat intersects all the structures of interest and defines the points to be sampledfrom the volume. To generate this manifold, we define a camera axis along thepatient’s superior-inferior axis. The axis is located inside the thorax. We use thesegmented spine and sternum to define a top coordinate and a bottom coordinateof the camera axis.
Along this axis, we define a collection of points O(1...n ). Each point corresponds to a horizontal row of pixels in the output image. For each point Oi ,we define a perpendicular plane Pi and tilt it based on an angle α. The tilting
makes the ribs more horizontal in the output image. We then define a collectionof rays in plane Pi , with origin Ai , rotating in the plane around Oi . Each raycorresponds to a column of pixels in the output image. We intersect each ray

40

Tobon-Gomez et al.

with the segmented ribs (see Sec. 3.1) and choose a mid point. At that position
the intensity is sampled and displayed in the two-dimensional (2D) image. Rays
that do not intersect are interpolated from neighbouring positions (see Fig. 3).

4

Evaluation

4.1

Qualitative Results

We evaluated the clinical application in 70 representative CT chest scans. The
qualitative evaluation was performed by a clinical expert who graded the specialized rib view on a 5 point Likert scale according to the level of diagnostic
confidence. This grading scale followed the criteria below.
• Gold standard - grade 5: The reader can confidently make a diagnosis
without any review of the MPRs. No artifacts present.
• Diagnostic confidence - grade 4: The reader can confidently make a diagnosis with limited or no review of the MPRs. Little to no artifacts present
and there are no artifacts obstructing any relevant anatomy.
• Moderate confidence - grade 3: Helpful to make a diagnosis but a clinician would need the MPRs for confirmation. Some artifacts are present but
do not obscure significant areas of relevant anatomy.
• Low level of confidence - grade 2: Unacceptable. Minimally helpful andartifacts render the projection unacceptable.
• Very low level of confidence - grade 1: Indisputably unacceptable.
The evaluation database was split in seen datasets and unseen datasets (as explained in Sec. 2). Results of the grading can be found in Tab. 1. Fig. 4 shows thelayout of the clinical application. This layout is maintained on Fig. 5 and Fig. 6
which include illustrative example cases.
Table 1. Summary of qualitative evaluation.
Scoring results on datasets∗
Grade
5
4
3
2
1
∗

Description
Gold standard
Diagnostic confidence
Moderate confidence
Low level of confidence
Very low level of confidenceSeen(n = 44)

Unseen(n = 26)All(n = 70)

0.0
58.7
28.3
10.9
2.2

0.0
69.2
7.7
7.7
15.4

0.0
63.4
21.1
9.9
5.6

in percentage w.r.t. total number of cases.
Bolded cases considered clinically useful = 84.5%.

OpenRib Clinical Application

41

Fig. 4. Layout of the clinical application: the top displays the unfolded OpenRib view.
The bottom displays three conventional MRPs (axial, sagittal, coronal) and a single
oblique multi-planar reconstruction. Use this figure as guidance for Fig. 5 and Fig. 6.

4.2

Quantitative Results

From the seen datasets, six cases were manually labelled to provide a quantitative reference. The labels were separated as ribs and spine. We compared theautomatically generated masks with the manually generated masks using thefollowing metrics: DICE (overlap), sensitivity and specificity. The masks wereclipped to the same thorax bounding box for metric calculation. The results are
summarized in Tab. 2.
4.3

Discussion

Results showed that overall 84% of the studies were clinically useful without
any user interaction (summing cases above grade 3 in all datasets). Low scores
were mainly due to segmentation inaccuracies. Some cases still include remainsof the scapulae or contrast bolus (vessels) that obscure the ribs. We plan to add
Table 2. Summary of quantitative evaluation.
Ribs
Case

DICE

1
2
3
4
5
6

0.692
0.748
0.663
0.758
0.688
0.722

Spine

Sensitivity Specificity
0.982
0.994
0.995
0.995
0.996
0.993

0.982
0.987
0.981
0.987
0.984
0.985

DICE

Sensitivity Specificity

0.843
0.897
0.910
0.936
0.920
0.923

Masks clipped to the same thorax bounding box for metric calculation.

0.837
0.944
0.932
0.933
0.933
0.915

0.974
0.983
0.989
0.991
0.986
0.990

42

Tobon-Gomez et al.

Distal fracture, grade 4

Proximal fracture, grade 4

Cartilage calcification, grade 3

Fig. 5. Snapshots of two cases with fractures (top and middle) and a case with cartilage
calcification (bottom).

OpenRib Clinical Application

43

Removed vessels, grade 4

Removed cables, grade 3

Scoliosis, grade 3

Fig. 6. Snapshot of a case with successfully removed vessels (top), a case with successfully removed cables (middle) and a case with scoliosis (bottom).

44

Tobon-Gomez et al.

Axial

CoronalSagittalFig. 7. Case 1 from Tab. 2. We show the manual labels of the spine (blue) and ribs
(green). The differences between the automatic and the manual labels are displayed inyellow. Notice that most of the differences are due to thickness, costal cartilage andmorphological operations (specially in the spine).

editing tools for the user to remove these type of segmentation errors. Finally,
the unfolding generates a new type of view unfamiliar to clinicians. This explains
the lack of gold standard cases (grade 5). Therefore, the application was designed
to use the specialized rib view for quick assessment and navigation, while thefractures can be confirmed on the conventional MPR views.
Regarding the quantitative results, the dice scores for the spine were higher
than scores for the ribs. However, the sensitivity and specificity values of the ribs
were high. This can be explained by two aspects: 1) the automatic segmentation
obtained slightly thicker ribs than the manual segmentation, and 2) the stopping
point of the rib (towards the sternum) can be arbitrarily decided, hence including
more or less of the costal cartilage (see Fig. 7).
As with any clinical application, computation time is a very important factor.
Our application runs in an average time of 24 s, as measured in 48 cases in apersonal computer with similar specifications to a radiology work-station.
The evaluation described in this paper gives positive initial feedback on theusefulness of the application. A multi-user clinical study is under way to evaluate:
1) sensitivity and specificity of fracture detection, and 2) reading time withrespect to conventional MPR views.

5

Conclusion

We developed an application that generates a 2D image that displays the entire
ribcage. The application was evaluated in 70 chest CT datasets.
The unfolding technique we developed preserves the relative size and locationof the ribs and surrounding tissue, providing a natural anatomical reference forthe reader. It also demonstrated usefulness to identify other musculoskeletal
conditions such us scoliosis, calcified cartilage, bone tumours.

OpenRib Clinical Application45

We believe using the specialized rib view the reader will see improvements
in diagnostic accuracy and a reduction in reading time relative to conventional
multi-planar reconstructions. Further studies are under way to validate this hypothesis.

References
1. Hospital ambulatory medical care survey: 2013 emergency department summary
tables (2013)
2. EuroSafe: Injuries in the European Union: summary of injury statistics for theyears 2008–2010 (2013)
3. Calhoon, J., Trinkle, J.: Pathophysiology of chest trauma. Chest Surg. Clin. N.
Am. 7(2), 199–211 (1997)
4. Gondim Teixeira, P., Blum, A.: Clinical application of musculoskeletal CT: trauma,
oncology, and postsurgery. pp. 1–27. Medical Radiology, Springer (2017)
5. Ringl, H., Lazar, M., Töpker, M., Woitek, R., Prosch, H., Asenbaum, U., Balassy,
C., Toth, D., Weber, M., Hajdu, S., Soza, G., Wimmer, A., Mang, T.: The ribs
unfolded - a CT visualization algorithm for fast detection of rib fractures: effect on
sensitivity and specificity in trauma patients. Eur. Radiol. 25(7), 1865–1874 (2015)
6. Dabbah, M., Murphy, S., Pello, H., Courbon, R., Beveridge, E., Wiseman, S.,
Wyeth, D., Poole, I.: Detection and location of 127 anatomical landmarks in diverse
CT datasets. In: Ourselin, S., Styner, M. (eds.) Proc. SPIE Medical Imaging 2014:
Image Processing, vol. 9034, p. 903415. SPIE (2014)
7. O’Neil, A.: Image Analysis of Noisy 3D/4D Medical Datasets. Ph.D. thesis, HeriotWatt University, UK (2015)
8. Criminisi, A., Shotton, J. (eds.): Decision Forests for Computer Vision and Medical
Image Analysis. Advances in Computer Vision and Pattern Recognition, Springer(2013)
9. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In:
Proc. IEEE Conference on Computer Vision and Pattern Recognition – CVPR
2005. vol. 1, pp. 886–893. IEEE (2005)
10. Crum, W., Griffin, L., Hill, D., Hawkes, D.: Zen and the art of medical image
registration: correspondence, homology, and quality. NeuroImage 20(3), 1425–1437(2003)
11. Frangi, A., Niessen, W., Vincken, K., Viergever, M.: Multiscale vessel enhancement
filtering. In: Wells, W., et al. (eds.) Proc. 1st International Conference on Medical
Image Computing and Computer-Assisted Intervention – MICCAI 1998, LNCS,
vol. 1496, pp. 130–137. Springer (1998)

3D Cobb Angle Measurements from Scoliotic
Mesh Models with Varying Face-Vertex Density
Uroš Petković1 , Robert Korez1( ) , Stefan Parent2 ,Samuel Kadoury3 , and Tomaž Vrtovec1
1

University of Ljubljana, Faculty of Electrical Engineering, Slovenia
2
CHU Sainte-Justine & University of Montréal, Canada
3
CHU Sainte-Justine & Polytechnique Montréal, Canada
robert.korez@fe.uni-lj.si

Abstract. To evaluate spinal deformities, the Cobb angle is the main
diagnostic parameter that is usually measured on two-dimensional coronal radiographic (X-ray) images. In this paper, we propose a method
for the evaluation of the three-dimensional (3D) Cobb angle from 3D
spine mesh models with varying face-vertex density. For the upper-end
and lower-end vertebra mesh models, the location of the vertebral body
center and mesh faces that belong to the vertebral body surface are identified by unsupervised classification of mesh faces of the vertebral body,which serve only as training data, and subsequent supervised classification of all mesh faces. Adjacent mesh faces are then labeled with thesame class, and after comparison to mesh faces in the training data, welabel the mesh faces of the superior and inferior vertebral endplate. Finally, planes are fitted to the superior endplate of the upper-end vertebra
and the inferior endplate of the lower-end vertebra, which define the 3D
Cobb angle. The method was tested on 60 triangular mesh models of
the scoliotic spine, and each mesh model was generated at 17 different
face-vertex densities. For meshes with the mean face edge length below
6 mm, the proposed method was accurate, with mean absolute error of
3.0◦ and corresponding standard deviation of 2.2◦ when compared toreference measurements.
Keywords: Spine modeling, adolescent idiopathic scoliosis, triangularmesh models, automated measurements.

1

Introduction

Scoliosis is one of the most common spinal deformities, described as an abnormal
lateral and rotational curvature of the spine [1]. One of the earliest methods forquantitative evaluation of spinal curvature was proposed by Cobb in 1948 [2],and is referred to as the Cobb angle. It is usually evaluated from two-dimensional(2D) anteroposterior radiographic (X-ray) images of the spine as the angle between the line along the superior endplate of the upper-end vertebra and the line
along the inferior endplate of the lower-end vertebra of the deformity (Fig. 1(a)).
Although several other methods for quantitative evaluation of spinal curvature
B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proc. 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. 46−55, 2017.

3D Cobb Angle Measurements

(a)

47

(b)

Fig. 1. (a) An illustration of the Cobb angle measurement between the superior endplate of the upper-end vertebra and the inferior endplate of the lower-end vertebra.
(b) A flowchart of the proposed algorithm. The inputs of the algorithm are triangularmesh models of the upper-end and lower-end vertebra. For each model we estimate
the parameters of the planes that are fitted to superior and inferior endplate of theupper-end and lower-end vertebra, respectively, and the Cobb angle is then estimatedfrom corresponding plane normal vectors.

have been proposed since [3], the Cobb angle is still the most preferred andestablished diagnostic parameter for evaluating scoliotic deformities.
Adolescent idiopathic scoliosis is the most common type of scoliosis, and it
appears more likely to girls than boys with a 6 : 1 ratio, and with a 10 : 1 ratio insevere cases [4]. The main diagnostic criterion for scoliosis is that the Cobb angle
exceeds 10◦ on an anteroposterior spine radiograph. Nowadays, however, withthe availability and common usage of three-dimensional (3D) imaging techniques,
such as computed tomography (CT) or magnetic resonance (MR), measurementsof the Cobb angle are often performed in 3D, which proved to be superior tomeasurements performed in 2D [3]. To address this issue, planes are, insteadof lines, defined along the vertebral endplates, which may be very challenging
considering the 3D nature of the images and a relatively complex structure of
the spine. On the other hand, automated measurements based on image analysis
and processing techniques usually require prior segmentation of vertebrae toperform quantitative measurements. As the results of vertebra segmentation arebinary masks that can be represented as triangular mesh models in 3D, it is alsovaluable to develop algorithms that perform measurements based on 3D meshmodels.
Recently, Huo et al. [5] described a method for the 3D Cobb angle from CT
images using a mesh model of the spine. However, their method is limited by

48

Petković et al.

the manual selection of end vertebrae of the deformity as well as by the manual
determination of the initial seed vertex of the mesh. In this paper, we propose
an alternative algorithm for the evaluation of the 3D Cobb angle from 3D spine
mesh models.

2

Methodology

The proposed algorithm for 3D Cobb angle measurement is divided into sevensteps that are shown in Fig. 1(b). The inputs of the algorithm are the upperend and lower-end vertebra mesh models. For each mesh model we estimate thelocation of the vertebral body center that is further used to identify the meshfaces that belong to the vertebral body surface. For this purpose, we perform
unsupervised classification of mesh faces of the vertebral body that serve only as
training data for the subsequent supervised classification of all mesh faces of thevertebra. Adjacent mesh faces are then labeled with the same class, and after
comparison to mesh faces in the training data, we label the mesh faces of thesuperior and inferior vertebral endplate. Finally, planes are fitted to the superior
endplate of the upper-end vertebra and the inferior endplate of the lower-end
vertebra, which define the 3D Cobb angle.
2.1

Vertebral Body Center Detection

The proposed detection of the vertebral body center is, with the difference that
it is adapted for a 3D mesh model, based on the work of Štern et al. [6], who
proposed an automated algorithm for the detection of the spinal centerline in
CT and MR spine images. As a 3D mesh model of the vertebral body is a closed
surface, any line orthogonal to any mesh face intersects with the vertebral surface
at least twice, i.e. in a pair of two opposite faces that intersect with this line
when it passes inside and outside of the vertebral body. As the vertebral body
is approximately a cylindrical structure, these opposite mesh faces have normal
vectors with approximately opposite directions. Moreover, it is expected that thecenterline of the vertebral body is in the middle of each line connecting any two
opposite vertebral body mesh faces. However, as an arbitrary line can intersectwith the vertebral surface more than twice (i.e. passes further through pedicles
and processes), more candidates for the opposite face can exist.
The 3D mesh model of the k-th vertebra is a triangular mesh structure composed of a vertex set Vk and a face set Fk :no
Vk = vi | i = 1, 2, . . . , Nv ,
n
o
n
o
(1)
Fk = fi | i = 1, 2, . . . , Nf , fi = vp , vq , vr .
For each mesh face fi we perform a search in 3D for the opposite mesh face fi∗ ,and the candidates for the opposite mesh face are all faces fj ∈ FC,i that fulfill
the following three criteria:

3D Cobb Angle Measurements

49• normal vector nj of the candidate opposite face fj is oriented in the approximate opposite direction of normal vector ni of face fi ;• the Euclidean distance from the center ci of the face fi to cj is between
dvb,min and dvb,max ;• the Euclidean distance from cj to line li = ci + t · ni , t ∈ R, is less thenrvb,min .
Among all candidates in FC,i , the opposite mesh face fi∗ is the one that yields
the minimal dot product between normal vectors ni and nj of faces fi and fj ,respectively:fi∗ = arg min dot(ni , nj ) , j = 1, 2, . . . , J,(2)
nj →fjwhere J is the number of candidate mesh faces in FC,i . The vertebral body center
is located where the lines connecting every pair {fi , fi∗ } of opposite mesh faces
most often intersect. For the purpose of finding the location these intersections,a new 3D image is initialized with zero values in the same coordinate system as
the observed vertebra – the 3D accumulator A. Each line connecting a pair of
opposite mesh faces {fi , fi∗ } is first transformed with the Bresenham algorithm[7] to determine M discrete points pm = (xm , ym , zm ), m = 1, 2, . . . , M , along
the line, and then assigned a weighting function, which is normally distributed
according to the Euclidian distance D between the centers ci and c∗i of the
opposite mesh faces, and scaled by the dot product dot(ni , n∗i ) between thecorresponding normal vectors ni and n∗i . The 3D accumulator value A(pm ) isthen increased by the value of the weighting function at each point pm , m =1, 2, . . . , M , along the connecting line:!
2(d(ci , pm ) − D/2)∗
A(pm ) = A(pm ) − dot(ni , ni ) · exp −,(3)
2(D/6)2

where d(ci , pm ) is the Euclidian distance between ci and pm .The resulting 3D accumulator A represents a probability map of likelihoods of image voxels to belong
to the vertebral body center. However, the highest likelihoods may be locatedalso within the vertebral processes due to their cylindrical shape. To address thisissue, the 3D accumulator is filtered with a mean filter of size KX × KY × KZ ,determined according to the minimal average size of the human vertebral body.
The center of the vertebral body cb is identified as the voxel with the highest
filtered value of the 3D accumulator.
2.2

Superior and Inferior Endplate Labeling

Mesh vertices are first transformed from the image-based coordinate system V
to the coordinate system of the mesh V ∗ by means of the principal component
analysis (PCA). Vectors {e∗x , e∗y , e∗z } that define the new coordinate system areprincipal components estimated by PCA, which rotates the vertebra mesh modelso that e∗z is nearly parallel with normal vectors to the vertebral body endplates.

50

Petković et al.

For the purpose of labeling the superior and inferior endplate of the given
3D vertebra mesh model, mesh faces that belong to the vertebral body surface
have to be identified. By following a set of rays that are represented as cylinders
originating from the detected vertebral body center, we can identify whether thevertebral body center is located within these cylinders. By using the spherical
Fibonacci mapping, we first yield a set of points on a unit sphere with a nearly
uniform distribution. These point sets represent unit vectors of the set of rays,
and each unit vector defines the orientation of the corresponding cylinder ofradius rc and height hc , where hc is the maximum Euclidean distance from thedetected vertebral body center cb . As mesh faces fcyl,j ∈ Fcyl with corresponding
centers ccyl,j that are located within the cylinder are also candidates for thevertebral body surface, we select for each i-th ray face fbody,i with the minimal
Euclidean distance between its center and the detected vertebral body center cb :fbody,i = arg min d(cb , ccyl,j ) , j = 1, 2, . . . , J.
(4)ccyl,i →fcyl,iThe resulting mesh faces fbody,i ∈ Fbody represent the boundaries of the vertebral
body, i.e. its surface.
However, as the vertebral body is not a fully closed surface, some faces may
not belong to the vertebral body but to other vertebral structures. On the other
hand, there are mesh faces of the vertebral body that may not be detected. We
can predict where the vertebral body surface is by least square ellipsoid fitting
to the set of centroids Cbody of mesh faces Fbody in the PCA-based coordinate
system, resulting in three eigenvectors {e1 , e2 , e3 } and ellipsoid radii {r1 , r2 , r3 }.
Due to the elliptical shape of vertebral endplates, ellipsoid fitting is most accurate
in the X ∗ − Y ∗ plane. From the cross products of all pairs of eigenvectors, wefind the pair for which its cross product is most parallel to the Z ∗ -axis, i.e. tothe unit vector (0, 0, 1), by:{e∗i , e∗j } = arg max |dot (cross(ei , ej ), (0, 0, 1))| , i, j ∈ {1, 2, 3}, i 6= j. (5){ei ,ej }From the projecting ellipse, defined by eigenvectors {e∗i , e∗j } and correspondingradii {ri∗ , rj∗ } to the X ∗ − Y ∗ plane, we determine the vertebral body surface.
Each mesh fbody,i ∈ Fbody (4), for which the projection of the center c∗body,i to theX ∗ − Y ∗ plane is located within the projecting ellipse, is included the training
set of mesh faces Ftrain for the determination of vertebral endplates (Fig. 2(a)).
Mesh faces Ftrain (5) can be, again by assuming that the vertebral body iscylindrically shaped, classified in three groups: faces FB,T OP of the top ellipticalplanar surface (i.e. the superior endplate), faces FB,BOT T OM of the bottom elliptical planar surface (i.e. the inferior endplate), and faces FB,SIDE of the curved
surface (i.e. the vertebral body wall). If the main axis of the cylinder representing the vertebral body is aligned with the direction of the Z ∗ axis, then normal
vectors (n∗B,x , n∗B,y , n∗B,z ) of FB,T OP , FB,BOT T OM and FB,SIDE are representedby (0, 0, 1), (0, 0, −1) and (x∗ , y ∗ , 0), respectively. The appropriate feature forclassification of mesh faces Ftrain is therefore n∗B,z , i.e. the Z ∗ component of

3D Cobb Angle Measurements

(a)

51

(b)

Fig. 2. (a) An illustration of mesh faces Fbody (green and blue) and Ftrain (gray andblue). [Note that the blue color represents the intersection of Fbody and Ftrain .] (b) An
illustration of mesh faces FB,T OP (blue), FB,BOT T OM (green) and FB,SIDE (gray)estimated with k-Means++.

the normal vector, which is performed by the unsupervised learning method kMeans++ [8]. However, as the vertebral body is not perfectly cylindrical andnot perfectly aligned with the Z ∗ axis, values n∗B,z are arranged around corresponding cluster centers with some variance. To avoid poor clustering, we seed
the initial values of clusters by choosing (1, −1, 0) for the initial cluster centers
as the main axis of the vertebral body is nearly aligned with the Z ∗ axis. To
emphasize the distances between cluster centers, we transform n∗B,z → n0B,z witha function f : n0B,z = f (n∗B,z ), which should be an odd function to preserve
the sign of n∗B,z . By applying k-Means++ classification we therefore yield threeclusters and corresponding cluster centers. The cluster with the highest value ofthe cluster center represents faces FB,T OP , while the cluster with lowest valueof the cluster center represents faces FB,BOT T OM (Fig. 2(b)).

(a)

(b)

Fig. 3. (a) An illustration of mesh faces FV B,T OP (blue), FV B,BOT T OM (green) and
FV B,SIDE that were estimated with a linear classifier. (b) An illustration of mesh faces
FSU P (blue) and FIN F (green) estimated from faces FV B,T OP and FV B,SIDE .

52

Petković et al.

(a)

(b)

Fig. 4. (a) An illustration of the plane estimated with the random sample consensus(RANSAC) algorithm on mesh faces FSU P of the superior endplate of the upper-end
vertebra (top) and FIN F of the inferior endplate of the lower-end vertebra (bottom).
(b) An illustration of the 3D Cobb angle measurement.

In the next step, we classify mesh faces that belong to the whole vertebra insets FV B,T OP , FV B,BOT T OM and FV B,SIDE according to n0V B,z (Fig. 3(a)). By
applying a linear classifier, adjacent faces (i.e. faces that share a common edge)that belong to the same class are collected into groups of faces FA,1 , FA,2 , . . . , FA,J .
The groups that results in the largest number of adjacent faces from the training
data represent the superior and inferior endplate (Fig. 3(b)):

FSU P = arg max count(FA,j ∩ FB,T OP ) , j = 1, 2, . . . , J,
FA,j



FIN F = arg max count(FA,j ∩ FB,BOT T OM ) ,(6)

j = 1, 2, . . . , J.

FA,j

2.3

3D Cobb Angle Measurement

By applying the random sample consensus (RANSAC) algorithm [9] with error
tolerance ed on mesh faces FSU P and FIN F (6), we estimate the parameters of
the planes along the superior endplate of the upper-end vertebra (Fig. 4(a)) andthe inferior endplate of the lower-end vertebra, resulting in normal vectors nsup
and ninf , respectively. The 3D Cobb angle is finally measured as (Fig. 4(b)):

αCobb = arccos dot(n0sup , n0inf ) ,(7)

3D Cobb Angle Measurements

53

where n0sup is the projection of nsup to the Y −Z plane of the upper-end vertebra,
and n0inf is the projection of vector ninf to the Y − Z plane of the lower-end
vertebra.

3

Experiments and Results

The performance of the proposed method was evaluated on 60 spines that werediagnosed with adolescent idiopathic scoliosis (Sainte-Justine Hospital ResearchCenter, Montréal, Quebec, Canada), with mean reference Cobb angle measurements of 48.8◦ (range: 15.2–80.9◦ ). The spines were imaged with biplanar radiography and corresponding triangular mesh models between vertebral levels T1
and L5 were obtained by biplanar stereo-reconstruction [10] and kriging interpolation method [11]. For each spine, 17 mesh models with different face-vertex
density (i.e. with a different mean face edge length ranging from 1.58 to 6.85 mm)
were generated and used for the performance evaluation.
For the vertebral body center detection (Sec. 2.1), dvb,min = 12 mm, dvb,max =60 mm, rvb,min = 1.2 × mean face edge length were used for the opposite meshface determination, and mean filter of size KX × KY × KZ = 15 × 9 × 9 mm3
for the accumulation of lines between the opposite mesh faces. For superior andinferior endplate labeling (Sec. 2.2), rc = 1.2 × mean face edge length for cylinders representing rays, function f : n0B,z = f (n∗B,z ) = (n∗B,z )5 for emphasizing
the distances between cluster centers, and ed = 1.35 mm for RANSAC error
tolerance.
The proposed method was successfully applied to the 3D mesh models of
all 60 spines. Detailed statistical analysis of the obtained results is presented in
Tab. 1.

4Discussion

In this work, we described a semi-automated method for measuring the Cobb
angle from 3D mesh models of the spine. The only manual input to the method is
the selection of the upper-end and lower-end vertebra in the (scoliotic) deformity.
Apart from that, the method is fully automated, and, according to the obtained
results, yields relatively accurate 3D Cobb angle measurements.
From the results in Tab. 1 we can conclude that the results of the proposed
method are comparable to reference manual measurements at different facevertex densities that correspond to the mean face edge length ranging between
1.58 and 5.90 mm, as the comparison resulted in an average mean absolute error(MAE) of 3.0◦ and corresponding standard deviation (SD) of 2.2◦ . Once themean face edge length exceeds approximately 6 mm, the agreement with reference
measurements is lower, which can be observed in a higher MAE, maximal error
and the corresponding SD. It appears that at such face-vertex density there
are few spine mesh models in the database that do not fit well the vertebral
body. Considering that the smallest vertebral body is T1 with a mean width of
23.5 mm, length of 15.0 mm and height of 15.1 mm height for females [12], it is54

Petković et al.

Table 1. Statistical analysis of the obtained 3D Cobb angle measurements at different
mean face edge lengths of corresponding 3D spine mesh models, and comparison againstreference Cobb angle measurements.
N = 60 scoliotic spines

Comparison to reference Cobb angles

Mean face edge
length (mm)Mean Cobb
angle (◦ )Mean absolute
error (◦ )Maximalerror (◦ )Standard
deviation (◦ )

1.58
1.91
2.26
2.60
2.93
3.28
3.61
3.94
4.27
4.60
4.93
5.25
5.57
5.90
6.21
6.53
6.85

48.0
48.3
48.2
48.1
48.1
48.3
47.9
48.5
48.7
48.5
48.5
48.9
47.9
48.1
49.0
48.5
47.0

2.9
3.2
3.0
3.1
2.7
2.8
2.7
2.9
2.8
3.3
3.3
2.8
2.9
3.5
4.0
4.0
4.1

7.7
9.0
7.1
8.1
7.6
9.4
9.0
8.4
10.3
9.7
9.0
8.7
10.2
11.8
25.4
13.7
31.3

2.1
2.1
2.0
1.9
2.0
2.0
2.0
2.0
2.1
2.2
2.4
2.3
2.3
2.9
4.0
3.0
4.7

expected the that the mesh model with a mean face edge length in the range ofvertebral body dimensions would be inaccurate, and consequently the resulting
Cobb angle measurements would also be incorrect. Nevertheless, the accuracy
and robustness of the proposed method at lower face-vertex densities, i.e. of up
to 6 mm of face edge length, are valuable in terms of reducing the running time of
the algorithm due to its quadratic time complexity. On average, the running time
was on average estimated to 40 s (Intel(R) Core(TM) i7 - 4720HQ processor) fora single spine mesh model at the highest face-vertex density.
A method that performs Cobb angle measurements from 3D mesh modelsof the spine was also proposed by Huo et al. [5], however, the obtained resultscannot be directly compared to their method due to the different spine database,different techniques for mesh model reconstruction, and different face-vertex densities of the mesh models. However, Huo et al. obtained SD between 4.56 and4.67◦ when measuring the Cobb angle on a database of 22 spines, which is nothigher than SD of around 2.0◦ that was obtained by the proposed method. Nevertheless, the most noticeable difference is in the level of automation of bothmethods. While both methods require manual identification of the upper-end
and lower-end vertebrae, the method of Huo et al. additionally requires manual
selection of the seed vertex to estimate the parameters of each plane that rep-3D Cobb Angle Measurements

55

resents a vertebral endplate. The proposed method, on the other hand, is after
identification of the upper-end and lower-end vertebrae completely automated.

5

Conclusion

We presented a method for measuring the 3D Cobb angle from a 3D triangularmesh model of the spine. The results obtained on a database for 60 scoliotic
spines indicate that the method is relatively accurate and robust, and up toa certain level also relatively insensitive to the face-vertex density of the spine
mesh model.
Acknowledgements. This work was supported by the Slovenian Research Agency
under grants P2-0232, J2-5473, J7-6781 and J2-7118.

References
1. Stedman’s Medical Dictionary, http://www.medilexicon.com/dictionary/80286,
online; accessed 2017-06-04
2. Cobb, J.: Outline for the study of scoliosis. Am. Acad. Orthop. Surg. Instr. Course
Lect. 5, 261–275 (1948)
3. Vrtovec, T., Pernuš, F., Likar, B.: A review of methods for quantitative evaluationof spinal curvature. Eur. Spine J. 18(5), 593–607 (2009)
4. Trobisch, P., Suess, O., Schwab, F.: Idiopathic scoliosis. Dtsch. Arztebl. Int.
107(49), 875–884 (2010)
5. Huo, X., Tan, J., Qian, J., Cheng, L., Jing, J., Shao, K., Li, B.: An integrative
framework for 3D Cobb angle measurement on CT images. Comput. Biol. Med.
82, 111–118 (2017)
6. Štern, D., Likar, B., Pernuš, F., Vrtovec, T.: Automated detection of spinal centrelines, vertebral bodies and intervertebral discs in CT and MR images of lumbar
spine. Phys. Med. Biol. 55(1), 247–264 (2010)
7. Bresenham, J.: Algorithm for computer control of a digital plotter. IBM Sys. J.
4(1), 25–30 (1965)
8. Arthur, D., Vassilvitskii, S.: k-means++: the advantages of careful seeding. In:
Proc. 18th Annual ACM-SIAM Symposium on Discrete Algorithms – SODA 2007.
pp. 1027–1035. SIAM (2007)
9. Fischler, M., Bolles, R.: Random sample consensus: a paradigm for model fittingwith applications to image analysis and automated cartography. Comm. ACM
24(6), 381–395 (1965)
10. Kadoury, S., Cheriet, F., Laporte, C., Labelle, H.: A versatile 3-D reconstruction
system of the spine and pelvis for clinical assessment of spinal deformities. Comm.
ACM 45(6), 591–602 (2007)
11. Kadoury, S., Cheriet, F., Labelle, H.: Personalized X-ray 3-D reconstruction of thescoliotic spine from hybrid statistical and image-based models. IEEE Trans. Med.
Imaging 28(9), 1422–1435 (2009)
12. Masharawi, Y., Salame, K., Mirovsky, Y., Peleg, S., Dar, G., Stemberg, N., Hershkovitz, I.: Vertebral body shape variation in the thoracic and lumbar spine:characterization of its asymmetry and wedging. Clin. Anat. 21(1), 46–54 (2008)

Automatic Localization of the Lumbar Vertebral
Landmarks in CT Images with Context Features
Dimitrios Damopoulos1(), Ben Glocker2 , and Guoyan Zheng1

1

Institute for Surgical Technology and Biomechanics,
University of Bern, Switzerland
2
Imperial College London, Faculty of Engineering, UK
dimitrios.damopoulos@istb.unibe.ch

Abstract. A recent research direction for the localization of anatomical
landmarks with learning-based methods is to explore ways to enrich thetrained models with context information. Lately, the addition of context
features in regression-based approaches has been tried in the literature.
In this work, a method is presented for the addition of context featuresin a regression setting where the locations of many vertebral landmarks
are regressed all at once. As this method relies on the knowledge ofthe centers of the vertebral bodies (VBs), an automatic, endplate-based
approach for the localization of the VB centers is also presented. The
proposed methods are evaluated on a dataset of 28 lumbar-focused computed tomography images. The VB localization method detects all of thelumbar VBs of the testing set with a mean localization error of 3.2 mm.
The multi-landmark localization method is tested on the task of localizing the tips of all the inferior articular processes of the lumbar vertebrae,in addition to their VB centers. The proposed method detects these landmarks with a mean localization error of 3.0 mm.
Keywords: Regression, localization, lumbar, vertebral body, inferior articular process.

1

Introduction

Back pain in general and low back pain in particular constitutes a major public health problem, exhibiting epidemic proportions [1]. The computer-assisted
diagnosis of pathologies of the lumbar spine involves the analysis of images coming from a series of standard imaging modalities. Computed tomography (CT)images can be used for the diagnosis of spondylolysis, spondylolisthesis andosteoporosis, as this imaging modality permits the measurement of the bonemineral density of the vertebral bodies (VBs). This work focuses on the taskof the localization of the lumbar VBs in CT images and the localization of key
land-marks on the vertebral processes.
The proposed framework can facilitate subsequent automated procedures,such as the segmentation of the vertebrae, the automatic assessment of skeletalvertebral pathologies and the analysis of the spinal shape. In the case of vertebral
B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proc. 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. 56−68, 2017.

Automatic Localization of the Lumbar Vertebral Landmarks

57

segmentation, a large number of proposed methods employ some form of active
shape models or active appearance models [2, 3]. The initialization step of thesemodel-based approaches is typically based on the localization of the centersof the VBs. Using more landmarks than just the center of the VBs can add
robustness to this initialization step. Furthermore, the detection of vertebral
landmarks can function as the building block for the automated assessmentof pathologies concerning the global spinal shape (scoliosis, lordosis) and thegrading of spondylolisthesis: In [4], an automated method for the measurementof spondylolisthesis was presented, based on the identification of the endplateregions. For this application, the localization of the edges of the endplates in thecoronal direction could provide a more direct method for the measurement of
the anterior shift.
Localization of anatomical landmarks is a fundamental problem in medical
image analysis and a plethora of methods have been proposed in the literature.
In recent years, these tend to be based on machine learning tools and they canbe roughly categorized into classification-based methods and regression-based
methods. A popular research direction is the addition of context information in
the model that is constructed by these learning-based methods. For the problemof object segmentation, a principled method for achieving so is the auto-context
framework [5]. Recently, there have been attempts to apply this framework forthe localization of landmarks with random forest regressors. In particular, in [6]the authors showed that the extraction of context features from the distancemaps of a traditional random forest regressor can improve the landmark localization accuracy. Following this research direction, in the present work thismethod of adding context information is applied to a multiple-landmark localization task, where the locations of more than one landmark are regressed all at
once by random forest regressors. We show that the proposed method is able todetect robustly key landmarks of the vertebrae, despite the similar appearanceof neigh-boring vertebra. As the proposed method assumes that the centers of
the VBs have been already detected, we also present an endplate-based method
for the detection of the lumbar VB centers. We evaluate both of the methods ina dataset of 28 lumbar-focused CT images.

2

MethodThe proposed framework consists of two modules. The first module deals withthe localization of the VBs and the estimation of the pose of the vertebrae. It
performs this task via the detection of the vertebral endplates on a spline-based
unwrapping of the input image. The second module deals with the localization of
key landmarks of the vertebrae, based on the estimation of the VB centers andthe vertebral pose by the first module. It employs two levels of random forest
regressors. The two modules are described in Sec. 2.1 and 2.2. For the rest of
this section, it is assumed that a number of CT images of the lumbar spine areavailable for training. A training image will be referred to using the notation:
Ii : Ωi ⊂ R3 → R, i ∈ {1, . . . , N }.

(1)

58

Damopoulos et al.

A testing image will be denoted with IT . Every training image is accompaniedwith annotations of the centers of all the VBs within the field of view. We denotethe set of the annotations of the training image Ii with:
Ai = (c1 , v1 ), . . . , (cmi , vmi ) ,(2)
where cj ∈ Ωi , vj ∈ L, L = {S2 , S1 , L5 , L4 , L3 , L2 , L1 , T1 2, T1 1, T1 0}. We reduce
the set of the spinal levels to T10, since the dataset that we used for the experiments does not capture any vertebrae at higher spinal levels. Finally, it isassumed that the field of view covers at least the S1 – L1 region.
2.1

Localization of Vertebral Bodies and Estimation of their Pose

The localization of the lumbar VBs is performed in four steps, summarized
in Fig. 1. Firstly, a first-level detection of the centers is performed using themethod proposed in [7]. This method employs a random forest classifier and in
the present work it is used for a first-level detection of the VB centers of thelevels L. At training time, it constructs a label-map for every training image,using the ground-truth annotations of the VB centers. A random forest multilabel classifier is trained on the label-maps, using as features the mean intensityof displayed boxes (Haar-like features). At testing time, the generated probability map for every vertebral level is assumed to follow a normal distribution.
The mode of the distribution for every generated probability map separately is
retrieved with the mean-shift mode-seeking algorithm [8].
Secondly, the original CT image is resliced along the curve that passes througha set of VB center locations by performing an image deformation known ascurved planar reformation [9]. At training time, these locations are the groundtruth VB center annotations whereas at testing time they are the first-level VB
center detections. The reslicing is carried out using the method of [10], which
firstly calculates a B-spline that passes through the first-level detections andthen constructs a local coordinate system (LCS) on every point of the B-spline.
In the rest of this paper, the resulting deformed image will be referred to as the
spline-unwrapped image.
Thirdly, for every slice of the spline-unwrapped image, the mean intensityof a region around the middle point of the slice is calculated. The result of
this operation is a univariate signal, referred to as the mean-intensity profile(shown in the top plot of Fig. 2). For the training phase, it is denoted withsi : Oi ⊂ N → R. For a testing image, it is denoted with sT .
Lastly, the locations of VB centers are inferred from the positions of thevertebral endplates in the mean-intensity profile, using an approach very similar
to those of [11] and [12]. Unlike [11], we do not attempt to detect periodic
patterns in the mean intensity profile but we just locate its local maxima. As in[12], the basic observation for the detection of the endplates is that their locations
correspond to local maxima (peaks) in the mean-intensity profile. Unlike [12],we do not make any assumptions concerning the orthogonal symmetry of thevertebrae in order to fine tune the VB center estimations and we just average

Automatic Localization of the Lumbar Vertebral Landmarks

59

Fig. 1. A flowchart of the steps of proposed pipeline for the localization of the lumbar
vertebral bodies (VBs). From left to right: a first-level detection of the VB centers is
performed using the method of [7]; the original image is resliced along the curve thatpasses through the first-level detections; a mean-intensity profile is calculated along
the axial center of the resliced image and the peaks of the mean-intensity profile that
correspond to endplate locations are identified using a k-nearest neighbors classifier.

the locations of the bottom and top endplates. Furthermore, we attempt to add
robustness to the identification of the peaks that correspond to endplates bytraining a k-nearest neighbors classifier specifically for this task. In detail:• At training time, for every mean-intensity profile si , we locate the positions of those peaks Pi = {p1 , . . . , pqi }, p1 ∈ Oi which are anatomicallysuperior to the annotation cS1 . Hence, S1 is used as an anchor vertebra. For
every peak position pξ we compute three simple features: (a) the value ofthe peak si (pξ ); (b) its left prominence eξ and (c) its right prominence Eξ .
The left prominence is defined as:eξ = max si (pξ ) − si (ρ), ρ ∈ Oi , si % [ρ, pξ ] ,(3)

where the % denotes that si is increasing in the specified interval. The rightprominence Eξ is defined symmetrically. A binary label is provided for every peak, marking whether it corresponds to an endplate position or not. A
k-nearest neighbors classifier is fit to this training set.

• At testing time, the peaks of mean-intensity profile sT after the S1 firstlevel detection are identified (Fig. 1) and the three features are computed asin training. The trained k-nearest neighbors classifier classifies these peaks
as corresponding to endplates or not. Finally, the estimates for the centers of
the lumbar VBs are given by simply averaging the endplate positions. The
pose of every vertebra is given by the LCS (computed at the second step) of
the point of the B-spline which is closest to the VB center estimation.

60

Damopoulos et al.

Fig. 2. The mean-intensity profile and the detection of the endplates. Top: The meanintensity profile of a testing image with all the local maxima (peaks) in black dots;
Middle: The out-put of the k-nearest neighbors classifier, which classifies the peaks in“endplate” and “non-endplate”. Bottom: The same endplate positions, on the splineunwrapped image. The prediction of the vertebral body center is the average positionof the bottom and top endplate on every lumbar level.

2.2

Localization of Vertebral Landmarks

The objective of the second module is to locate a given number of vertebral
landmarks on each level of the lumbar spine separately. We are interested in the
lumbar spinal levels L0 = {L5 , L4 , L3 , L2 , L1 }. Let v ∈ L0 be one such level. For
simplicity, it is assumed that same number Mv = M of landmarks is desired
to be found on all the levels. Therefore, it is assumed that the annotations Bivof the M landmarks of the vertebra at level v of every training image Ii areavailable. This ordered set of annotations is denoted as:
Biv = (cv1 , . . . , cvM ) , cj ∈ Ωi .

(4)

The VB center annotations for the lumbar region of (2) are incorporated in theordered sets Bi as its first elements, i.e. cv1 = cv for all the levels, where cv isannotation for the VB center. Hence, given a testing image IT , the task is tolocalize the M landmarks on each level v ∈ L0 . This is accomplished with twolayers of random forest regressors, combined in an auto-context fashion [5]. The
next two sections describe these two layers.
First Layer: Multi-Landmark Localization Using Appearance Features.
The first layer employs a traditional multi-landmark regression-based method.
One random forest regressor is trained for each lumbar level. Since each of these

Automatic Localization of the Lumbar Vertebral Landmarks

61

regressors is being utilized independently of all the others, let’s assume thatinterest is on a specific level v ∈ L0 .
• At training time, the images Ii are rotated according to the poses of thevertebrae at level v, so that all the vertebrae at level v are aligned, and aregion of interest ROI is constructed around the VB center. The training set
is sampled from all the ROIs. For every training sample, the displacements
to the M landmarks are computed, hence it is paired with 3 ∗ M continuous values. A random forest is trained to regress these displacements. The
traditional Haar-like features are used (as in [6, 7, 13]), which are based on
the mean intensity value of randomly displayed boxes. The following feature
types are considered:PPy∈B2 I(x + o2 + y)y∈B1 I(x + o1 + y)−s,f (x; B1 , B2 , o1 , o2 , s) =
|B1 |
|B2 |
(5)
where s ∈ {0, 1}, B1 , B2 are the sizes of two 3D boxes and o1 , o2 are 3D
offsets. A specific number of these features is sampled at the beginning of
the random forest training and the parameters of the features are sampled
uniformly from an interval of allowed values. In each leaf node of the decision
trees of the trained random forest, two vectors of dimension 3∗M are stored:
the mean displacements of the training samples that arrived on this leaf andtheir variance along every dimension.
• At testing time, the first module estimates the VB center of IT at level v
and the relevant pose. Then, IT is aligned according the detected pose, a ROI
is generated around the detected VB center and a testing set is sampled frominside this ROI. In a traditional single-layer approach, every testing sample
would be parsed by every tree of the forest and it would cast M votes for thelocations of each of the M landmarks. The aggregation of the votes from all

(a)

(b)

(c)

Fig. 3. Illustration of the problem with the single regressor approach on three testing
images. The semi-transparent red layer represents the region of interest testing region
and the blue-red colormap the vote maps. All the testing images are rotated aroundthe respective vertebra of interest, L1 for (a) and (c) and L2 for (b). The white arrow
point to the ground-truth location of the corresponding landmark. On (a), the votemap retains its maximum value around the correct location. On (b), while the votemap still has a higher value around the correct location, it can no longer be considered
unimodal. On (c), the problem is even clearer, as the mode with the highest value does
not correspond to the ground truth location.

62

Damopoulos et al.

the testing samples results in M maps, which in this work will be referred
to as vote maps. The location of the each of the M landmarks would be
inferred from its vote map, via a mode-seeking algorithm. A drawback ofthis approach is that the vote maps are not guaranteed to be unimodal. In
fact, it is to be expected that the vote map will have a high value on not only
the target landmark at level v but possibly on the homologous location ofneighboring vertebra with similar appearance. This is partially addressed by
the fact that only a ROI around the detected VB center of the testing image
is considered. However, the problem is not fully eliminated, since it is notpossible to know in advance how large this ROI should be. This is illustrated
in Fig. 3, where sagittal slices from three different testing cases are depicted.
The problem is most apparent in Fig. 3(c), where the maximum of the votemap occurs on the incorrect mode. A mode-seeking algorithm, without any
additional post-processing, would fail in that case.
Second Layer: Addition of Context Features. The problem of the concurrent appearance of modes on neighboring spinal levels is addressed by theaddition of context features. These context features are similar to the ones introduced in [6], where context information is extracted from the distance maps.
In [6] one random forest regressor is constructed for every landmark. This would
not scale well to the current task, as |L0 | ∗ M regressors would have to be trainedon every layer. Therefore, the context features are used here by a multi-landmark
regressor.
• At training time, every tree of the first layer makes three predictions (onefor each spatial dimension) for the displacement of each training sample toeach of the M landmarks. The mean value of Euclidean distance of thesethree predictions over all the trees is called a distance map. At this point, an
important decision to be made is which part of the training image should beconsidered for the computation of the distance maps. A simple choice is to
use the same ROI as the one used for the training of the first layer. However,such a setup will bias the second layer into expecting that the VB centerlies exactly at the center of the sampling ROI. In order to remove this bias,
a modification to the standard Auto-context framework is introduced. For
every ground truth annotation cv1 of the VB centers, W randomly displaced
locations are generated:c̃v1,w = cv1 + dw , dw ∈ [−d, d]3 , w ∈ {1, . . . , W }.

(6)

The displacements dw are sampled randomly from the space [−d, d]3 . Then,W ROIs Rw ⊆ Ωw , w ∈ {1, . . . , W } around each c̃v1,w are generated. The
regions Rw of the training image Ii are parsed by the random forest of themfirst layer in order to compute the W ∗ M distance maps Di,w
: Ωi → R+ .
m
The value of distance maps Di,w outside of Rw is set to a fixed, large value.
For the training of the second layer, each training image Ii is taken intomaccount W times, each time paired with the M distance maps Di,w
. As inAutomatic Localization of the Lumbar Vertebral Landmarks

63

the first layer, a pool of Haar-like features is sampled before the training of
the forest starts, with the difference that the intensity image I(·) of (5) canmbe replaced by one of the Di,w
. When this happens, the resulting Haar-likefeature is able to capture context information from the distance maps ofthe first layer (context feature). The total number of context features is set
beforehand as a hyperparameter.
• At testing time, the distance maps of the testing image are generated by
the first layer, using the same testing ROI as in the first layer. The testing
image is paired with the generated distance maps and it is passed to thesecond layer, so that both appearance features (computed on the original
testing image) and context features (computed on the distance maps) can
be calculated. The testing pipeline proceeds with the computation of thevote maps. For every testing sample, the vote generated by each tree of theforest for each landmark is taken into account separately, provided that thevariance of the displacement is below a certain threshold. As it is illustrated
in the second row of Fig. 4, the resulting vote map is unimodal. The modeof every vote map is estimated using the mean-shift algorithm [8]. Finally,
the estimated modes are rotated back to the original image space in orderto provide the localization of the landmarks.

Fig. 4. Qualitative comparison of the voting maps of a single random forest layer (toprow) with the voting maps after the second layer of the proposed method (bottomrow). From left to right: sagittal cuts of different testing images for levels L1 to L5,
respectively. Notice that the images have been automatically aligned around the vertebra of interest. In can be observed that the voting map is more concentrated in the
two-layered approach and it is has exactly one mode around the correct landmark
location.

64

Damopoulos et al.

3

Experiments and Results

3.1

Dataset and Experimental Setup

The proposed methods are evaluated on a dataset of 28 CT images. The intraslice slice spacing is in the 0.29–0.42 mm range, the inter-slice spacing is 0.7 mm
and the slice size is 512 × 512. All of the images capture at least the S1 –L1 levels, which is typical for scans of the lumbar spine. The thoracic region is
captured up to the T10 level some cases. No implants are presented in any of theimages. There are cases with mild scoliosis, osteophytes and fractures vertebrae.
For every lumbar vertebra, five manual annotations are made: the center of the
VB and the tips of the four inferior articular processes. There are four inferior
articular process on a typical lumbar vertebra: a bottom-left, a bottom-right, a
top-right and a top-left. We will refer to their tips as landmarks A, B, C and D
respectively. Sagittal and coronal view of two example annotations for landmarks
A and B are shown on Fig. 5(a) and Fig. 5(b).
All of the images are resampled to an isotropic spacing of 1 × 1 × 1 mm. We
randomly select 20 images to be used for the training phase of the proposed
methods. The held-out eight images will be used for evaluation.
The hyperparameters of the proposed methods are set through a leave-oneout cross-validation iteration on the training set. The K parameter of the knearest neighbors classifier of the first module (localization of VBs) is set to 15.
For the second module (localization of vertebral landmarks) the hyperparameters
are as following: for the random forest of the first layer, 50 trees are trained, thesize of the feature pool is 10000 and on every node the search space is 200
features. The size of the ROI around every vertebra, during both training andtesting, is 120×150×80 mm. The training set of every tree is a random 1% subsetof all the voxels inside the ROIs of the images. At testing time, all of voxels inside
the ROIs are used. For the second layer, the parameters W , d of (6) are set to
5 and 8 mm respectively. 50 trees are trained with a depth of 25. The size of thefeature pool is 11000 features: 10000 features plus exactly 200 from the each of

(a)

(b)

(c)

Fig. 5. (a) the sagittal and the coronal views of the annotation of the tip of the bottomleft inferior articular process (landmark A) of an L3 vertebra. (b) The sagittal andcoronal views of the annotation of a bottom-right inferior articular process (landmarkB) of an L2 vertebra. (c) The three-dimensional bounding boxes of the manual annotations for the inferior articular process (blue dots) and their detections (red dots). The
localization errors have been exaggerated.

Automatic Localization of the Lumbar Vertebral Landmarks

65

Table 1. Localization performance of the first module for the vertebral body centers.
A detection is considered successful if it lies within 10 mm from the manual annotation(Id. rate). The mean, standard deviation and median of the localization errors arecomputed on the successful detections only. The first-level detections are the output of
the method [7]. The endplate-based detections are the output of the first module.
First-level detections

Id. rate (%)
Mean (mm)Std. (mm)Median (mm)Endplate-based detections

L1

L2

L3

L4

L5

L1

L2

L3

L4

L5

75.0
3.7
2.1
3.4

62.5
6.7
3.5
6.1

62.5
8.1
3.3
6.9

75.0
7.6
3.2
7.6

100
5.0
1.8
5.2

100
2.8
1.3
2.4

100
3.4
2.3
2.4

100
3.8
1.5
4.0

100
3.1
2.6
1.7

100
2.9
1.5
2.7

the five distance maps. The sampling of the training samples is again 1%. At
testing time, all the voxels inside the ROI are tested, but only the votes with apredicted variance of less than 15 mm in every spatial dimension (from each tree
independently) are taken into account for the construction of the vote maps.
3.2

Evaluation

For the evaluation of the first module, two metrics are used: (a) the rate ofsuccessful detections and (b) the displacements to the manual annotations of
the VB centers of the lumbar spine (localization error). A VB center detection
is considered successful when it lies within 10 mm from the respective manual
annotation. The detailed evaluation for every lumbar spinal level is presented
in Tab. 1, where the rate of successful detections is labeled as “Id. rate”. All of
the lumbar VB of the eight testing images are detected successfully. The mean
localization error is 3.2 mm, with a standard deviation of 2.0 mm and a medianvalue of 2.8 mm. The evaluation of the first-level detections obtained with themethod of [7] is also presented in Tab. 1.
For the evaluation of the second module, the localization error metric isalso used. The localization errors for each lumbar level are presented in Tab. 2,
along with the rate of the detections with a localization error of less than 6 mm.
Overall, the proposed method achieves a mean localization error of 3.0 mm,
with a 1.6 mm standard deviation and a median value of 2.7 mm. 95.4% of thedetections have a localization error of below 6 mm. Regarding the training of
the second layer, we experimented with removing the randomly displaced ROIsof (6) and train instead using ROIs centered around the VB centers. With that
setup, the localization error increases to 3.4 ± 1.8 mm.
As an additional metric for the quality of the detections, their boundingboxes are also considered. In particular, the extreme locations of the five landmarks in the each of the three spatial dimensions define six bounding planes
and therefore a 3D bounding box. Coronal projections of such bounding boxes
are depicted in Fig. 5(c) for both the manual annotations (blue box) and the

66

Damopoulos et al.

Table 2. Localization errors of the second module for the five landmarks of L1 – L5.
In the case of L4, one testing case has been omitted because it was not possible toannotate all its articular processes due to a vertebral fracture. Hence, there are seventesting images on this spinal level.
VB center

A

B

C

D

Overall

100
1.8
0.6
1.7

100
2.8
1.4
2.6

100
3.0
1.3
2.7

100
2.6
1.1
2.6

100
3.4
1.5
2.9

100
2.7
1.3
2.7

100
2.4
1.1
2.8

100
2.2
0.8
2.2

100
2.1
0.8
2.0

100
3.1
1.2
2.8

100
2.6
1.4
2.4

100
2.5
1.1
2.3

100
2.7
0.9
2.4

100
3.4
1.6
3.1

87.5
3.7
1.5
3.1

87.5
3.6
1.9
3.1

100
2.8
0.9
2.7

95.0
3.2
1.5
2.9

100
1.9
1.0
1.7

87.5
4.0
1.7
3.2

87.5
3.4
2.3
2.1

87.5
3.5
1.9
3.5

87.5
3.4
2.9
2.4

88.6
3.2
2.2
2.4

100
2.7
1.6
2.4

75.0
4.5
1.6
4.7

100
3.4
1.3
3.8

100
3.4
1.3
3.3

100
2.9
1.1
2.9

92.5
3.4
1.5
3.0

L1-level landmarks
Loc. error < 6 mm (%)
Mean (mm)Std. (mm)Median (mm)L2-level landmarks
Loc. error < 6 mm (%)
Mean (mm)Std. (mm)Median (mm)L3-level landmarks
Loc. error < 6 mm (%)
Mean (mm)Std. (mm)Median (mm)L4-level landmarks
Loc. error < 6 mm (%)
Mean (mm)Std. (mm)Median (mm)L5-level landmarks
Loc. error < 6 mm (%)
Mean (mm)Std. (mm)Median (mm)automatic detections (red box). The evaluation metric is the Dice overlap coefficient of the bounding box of the manual annotations and the bounding box ofthe detections. The achieved scores on this metric are presented in Tab. 3. The
mean dice coefficient, across all the spinal levels, is 88.8%.

4Conclusion

The repetitive nature of the spine poses an additional difficult to the task oflandmark localization, as neighboring vertebrae often have very similar appearance. However, a fully automatic method for localizing vertebral landmarks isAutomatic Localization of the Lumbar Vertebral Landmarks

67

Table 3. Dice coefficients for the bounding boxes from the five landmarks over eachspinal level.
Dice coefficients

L1

L2

L3

L4

L5

Overall

Mean
Min.
Max.

0.87
0.80
0.94

0.90
0.86
0.95

0.90
0.87
0.96

0.90
0.80
0.95

0.87
0.82
0.92

0.89
0.80
0.96

highly desirable, as it can provide as a robust initialization step for model-based
segmentation methods and it can facilitate the assessment of certain vertebral
pathologies. In this work, a pipeline for the detection of lumbar vertebral landmarks is proposed. The proposed pipeline starts with the detection of VB centersand proceeds with the localization of landmarks on each lumbar level. For evaluation, the pipeline was applied for the localization of the VB centers and theinferior articular processes on a dataset of lumbar-focused CT images. The experimental results suggest that the proposed method can detect reliably thevertebral landmarks on all the levels of the lumbar spine. Even though in our
experiments we focused on the articular processes, we expect that the proposed
method can be applied for different vertebral landmarks as well, such as key
endplate landmarks for the measurement of spondylolisthesis. In the future, weplan to explore such a direction. Future research also includes the more extensive
evaluation of the proposed methods on larger datasets and the investigation ofways to improve the localization accuracy, for example by fine-tuning the detections in a multi-scale fashion and by introducing context features from different
vertebral levels.

References
1. Reginster, J.: The prevalence and burden of arthritis. Rheumatol. 41(Supp 1), 3–6(2002)
2. Al Arif, S., Gundry, M., Knapp, K., Slabaugh, G.: Improving an active shape modelwith random classification forest for segmentation of cervical vertebrae. In: Yao, J.,et al. (eds.) Proc. 4th MICCAI Workshop & Challenge on Computational Methods
and Clinical Applications for Spine Imaging – CSI 2016, LNCS, vol. 10182, pp. 3–
15. Springer (2016)
3. Roberts, M., Cootes, T., Adams, J.: Automatic location of vertebrae on DXA
images using random forest regression. In: Ayache, N., et al. (eds.) Proc. 15th
International Conference on Medical Image Computing and Computer-Assisted
Intervention – MICCAI 2012, LNCS, vol. 7512, pp. 361–368. Springer (2012)
4. Liao, S., Zhan, Y., Dong, Z., Yan, R., Gong, L., Zhou, X., Salganicoff, M., Fei,
J.: Automatic lumbar spondylolisthesis measurement in CT images. IEEE Trans.
Med. Imaging 35(7), 1658–1669 (20016)
5. Tu, Z.: Auto-context and its application to high-level vision tasks. In: Proc. IEEE
Conference on Computer Vision and Pattern Recognition – CVPR 2008. IEEE
(2008)

68

Damopoulos et al.

6. Gao, Y., Shen, D.: Context-aware anatomical landmark detection: application to
deformable model initialization in prostate CT images. In: Wu, G., et al. (eds.)
Proc. 5th MICCAI International Workshop on Machine Learning in Medical Imaging – MLMI 2014, LNCS, vol. 8679, pp. 165–173. Springer (2014)
7. Glocker, B., Zikic, D., Konukoglu, E., Haynor, D., Criminisi, A.: Vertebrae localization in pathological spine CT via dense classification from sparse annotations.
In: Mori, K., et al. (eds.) Proc. 16th International Conference on Medical Image
Computing and Computer-Assisted Intervention – MICCAI 2013, LNCS, vol. 8150,
pp. 262–270. Springer (2013)
8. Cheng, Y.: Mean shift, mode seeking, and clustering. IEEE Trans. Pattern Anal.
Mach. Intell. 17(8), 790–799 (1995)
9. Kanitsar, A., Fleischmann, D., Wegenkittl, R., Felkel, P., Gröller, M.: CPR: curved
planar reformation. In: Proc. IEEE Visualization Conference – VIS 2002. pp. 37–
44. IEEE (2002)
10. Velut, J.: A spline-driven image slicer. VTK J. (2011)
11. Štern, D., Likar, B., Pernuš, F., Vrtovec, T.: Automated detection of spinal centrelines, vertebral bodies and intervertebral discs in CT and MR images of lumbar
spine. Phys. Med. Biol. 55(1), 247–264 (2010)
12. Forsberg, D., Lundström, C., Andersson, M., Vavruch, L., Tropp, H., Knutsson, H.:
Fully automatic measurements of axial vertebral rotation for assessment of spinal
deformity in idiopathic scoliosis. Phys. Med. Biol. 58(6), 1775–1787 (2013)
13. Criminisi, A., Robertson, D., Konukoglu, E., Shotton, J., Pathak, S., White, S.,
Siddiqui, K.: Regression forests for efficient anatomy detection and localization incomputed tomography scans. Med. Image Anal. 17(8), 1293–1303 (2013)

Joint Multimodal Segmentation of Clinical CT
and MR from Hip Arthroplasty Patients
Marta B. M. Ranzini1( ) , Michael Ebner1 , M. Jorge Cardoso1 ,Anastasia Fotiadou2 , Tom Vercauteren1,3 , Johann Henckel2 ,Alister Hart2,3 , Sébastien Ourselin1,3 , and Marc Modat1
1

Translational Imaging Group, CMIC, University College London, UK
Royal National Orthopaedic Hospital NHS Foundation Trust, UK
3
Wellcome/EPSRC Centre for Interventional and Surgical Sciences, UCL, UK
marta.ranzini.15@ucl.ac.uk
2

Abstract. Magnetic resonance imaging (MRI) is routinely employed
to assess muscular response and presence of inflammatory reactions inpatients treated with metal-on-metal hip arthroplasty, driving the decision for revision surgery. However, MRI is lacking contrast for bonystructures and as a result orthopaedic surgical planning is mostly performed on computed tomography images. In this paper, we combine thecomplementary information of both modalities into a novel framework
for the joint segmentation of healthy and pathological musculoskeletal
structures as well as implants on all images. Our processing pipeline isfully automated and was designed to handle the highly anisotropic resolution of clinical MR images by means of super resolution reconstruction.
The accuracy of the intra-subject multimodal registration was improvedby employing a non-linear registration algorithm with hard constraints
on the deformation of bony structures, while a multi-atlas segmentation
propagation approach provided robustness to the large shape variability
in the population. The suggested framework was evaluated in a leaveone-out cross-validation study on 20 hip sides. The proposed pipeline has
potential for the extraction of clinically relevant imaging biomarkers for
implant failure detection.
Keywords: Muscoloskeletal imaging, multimodal segmentation, multimodal registration, CT, MR, arthroplasty.

1

Introduction

In the past 20 years, metal-on-metal (MoM) hip arthroplasty has been one of themost effective surgical interventions for improving life quality. However, this implant type is associated with a non-negligible rate of failure (8% at 12 years fromprimary surgery [1]), due to adverse tissue inflammatory reactions and increased
muscle atrophy [2]. Routine assessment of periprosthetic muscle response to theimplant is performed on magnetic resonance (MR) images [3], whereas computed
tomography (CT) imaging is preferred for surgical planning and post-operative
follow-up, thanks to its improved contrast for bone and implant [4]. The two
B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proc. 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. 69−81, 2017.

70

Ranzini et al.

modalities provide complementary skeletal and muscular information, which arepresently assessed independently in clinical practice. In this context, a single
framework merging this information by means of joint automated segmentation
could be beneficial for both early detection of implant failure and planning ofrevision surgery. By providing spatial relationship between muscle, bone and implant simultaneously, the combination of the two imaging modalities could help
link implant position (not MR visible) with muscle damage (estimated on MR)
to better characterise pain origin. Moreover, it could favour a patient-specificplanning of surgical approach to minimise damage to healthy bone and muscular tissue.
In the musculoskeletal clinical field, manual segmentation is still the mostfrequently adopted solution in clinical routine for delineating regions of interest [5], despite the variety of image-based anatomical models and segmentation
techniques presented in the literature. Methods for automated segmentation of
hip bony structures in CT images are typically based on statistical shape models [6, 7], atlas-based segmentation propagation [8] or, more recently, hybrid approaches [9]. Segmentation of muscles on MR images is more problematic, because of their large inter-subject shape variability and the lack of image contrast
between different muscular structures. A common approach for thigh muscles
is the incorporation of atlases as priors into conventional segmentation techniques such as active contours or level-set algorithms [10,11]. Remarkable results
were also presented by Gilles et al. [12], who introduced a method to automatically segment hip muscles and bones on MR images by means of deformablemulti-resolution simplex meshes. The performances of all discussed methods arestrongly reliant on the variability encompassed in the training data set and they
are often not suitable for pathological conditions. Klemt et al. [13] addressed
this issue by developing a robust automated segmentation framework for abductor muscles on MR in both healthy subjects and patients with MoM prostheses. However, little work combining multimodal imaging for the segmentation of
musculoskeletal structures has been proposed so far and it is often limited only
to spine applications. An example is the method presented by Castro-Mateos
et al. [14], which is based on a fast mesh-to-image registration to extract a surface model of CT-derived vertebrae and MR-derived intervertebral discs. Whilst
being very suitable for bony structures, the applicability of this method to patients with hip arthroplasty would be hindered by the presence of metal artefact
in the images and by the greater morphological and textural variability of muscles.
Taking advantage of the complementary information derived from CT and
MR, we propose a fully automated joint segmentation framework of both modalities from patients treated with MoM arthroplasty. Our processing pipeline wasdesigned to handle clinical data, characterised by highly anisotropic resolution
and presence of severe metal artefact induced noise, and allows for a threedimensional representation of patient-specific musculoskeletal hip anatomy. Key
contributions of this work include the use of super resolution reconstruction(SRR) to improve clinical MR image quality; moreover, the development of a

Segmentation of Clinical CT and MR from Hip Arthroplasty Patients

71

robust intra-subject multimodal registration allowed preservation of the rigid
structure of bones, while deforming the muscles. Finally, a multi-channel multiatlas based segmentation propagation guaranteed robustness to the large shapevariability in the population.

2

Materials and Methods

2.1

Dataset and Templates Creation

Our dataset includes retrospectively collected images of 11 MoM hip implanted
patients (7 females and 4 males, 10 unilateral and 1 bilateral replacement) who
had both a CT and an MR scan acquired on the same day. For the MR acquisitions, a Siemens MAGNETON Avanto 1.5T scanner was employed forall patients, using the MARS MRI protocol proposed in [15], which is characterised by rapid 2D MRI acquisition but high voxel resolution anisotropy.
This includes the collection of two T1-weighted Turbo Spin Echo (TSE) images: a high-resolution axial acquisition (TE = 8 ms, TR = 509 ms, typical imaging resolution = 0.78×0.78×7.02 mm3 ) and a high-resolution coronal acquisition(TE = 7.1 ms, TR = 627 ms, typical imaging resolution = 1.25 × 1.25 × 6.00 mm3 ).
Eight CT images were acquired on a Siemens SOMATOM Sensation 16 CT Scanner, while three on a Siemens SOMATOM Definition AS machine (tube voltage
in [80, 120] kVp). The images were processed (see Sec. 2.2), split along the leftright axis of symmetry and separated according to the presence of implant. Manual segmentation of pelvic bones, femora and implant were performed on CT,
while Gluteus Maximus (GMAX), Gluteus Medius (GMED), Gluteus Minimus(GMIN) and Tensor Fasciae Latae (TFL) were individually manually delineatedon the MR. As a result of these processes, we built two template data sets,composed of 10 implanted and 10 non-implanted hip sides respectively - for thesake of simplicity we will refer to the latter as the healthy data set despite thepresence of metal artefact generated by the implanted side.Each template includes a CT image, a registered super-resolution reconstructed MR image andthe respective joint manual segmentation of bones, muscles and implant. Withineach dataset, the templates were robustly aligned onto the average space basedon the method proposed in [13].
2.2

Pipeline for Automated Segmentation

A schematic representation of our processing framework is presented in Fig. 1.
The pipeline was implemented in NiPype [16], combining registration and segmentation utilities of NiftyReg1 , NiftySeg2 and FSL3 software packages withsuper-resoluton reconstruction and the proposed novel multimodal registration
framework. Our method is composed of three main blocks which are performed
1
2
3

https://cmiclab.cs.ucl.ac.uk/mmodat/niftyreg
http://cmictig.cs.ucl.ac.uk/wiki/index.php/NiftySeg
https://fsl.fmrib.ox.ac.uk/fsl/fslwiki

72

Ranzini et al.

Fig. 1. Proposed pipeline for joint automated segmentation of computed tomography (CT) and magnetic resonance (MR) pelvic images. The two modalities are firstprocessed independently to enhance the image quality. Intra-subject multimodal registration is then performed to align them through a non-linear deformation with rigid
constraints in bony structures. The registered CT and MR are split along the axis ofsymmetry and a multi-atlas based segmentation propagation approach is applied toobtain the automated segmentations of each side, which are finally recombined into
the full field of view.

sequentially: image quality enhancement of each modality, intra-subject MR-toCT registration, and atlas-based segmentation.
Image Quality Enhancement. In the first block, we aim at improving thequality of the clinical images for improved registration steps. The axial and thecoronal MR images are first corrected for bias field effects [17]. In order to compensate for the highly anisotropic resolution of clinical MR images (up to a factorof 10), we combine both MR acquisitions into a 1 × 1 × 1 mm3 resolution image
using the SRR algorithm presented in [18]. To ease the subsequent registration,
the CT is also resampled to the same resolution using a cubic interpolation
scheme. An initial estimate of bones segmentation on the CT is extracted byregistering the templates to the target space and consequently propagating andfusing their segmentation, allowing the creation of masks for femur, pelvis andimplant to be used in the intra-subject registration.

Segmentation of Clinical CT and MR from Hip Arthroplasty Patients73

Intra-Subject MR-CT Registration with Bone Rigid Constraints. The
subsequent step in our processing pipeline is the registration of the SRR MR
image to the respective CT. Multimodal registration for hip musculoskeletal
structures is challenging and no standard method has been proposed yet. A
simple affine transformation is not sufficient to guarantee an accurate alignment of the images, due to differences in the patient’s pose in the scanners. On
the other hand, high frequency deformations should be curbed when dealing
with intra-subject registration to prevent non-physiological deformation. The
applied transformation should embed a rigid behaviour for bones to preserve
their shape, while allowing non-linear deformation of fat and muscular tissue.
To tackle the discussed issues, we designed a registration pipeline composedof two steps. Firstly, the two images are affinely registered using a symmetricblock-matching algorithm [19], in order to provide an initial global alignment.
Subsequently, the non-linear registration is performed by imposing locally rigid
hard constraints directly on the transformation through the following method,which we developed from the mathematical formulation proposed in [20]. Givena reference space X with the associated intensities R(X) (i.e. a reference imageR), a set of masks Mj defined in the reference space labelling the rigid structures, and a floating image F defined in the floating space Y , we defined our
registration problem as the optimisation of the transformation φ : X → Y such
that:


max (1− α− β) D(F (φ(X)), R(X)) − αPL − βPB
φ(1)subject to φ(x) − Aj x = 0 ∀x ∈ Mj ⊂ X,
D is a measure of similarity between the reference and the warped floating image, while PL and PB represent the linear elasticity and the bending energy
penalty terms [21], whose contribution to the total cost function is weighted
by α and β respectively; Aj refers to a rigid transformation applied within thej-th mask. In order to guarantee inverse-consistency and symmetry of the registration [22], we exploit a scaling-and-squaring exponentiation of a stationary
velocity field encoded by a cubic B-spline parametrisation defined over a setof control points {µ}. The transformation is optimised within a conjugate gradient scheme, and the rigid behaviour in the mask areas is ensured throughthe following steps: We stress the fact that the rigid constraints are embedded

Algorithm 1 Apply rigid constraints
Compute the gradient G(µ) of the cost function ∀µ ∈ {µ},for each mask Mj do
Least square regression of G(µ), ∀µ ∈ Mj to fit a rigid transformation Aj
Set the gradient to Aj (µ) ∀µ ∈ Mj
end for
Perform a line search along the direction of G.

into the transformation model, not in the optimisation scheme. Thus, chain rule

74

Ranzini et al.

provides an analytical formulation of the conjugate gradient thereby avoiding
constrained optimisation. Using the proposed method on a coarse-to-fine pyramidal approach, smooth transitions in the deformation field are maintained by
the cubic B-spline parametrisation and the stationary velocity field exponentiation, while forcing rigid behaviours within the masks. To reduce the effect ofundesired high-frequency components in the transformation, we set one control
point every five voxels, and the masks are dilated at each pyramidal level toaccount for the local support of the control points. We underline that we extract
the robust range of the intensity distributions for both the reference and thefloating image, and we perform all the registration steps by flooring or ceilingall intensities outside this range, so as to decrease the influence of metal artefact
induced noise.
Once registered with the proposed method, the CT and the MR are merged
into a single four-dimensional (4D) volume. In order to employ the appropriate
template dataset for the atlas-based segmentation – i.e. healthy or implanted –we developed a symmetry and implant detection algorithm. Based on left-rightaxis flip and rigid registration, it extracts the sagittal axis of symmetry from theinertia tensor of the image intensities. The 4D volume is split along this axis andeach hip side is automatically classified according to the presence of implants.
Atlas-Based Segmentation. Each split hip side is segmented by means ofmulti-atlas segmentation propagation and label fusion. All the templates areregistered to the target 4D image in a three-step process (rigid, affine and nonlinear registration as implemented in NiftyReg). The transformation of the affine
and the non-linear steps is initialised as the least trimmed squares average affinefrom all the template transformations estimated at the previous step. Since ourtemplates were previously aligned to their mid-space (Sec. 2.1), this initialisation
provides robustness against global failed registration. Notably, the non-linear
step is a multi-channel registration, where both modalities contribute jointly andequally to the optimisation of the transformation. Using the estimated transformation, the segmentation of each template is propagated onto the target space.
The candidate segmentations are then fused into a consensus through the STEPS
algorithm [23], specifically modified to manage a multi-channel local similarity
measure. The final segmentation is obtained by merging back the two hip sides
and their estimated segmentation, providing a multi-label image that highlights
different bones, muscles and implants on both the CT and the MR.

3

Validation and Experiments

3.1

Intra-Subject Registration Evaluation

The first set of experiments we performed aimed at identifying the optimal set of
regularisation parameters α and β as shown in (1) for the intra-subject registration. Normalised mutual information (NMI) was used as measure of similarity,
since it is best suited for multimodal registration. For the sake of comparison,Segmentation of Clinical CT and MR from Hip Arthroplasty Patients

75

Fig. 2. Example of qualitative registration assessment with default NiftyReg regularisation parameters. The same axial and coronal slices are reported for the referencecomputed tomography, the super-resolution reconstructed magnetic resonance (MR)after affine registration, the super resolution reconstruction (SRR) MR after rigidly
constrained non rigid registration and the SRR MR after standard non-linear registration. For these latter cases, the transformation Jacobian determinant maps are also
displayed, showing the effect of the rigid constraints. Yellow arrows indicate exemplary areas where the proposed approach visually recovers a better alignment than thestandard fully non-linear registration (e.g. in the femoral head size).

we performed the same study using the standard non-linear registration without
the application of the rigid constraints, while keeping all the other parameters
unchanged. Although this variant would assume non-rigid deformation of thebones, which is neither anatomically nor clinically correct, such a comparison
allows us to verify whether our implementation also improved the registration
results compared to the classical approach.
The choice of the best parameters was based on both qualitative and quantitative analysis. The former included visual assessment of the alignment between
the CT and the registered MR and of the transformation Jacobian maps. An
example of this comparison is reported in Fig. 2, where the Jacobian determinantmaps clearly show how the standard registration algorithm fails in recovering arigid behaviour within the bones, as opposed to the proposed method.

76

Ranzini et al.

Fig. 3. Target registration error (TRE) analysis. Top figure: comparison of TRE root
mean square error (RMSE) values obtained from the rigidly constrained non-linear
registration and the standard one with varying regularisation parameters α - linear
elasticity weight - and β - bending energy weight. TRE RMSE from affine registrationis shown as well. Table: highest RMSE for each set of parameters. Starred values
indicate the sets for which the rigidly constrained registration provided significantly
lower RMSE than the standard (Wilcoxon rank sum test, p < 0.05). Highlighted in
red are the results for the selected best set of parameters. Bottom figure: manual
selection reproducibility error for the 10 landmarks and for the two modalities. List oflandmarks abbreviations: greater trochanter (GT), tensor fasciae latae (TFL), anteriorinferior iliac spine (AIS), gluteus maximus (GMAX), ischium (Isc). Each landmark is
identified in each side and it is categorized as healthy (H) or implanted (I) side.

A quantification of the registration accuracy was obtained through landmarks
analysis. Specifically, we labeled 5 landmarks (3 in bone, 2 in muscles) per hipside which could be conveniently located in both modalities and which cover
the full field of view. The target registration error (TRE) was computed as the
distance between the CT and the respective warped MR landmark. In order tolimit the bias from the manual landmark choice, we repeated the selection twice
at different times, we estimated the TRE for each selection and then computed
the average TRE for each landmark and for each subject (reproducibility errors
for the manual selection are shown in the bottom-left panel of Fig. 3). For eachsubject we extracted the root mean square error (RMSE) of the TRE across theten landmarks and we compared the distribution of the RMSE with respect tothe registration parameters. A summary of the obtained results is presented inFig. 3. Overall the proposed method not only provided clinically plausible regis-

Segmentation of Clinical CT and MR from Hip Arthroplasty Patients

77

Fig. 4. Example of automated segmentation obtained with the proposed automatedpipeline. Top row shows the central axial and coronal slices for one of the subjects forboth computed tomography and magnetic resonance, while the second row reports thesame images overlaid with the segmentation result. A three-dimensional rendering of
the full segmentation is also displayed on the bottom left for the same subject.

tration results, but also produced a more accurate alignment of the considered
landmarks compared to a standard non-linear registration algorithm. The best
set of parameters was identified as the one minimising the highest TRE RMSE
among all the landmarks, so as to guarantee a reasonably good alignment acrossthe whole field of view. We therefore concluded that the optimal results for the
intra-subject registration resulted from the use of normalised mutual information with α = 0.2 and β = 0.01.

3.2

Leave-One-Out Cross Validation

The proposed pipeline was validated through a leave-one-out cross-validation(LOOCV) experiment on the template datasets, by calculating the Dice score between the automated segmentation result and the corresponding manual groundtruth for each label and for each subject. The goal of the LOOCV was to compare the achieved results using both modalities jointly to those obtained using
only the CT or only the MR images. We recall that the manual segmentation

78

Ranzini et al.

of muscles was not available for the CT, and similarly bones and implant labelling on the MR. Therefore only the available labels were considered in the
single-modality experiments. For each analysed type – i.e. only CT, only MR,
combined modalities for healthy and for implanted sides – the segmentation
propagation and label fusion parameters were tuned to maximise the lowest average Dice Score across subjects and across labels. An example of the obtained
automated segmentation is shown in Fig. 4.
The median Dice score for bones, muscles and implant extracted from thethree LOOCV experiments are reported in Tab. 1. It can be observed that themultimodal and the single-modality approaches perform similarly with comparable Dice score values. Overall bone structures were better segmented than
muscular ones, due to their lower shape and texture variability. Although theobtained results appear slightly lower with the proposed approach, the observed
differences were statistically significant only in one case – i.e. for muscular structures in the healthy side – while for all the other cases the null hypothesis ofsame underlying distribution was accepted (Wilcoxon rank sum test with 5%
significance level). This difference could arise from the need of finding a tradeoff in the segmentation propagation and label fusion parameters to achieve agood accuracy in both skeletal and muscular structures for the 4D case. Thismight go at the expense of a slight reduction of performances with respect tothe single-modality case, where the parameters are tuned only for the bones andimplant (CT) or for the muscles (MR). Nonetheless, only the proposed framework is able to provide consistent and unified solution to the segmentation ofboth the CT and the MRI. The use of independent approaches to segment themuscles in the MR images and the bones and implants in the CT image would
indeed not guarantee non-overlapping regions of interest. As an example, on ourdataset we evaluated that on average 2% of the voxels labeled as muscle on
the MR overlapped with CT-labeled bone voxels in our manual segmentation,
while the proposed method guarantees no overlap by design. Without the use ofa registration framework able to deal with the rigid nature of the bones while
non-linearly deforming the surrounding soft tissue, it would be more challenging
to accurately highlight the muscles in the CT space or the bones in the MR
space, due to their poor contrast for these structures.

4

Conclusion

We presented a fully automated processing pipeline for the joint segmentationof bones, abductor muscles and implant on CT and MR images from hip arthroplasty patients. The combination of the two modalities enables accurate joint
delineation of healthy and pathological musculoskeletal structures and of their
spatial relationship.
As for other atlas-based approaches, the performance of our method could be
improved by enlarging the template data sets to better encompass the population variability. Moreover, the presence of metal artefact-induced noise strongly
affects the accuracy of both intra- and inter-subject registration; hence futureSegmentation of Clinical CT and MR from Hip Arthroplasty Patients

79

Table 1. Median Dice score values and 95% confidence intervals for bones, implant andmuscles: comparison between single- and multimodality results. Wilcoxon rank sum test
was performed to test the null hypothesis of same distribution for the multimodalityand the respective single-modality-derived Dice scores (obtained p-values are reported
and starred are the cases of rejection of the null hypothesis with 5% significance level).
N.A. indicates cases where the manual segmentation was not available.
Healthy side

Bones
MusclesCT

MR

Multimodal

p-value

0.95 [0.74, 0.97]
N.A.

N.A.
0.88 [0.74, 0.95]

0.94 [0.74, 0.96]
0.85 [0.70, 0.92]

0.164
0.007∗

Implanted side

Bones
Muscles
Implant

CT

MR

Multimodal

p-value

0.87 [0.63, 0.93]
N.A.
0.91 [0.77, 0.95]

N.A.
0.84 [0.60, 0.93]
N.A.

0.85 [0.53, 0.90]
0.77 [0.41, 0.90]
0.91 [0.69, 0.93]

0.365
0.054
0.970

developments of the processing pipeline will introduce novel metal artefact reduction techniques as an image quality enhancement step for the CT. In conclusion,
the proposed pipeline is a promising tool towards patient-specific 3D visualisation of musculoskeletal structures, and towards the extraction of clinically
relevant imaging biomarkers to detect implant failure. Thanks to our processing
steps, the implant can be outlined also on the MR image, where it is typically
obscured by the metal artefact. This could help identify the muscles that are
at greater risk of developing atrophy due to the presence of the implant, andtherefore inform the decision-making process for revision surgery.
Acknowledgements. This work is supported by the EPSRC-funded UCL Centre
for Doctoral Training in Medical Imaging [EP/L016478/1], the Royal National Orthopaedic Hospital NHS Trust, the Department of Healths NIHR-funded Biomedical
Research Centre at University College London Hospitals and Innovative Engineeringfor Health award by the Wellcome Trust [WT101957] and EPSRC [NS/A000027/1],
and by Wellcome/EPSRC [203145Z/16/Z].

References
1. UK National Joint Registry: 13th Annual Report (December 2015) (2016)
2. Berber, R., Khoo, M., Cook, E., Guppy, A., Hua, J., Miles, J., Carrington, R.,
Skinner, J., Hart, A.: Muscle atrophy and metal-on-metal hip implants: a serial
MRI study of 74 hips. Acta Orthop. 86(3), 351–357 (2015)
3. MHRA of the Department of Health: Medical Device Alert. MHRA Database pp.
1–7 (2012), http://www.mhra.gov.uk
4. Bogner, E., Sofka, C.: CT evaluation of total hip arthroplasty complication: Dissociation of acetabular component. HSS J. 3(1), 112–114 (2007)

80

Ranzini et al.

5. Pedoia, V., Majumdar, S., Link, T.: Segmentation of joint and musculoskeletal
tissue in the study of arthritis. Magn. Reson. Mater. Phy. 29(2), 207–221 (2016)
6. Seim, H., Kainmüller, D., Heller, M., Lamecker, H., Zachow, S., Hege, H.C.: Automatic segmentation of the pelvic bones from CT data based on a statisticalshape model. In: Proc. 1st Eurographics Conference on Visual Computing for
Biomedicine – EG VCBM 2008. pp. 93–100 (2008)
7. Yokota, F., Okada, T., Takao, M., Sugano, N., Tada, Y., Tomiyama, N., Sato,Y.: Automated CT segmentation of diseased hip using hierarchical and conditional statistical shape models. In: Mori, K., et al. (eds.) Proc. 16th International
Conference on Medical Image Computing and Computer-Assisted Intervention –MICCAI 2013, LNCS, vol. 8150, pp. 190–197. Springer (2013)
8. Pettersson, J., Knutsson, H., Borga, M.: Automatic hip bone segmentation using
non-rigid registration. In: Proc. 18th International Conference on Pattern Recognition – ICPR 2006. vol. 3, pp. 946–949. IEEE (2006)
9. Chu, C., Chen, C., Liu, L., Zheng, G.: FACTS: fully automatic CT segmentationof a hip joint. Ann. Biomed. Eng. 43(5), 1247–1259 (2015)10. Prescott, J., Best, T., Swanson, M., Haq, F., Jackson, R., Gurcan, M.: Anatomicallyanchored template-based level set segmentation: application to quadriceps muscles
in MR images from the osteoarthritis initiative. J. Digit. Imaging 24(1), 28–43(2011)
11. Ahmad, E., Yap, M., Degens, H., McPhee, J.: Atlas-registration based image segmentation of MRI human thigh muscles in 3D space. In: C.R., M.T., Kupinski, M.
(eds.) Proc. SPIE Medical Imaging 2014: Image Perception, Observer Performance,
and Technology Assessment, vol. 9037, p. 90371L. SPIE (2014)
12. Gilles, B., Magnenat-Thalmann, N.: Musculoskeletal MRI segmentation using
multi-resolution simplex meshes with medial representations. Med. Image Anal.
14(3), 291–302 (2010)
13. Klemt, C., Modat, M., Pichat, J., Cardoso, M., Henckel, J., Hart, A., Ourselin,
S.: Automatic assessment of volume asymmetries applied to hip abductor muscles
in patients with hip arthroplasty. In: Ourselin, S., Styner, M. (eds.) Proc. SPIE
Medical Imaging 2015: Image Processing, vol. 9413, p. 94131M. SPIE (2015)
14. Castro-Mateos, I., Pozo, J., Lazary, A., Frangi, A.: Automatic construction ofpatient-specific finite-element mesh of the spine from IVDs and vertebra segmentations. In: Gimi, B., Krol, A. (eds.) Proc. SPIE Medical Imaging 2016: Biomedical Applications in Molecular, Structural, and Functional Imaging, vol. 9788, p.
97881U. SPIE (2016)15. Sabah, S., Mitchell, A., Henckel, J., Sandison, A., Skinner, J., Hart, A.: Magnetic
resonance imaging findings in painful metal-on-metal hips: a prospective study. J.
Arthroplasty 26(1), 71–76 (2011)
16. Gorgolewski, K., Burns, C., Madison, C., Clark, D., Halchenko, Y., Waskom, M.,
Ghosh, S.: Nipype: a flexible, lightweight and extensible neuroimaging data processing framework in python. Front. Neuroinf. 5, 13 (2011)
17. Van Leemput, K., Maes, F., Vandermeulen, D., Suetens, P.: Automated modelbased bias field correction of MR images of the brain. IEEE Trans. Med. Imaging
18(10), 885–896 (1999)
18. Ebner, M., Chouhan, M., Patel, P., Atkinson, D., Amin, Z., Read, S., Punwani,
S., Taylor, S., Vercauteren, T., Ourselin, S.: Point-spread-function-aware slice-tovolume registration: application to upper abdominal MRI super-resolution. In:
Zuluaga, M., et al. (eds.) Proc. MICCAI International Workshop on Reconstruction
and Analysis of Moving Body Organs – RAMBO 2016, LNCS, vol. 10129, pp. 3–13.
Springer (2017)

Segmentation of Clinical CT and MR from Hip Arthroplasty Patients81

19. Modat, M., Cash, D., Daga, P., Winston, G., Duncan, J., Ourselin, S.: Global
image registration using a symmetric block-matching approach. J. Med. Imaging
1(2), 024003 (2014)20. Haber, E., Heldmann, S., Modersitzki, J.: A computational framework for imagebased constrained registration. Linear Algebra Appl. 431(3–4), 459–470 (2009)
21. Ashburner, J., Ridgway, G.: Symmetric diffeomorphic modeling of longitudinal
structural MRI. Front. Neurosci. 6, 197 (2013)
22. Modat, M., Daga, P., Cardoso, M., Ourselin, S., Ridgway, G., Ashburner, J.: Parametric non-rigid registration using a stationary velocity field. In: Proc. IEEE Workshop on Mathematical Methods in Biomedical Image Analysis – MMBIA 2012. pp.
145–150 (2012)
23. Cardoso, J., Leung, K., Modat, M., Keihaninejad, S., Cash, D., Barnes, J., Fox,
N., Ourselin, S.: STEPS: similarity and truth estimation for propagated segmentations and its application to hippocampal segmentation and brain parcelation.
Med. Image Anal. 17(6), 671–684 (2013)

Reconstruction of 3D Muscle Fiber Structure
Using High Resolution Cryosectioned Volume
Yoshito Otake1( ) , Kohei Miyamoto1 , Axel Ollivier1,2 , Futoshi Yokota1 ,Norio Fukuda1 , Lauren J. O’Donnell3 , Carl-Fredrik Westin3 , Masaki Takao4 ,Nobuhiko Sugano4 , Beom Sun Chung5 , Jin Seo Park6 , and Yoshinobu Sato1
1

Graduate School of Information Science, Nara Institute of
Science and Technology, Japan
2
Ecole Nationale Supérieure d’Ingénieurs de Caen, France
3
Brigham and Women’s Hospital and Harvard Medical School, USA
4
Graduate School of Medicine, Osaka University, Japan
5
Department of Anatomy, Ajou University School of Medicine, South Korea
6
Dongguk University School of Medicine, South Korea
otake@is.naist.jp

Abstract. Three-dimensional (3D) muscle fiber architecture is important in patient-specific biomechanical simulation. While several in-vivo
methods using diffusion tensor imaging and ultrasound have been demonstrated their feasibility in reconstruction of the fiber architecture, themain challenge is the lack of gold standard. Although physical measurement from cadavers has been considered as the accurate way of determining 3D muscle fiber architecture, its downsides include error in
the manual tracing and the labor intensive process allowing only sparse
sampling of a particular muscle. We propose an alternative method of
obtaining a dense fiber architecture of multiple muscles in close proximity using high resolution cryosectioned images. Similar to the diffusion
tensor imaging, we first extract the local orientation at each voxel using
the structure tensor analysis and then tractography algorithm is applied
to obtain stream lines. The proposed method was applied to all muscles
around the hip joint and the masticatory muscles. Qualitative comparison with the anatomy textbook indicated that the proposed method
reconstructed a plausible muscle fiber architecture. We plan to make thereconstructed fiber architecture of whole body muscles publicly available
in order to serve for the biomechanics community.
Keywords: Muscle fiber architecture, gold standard, high resolution
cryosectioned images.

1

Introduction

While the Hill-type muscle model that simplifies three-dimensional (3D) muscle
architecture as a few strings with the mechanical property of muscle-tendon
complex has successfully demonstrated usefulness of biomechanical simulation
in a number of applications including human gait [1] and surgical simulation [2],B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proc. 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. 82−90, 2017.

Reconstruction of Muscle Fiber Structure from Cryosectioned Images

83

limitations of the simplification include 1) the lack of ability to represent changesof the line-of-action during movement due to wrapping over the bones or othermuscles [3] and 2) difficulty in determining the parameters of each string which
in reality represents a cluster of muscle fibers [4]. Thus, 3D volumetric muscle
models containing the muscle fiber architecture [3, 5] have been investigated to
improve accuracy in musculoskeletal simulation.
In order to model the patient-specific muscle fiber architecture, diffusion
tensor imaging (DTI) and ultrasound [6, 7] have been employed and evaluatedagainst a ground truth obtained from cadaveric specimens [8,9]. The direct measurement of a fixed cadaver using a coordinate measurement machine (CMM)
provides reliable ground truth, however, it has the following downsides: 1) the
number of measurable fibers is limited due to the large amount labors, resulting
in a sparsely sampled fiber model, and 2) difficulty in measuring multiple contiguous muscles while keeping their original shape since dissection of the muscle
is inevitable. Since muscles often function as a group, the spatial relationship ofthe fiber architecture between neighboring muscles is important.
In this study, we propose an approach to reconstructing dense muscle fiberarchitecture of multiple muscles in one subject using a dataset of high resolution cryosectioned images. The method computes the gradient-based structure
tensor to obtain local orientation and computes tractography similar to the one
proposed by Wang et al. [10] in the optical coherence images of the brain. We
demonstrate and qualitatively evaluate the proposed method for muscles in the
hip and head regions.

(d)

(a)

(b)

(c)

(e)

Fig. 1. Cryo-section images contained in the Visible Korean Human [11, 12]. (a) axial
and coronal slices of the female pelvis dataset, (b) male whole body dataset, and (c) its
segmentation mask. The artifact correction (see text for details) has been applied to (a)and (b). Volume rendering and its projection view (ray-sum rendering) of (d) thegluteus maximus and (e) gluteus medius muscles.

84

Otake et al.

2

Method2.1

DatasetThe dataset we used in this study is the Visible Korean Human (VKH) [11, 12],which consists of a series of photographs of cryosection of human cadavers.
Among several datasets in VKH (different genders and type of anatomies), we
used two datasets in this study: 1) The male whole body dataset containing 8506
RGB images with 0.2 mm slice interval, 0.2 × 0.2 mm2 in-plane resolution, thematrix size of 2468 × 1407 and covering the entire body from the top of the headto the tip of the foot [11], and 2) female pelvis dataset containing 2508 images
with 0.1 mm interval, 0.1 × 0.1 mm2 in-plane resolution with 3520 × 1943 matrix
and covering from the top of the pelvis to about 8 cm inferior to the greater
trochanter [12]. Both datasets contain segmentation masks of anatomical structures including muscles, bones, vessels and other organs (937 and 154 structures
for the male whole body and female pelvis datasets, respectively) which weremanually traced by experts at 1.0 mm interval (i.e., every 5 slices for the maledataset and every 10 slices for the female dataset). B-spline interpolation was
performed to the segmentation mask to obtain a slice interval corresponding to
the RGB images and then the mask was applied to extract the target muscle
and stuck to obtain a volume with isotropic voxels (Fig. 1(d),(e)).
2.2Correction of Artifact

Despite careful calibrations in terms of geometry and color to keep the pixel
size and color response constant throughout the dataset, we observed a slight
discontinuity of the color balance at every few slices as shown in Fig. 2(a), whichappears as an artifact in the coronal slice and was enhanced when we observed
the dataset in a projection view (Fig. 2(b)). This artifact is likely to be originated from the slice-by-slice color adjustment process. In order to avoid accuracy
degradation in our texture analysis to extract the 3D orientation, we propose an
artifact correction process as detailed in the following. The repetitive artifact inz-direction (out-of-plane direction) is corrected by applying a low-pass filter inthe frequency domain.
We denote the original image and its Fourier transform as h(i, j, k) andH(x, y, z), respectively. The high frequency region in z-direction near the zaxis of H(x, y, z) was zero-masked by multiplying a mask volume defined as thefollowing:(p0, if x2 + y 2 < tr and |z| > tz ,(1)
M (x, y, z) =1, otherwise,
which represents a zero-valued cylinder along z-axis with radius of tr and filled
by one in between −tz < z < tz . tr and tz are the cut-off frequency which we
experimentally determined as 0.2 and 0.08 (cycles per millimeter) in this study.
An inverse Fourier transform of the masked volume, F −1 (H(x, y, z)M (x, y, z)),provides a volume after the artifact correction. In this study, we corrected artifact
in the R-, G-, and B-plane separately.

Reconstruction of Muscle Fiber Structure from Cryosectioned Images

(a)

(b)

85

(c)

Fig. 2. Example demonstration of the effect of the proposed artifact correction method.
(a) Coronal slice of the original RGB volume and a zoomed-in view of the yellow
rectangular region. The red arrows indicate the artifact especially distinct in the view.
(b) Anterior-posterior projection view, where the artifact appears more distinct. (c) The
grayscale converted volume after the proposed artifact correction. The effect of theartifact correction is clearly demonstrated.

2.3

Computation of Structure Tensor

The RGB volume after the artifact correction was then converted into gray-scaleusing rgb2gray() function in Matlab 2016a (The MathWorks Inc., Natick, MA,
USA) which follows the equation: grayvalue = 0.2989R + 0.5870G + 0.1140B.
The local orientation of the muscle fiber around each voxel was estimated by thegradient-based structure tensor [13]. The structure tensor of a volume-of-interest
centering at (i, j, k) is defined byJxx (i, j, k) Jxy (i, j, k) Jxz (i, j, k)
J(i, j, k) = Jyx (i, j, k) Jyy (i, j, k) Jyz (i, j, k) ,(2)
Jzx (i, j, k) Jzy (i, j, k) Jzz (i, j, k)ddwhere Jxy = { dxG(i, j, k; σ1 ) ∗ I(i, j, k)}{ dyG(i, j, k; σ1 ) ∗ I(i, j, k)} represents a
multiplication of Gaussian weighted first derivative with respect to the x and yaxes. The other element follows the same notation. In order to obtain a smoothtensor field, we apply another Gaussian filter to each element of J(i, j, k) witha standard deviation of σ2 . While σ1 needs to be adjusted based on the amountof image noise, σ2 determines smoothness of the tensor field which affects thefollowing tractography algorithm. We empirically found that σ1 = 1 mm andσ2 = 3 mm are suitable in our dataset, thus are used in the experiment below. An
example visualization of the computed structure tensor field is shown in Fig. 3(b).
The structure tensor becomes a pin-like shape when the volume-of-interest exhibits a sheet-like texture (the “pin” directs perpendicular to the sheet), while
it becomes a disk-like shape for a line-like structure (the disk spreads in theplane perpendicular to the line). When comparing with DTI, since the diffusion
happens in the direction along the line for a line-like structure, we proposed
the following conversion process of the structure tensor field in order to employ

86

Otake et al.

(a)

(b)

(c)

Fig. 3. Computation and visualization of the structure tensor. (a) Grayscale volume
after artifact correction. Visualization of the (b) structure tensor and (c) its inverse
computed in the right gluteus maximus muscle. Each tensor is represented as an ellipsoid whose principal axes correspond to the tensor’s eigenvector system (its lengthshows the absolute value of the tensor’s eigenvalue and the orientation is its eigenvector). The color indicates the linear measure, (λ1 − λ2 )/λ1 [14], of each tensor.

tractography algorithms commonly used in DTI, which we call an inverse of thestructure tensor.
λ01 = λ1 ,λ02 = λ1 + λ3 − λ2 ,λ03 = λ3 ,(3)where λ1 , λ2 , λ3 represent eigenvalues of the original structure tensor (λ1 >λ2 > λ3 ) and λ01 , λ02 , λ03 represent eigenvalues of the inverted structure tensor(which ensures λ01 > λ02 > λ03 ). Then, the eigenvectors corresponding to thelargest and smallest eigenvalues are swapped. The idea behind this inversion
process is to generate a diffusion-tensor-like tensor from the structure tensor by
converting the “pen”-tensor to the disk and the “disk”-tensor to the pen (notethat the metric called linear measure [14], cl = (λ1 − λ2 )/λ1 , becomes the planarmeasure, cp = (λ2 − λ3 )/λ1 , and vice versa after this inversion).
2.4

Computation of Tractography

From the inverted structure tensor field, tractography is computed. In this study,
we performed the tractography algorithm and visualization of the results in 3D
Slicer1 [15] via the SlicerDMRI project2 .

3Results

The tractography computed on 24 muscles around the hip joint in the female
pelvis dataset is shown in Fig. 4 together with the list of each muscle’s name.
Fig. 5 shows the results of four representative masticatory muscles (masseter,
buccinator, lateral and medial pterygoid) from the male whole body dataset. The
illustrations from an anatomy textbook [16] are shown next to each visualizationin order for the visual evaluation of the validity of the computed fiber bundles. In
1
2

www.slicer.org
dmri.slicer.org

Reconstruction of Muscle Fiber Structure from Cryosectioned Images

(a)

(b)

87

(c)

Fig. 4. Results of the computed fiber architecture of the muscles around the hip joint.
(a) Anterior and (b) posterior view. (c) List of muscles and the corresponding color
shown in (a) and (b). The relationship of the three-dimensional fiber architecture ofmultiple muscles is able to be analyzed.

order to further explore an application of the proposed method, we performed afiber clustering method [17] which applies the spectral clustering on a similarity
metric between fibers. The similarity metric used here is the fiber distance error
defied as the mean distance between pairs of corresponding points on the fibers.
Fig. 6 shows an example where the fibers in the gluteus medius muscle wereclustered into five clusters and visualized fibers in each cluster with one color.

(a) Masseter(b) Buccinator and pterygoid(c) Lateral and medial pterygoid(view from the left side)(d) Lateral and medial pterygoid(view from the back)Fig. 5. Results of the computed fiber architecture of the masticatory muscles. The
tractography (right) was visually compared with an anatomy textbook [16] (left). The
color of the fibers indicates the fiber’s orientation averaged over the entire line (red,green and blue components correspond to the x, y, and z direction, respectively).

88

Otake et al.

(a)

(b)

(c)

(d)

Fig. 6. Result of the fiber clustering [17] of the right gluteus medius muscle. The muscle
fiber bundles computed by the proposed method were clustered into five clusters. (a) to(d) show the views from different angles rotated by 45 degrees each. All fiber bundles
in each cluster was visualized with one color. The anterior region acting for flexion andinternal rotation is represented as clusters 3, 4 and 5, while the posterior region acting
for extension and external rotation is represented as clusters 1 and 2.

4

Discussion and Conclusion

We proposed a method to reconstruct a dense 3D fiber architecture of multiple
muscles in one functional unit using a high resolution cryosectioned volume.
The manual tracing in cadaver specimens, which is currently considered as agold standard, requires a large amount of labor and is subjected to the error
in the tracing by the CMM and imposes deformation that is implausible in invivo situation. On the other hand, the proposed method allows to reconstruct
a large number of fibers of multiple neighboring muscles without the effect of
deformation.
A straightforward application of the reconstructed fiber architecture is biomechanical simulation. The simulation with volumetric muscle models has beendrawing attention partly due to the rapid increase of computing power allowing
simulation of behavior of more detailed structure. The patient-specific characteristics of fiber arrangement is critical in such simulation environment. The resultof fiber clustering in Fig. 6 shows several clusters which may be corresponding
to each functional unit within a muscle. For example, the anterior fibers in thegluteus medius are argued to help to rotate the hip joint medially, while theposterior fibers help to rotate laterally. Such detailed analysis of the fiber architecture in each muscle may elucidate other functional units which are yet to be
identified.
Limitations of this study include the lack of the analysis of variability among
subjects. It is clearly not easy to obtain high resolution cryosectioned images ofmany subjects. We plan to use two subjects (male and female) [18] and analyze
inter-subject variability, which will allow us to explore possibility for performinga patient-specific adaptation using a simple local scaling based on a few measurements (e.g., height, weight, thigh length, etc.) similar to [1]. We also note

Reconstruction of Muscle Fiber Structure from Cryosectioned Images

89

that a non-rigid registration of the muscle’s outer shape extracted automaticallyfrom patient-specific images such as CT [19] may allow a more accurate patientspecific adaptation of the muscle fiber arrangement. Another potential limitation
concerns a quantitative evaluation. Due to the lack of gold standard, quantitative evaluation of the proposed method is quite challenging. The manual tracingof each fiber in the cryo-section volume is one possible solution to obtain the
ground truth, though visually tracking one muscle fiber over a number of slices
is troublesome due to ambiguous boarders between the neighboring fibers. Animal or cadaver studies subsequent cryo-section could be another option though
the accuracy of physical tracing of fibers and muscle’s deformation become error
source in that case. Further investigation on quantitative evaluation is in our
future work.
Acknowledgements. This research was supported by MEXT/JSPS KAKENHI 26108
004, JST PRESTO 20407, and AMED/ETH the strategic Japanese-Swiss cooperative
research program, NIH grant U01CA199459 and P41EB015902.

References
1. Rajagopal, A., Dembia, C., DeMers, M., Delp, D., Hicks, J., Delp, S.: Full-body
musculoskeletal model for muscle-driven simulation of human gait. IEEE Trans.
Biomed. Eng. 63(10), 2068–2079 (2016)
2. Marra, M., Vanheule, V., Fluit, R., Koopman, B., Rasmussen, J., Verdonschot, N.,
Andersen, M.: A subject-specific musculoskeletal modeling framework to predict invivo mechanics of total knee arthroplasty. J. Biomech. Eng. 137(2), 020904 (2015)
3. Webb, J., Blemker, S., Delp, S.: 3D finite element models of shoulder muscles forcomputing lines of actions and moment arms. Comput. Methods Biomech. Biomed.
Engin. 17(8), 829–837 (2014)
4. Hicks, J., Uchida, T., Seth, A., Rajagopal, A., Delp, S.: Is my model good enough?
best practices for verification and validation of musculoskeletal models and simulations of movement. J. Biomech. Eng. 137(2), 020905 (2015)
5. Blemker, S., Delp, S.: Three-dimensional representation of complex muscle architectures and geometries. Ann. Biomed. Eng. 33(5), 661–673 (2005)
6. Damon, B., Froeling, M., Buck, A., Oudeman, J., Ding, Z., Nederveen, A., Bush,E., Strijkers, G.: Skeletal muscle diffusion tensor-MRI fiber tracking: rationale,data acquisition and analysis methods, applications and future directions. NMR
Biomed. 30(3), e3563 (2017)
7. Bolsterlee, B., Veeger, H., van der Helm, F., Gandevia, S., Herbert, R.: Comparison of measurements of medial gastrocnemius architectural parameters fromultrasound and diffusion tensor images. J. Biomech. 48(6), 1133–1140 (2015)
8. Schenk, P., Siebert, T., Hiepe, P., Güllmar, D., Reichenbach, J., Wick, C., Blickhan,R., Böl, M.: Determination of three-dimensional muscle architectures: validation of
the DTI-based fiber tractography method by manual digitization. J. Anat. 223(1),61–68 (2013)
9. Engelina, S., Robertson, C., Moggridge, J., Killingback, A., Adds, P.: Using ultrasound to measure the fibre angle of vastus medialis oblique: a cadaveric validation
study. Knee 21(1), 107–111 (2014)

90

Otake et al.

10. Wang, H., Lenglet, C., Akkin, T.: Structure tensor analysis of serial optical coherence scanner images for mapping fiber orientations and tractography in the brain.
J. Biomed. Opt. 20(3), 036003 (2015)11. Park, J., Chung, M., Hwang, S., Lee, Y., Har, D., Park, H.: Visible Korean human:
improved serially sectioned images of the entire body. IEEE Trans. Med. Imaging
24(3), 352–360 (2005)
12. Shin, D., Jang, H., Hwang, S., Har, D., Moon, Y., Chung, M.: Two-dimensional
sectioned images and three-dimensional surface models for learning the anatomyof the female pelvis. Anat. Sci. Educ. 6(5), 316–323 (2013)
13. Bigun, J.: Optimal orientation detection of linear symmetry. In: Proc. 1st IEEE
International Conference on Computer Vision – ICCV 1987. pp. 433–438. IEEE
(1987)
14. Westin, C., Maier, S., Mamata, H., Nabavi, A., Jolesz, F., Kikinis, R.: Processing
and visualization for diffusion tensor MRI. Med. Image Anal. 6(2), 93–108 (2002)
15. Fedorov, A., Beichel, R., Kalpathy-Cramer, J., Finet, J., Fillion-Robin, J., Pujol, S., Bauer, C., Jennings, D., Fennessy, F., Sonka, M., Buatti, J., Aylward, S.,
Miller, J., Pieper, S., Kikinis, R.: 3D Slicer as an image computing platform for the
Quantitative Imaging Network. Magn. Resonan. Imaging 30(9), 1323–1341 (2012)
16. Netter, F.: Atlas of Human Anatomy. Netter Basic Science, Elsevier Health Sciences (2010)
17. O’Donnell, L., Westin, C.F.: Automatic tractography segmentationu using a highdimensional white matter atlas. IEEE Trans. Med. Imaging 26(11), 1562–1575(2007)
18. Park, H., Choi, D., Park, J.: Improved sectioned images and surface models of thewhole female body. Int. J. Morphol. 33(4), 1323–1332 (2015)19. Yokota, F., Takaya, M., Okada, T., Sugano, N., Tada, Y., Tomiyama, N., Sato, Y.:
Automated muscle segmentation from 3D CT data of the hip using hierarchicalmulti-atlas method. In: Proc. 12th Annual Meeting of the International Society for
Computer Assisted Orthopaedic Surgery – CAOS 2012 (2012)

Segmentation of Pathological Spines in CT
Images Using a Two-Way CNN and a
Collision-Based Model
Robert Korez(), Boštjan Likar, Franjo Pernuš, and Tomaž Vrtovec

University of Ljubljana, Faculty of Electrical Engineering, Slovenia
robert.korez@fe.uni-lj.si

Abstract. Accurate boundary delineation and segmentation of pathological spines is indispensable in spine-related applications that rely on
the knowledge of vertebral shape. However, exact vertebral boundaries
are often difficult to determine due to articulation of vertebrae with eachother that may cause vertebral overlaps in segmentations of adjacentvertebrae. To solve this problem, we propose a novel method that consists of two steps. In the first step, the probability maps that determinevertebral boundaries are obtained from a two-way convolutional neuralnetwork, trained on normal thoracolumbar spines. In the second step, acollision-based model that consists of (at least two) consecutive vertebra
mesh models is initialized close to the observed vertebrae and verticesof each mesh are displaced towards the detected boundaries. As this canlead to mesh collisions in the form of vertices of one mesh penetrating the adjacent one (and/or vice versa), these vertices are efficiently
detected and then driven out of the adjacent mesh while locally preserving the shape of the corresponding mesh. By applying the proposed
method to 15 three-dimensional computed tomography images of thelumbar spine containing 75 normal and fractured vertebrae, quantitative
comparison against reference vertebra segmentations yielded an overallmean Dice similarity coefficient of 93.2%, mean symmetric surface distance of 0.5 mm, and Hausdorff distance of 8.4 mm.
Keywords: Image segmentation, computed tomography, pathologicalspine, two-way convolutional neural network, collision-based model.

1

Introduction

Accurate boundary delineation and segmentation of vertebrae in three-dimensional (3D) computed tomography (CT) images is essential to the clinical diagnosis and treatment of pathological conditions that affect the spine. However,exact vertebral boundaries are often difficult to determine due to articulation ofvertebrae with each other that may cause vertebral overlaps in segmentations ofadjacent vertebrae. Thus, despite an increasing interest in vertebra segmentationin recent years, accurate segmentation methods of pathological spines are stilllacking.
B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proc. 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. 91−103, 2017.

92

Korez et al.

Related Work. In the past decade, several automated and semi-automatedvertebrae segmentation methods that directly or indirectly address the vertebral overlap have been developed. Kim and Kim [1] proposed a fully automatedmethod that was based on constructing 3D fences to separate vertebrae, and thenthe region growing algorithm was applied within the constructed 3D fences toobtain the final segmentation. Klinder et al. [2] carried out segmentation of vertebrae by using an ensemble of shape-constrained deformable models [3], wherevertebrae models were interacting with each other throughout the adaptation
to target vertebrae. A statistical multi-vertebrae anatomical model that treatedthe shape and pose of each vertebra independently was developed by Rasoulian et al. [4], and by the nature of the model definition, vertebral overlap was
avoided. Recently, in the work of Castro-Mateos et al. [5], vertebral overlap wasaddressed with the statistical interspace model, which provides the neighboring
relationship between regions in different objects of a multi-object structure bylearning the statistical distribution of the interspace between them. The limitations of the aforementioned studies are: (i) manual interactions were often
required [4, 5], which can be labor-intensive, time-consuming and error-prone,especially when observing large datasets, (ii) statistical shape modeling cannot
always capture shape features of pathological samples as they may be suppressedby shape features of a considerably larger number of healthy samples [2, 4, 5],(iii) the complexity and variability in the appearance of vertebrae and theirsurroundings cannot be handled by low-level appearance representations, suchas intensities [4], valley-emphasized Gaussian intensities [1] or intensity gradients [2, 5], and (iv) vertebral overlaps were not completely prevented [2].
To overcome the above mentioned limitations, we combine recent developments in machine learning (i.e. deep learning) and shape modeling (i.e. deformable models), and propose a novel method for fully automated segmentationof vertebrae from 3D CT pathological spine images. In the first step, the probability maps that determine vertebral boundaries are obtained from a two-way
convolutional neural network (CNN), trained on normal thoracolumbar spines.
In the second step, a collision-based model that consists of (at least two) consecutive vertebra mesh models is initialized close to the observed vertebrae andvertices of each mesh are displaced towards the determined boundaries. As thiscan lead to mesh collisions in the form of vertices of one mesh penetrating theadjacent one (and/or vice versa), these vertices are efficiently detected and thendriven out of the adjacent mesh while locally preserving the shape of the corresponding mesh. The proposed method was evaluated on a publicly available
database of 15 CT images of the lumbar spine and demonstrated highly accuratesegmentations of 75 normal and fractured vertebrae.

2

Methodology

The proposed method consists of two consecutive steps. In the first step, a testing image of the pathological spine is run through a two-way CNN that is trainedon a repository of training images of normal spines, resulting in three probability

Segmentation of Pathological Spines

93

Fig. 1. A schematic illustration of the applied two-way convolutional neural networkarchitecture.

maps that represent likelihoods of image voxels to belong to the anterior arch(i.e. vertebral body), posterior arch (i.e. spinous, transverse, superior articularand inferior articular processes, and pedicles) and background. By thresholding
the anterior arch probability map and using simple morphological operations
on the obtained binary image, centroids of vertebral bodies are coarsely determined. In the second step, a collision-based model is initialized as a mesh unionof (at least two) consecutive vertebrae with the mean pose and shape obtained
from the repository of training meshes of normal vertebrae, and placed in thevicinity of the target vertebrae using the previously determined centroids. The
segmentation of target vertebrae is performed by an iterative approach where
the model is deformed toward high-magnitude gradients of the augmented background probability map, and mesh collisions in the form of one mesh penetrating
the adjacent one (and/or vice versa) are prevented.
2.1

Two-Way CNN

In the field of medical imaging, CNNs are currently one of the main tools used
for solving challenging tasks that have traditionally depended on experts forsolution [6]. Such tasks include object detection and identification [7], image
segmentation [8] and assessment of condition severity [9]. Without relying on
handcrafted features, CNNs have the ability to learn features for a specific task
directly from the raw image data, and since features are learned in hierarchy ofincreasing complexity and abstraction, they provide robustness and flexibility.
The proposed two-way CNN architecture is, similarly as first proposed in [10]and later adopted in [11–13], made of two paths in order to perform voxelwise classification based on appearance (i.e. local path) and spatial location (i.e.
global path) details around the voxel of interest.
Training. In each epoch of the training stage, thousands of pairs of image
patches are extracted around randomly selected voxels from the repository of
training images of the normal spines, and sent through a two-way CNN in orderto learn its weights and biases θ. In detail, let pair (L, G) denote the collectionof all extracted pairs of image patches in the current epoch. For each randomly
selected voxel v, two image patches L ∈ L and G ∈ G of size 27 × 27 × 27 and94

Korez et al.

(a)

(b)

Fig. 2. The results of the trained two-way convolutional neural network applied on atesting image of the pathological spine. (a) A testing image, shown in the mid-sagittal
cross-section. (b) The corresponding probability maps that represent likelihoods ofimage voxels to belong to the anterior arch (left), posterior arch (middle) and background (right). The colors range from dark blue for low probabilities to dark red forhigh probabilities of voxels to belong to each segmentation class.

55 × 55 × 55, respectively, are extracted around the voxel v and sent through twopaths of different operations (i.e. convolution, pooling). In the first (local) path,
patch L is sent through nine convolutional layers with small kernels (i.e. of size
3 × 3 × 3) and residual connections, each followed by a parametric rectified linear
unit (PReLU). In the second (global) path, patch G is sent consecutively twotimes through three convolutional layers with kernels of size 5 × 5 × 5, 4 × 4 × 4
and 3 × 3 × 3, respectively, each followed by a PReLU, and a (max) poolinglayer with pool of size 2 × 2 × 2 and stride of size 2 × 2 × 2. To achieve that
outputs of both paths have the same size, the final layer of the second path
is a transposed (convolutional) layer with kernels of size 3 × 3 × 3 and strideof size 2 × 2 × 2, followed by a PReLU. The resulting outputs of both paths
are jointly sent through three fully connected layers, each seen as a convolution
one with kernels of size 1 × 1 × 1 and followed by a PReLU. The outputs of
the last fully connected layer correspond to particular segmentation classes (i.e.
anterior arch, posterior arch and background) and are finally fed to a softmax
function to obtain the predicted posterior probabilities that the group of voxels(of size 9 × 9 × 9) around the randomly selected voxel v belongs to each of
the segmentation classes. The two-way CNN architecture is in detail illustrated
in Fig. 1. By interpreting the probabilities of the CNN as distributions over thesegmentation classes, a natural training criteria is to maximize the probability of
all classes in the training repository, i.e. to minimize the negative log-likelihood
probability for each segmented image in the training repository. The optimal
weights and biases θ of the CNN are obtained using the hybrid scheme [13]between the dense training scheme on a whole image [14] and the commonlyused training scheme on individual patches.
Testing. In the testing stage, the two-way CNN with optimally learned weights
and biases θ is applied on the testing image I of the pathological spine in orderSegmentation of Pathological Spines95

to obtain three probability maps A, P and B that represent likelihoods of image
voxels to belong to the anterior arch, posterior arch and background, respectively.
In detail, the input image I is first fragmented into I non-overlapping blocks ofsize 9 × 9 × 9. For each block, two image patches Li and Gi , i ∈ I, of size
27 × 27 × 27 and 55 × 55 × 55, respectively, are extracted around its central
voxel. By sending Li and Gi through the trained two-way CNN, probability
maps Ai , P i and B i are obtained and assembled into probability maps A, P
and B, respectively. The testing image with corresponding probability maps isshown in Fig. 2.
2.2

Collision-Based Model

Initialization. A collision-based model is initialized as a mesh union of twoconsecutive vertebrae with the mean pose and shape obtained from the repository of training meshes of normal vertebrae, and denoted by M = M1 ∪ M2
with vertices M = M 1 ∪ M 2 and faces F = F 1 ∪ F 2 . In order to place mesh M
in the vicinity of target vertebrae, the probability map A (Sec. 2.1, Fig. 2(b)),
which is obtained from the two-way CNN, is thresholded into vertebral bodies and background. Then, the obtained binary mask is processed with simple
morphological operations, such as hole filling, dilation and erosion, to obtain
multiple connected components with their centers of masses coarsely defining
the centroids of all visible vertebral bodies in image I. Finally, the centers ofmasses of M1 and M2 are jointly repositioned near previously determined centroids of two consecutive vertebral bodies, and the repositioned mesh union isdenoted by X = X1 ∪ X2 with vertices X = X 1 ∪ X 2 and faces F .
Segmentation. The segmentation of the target vertebrae is performed by an
iterative approach, where vertices X of mesh X are deformed toward highmagnitude gradients of probability map B. However, along the high-magnitude
gradients that represent vertebral boundaries, holes or protuberances that are
not part of the vertebrae may appear due to the two-way CNN mispredictions.
Therefore, probability map B is augmented with image I as B 0 = (1+α·B) I,
where is the element-wise multiplication and α is the weighting factor.
In each iteration, the deformation of vertices X is represented as an optimization problem, where each vertex is iteratively driven by the informationfrom B 0 towards vertebral boundaries and constrained by the topology of mesh
M with mean pose and shape. The optimization problem is formulated as∗

X = arg min|X|
X

X∈R|X|×3 i=1

wi projg(x∗i ) (x∗i − xi )βXj∈N (xi )2+2k(xi − xj ) − (mi − mj )k

!

(1)

96

Korez et al.

andx∗i = xi + arg max Fi (xi + j δ n(xi )) − D δ 2 j 2 δ n(xi ),

(2)

j∈J

Fi (x) = hn(xi ), g(x)i ,wi = max 0, arg max Fi (x∗i ) − D δ 2 j 2
,


(3)
(4)

j∈J

where xi ∈ X is the current vertex position, n(xi ) is the mesh outwards normal at xi , J is the sampling parcel that represents the discrete search profile
along n(xi ), D and δ serve to penalize large distances between xi and x∗i , g(x)is the gradient of the augmented probability map B 0 at location x, N (xi ) is
the set of vertices in the one-ring neighbourhood of xi (i.e. set of vertices connected to vertex xi ), mi ∈ M , β is the weighting factor and proj is the vector
projection [3].
The optimization problem can however lead to mesh collisions in the form of
one mesh (e.g. X1 ) penetrating the adjacent one (i.e. X2 ), and/or vice versa. To
avoid such a scenario, we introduce a novel step in the iterative approach wherevertices from X 1 that penetrate X2 are efficiently detected and then driven outof X2 by the as-rigid-as-possible modeling [15] that locally preserves the shapeof X1 .
Let vertices Y = Y 1 ∪Y 2 denote the solution of the optimization problem (1)at each iteration, and let Y = Y1 ∪ Y2 denote the corresponding mesh with faces
F . To determine if Y1 penetrates Y2 , vertices from Y 1 have to be tested whether
they are inside Y2 . Firstly, for a vertex y 1,i ∈ Y 1 , a ray r 1,i (t) = y 1,i + td1,i isshot out of y 1,i in a random direction d1,i . Secondly, for each face f 2,j ∈ F 2 ,j = 1, 2, . . . , |F 2 |, the matrix equation in the form
h
i
(2)
(1)
(3)
(1)
(1)
T
−d, y 2,j − y 2,j , y 2,j − y 2,j · [t, u, v] = y 1,i − y 2,j(5)(k)

is solved for t, u, v ∈ R, where y 2,j ∈ Y 2 , k = 1, 2, 3, are three vertices thatdefine face f 2,j [16]. It can be concluded that if u, v ≥ 0 and u + v = 1, then
ray r 1,i intersects face f 2,j . Let N1,i denote the number of faces that ray r 1,iintersects. Finally, it can be concluded that if N1,i ≡ 1 mod 2, then vertex y 1,iis inside Y2 (Alg. 1).
Based on the above, let Y 01 ⊂ Y 1 denote vertices that are inside Y2 . The
most straightforward way to drive vertices Y 01 out of Y2 is an iterative approach
where vertices Y 01 are firstly displaced for a small distance in the inward direction
along the corresponding vertex normals and then tested whether they are stillinside Y2 . However, such primitive deformations of mesh Y1 can significantly
change its topology (e.g. spikes jutting out of the mesh) and the outcome isa mesh that may not represent anatomically correct vertebral boundaries. To
avoid such a scenario, vertices Y 01 are rather displaced by the as-rigid-as-possiblemodeling [15] that locally preserves the shape of Y1 .
The as-rigid-as-possible modeling is a powerful deformation technique that
produces anatomically natural-looking results with guaranteed convergence. The

Segmentation of Pathological Spines

97

Algorithm 1 Test if a vertex is inside a mesh
input: vertex y 1,i ∈ Y 1 , mesh Y2 with vertices Y 2 and faces F 2
output: boolean b
1: set b = false, N1,i = 0
2: shot a ray r(t) = y 1,i + td, t ∈ R, out of vertex y 1 in a random direction d
3: for all faces
f 2,j ∈ F 2 do
h
i
4:

(2)

(1)

(3)

(1)

(1)

solve −d, y 2,j − y 2,j , y 2,j − y 2,j · [t, u, v]T = y 1,i − y 2,j for t, u, v ∈ R, where(k)y 2,j ∈ Y 2 , k = 1, 2, 3, are three vertices that define face f 2,j [16]5:if u, v ≥ 0 and u + v ≤ 1 then6:
set N1,i = N1,i + 1
7:
end if
8: end for
9: if N1,i ≡ 1 mod 2 then10:
set b = true
11: end if

main principle used is that small local parts of the mesh change smoothly andas rigidly as possible. Let Z 1 denote initial vertices, i.e. Y 1 , and let Z 01 denote
displaced vertices, i.e. (Y 1 \Y 01 ) ∪ Y 001 , where Y 001 denotes vertices Y 01 that weredisplaced for a small distance in the inward direction along the corresponding
vertex normals. To achieve that the topology is preserved in small local neighbourhoods of displaced vertices, the as-rigid-as-possible modeling of vertices Z 01

(a)

(b)

Fig. 3. (a) One of the meshes in the mesh union configuration with few vertices (inred) that were penetrating the adjacent mesh (not visible). (b) The resulting mesh after
iteratively displacing the penetrated vertices for a small distance in the inward direction
along the corresponding vertex normal using the as-rigid-as-possible modeling. [Notethat for a clear visualization, we manually displaced a few vertices on the superior
endplate, and the demonstrated situation is unlikely to occur.]98

Korez et al.

is represented as an optimization problem and formulated as∗
Z 0∗
1 , {Ri } =arg min0
Z 01 ∈R|Z 1 |×3{Ri },Ri ∈R3×3

Z 01 ||Xi=1

X

z 01,i − z 01,j −wijj∈N (z 01,i )Ri (z 1,i − z 1,j )

wij =

1
(cot αij + cot βij ) ,
2

2

!

(6)

,

(7)

where z 1,i ∈ Z 1 , z 01,i ∈ Z 01 , αij and βij are the opposite angles of the meshedge defined between vertices z 1,i and z 1,j , N (z 01,i ) is the set of vertices in the
one-ring neighbourhood of z 01,i and Ri is the rotation that best approximates
the transformation that takes vertex z 1,i to z 01,i (Fig. 3). To solve (6) for Z 01 and{Ri }, a two-step alternating minimization strategy is used [15]. In the first step,
vertices Z 01 are considered as fixed so that the rigid transformations are the only
unknowns, and they are determined byXSi =wij (z 1,i − z 1,j )(z 01,i − z 01,j )T = U i Σi V Ti ,(8)j∈N (z 01,i )Ri = U i V Ti ,(9)where U i Σi V Ti is the singular value decomposition of S i . In the second step,with the given set of rigid transformations, vertices Z 01 are determined by minimizing (6). These interleaved iterations are repeated until the minimum isreached. The resulting mesh Z1 ∪Z2 with mesh vertices Z 1 ∪Z 2 finally represents
the segmentation of two consecutive vertebrae.

3

Experiments and Results

3.1

Spine Image Database

The proposed method for segmentation of pathological spines in CT images wastrained (i.e. the two-way CNN training and the creation of mesh unions of twoconsecutive vertebrae with the mean pose and shape) on two and tested on one
publicly available databases of 3D CT images that are part of the SpineWeb,1
a collaborative platform for research on spine imaging and image analysis. The
first (training) database consists of 10 axially reconstructed CT images of thelumbar spine with a total of 50 normal lumbar vertebrae, in-plane voxel size of
0.282 − 0.791 mm and cross-sectional thickness of 0.725 − 1.530 mm [17]. The
1

Accessible via http://spineweb.digitalimaginggroup.ca.

Segmentation of Pathological Spines

99

Table 1. The segmentation results of the proposed method with and without collision
detection in terms of mean ± standard deviation of the Dice similarity coefficient(DSC), mean symmetric surface distance (MSD) and Hausdorff surface distance (HD).

With collision detection

Without collision detection

Level

DSC (%) MSD (mm) HD (mm)DSC (%) MSD (mm) HD (mm)L1

94.3 ± 2.2 0.4 ± 0.1

6.4 ± 2.2

94.1 ± 2.4 0.4 ± 0.1

L293.8 ± 2.0 0.5 ± 0.2

8.0 ± 2.6

93.6 ± 2.2 0.5 ± 0.2 11.0 ± 3.3

L393.9 ± 1.6 0.5 ± 0.1

7.4 ± 3.5

93.4 ± 2.3 0.5 ± 0.2 10.7 ± 3.6

L492.2 ± 2.2 0.6 ± 0.2 10.0 ± 3.9

91.3 ± 3.0 0.7 ± 0.2 14.1 ± 4.5

L591.8 ± 1.8 0.6 ± 0.1 10.4 ± 3.2

91.4 ± 2.8 0.6 ± 0.2 12.8 ± 5.0

All93.2 ± 2.2 0.5 ± 0.2

92.7 ± 2.8 0.5 ± 0.2 11.1 ± 4.6

8.4 ± 3.4

6.8 ± 2.7

second (training) database consists of 10 axially reconstructed CT images of thethoracolumbar spine with a total of 170 normal thoracic and lumbar vertebrae,
in-plane voxel size of 0.313−0.361 mm and cross-sectional thickness of 1 mm [18].
The third (testing) database consists of 15 axially reconstructed CT images of thelumbar spine with a total of 75 normal and fractured lumbar vertebrae, in-planevoxel size of 0.289 − 0.803 mm and cross-sectional thickness of 1.105 − 1.892 mm[19]. For all databases, a reference manually defined binary segmentation mask
was available for each vertebra in each image.
3.2

Experimental Setup

Two-Way Convolutional Neural Network. Since the two-way CNN is able
to learn useful features from scratch, only minimal image preprocessing wasapplied (on training and testing databases). Firstly, all images were resampled
to an isotropic resolution of 1 × 1 × 1 mm3 per voxel (using linear interpolation)
since the kernels of the CNN need to correspond to the same real-size patterns
for all images. Secondly, images were smoothed using a Gaussian kernel of size
0.5 mm. Finally, the obtained intensities were linearly mapped to interval [−5, 5].
To train the two-way CNN, in each out of 40 epochs, 80k pairs of patches wereextracted around randomly selected voxels from images in the first and second
database. The CNN was trained via the stochastic gradient descent method andaccelerated by the Nesterov momentum with its parameter set to 0.9. To avoid
overfitting, training was augmented by the dropout (the outputs of last two fully
connected layers were randomly set to zero with probabilities of 0.5) and batchnormalization.

100

Korez et al.

(a)

(b)

Fig. 4. The segmentation results of the proposed method (a) with and (b) without
collision detection, shown in mid-sagittal cross-sections (top) and in three dimensions
with superimposed color-coded symmetric surface distances against corresponding reference segmentations (bottom). The colors range from blue for short to red for large
surface distances (up to 11 mm). The arrow indicates inaccurate delineation of the posterior arch due to the collisions that occur during the deformation of vertices towards
vertebral boundaries.

Mean Pose and Shape Mesh Union. For each i-th lumbar vertebrae pair(i = 1, 2, . . . , 5, corresponding to pairs T11-L1, L1-L2, L2-L3, L3-L4 and L4L5), the mean pose and shape mesh union Mi ∪ Mi+1 of corresponding two
consecutive vertebrae was constructed. Firstly, the marching cubes algorithm
was applied to each binary mask B jk that represents the reference segmentationof j-th vertebra (j = 1, 2, . . . , 6, corresponding to one thoracic and five lumbar
vertebrae) in k-th training image (k = 1, 2, . . . , 10, corresponding to 10 images
in the second database). Secondly, each resulting mesh Mj,k was isotropicallyremeshed [20] so that the number of vertices was independent of the size of jth vertebra and voxel size of k-th image. Thirdly, the coherent point drift wasapplied to establish pointwise vertex correspondences among meshes {Mj,k }10
k=1
of j-th vertebra. Finally, the joint pose+shape modeling [21] was used to include,in addition to shape variations, pose variations in i-th vertebrae pair {Mi,k ∪Mi+1,k }10
k=1 , yielding the mean pose and shape mesh union Mi ∪ Mi+1 for eachvertebrae pair.
Collision-Based Model. The collision-based model was initialized as the meanpose and shape mesh union Mi ∪ Mi+1 corresponding to i-th observed lumbar
vertebrae pair was aligned with the centroids of corresponding vertebral bodies

Segmentation of Pathological Spines

101

that were determined by processing two-way CNN probability maps. The segmentation parameters that guide the model toward vertebra boundaries wereexperimentally set to α = 1, δ = 1 mm, |J | = 51, i.e. the length of the discrete
search profile was 25 mm in the inward and outward direction, D = 2.5 mm−2
and β = 25. The number of iterations was set to 5 for the deformation (2)-(4)and as-rigid-as-possible modeling (7)-(9) of mesh vertices. For each resulting
mesh pair Zi ∪ Zi+1 , only Zi+1 was used to generate the binary mask of the final
segmentation since Zi was used to eliminate mesh collisions.
3.3

Results

The vertebrae segmentation performance was evaluated on the third databasethat consists of lumbar spine images with normal and fractured vertebrae bycomputing (with respect to the reference manual segmentations) the Dice similarity coefficient (DSC), mean symmetric surface distance (MSD) and Hausdorff
surface distance (HD) in terms of mean ± standard deviation. For all vertebrae,
DSC of 93.2 ± 2.2%, MSD of 0.5 ± 0.2 mm and HD of 8.4 ± 3.4 mm were obtained.
Detailed results of the proposed method with and without collision detection arepresented in Tab. 1 for all vertebrae levels and in Fig. 4 for a randomly selectedvertebra.

4

Discussion

In this paper, we propose a novel method for fully automated segmentation ofvertebrae from 3D CT pathological spine images that combines recent developments in machine learning and shape modeling. A two-way CNN is first used todetermine vertebral boundaries, and then a collision-based model is used to obtain final segmentations without vertebral overlaps. Similarly as in the method
proposed by Castro-Mateos et al. [5], a limitation of the proposed method is thatduring the segmentation of vertebra L5 (i.e. pair L4-L5), there is no guaranteethat L5 segmentation results did not over-segment sacral vertebra S1. A feasible solution would be either to include vertebra S1 as an additional label into
the training of the two-way CNN or the lumbosacral vertebra pair L5-S1 into
the collision-based model. Despite this minor limitation, the proposed method
demonstrated highly accurate segmentations of normal and fractured vertebrae.
Directions for future work include the application of the proposed method forthe evaluation of vertebral fractures, as well as its extension to other anatomical structures (e.g. carpal bones or multiple abdominal organs) and imaging
modalities (e.g. magnetic resonance).
Acknowledgements. This work was supported by the Slovenian Research Agency(ARRS) under grants P2-0232, J2-5473, J7-6781 and J2-7118.

102

Korez et al.

References
1. Kim, Y., Kim, D.: A fully automatic vertebra segmentation method using 3D
deformable fences. Comput. Med. Imag. Graph. 33(5), 343–352 (2009)
2. Klinder, T., Ostermann, J., Ehm, M., Franz, A., Kneser, R., Lorenz, C.: Automatedmodel-based vertebra detection, identification, and segmentation in CT images.
Med. Image Anal. 13(3), 471–482 (2009)
3. Weese, J., Kaus, M., Lorenz, C., Lobregt, S., Truyen, R., Pekar, V.: Shape constrained deformable models for 3D medical image segmentation. In: Insana, M.,
Leahy, R. (eds.) Proc. 17th International Conference on Information Processing in
Medical Imaging – IPMI 2001, LNCS, vol. 2082, pp. 380–387. Springer (2001)
4. Rasoulian, A., Rohling, R., Abolmaesumi, P.: Lumbar spine segmentation using astatistical multi-vertebrae anatomical shape+pose model. IEEE Trans. Med. Imaging 32(10), 1890–1900 (2013)
5. Castro-Mateos, I., Pozo, J., Pereañez, M., Lekadir, K., Lazary, A., Frangi, A.:
Statistical interspace models (SIMs): application to robust 3D spine segmentation.
IEEE Trans. Med. Imaging 34(8), 1663–1675 (2015)
6. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444(2015)
7. Chen, H., Shen, C., Qin, J., Ni, D., Shi, L., Cheng, J., Heng, P.A.: Automatic
localization and identification of vertebrae in spine CT via a joint learning modelwith deep neural networks. In: Navab, N., et al. (eds.) Proc. 18th International
Conference on Medical Image Computing and Computer-Assisted Intervention –MICCAI 2015, LNCS, vol. 9349, pp. 515–522. Springer (2015)
8. Chen, H., Dou, Q., Wang, X., Qin, J., Cheng, J., Heng, P.A.: 3D fully convolutional
networks for intervertebral disc localization and segmentation. In: Zheng, G., et al.
(eds.) Proc. 7th International Conference on Medical Imaging and Augmented
Reality – MIAR 2016, LNCS, vol. 9805, pp. 375–382. Springer (2016)
9. Jamaludin, A., Kadir, T., Zisserman, A.: SpineNet: automatically pinpointing classification evidence in spinal MRIs. In: Ourselin, S., et al. (eds.) Proc. 19th International Conference on Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016, LNCS, vol. 9901, pp. 166–175. Springer (2016)
10. Davy, A., Havaei, M., Warde-Farley, D., Biard, A., Tran, L., Jodoin, P.M.,
Courville, A., Larochelle, H., Pal, C., Bengio, Y.: Brain tumor segmentation withdeep neural networks. In: Proc. 3rd MICCAI Multimodal Brain Tumor Segmentation Challenge – BRATS 2014. pp. 1–5 (2014)
11. Schlegl, T., Waldstein, S., Vogl, W.D., Schmidt-Erfurth, U., Langs, G.: Predicting
semantic descriptions from medical images with convolutional neural networks. In:
Ourselin, S., et al. (eds.) Proc. 24th International Conference on Information Processing in Medical Imaging – IPMI 2015, LNCS, vol. 9123, pp. 437–448. Springer(2015)
12. Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal,C., Jodoin, P.M., Larochelle, H.: Brain tumor segmentation with deep neural networks. Med. Image Anal. 35, 18–31 (2017)
13. Kamnitsas, K., Ledig, C., V.F.J., N., Simpson, J., Kane, A., Menon, D., Rueckert, D., Glocker, B.: Efficient multi-scale 3D CNN with fully connected CRF for
accurate brain lesion segmentation. Med. Image Anal. 36, 61–78 (2017)
14. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic
segmentation. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition – CVPR 2015. pp. 3431–3440 (2015)

Segmentation of Pathological Spines

103

15. Sorkine, O., Alexa, M.: As-rigid-as-possible surface modeling. In: Proc. 5th Symposium on Geometry Processing – SGP 2007. pp. 1–8 (2007)
16. Möller, T., Trumbore, B.: Fast, minimum storage ray-triangle intersection. J.
Graph. Tools 2(1), 21–28 (1997)
17. Ibragimov, B., Likar, B., Pernuš, F., Vrtovec, T.: Shape representation for efficient
landmark-based segmentation in 3-D. IEEE Trans. Med. Imaging 33(4), 861–874(2014)
18. Yao, J., Burns, J.E., Munoz, H., Summers, R.: Detection of vertebral body fractures based on cortical shell unwrapping. In: Ayache, N., et al. (eds.) Proc. 15th
International Conference on Medical Image Computing and Computer-Assisted
Intervention – MICCAI 2012, LNCS, vol. 7512, pp. 509–516. Springer (2012)
19. Korez, R., Ibragimov, B., Likar, B., Pernuš, F., Vrtovec, T.: A framework for automated spine and vertebrae interpolation-based detection and model-based segmentation. IEEE Trans. Med. Imaging 34(8), 1649–1662 (2015)20. Botsch, M., Kobbelt, L.: A remeshing approach to multiresolution modeling. In:
Proc. 2nd Eurographics Symposium on Geometry Processing – SGP 2004. pp.
189–196 (2004)
21. Bossa, M., Olmos, S.: Multi-object statistical pose+shape models. In: Proc. 4thIEEE International Symposium on Biomedical Imaging – ISBI 2007. pp. 1204–
1207. IEEE (2007)

Attention-Driven Deep Learning for
Pathological Spine Segmentation
Anjany Sekuboyina1,2?( ) , Jan Kukačka1,2? , Jan S. Kirschke2 ,Bjoern H. Menze1 , and Alexander Valentinitsch1,21Department of Informatics, Technische Universität München, Germany
2
Department of Diagnostic and Interventional Neuroradiology,
Klinikum rechts der Isar, Germany
anjany.sekuboyina@tum.de

Abstract. Accurate segmentation of the spine in computed tomography(CT) images is mandatory for quantitative analysis, e.g. in osteoporosis,
but remains challenging due to high variability in vertebral morphology
and spinal anatomy among patients. Conventionally, spine segmentation
was performed by model-based techniques employing spine atlases or
statistical shape models. We argue that such approaches, even though
intuitive, fail to address clinical abnormalities such as vertebral fractures,
scoliosis, etc. We propose a novel deep learning-based method for segmenting the spine, which does not rely on any pre-defined shape model.
We employ two networks: one for localisation and another for segmentation. Since a typical spine CT scan cannot be processed at once owing to
its large dimensions, we find that both nets are essential to work towards
a perfect segmentation. We evaluate our framework on three datasets
containing healthy and fractured cases: two private and one public. Our
approach achieves a mean Dice coefficient of ∼0.87, which is comparable
but not higher than the state-of-art model-based approaches. However,
we show that our approach handles degenerate cases more accurately.
Keywords: Spine segmentation, automated segmentation, deep learning, fully convolutional network.

1

Introduction

Spine segmentation is a crucial component in quantitative medical image analysis. It directly allows detection and assessment of vertebral fractures and indirectly supports modelling and monitoring of the spinal ageing process. In thiswork we propose a method based on “deep-learning”, that generates precise
spine segmentations on computed tomography (CT) images. It overcomes the
drawbacks of earlier segmentation approaches and thus can be used in clinicalsettings. Particularly, our approach is capable of handling scans with varying
fields-of-view (FOV) and degenerate spine or vertebrae.
?

Both the authors have contributed equally.

B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proc. 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. 104−115, 2017.

Attention-Driven Deep Learning for Spine Segmentation

105

Fig. 1. Schematic outline of the proposed approach.

Previous works often deal with spine segmentation in a multi-step approach
incorporating spine localisation and vertebra detection followed by the segmentation [1,2]. Various traditional computer vision techniques have been successfully
applied, such as active shape models and snake-based methods [3,4], level sets [5]or graph-based approaches using normalized-cuts [6]. Most of these methods rely
on prior knowledge in the form of spine atlases or statistical shape models which
are used to provide a good initialisation. Such models reach state-of-the-art performance on healthy spines with no signs of osteoporotic fractures, attaining
Dice coefficients (DICE) of over 0.9. However, osteoporotic patients often sufferfrom severe vertebral fractures in various stages and spinal deformities such as
scoliosis. In such cases, model-dependent segmentation might fail due to the highvariability of the unique shape of a fracture or deformity that does not resemble
a mean shape model. A shape model is also restricted by its mesh interpolation algorithm, which makes extreme deformations unfeasible. Moreover, CT
images acquired for preoperative planning due to other diseases in the thoracic
or abdominal area have the spine in them as a consequence. Such opportunisticscans have varying FOVs, spatial resolution, and image reconstruction, in addition to variations in scan enhancements due to contrast agents. Model-based
approaches, which rely on good initialisations, could fail in such cases either
due to lack of landmarks for registration, uneven intensities, and noise. This
calls for data-driven approach based on supervised learning that does not rely
on pre-defined models, but learns the variability by training on several kinds ofcontingencies.
Machine learning-based approaches have proven to fulfill these requirements,
given that enough data is available for their training. Glocker et al. [7] andSuzani et al. [8] attempt the vertebra detection problem on arbitrary FOVs
using random forests and multi-layer perceptrons respectively. More recently,

106

Sekuboyina et al.

Chen et al. [9] try to make use of the omni-present convolutional neural networks (CNN), with a clever cost formulation, to detect vertebrae. Eventually
the CNNs have gained large popularity also for image segmentation throughthe concept of fully-convolutional networks (FCN) allowing pixel-to-pixel training and inference on (nearly) arbitrary sized inputs [10]. The standard FCN
architecture of a contracting and an expanding path with shortcut connections
is exploited in recent works on segmentation in the context of medical imaging [11–13]. However, these approaches cannot be directly extended to obtaina dense segmentation of a spine scan due to the sheer spatial resolution of ascan. For instance, the segmentation net used in [10] works on inputs containing∼2.5×105 pixels; a typical whole spine CT scan is about 100−1000 times larger,
thereby making a straightforward extension of an FCN non-viable.
We combine the FCN architecture with a domain-specific data-preprocessing
pipeline and data-augmentation scheme to propose a robust and scalable framework for spine segmentation in CT images. The method builds on the following
key elements:
1. A low-resolution attention FCN for spine localisation that works on twodimensional sagittal slices of a scan.
2. A high-resolution segmentation FCN for fine segmentation that takes threedimensional patches as input.
3. A smart patch extraction strategy to incorporate the FOV invariance andbypass memory limitations.
4. A domain-specific data augmentation to increase the training set size andincorporate the typical biological variance.
Our approach is free from predefined shape models and is purely data-driven,and is thus highly generalisable across varying FOVs, spinal deformities, andspatial resolutions given sufficiently diverse training data. Methodological details are presented in Sec. 2. We evaluate our method on a large private datasetof 56 (a) healthy and (b) fractured patients. We also compare our approachagainst the state-of-the-art methods on a publicly available dataset from the2014 MICCAI Workshop on Computational Methods and Clinical Applicationsfor Spine Imaging (CSI2014) [14]. Our approach achieves a comparable mean
DICE of around 87%, while perfectly segmenting fine details of normal as well
as deformed vertebrae. Details of the experiments follow in Sec. 3.

2

Methodology

We present our approach to spine segmentation in two stages. Firstly, a twodimensional (2D) FCN which provides a low-resolution localisation of the spine.
Secondly, we present the 2D-three-dimensional (3D) FCN that generates highresolution binary segmentations. Fusing the predictions of both the networks
results in a good segmentation of a spine volume. An overview of our approachis shown in Fig. 1.

Attention-Driven Deep Learning for Spine Segmentation

107

Fig. 2. Attention-net. The network utilizes 10 convolutional layers with stride 1, kernel
sizes and dilation factors are denoted in the image. Moreover, the first 4 conv-layers
are followed by max-pooling. Numbers above blobs represent number of features in
the hidden feature space. Observe the drop in spatial resolution from the input tothe output slice, thereby providing a low-resolution attention map for a given sagittal
slice. Every n-th sagittal slice from the scan volume is considered for inference, thereby
reducing the number of forward passes for scan, which makes the attention generation
very fast. Consequently, the original scan resolution is restored by interpolation in all
three directions.

2.1

Localization: Attention-Net

We exploit the the structure and position of the spine in a scan, that are generally
invariant, to obtain a rough localisation of the spine. The network performing thistask is called the attention net. Since sagittal view provides significant context
on the spine’s location, the attention net operates fully in 2D on sagittal slices.
The net is fully convolutional and outputs a 2D map of lower dimension than the
input. Every value (∈ [0, 1]) in the predicted 2D map corresponds to a 16 × 16
region in the input, and represents the percentage of foreground voxels (‘spine’voxels) in that region. Fig. 2 illustrates the architecture of the attention-net.
A 2D patch of 160 × 160 (padded to size 720 × 720) predicts a maps of size
10 × 10. Since context is of utmost importance for determining the presenceof spine, we increase the receptive field with dilated convolutions [15] in thedownstream convolutional layers. We incorporate the resolution reduction in
the third dimension by working only on every n th sagittal slice. At the end,
given an input volume, the attention net works on sagittal slices and predicts
a lower-resolution volume (called the attention map) whose values indicate thepresence of spine. The attention map is then up-sampled to the input dimension
for further use.
Training and Inference. The ground truth for training is obtained from theavailable spine segmentations. Every volume is down-sampled by a factor of
16 × 16 × n (n = 8, in our experiments), each voxel representing the ratio ofspine-voxels to total-voxels in its corresponding 16 × 16 region. The network is
trained to minimise the mean-squared error between predicted map and groundtruth. During training, given a scan volume, we train on large patches randomly
sampled from sagittal slices with data augmentation through rigid transformations; we advocate the use of patches as incorporation of invariance to arbitrary

108

Sekuboyina et al.

(a)

(b)

Fig. 3. Attention maps. (a) and (b) show two computed tomography scans overlaid
with the response of the Attention-Net aggregated over all slices in the direction ofview. Notice that the response is focused predominantly on the spine. This attentionmap is Gaussian smoothened, thresholded, and converted to a binary mask, which isthen fused with the Segmentation-Net’s response.

FOVs. During test time, the patches are sampled from every n-th sagittal slicewith overlap such that the entire slice is covered. The predicted low-resolution attention map for each of these patches is up-sampled to the resolution of the scan
volume, and filtered with a 3D Gaussian kernel, N (0, Σ). Fig. 3 shows an attention map of two test cases. For better visualisation of the 3D map, the response
is aggregated in the direction of view and overlaid on the mid-slice. Observe thatthe net succeeds in localising the spine. This attention-map is thresholded andconverted to a binary mask, which is then fused with the Segmentation-Net’s
response as elaborated in the following sections. The threshold-value and thecovariance of the Gaussian smoothing (Σ) are tuned on the validation set.
2.2

Spine Segmentation: Segmentation-Net

Precise segmentation can be obtained when the receptive field of the network
if small enough that it focuses on minute details, while being large enough tocapture sufficient context. We achieve this by incorporating a patch-based approach. Such an approach also alleviates the restriction that the limited memoryof a GPU imposes on the volume that can be processed by the network. We propose a segmentation net that is fully convolutional and a combination of 2D
and 3D convolutions, building on the versatile ‘U’ architecture commonly used
for segmentation [12, 16]. A detailed view of the network’s architecture is shownin Fig. 4. The input to the network is a 3D block from the scan, having larger
receptive field in the sagittal view (for example, an input block could be of size
188 × 188 × 12, with the first two dimensions corresponding to the sagittal view).
At an isotropic resolution of 1 mm3 , the receptive field for predicting one voxel’s

Attention-Driven Deep Learning for Spine Segmentation

109

Fig. 4. Segmentation-net. The network has a five-level downsampling path, each levelconsisting of two convolutions with a ReLU activation and a max-pooling, and asymmetric upsampling path, which are connected by skip connections for recovery ofhigh resolution. Notice the utilisation of a combination of two-dimensional and threedimensional convolutions to process higher information in the sagittal direction, whileconglomerating information from the adjacent slices. The size of the inputs and outputs
is parameterised by the dimensions of the smallest blob on the path as x, y ≥ 4, z ≥ 3.

label has a size of ∼ 18.4 × 18.4 × 0.8 cm3 . The output is a dense pixel-wisesegmentation of dimensions equal to that of the input.
Training and Inference. We use a segmentation-centric loss function of DICE [13] as the objective function. At the training time we randomly sample
patches from the input volumes and apply online generated elastic and rigid
deformations to them as a form of data augmentation. Employing dropout after
the convolutions on the lowest level of the contracting path gave a significant
performance improvement. At the test time, the patches are extracted uniformly
with overlap such that the resulting segmentations cover the entire test scan.
To conclude the approach: given a scan volume, the segmentation-net provides its dense segmentation. Since the latter splits the volume in sub-blocks, weobserve that the context in these blocks is insufficient for perfect segmentation.
We therefore observe several false positives in the form of stray segmentation,
wherein, in addition to spine, other regions such as parts of the rib cage, pelvic
bone, sacrum etc. are also segmented. Now, the thresholded binary attentionmap, which localises only the spine, is fused with the segmentation-net’s output
to clear away the stray segmentations1 , resulting in the final segmentation map.
1

A cascaded fusion of these nets was also tried where the patches for the segmentation
net are obtained only from the region proposed by the attention map. We observedthat this approach’s accuracy was not superior to our approach of late-fusion.

110

3

Sekuboyina et al.

Experiments and Results

Implementation Details. The two networks in our approach are implemented
as standalone modules. Given a scan, the inference in both the nets run inparallel, and their responses are combined. As a preprocessing step, the CT
volumes are subjected to anisotropic diffusion filtering in order to smoothen thehomogeneous regions and improve the details around the edges. Both the nets
were implemented in Caffe [17]. Adam solver was employed for optimising theloss. The nets were trained till convergence with initial learning rates of 10−5 forsegmentation net and 10−4 for the attention net. When implemented on a Nvidia
Titan X GPU with 12GB RAM, the training of the segmentation net convergedin two days while that of the attention net took one day. The convergence was
faster in the latter case owing to a significantly lower number of parameters.
Experiments. We evaluate our framework on three datasets: (Dataset 1) forty
five patients without fractures but varying age from 25 to 69 years, (Dataset 2)
eleven cases with vertebral fractures, to evaluate the performance on deformed
vertebrae, and (Dataset 3) the spine segmentation challenge of the CSI2014
dataset containing fifteen scans2 , for comparison with other model-based techniques. Datasets 1 and 2 are private in-house datasets gathered from our picturearchiving and communication system. The scans were acquired over a periodof two years for various patient examinations and not specifically for the spine
analysis, which results in a high diversity in terms of patient-age, abnormalities,
FOV, and scanner calibrations. The ground truth segmentation for this data was
obtained by first using the approach in [1]3 and then manually corrected by a
medical expert. Experiments on these datasets are therefore close to the clinicalscenario.
Dataset 1. This is employed to validate the generalisability towards varying
BMDs and scanner calibrations. It includes healthy controls (HC) with no fractures, which have been acquired with varying scanner settings (fields-of-view,
spatial resolution, etc.). Thus, there is a significant variation in the Hounsfield
units (HU) of the scans. We reserve three volumes as a validation set and anotherthree volumes for testing, the remaining thirty nine are used for training.
Dataset 2. We utilise this to evaluate the performance on fracture cases (Fx).
For this experiment we fused the forty five cases of Dataset 1 with eleven cases
from this dataset, all of which have fractures. On top of the splits defined for
Dataset 1, we add six fractured cases to the training set, two to the validation
set, and two to the test set.
2

According to [14], the complete test set contains ten volumes. However, we obtained
only five volumes from the organisers.
3
We would like to thank Klinder et al., the authors and our industry partners (Philips,
Hamburg, Germany), for providing us with the segmented images.

Attention-Driven Deep Learning for Spine Segmentation

111

Dataset 3. As a final experiment, we intend to compare our algorithm withthe other best-performing segmentation algorithms. For this task, we choose thepublic dataset of the segmentation challenge in CSI2014. This contains a total of
fifteen scans, ten for training phase and five for testing2 . From the ten training
volumes we reserve one for validation and use the remaining nine for fine-tuningof the model pre-trained on the Dataset 2. We refer the reader to [14] for a
detailed description of the dataset.
Results. The results of the experiments on the Datasets 1 and 2 are reported in
the Tab. 1. Our method reaches good results both for healthy and fractured cases,
proving to generalise well to a wide range of data. In both cases we observed thesegmentation net suffer from stray segmentations which were mostly filtered out
by the attention net, steadily improving the DICE by 1 − −10%. The resulting
segmentation of our method are visualised in Fig. 5, comparing it to a wellknown model-based segmentation approach [1]. Notice the over-segmentationof the vertebral process regions (top row, HC) in the model-based approach.
This is expected as the process of a vertebra has very high variability frompatient-to-patient. Such a variability cannot be captured by an atlas or a shapemodel. Such over-segmentation does not occur in our approach. The bottom row
illustrates our approaches performance on a fractured vertebra. Observe how
the model-based segmentation fails to capture the deformities. This illustrates
the restriction the interpolation algorithm in such approaches fails to capture
extreme deviations from the mean shape. Our approach, however, successfully
segments the degenerate vertebra as it learns to segment on the edges and is nothindered by any shape priors. We attribute the bleeding in the vertebral process
regions (Fig. 5, bottom row) to predictions of the neighbouring slices; we do not
observe this artefact consistently in our results.
The results of the experiment on Dataset 3 are reported in Tab. 2. We compare our approach with two methods that have been deemed best performing
according to the challenge organisers. Korez et al. [4] use a mean shape model
based strategy followed by an efficient interpolation theory oriented mesh deformation, and performed best on healthy cases. Forsberg [2] performed well on
fractured cases. It uses a multi-atlas based segmentation followed by a B-spline
relaxation to adapt for the variability of vertebral structures. Observe that ouralgorithm achieves a comparable performance with the state-of-art in the healthy
cohort. The performance on the fractured cases is not up to the mark; we identify the cause to be following: The fractured cases in the test set are all fractures
treated by cement injection inside the vertebra. This procedure causes distinctive artefacts in the image, as opposed to untreated fractures. In the training
data for our model there was not a single case of a cement-treated fracture andsince our approach is purely data-driven, the method failed on these cases.

112

Sekuboyina et al.

Table 1. DICE (in %) for Datasets 1 and 2. The performance of our method is consistent among HC and Fx cases.
Dataset

DICE
87.60 ± 5.0
85.91 ± 4.8

Dataset 1 (HC)
Dataset 2 (HC+Fx)

4

Conclusion

In this paper we propose a model-free deep learning based framework for pathological spine segmentation in CT images. Our method uses a pair of fullyconvolutional networks that complement one another: The first network provides a coarse localisation of the spine in the form of an attention map, while
the second network provides precise high-resolution segmentations. Both theseare fused to obtain the final segmentation map of the spine. We evaluate themethod on three datasets and obtain promising results indicating applicability
in a clinical setting. Our main conclusions are the following: (1) Our approachbased on neural networks is far more robust, generalisable, and precise on finedetails as a consequence of its dependence on every voxel and its surrounding,unlike the traditional models that depend on predefined shapes and edges forfine-tuning, (2) our approach successfully segments healthy as well as fractured
vertebrae, given both cases are sufficiently represented in the training set, (3) our
approach achieves Dice coefficients of above 90% for healthy cases and above
80% for fractured cases on the CSI2014 dataset. We believe that its performancecan be further improved with a larger and more representative dataset. Lastly,(4) we remark that our approach fails to incorporate information pertaining tostructural consistency of a spine. This results in a peculiar behaviour where ourmethod fails to segment parts of a vertebra or sometimes entire vertebrae at
the start or end of a spine. This can be observed in Fig. 5 (top row). We intend
to investigate ways in which such global structural regularity can be imposed
during the training phase of our networks.
Table 2. Comparison of our approach’s performance with other state-of-the-art methods on the benchmark dataset of the 2014 MICCAI Workshop on Computational Methods and Clinical Applications for Spine Imaging. (∗ In these methods, DICE was computed at vertebra level and aggregated; our DICE is computed on voxel-level.)
DICE

Approach∗

Forsberg [2]Korez et al. [4] ∗Our approachHC

Fx

92.1 ± 3.0
94.7 ± 4.0
90.84 ± 4.4

87.8 ± 1.5
88.2 ± 3.0
80.41 ± 4.1

Attention-Driven Deep Learning for Spine Segmentation

113

Fig. 5. Comparison of the proposed method (red) and Klinder et al. [1] (blue) to theground truth (yellow contour). Top row: Test cases from the Dataset 1. Our method
performs better in matching the actual vertebra shape and does not suffer from oversegmentation. However, it does not always provide structurally consistent results; forexample, the first and the last vertebra in the left-most image are not fully segmented.
Bottom row: Fractured cases from the Dataset 2. Our model-free approach is able tocapture the unique deformation of the fractured vertebra, opposed to the model-based
method.

Moreover, as part of future work, we plan to employ a graphical model to split
the binary segmentation into different vertebrae thereby labeling the segmentation. As the graph needs to work only on the segmented regions, we hypothesisethat it will be light in weight and easily trainable.
Acknowledgements. This work was funded from the European Research Council(ERC) under the European Union’s Horizon 2020 research and innovation programme(GA637164–iBack–ERC-2014-STG).

References
1. Klinder, T., Ostermann, J., Ehm, M., Franz, A., Kneser, R., Lorenz, C.: Automatedmodel-based vertebra detection, identification, and segmentation in CT images.

114

Sekuboyina et al.

Med. Image Anal. 13(3), 471–482 (2009)
2. Forsberg, D.: Atlas-based segmentation of the thoracic and lumbar vertebrae. In:
Yao, J., et al. (eds.) Proc. 2nd MICCAI Workshop on Computational Methods andClinical Applications for Spine Imaging – CSI 2014, LNCVB, vol. 20, pp. 215–220.
Springer (2015)
3. Kadoury, S., Labelle, H., Paragios, N.: Spine segmentation in medical images using
manifold embeddings and higher-order MRFs. IEEE Trans. Med. Imaging 32(7),1227–1238 (2013)
4. Korez, R., Ibragimov, B., Likar, B., Pernuš, F., Vrtovec, T.: Interpolation-based
shape-constrained deformable model approach for segmentation of vertebrae from
CT spine images. In: Yao, J., et al. (eds.) Proc. 2nd MICCAI Workshop on Computational Methods and Clinical Applications for Spine Imaging – CSI 2014, LNCVB,
vol. 20, pp. 235–240. Springer (2015)
5. Huang, J., Jian, F., Wu, H., Li, H.: An improved level set method for vertebra CT
image segmentation. Biomed. Eng. Online 12, 48 (2013)
6. Lootus, M., Kadir, T., Zisserman, A.: Automated radiological grading of spinal
MRI. In: Yao, J., et al. (eds.) Proc. 2nd MICCAI Workshop on Computational
Methods and Clinical Applications for Spine Imaging – CSI 2014, LNCVB, vol. 20,
pp. 119–130. Springer (2015)
7. Glocker, B., Feulner, J., Criminisi, A., Haynor, D., Konukoglu, E.: Automatic
localization and identification of vertebrae in arbitrary field-of-view CT scans. In:
Ayache, N., et al. (eds.) Proc. 15th International Conference on Medical Image
Computing and Computer-Assisted Intervention – MICCAI 2012, LNCS, vol. 7512,
pp. 590–598. Springer (2012)
8. Suzani, A., Rasoulian, A., Seitel, A., Fels, S., Rohling, R., Abolmaesumi, P.: Deeplearning for automatic localization, identification, and segmentation of vertebral
bodies in volumetric MR images. In: Webster, R., Yaniv, Z. (eds.) Proc. SPIE Medical Imaging 2015: Image-Guided Procedures, Robotic Interventions, and Modeling,
vol. 9415, p. 941514. SPIE (2015)
9. Chen, H., Shen, C., Qin, J., Ni, D., Shi, L., Cheng, J., Heng, P.A.: Automatic
localization and identification of vertebrae in spine CT via a joint learning modelwith deep neural networks. In: Navab, N., et al. (eds.) Proc. 18th International
Conference on Medical Image Computing and Computer-Assisted Intervention –MICCAI 2015, LNCS, vol. 9349, pp. 515–522. Springer (2015)10. Shelhamer, E., Long, J., Darrell, T.: Fully convolutional networks for semantic
segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 640–651 (2017)
11. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., et al. (eds.) Proc. 18th International Conference on Medical Image Computing and Computer-Assisted Intervention – MICCAI2015, LNCS, vol. 9351, pp. 234–241. Springer (2015)
12. Çiçek, O., Abdulkadir, A., Lienkamp, S., Brox, T., Ronneberger, O.: 3D U-Net:
learning dense volumetric segmentation from sparse annotation. In: Ourselin, S.,et al. (eds.) Proc. 19th International Conference on Medical Image Computing andComputer-Assisted Intervention – MICCAI 2016, LNCS, vol. 9901, pp. 424–432.
Springer (2016)
13. Milletari, F., Navab, N., Ahmadi, S.A.: V-Net: fully convolutional neural networks
for volumetric medical image segmentation. In: Proc. 4th International Conference
on 3D Vision – 3DV 2016. pp. 565–571. IEEE (2016)
14. Yao, J., Burns, J., Forsberg, D., Seitel, A., Rasoulian, A., Abolmaesumi, P., Hammernik, K., Urschler, M., Ibragimov, B., Korez, R., Vrtovec, T., Castro-Mateos, I.,Attention-Driven Deep Learning for Spine Segmentation

115

Pozo, J., Frangi, A., Summers, R., Li, S.: A multi-center milestone study of clinicalvertebral CT segmentation. Comput. Med. Imaging Graph. 49, 16–28 (2016)15. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. arXiv
p. arXiv:1511.07122 (2015)
16. A three-dimensional U-Net for synaptic cleft detection (2016), https://github.
com/zudi-lin/pse-unet
17. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding.
In: Proc. 22nd ACM International Conference on Multimedia. pp. 675–678. ACM(2014)

Automatic Full Femur Segmentation from
Computed Tomography Datasets
Using an Atlas-Based Approach
Bryce A. Besler1,2,3 , Andrew S. Michalski2,3 ,Nils D. Forkert3,4 , and Steven K. Boyd2,3( )12Biomedical Engineering Graduate Program, University of Calgary, Canada
McCaig Institute for Bone and Joint Health, University of Calgary, Canada
3
Department of Radiology, Cumming School of Medicine,
University of Calgary, Canada
4
Hotchkiss Brain Institute, University of Calgary, Canada
skboyd@ucalgary.ca

Abstract. Automatic segmentation of femurs in clinical computed tomography remains a challenge. Joints degraded by old age are a particularly challenging dataset to segment. The objective of this study is toevaluate existing methods and propose an alternative method for segmentation of femurs in clinical computed tomography datasets for joints degraded by old age. Bilateral hip computed tomography scans of three cadaveric specimens (six femurs) were available for this study. Deformableregistration using an affine selection criterion was used for atlas-based
segmentation. For comparison, the six femurs were also segmented with
two graph-cut algorithms. An automatic graph-cut segmentation algorithm was only able to separate the femur from the pelvis in two of thesix femurs due to a limitation of graph-cuts. The atlas-based method
produced consistent automatic segmentations for all degraded joints. In
conclusion, atlas-based femur segmentation performs considerably better
than an automatic graph-cut algorithm when applied to degraded joints.
Keywords: Atlas-based segmentation, graph-cut, femur, computed tomography.

1

Introduction

Clinical computed tomography (CT) is a versatile imaging modality used tostage surgery and investigate the source of abdominal pain. During these common applications, quantitative image data of bones are inadvertently captured.
As a consequence of its applicability, CT generates a large collection of clinically
relevant bone data that is not yet effectively used. This collection of bone image
data can be used to answer population-level questions about bone health anddevelop novel screening technologies. Of particular interest is the automatic assessment of femurs in clinical CT to opportunistically screen for osteoporosis [1].
Femoral fractures account for 72% of all fracture-related healthcare costs [2],B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proc. 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. 116−127, 2017.

Automatic Full Femur Segmentation

117

posing a strong incentive to help detect fractures before they happen. A robust
and automatic segmentation of the femur would be required for high-throughput
assessment of bone health.
Many techniques for segmenting femurs in medical image data have been
proposed in the past. For example, Zoroofi et al. [3] proposed a method employing standard image processing techniques. More precisely, the image data
is smoothed with a Gaussian filter, histogram-based thresholding is applied,
and the segmentation is cleaned using morphological operators. Kang et al. [4]employed a region growing technique using locally adaptive thresholding. The
resulting segmentation was cleaned using both morphological closing and slicewise contour closing.
More advanced techniques based on the energy minimization framework ofgraph-cuts [5] yield improved segmentations compared to simple intensity thresholding. In Pauchard et al. [6], a user interactively applies brush strokes to theimage data enforcing hard constraints on foreground and background voxels. The
edges of the bone are segmented using the standard boundary term for graphcuts. Krčah et al. [7] proposed an automatic graph-cut segmentation method.
Here, a modified sheetness measure [8] is used to increase the contrast at the surface of the bone. The graph-cut algorithm makes use of this enhanced contrast
and the quantitative nature of Hounsfield units to segment bone structures. A
morphological erosion is applied to disconnect the femur and the pelvis if they
are connected. The boundary between the two articulating bones is determinedby another, separate graph-cut algorithm.
However, the aforementioned techniques have difficulty separating well-connected joints, a problem generally seen in older populations. Extensive manual
correction is often necessary but not feasible for high-throughput applications.
Within this context, atlas-based approaches could overcome the problem of wellconnected joints and improve segmentation accuracy by including shape information. The basic idea of atlas-based methods is to register an atlas datasetwith a corresponding segmentation of the structure of interest to an individual
dataset. The result of the registration is used to transform the segmentation to
the reference image.
Previously, Ehrhardt et al. [9] used Thirion’s demons [10] to segment full
pelvis and femur CT scans and to identify anatomical landmarks. The authors
pointed out that the femur-acetabulum connection was particularly difficult tosegment. Whitmarsh et al. [11] proposed a multi-atlas-based method using freeform deformations with label fusion to produce a highly accurate segmentation.
Local-weighted voting was used to generate the final segmentation. However, theimage volumes were artificially restricted to the joint reducing the applicabilityof the algorithm to variable clinical scans. Additionally, it has been shown thatlabel fusion with an increasing number of atlas images can have both decreasing
performance and increased computation time [12], which was not considered in
the work of Whitmarsh et al.
This highlights the importance of atlas selection and composition for the
purpose of segmentation. Therefore, the objective of this study was to develop

118

Besler et al.

an automatic method for segmenting femurs in clinical CT datasets with a special
focus in degraded joints.

2

Material and Methods

2.1

Data Acquisition

Bilateral hip CT scans were acquired from three cadavers using a GE Revolution
CT Scanner (GE Healthcare). In-plane resolution was 0.684 mm by 0.684 mm
with a slice thickness of 0.625 mm. In all cases, the scan volume begins above
the iliac crest of the pelvis and stops in the proximal tibia containing the entire
femur. A slice of the greyscale image data and a direct volume rendering of theimage data is shown in Fig. 1.
The age and sex of the three cadavers were 86, male; 72, female; and 90,
female. The cause of death of all cadavers was respiratory and related to oldage. No cadavers had metal in the scan volume (such as an implant). All ofthese specimens display degraded joints due to age (see Fig. 2). As a result of
this, there is large contact between the femoral head and the acetabulum of thefemur in two of the three specimens. In the third case, the contact between the
femur and acetabulum is very small, but present. A quantitative measure of joint
degeneration was not made.
2.2

Pre-Processing

Each CT image was resampled to an isotropic resolution of 0.625 mm using
cubic interpolation. After this, the image volume was split sagittally creating(a) Coronal slice of the greyscale data.

(b) Direct volume rendering.

Fig. 1. Computed tomography scan volume.

Automatic Full Femur Segmentation(a) Axial.

119(b) Coronal.

(c) Sagittal.

Fig. 2. Contact between femur and pelvis.

two volumes with one volume containing the left femur and the other volumecontaining the right femur. The left femurs were mirrored in the lateral-medial
direction to match the anatomical orientation of right femurs. Similar to a previous study [11], a mask of the subject’s body was defined using a threshold of−200 Hounsfield units. This mask of the body was used to restrict the registration domain.
2.3

Manual Segmentations

Each femur was semi-automatically segmented using the graph-cut method described by Pauchard et al. [6], implemented in the freely available software
MITK-GEM (https://simtk.org/projects/mitk-gem). In this interactive method, a brush is used to define regions of foreground (femur) and background(tibia, pelvis, muscle, fat, and air). Foreground strokes were interactively placedin each dataset in the diaphyseal cortical bone and in the sagittal, coronal, andaxial slices of the proximal and distal femur. Background strokes were placedin the sagittal, coronal, and axial slices of the tibia and pelvis including joint
space. Background strokes were also places on muscle, fat, and air. After roughfore- and background definition, a graph-cut algorithm is used to obtain the final
segmentation.
Because the segmentations produced by MITK-GEM included joint space(see Fig. 3), the six MITK-GEM segmentations were manually corrected bya medical imaging expert. Furthermore, rough edges in the manual segmentation were smoothed with an image-closing operator (spherical kernel of diameter
three voxels). These manual segmentations were used as the ground truth forevaluation as well as for multi-atlas segmentation.
2.4

Atlas-Based Segmentations

The objective of atlas-based segmentation is to find a mapping of voxels in anatlas image to voxels in a fixed image. If a correspondence can be found, then asegmentation defined in the atlas image can be mapped to the fixed image andused as a segmentation of the fixed image. This mapping is typically solved forusing linear and non-linear (deformable) intensity-based image registration.

120

Besler et al.

(a) Manual segmentation (white) overlaid with the MITK-GEM segmentation (red).

(b) Axial slice of the MITK-GEM segmentation overlaid on the image data.

Fig. 3. Visualization of MITK-GEM segmentation for the specimen with the least
contact between femur and pelvis.

The previously described CT datasets of the six femurs and the corresponding
manual segmentations were used as the basis for the multi-atlas-based segmentation. The deformation field was solved by applying an affine transform for a roughinitialization followed by a deformable registration using B-spline free-from deformation. Normalized cross correlation was used as the image similarity-metric.
A selection criterion was used to rank the affine registered atlases comparedto the query image, where only the atlas with the best registration metric after
affine registration was selected for subsequent non-linear registration to savecomputation time while increasing the chance of achieving a good non-linear
registration. The final spacing of the B-spline control points was 5 mm. A multiresolution registration scheme with six levels was used and the spacing between
B-spline control points was decreased by half at each level of the pyramid. The
registration domain was restricted to a mask of the bone and soft tissue as
described in Sec. 2.2. The open-source software Elastix [13] version 4.81 wasused for registration. Parameter files can be found in the Elastix database2 ).
2.5

Evaluation

Leave-one-out-cross-validation was used to evaluate the multi-atlas-based segmentation method described in this work. Given a query femur, the remaining
five atlas images were used as described in Sec. 2.4 to produce an atlas segmentation.
Obviously, a contralateral femur exists for any query image in the multi-atlas
dataset, which might bias the quantitative results. To investigate this potential
bias, leave-one-out-cross-validation of atlas-based segmentation was conducted12

http://elastix.isi.uu.nl
http://elastix.bigr.nl/wiki/index.php/Par0046

Automatic Full Femur Segmentation

121

with and without the contralateral femur in the atlas. Thus, two atlas-based
segmentations were generated for each case, one with the contralateral femur inthe atlas (con.) and one without the contralateral femur in the atlas (alt.).
For comparison purposes, the segmentations resulting from applying thegraph-cut algorithm described by Pauchard et al. [6] (MITK-GEM) - prior to
manual correction - were also quantitatively evaluated. Finally, all six femurs
were segmented using the graph-cut method of Krčah et al. [7], the source codeof which is publicly available3 . Thus, four segmentation results – Atlas (con.),Atlas (atl.), MITK-GEM, and Krčah – were available for each femur and compared to the manual segmentation.
For all cases, the Dice similarity coefficient [14] and the Hausdorff distance[15] were used for quantitative comparison to the ground truth segmentation.
The Dice similarity coefficient is a measure of mean overlap between two segmentations and is expected to be very large (close to 1.0) given the small surface
area to volume ratio of a femur [16]. The Hausdorff distance is a complementary
statistic to the Dice similarity coefficient because it measures the worst possible
error between two segmentations. The Hausdorff distance is regarded as a veryharsh metric and identifies segmentations that may have large local differences
but are otherwise well segmented.

3

Results

The Dice similarity coefficient and Hausdorff distance comparing the four algorithm segmentations to the manual segmentation are presented in Tab. 1 and 2,
respectively. The MITK-GEM method had the best results with an average Dice
similarity coefficient of 0.994 and an average Hausdorff distance of 4.58 mm. The
Krčah method had the worst results with an average Dice similarity coefficientof 0.801 and an average Hausdorff distance of 102.40 mm. For the atlas-based
method, removing the contralateral femur from the atlas increased the average
Hausdorff distance from 5.95 mm to 8.53 mm and decreased the average Dice
similarity coefficient from 0.978 to 0.969. The contralateral femur always had
the best image similarity metric after the affine registration (data not shown).
All four algorithms have Dice similarity coefficients above 0.95 when the pelvis
and the femur are separated in the final segmentation.
The Krčah method successfully separated the femur from the pelvis in twoof the six femurs, both in the specimen with the least contact between the femur
and pelvis. Fig. 4 shows an axial slice of the successful Krčah segmentations
overlaid on the greyscale data. No joint space between the femur and acetabulumof the pelvis is included in the successful Krčah segmentations. Fig. 5 showseach segmentation method overlaid on the image data at the proximal, middle,
and distal femur for a specimen with a well-connected joint. Joint space wasincluded in all MITK-GEM segmentations including the specimen with the least
contact between the femur and pelvis, visualized in Fig. 2. Fig. 5(m) shows a3

https://github.com/mkrcah/bone-segmentation

122

Besler et al.

Table 1. Dice similarity coefficient for each method.
SampleLeg

Atlas (con.)Atlas (alt.)

MITK-GEM

Krčah

1
1

Left
Right

0.982
0.983

0.976
0.972

0.997
0.997

0.988
0.987

2
2

Left
Right

0.978
0.977

0.969
0.968

0.996
0.993

0.674
0.657

3
3

Left
Right

0.974
0.973

0.967
0.964

0.994
0.988

0.745
0.755

large volume of joint space included in the Krčah segmentation when it fails toseparate the femur and pelvis. Fig. 5(g) shows a poor atlas-based segmentation
result where the segmentation leaks from the femur into the pelvis. A surface
rendering for each segmentation overlaid on the manual segmentation is shownin Fig. 6 for a femur that is well-connected to the pelvis. Fig. 6(d) shows the full
pelvis connected to the femur when the Krčah method fails.
Table 2. Hausdorff distance for each method (in mm).
SampleLeg

Atlas (con.)Atlas (alt.)

MITK-GEM

Krčah

1
1

Left
Right

4.19
3.59

12.56
3.80

4.59
4.19

3.42
4.38

2
2

Left
Right

6.03
9.56

7.29
6.16

3.95
5.04

171.45
180.64

3
3

Left
Right

5.63
6.73

7.02
14.38

4.68
5.00

127.13
127.36(a) Right femur.

(b) Left femur.

Fig. 4. Visualization of the successful Krčah segmentations.

Automatic Full Femur Segmentation

123(a) Manual.

(b) Manual.

(c) Manual.

(d) Atlas (con.).

(e) Atlas (con.).

(f) Atlas (con.).

(g) Atlas (alt.).

(h) Atlas (alt.).

(i) Atlas (alt.).

(j) MITK-GEM.

(k) MITK-GEM.

(l) MITK-GEM.

(m) Krčah.

(n) Krčah.

(o) Krčah.

Fig. 5. An axial slice from the proximal femur (left), the diaphysis (middle), and the
distal femur (right) for each segmentation overlaid on the image data.

124

Besler et al.

(a) Atlas (con.).

(b) Atlas (atl.).

(c) MITK-GEM.

(d) Krčah.

Fig. 6. Visualization of algorithm segmentations (red) overlaid with manual segmentation (white).

4

Discussion

In this study, an automatic method for segmenting femurs in clinical computed
tomography using a multi-atlas-based approach is proposed. The age of the cadavers presented a particularly difficult dataset due to contact between the femur and pelvis. Segmentations obtained using graph-cut methods consistently
include joint space in the segmentation. The automatic graph-cut method ofKrčah et al. [7] was unsuccessful in separating the femur from the pelvis in fourof six femurs. In contrast, the atlas-based segmentation method with a selection
criterion generated more consistent results with respect to separating the femur
from the pelvis.
Screening technologies and population-level health assessment require robust
and automatic segmentation of the femur for high-throughput assessment of bonehealth. Algorithms for segmenting femurs from clinical CT must be accurate forthe full range of anatomical variation expected in the population. The CT dataset
used in this study consisted of three cadavers with degraded joints due to age,an expected population when performing opportunistic screening. Two of thethree cadavers had large contact areas between the femur and the acetabulumof the pelvis.
The difficulty of segmenting the femur arose from the connection in thefemoroacetabular joint. The femoroacetabular joint is a ball-and-socket joint
where contact between the femur and the acetabulum of the pelvis has a large
surface area with identical voxel intensities. Conceptually, this surface can bevisualized as a surface patch on a sphere with a large solid angle. The geometry of the joint makes it particularly difficult to separate the femur from theacetabulum when the joint has been degraded due to age.

Automatic Full Femur Segmentation

125

Two existing methods using the graph-cut framework were used for femur
segmentation in this challenging dataset. The automatic graph-cut method ofKrčah et al. [7] failed to separate the femur from the pelvis in four of the six
femurs. Both successful segmentations were from the specimen with the least
contact between the femur and the acetabulum of the pelvis. No joint space
was included in successful Krčah segmentations. The semi-automatic graph-cut
method of Pauchard et al. [6] produced femur segmentations with joint spacein the segmentation for all six femurs, including the less connected specimen.
The discrepancy between graph-cut methods can be explained by the modified
sheetness filter [8] used as a pre-processing step in the Krčah method. This
sheetness filter increases the contrast at the surface of the bone used by thegraph-cut framework to segment bone structures.
However, bone structures in contact have identical voxel intensities and contrast cannot be enhanced. This is a limitation of the graph-cut framework whereconnected bone structures cannot be separately segmented. To overcome thislimitation, Krčah et al. [7]. implemented a post-processing step to separate connected bone structures. The initial segmentation is eroded with a spherical kernel to separate connected bones. If a segmentation is split into multiple parts
after erosion, the boundary between the separated parts is determined by another, separate graph-cut algorithm. However, when the femur and pelvis aresegmented as one bone structure, an annulus of joint space can be included in
the initial segmentation if the joint space is narrow (see Fig. 5(m)). The thin annulus of joint space between the femur and pelvis is segmented similar to marrow
space in the diaphysis. This greatly increases the effective contact area in the
joint and makes it impossible for the spherical kernel to separate the connectedbone structures. Even if the bone structures could be separated, this large annulus of joint space would be classified as belonging to either the femur or pelvis
segmentation, or possibly split between both. The inclusion of joint space is alimitation of graph-cut based femur segmentation when the femoroacetabular
joints is well-connected.
The proposed atlas-based method using an affine-registration selection criterion was specifically designed as an alternative to graph-cut based methods
and to overcome the segmentation problems arising from large contact areas byincluding shape information. Overall, the atlas-based method produced morereliable results for the challenging CT dataset than the automatic graph-cut
method. On the contrary, the atlas method results in segmentation errors where
the femur segmentation leaks into the pelvis (see Fig. 5(g)). To overcome thisproblem, the atlas-based method could be improved by adding a higher regularization to prevent leakage of the femur into the pelvis.
The two graph-cut methods and the atlas-based method for femur segmentation with and without a contralateral femur were compared with a corresponding
manual segmentation. The semi-automatic method of Pauchard et al. [6] showed
the best performance both in terms of Dice similarity coefficient and Hausdorff
distance. However, the method of Pauchard et al. had a systematic bias to include joint space in the femur segmentation. Additionally, this method was used

126

Besler et al.

as the basis for generating the ground truth segmentations, thus the metrics arebiased towards favouring the method of Pauchard et al. Finally, this method is
not automatic and requires user interaction while the proposed atlas-based approach is fully automatic. For high-throughput application to clinically relevant
CT datasets with joints degraded by old age, the proposed atlas-based method
outperformed the automatic method of Krčah et al. [7]. Atlas-based segmentation with a selection criterion showed more consistent automatic separationof the femur from the pelvis than the automatic graph-cut method of Krčah
et al. [7].
The accuracy of the atlas-based segmentation was seen to depend on thecomposition of the atlas. The average Hausdorff distance increased by 2.58 mm
when the contralateral femur was removed from the atlas. Decreased performanceis expected since contralateral femurs are anatomically similar in size and shape,
but will vary locally in density, osteophytic growths, and geometric conformity.
It has to be highlighted that this is not the first work proposing an atlasbased method for femur segmentation [9, 11]. Atlas are a favoured technique forfemur segmentation because they can use shape information to better separate
the femur from the pelvis. However, using a selection criterion for ranking atlases
against a query image is a computationally fast method of utilizing all atlases
in a multi-atlas. A selection criterion avoids a full-resolution registration of eachatlas image to the query image while selecting the best atlas for the query image.
To improve segmentation accuracy, a larger number of images with a broader
anatomical range could be included in the atlas. Furthermore, combining atlas selection with label fusion could increase the segmentation accuracy of the
proposed atlas-based segmentation algorithm while decreasing the computation
time relative to naive label fusion techniques [12].
In conclusion, multi-atlas-based segmentation with a selection criterion is apromising alternative and provides more consistent femur segmentations thantypically used graph-cut methods in a challenging CT cadaveric dataset.
Acknowledgements. The authors would like to thank NSERC CGS-M for funding,the Body Donation Program at the Gross Anatomy Laboratory for access to cadaveric
specimens, and the individuals who graciously contributed their bodies. The authors
would also like to thank Dr. Sonny Chan from the Department of Computer Scienceat the University of Calgary for guidance.

References
1. Pickhardt, P., Pooler, B., Lauder, T., del Rio, A., Bruce, R., Binkley, N.: Opportunistic screening for osteoporosis using abdominal computed tomography scansobtained for other indications. Ann. Intern. Med. 158(8), 588–595 (2013)
2. Burge, R., Dawson-Hughes, B., Solomon, D., Wong, J., King, A., Tosteson, A.: Incidence and economic burden of osteoporosis-related fractures in the United States,
2005-2025. J. Bone Miner. Res. 22(3), 465–475 (2007)
3. Zoroofi, R., Sato, Y., Sasama, T., Nishii, T., Sugano, N., Yonenobu, K., Yoshikawa,
H., Ochi, T., Tamura, S.: Automated segmentation of acetabulum and femoral headfrom 3-D CT images. IEEE Trans. Inf. Technol. Biomed. 7(4), 329–343 (2003)

Automatic Full Femur Segmentation

127

4. Kang, Y., Engelke, K., Kalender, W.: A new accurate and precise 3-D segmentation
method for skeletal structures in volumetric CT data. IEEE Trans. Med. Imaging
22(5), 586–598 (2003)
5. Boykov, Y., Funka-Lea, G.: Graph cuts and efficient N-D image segmentation. Int.
J. Comput. Vision 70(2), 109–131 (2006)
6. Pauchard, Y., Fitze, T., Browarnik, D., Eskandari, A., Pauchard, I., Enns-Bray,
W., Pálsson, H., Sigurdsson, S., Ferguson, S., Harris, T., Gudnason, V., Helgason,
B.: Interactive graph-cut segmentation for fast creation of finite element models
from clinical CT data for hip fracture prediction. Comput. Methods. Biomech.
Biomed. Engin. 19(16), 1693–1703 (2016)
7. Krčah, M., Székely, G., Blanc, R.: Fully automatic and fast segmentation of thefemur bone from 3D-CT images with no shape prior. In: Proc. 8th IEEE International Symposium on Biomedical Imaging – ISBI 2011. pp. 2087–2090. IEEE
(2011)
8. Descoteaux, M., Audette, M., Chinzei, K., Siddiqi, K.: Bone enhancement filtering: application to sinus bone segmentation and simulation of pituitary surgery.
Comput. Aided Surg. 11(5), 247–255 (2006)
9. Ehrhardt, J., Handels, H., Malina, T., Strathmann, B., Plötz, W., Pöppl, S.: Atlasbased segmentation of bone structures to support the virtual planning of hip operations. Int. J. Med. Inform. 64(2–3), 439–447 (2001)
10. Thirion, J.: Image matching as a diffusion process: an analogy with Maxwell’s
demons. Med. Image Anal. 2(3), 243–260 (1998)
11. Whitmarsh, T., Treece, G., Poole, K.: Automatic segmentation and discrimination of connected joint bones from CT by multi-atlas registration. In: Yao, J.,et al. (eds.) Computational Methods and Clinical Applications for Spine Imaging,
LNCVB, vol. 17, pp. 199–207. Springer (2014)
12. Aljabar, P., Heckemann, R., Hammers, A., Hajnal, J., Rueckert, D.: Multi-atlas
based segmentation of brain images: atlas selection and its effect on accuracy.
NeuroImage 46(3), 726–738 (2009)
13. Klein, S., Staring, M., Murphy, K., Viergever, M., Pluim, J.: Elastix: a toolbox
for intensity-based medical image registration. IEEE Trans. Med. Imaging 29(1),196–205 (2010)
14. Dice, L.: Measures of the amount of ecologic association between species. Ecology
26(3), 297–302 (1945)
15. Huttenlocher, D., Klanderman, G., Rucklidge, W.: Comparing images using theHausdorff distance. IEEE Trans. Pattern Anal. Mach. Intell. 15(9), 850–863 (1993)
16. Rohlfing, T., Brandt, R., Menzel, R., Maurer, C.: Evaluation of atlas selection
strategies for atlas-based image segmentation with application to confocal microscopy images of bee brains. NeuroImage 21(4), 1428–1442 (2004)

Classification of Osteoporotic Vertebral
Fractures Using Shape andAppearance Modelling
Paul A. Bromiley1( ) , Eleni P. Kariki2 ,Judith E. Adams2 , and Timothy F. Cootes1
1

Centre for Imaging Sciences, School of Health Sciences,
University of Manchester, UK
2Radiology & Manchester Academic Health Science Centre,
Central Manchester University Hospitals NHS Foundation Trust, UK
paul.bromiley@manchester.ac.uk

Abstract. Osteoporotic vertebral fractures (VFs) are currently underdiagnosed, creating an opportunity for computer-aided, opportunistic
fracture identification in clinical images. VF diagnosis and grading inclinical practice involves comparisons of vertebral body heights. However,machine vision systems can provide a high-resolution segmentation of
the vertebral bodies and fully characterise their shape and appearance,
potentially allowing improved diagnostic accuracy compared to heightmeasurement alone. We compare approaches based on vertebral heights
to shape/appearance modelling combined with k-nearest neighbour andrandom forest (RF) classifiers, on both dual-energy X-ray absorptiometry
images and computed tomography image volumes containing the spine.
We conclude that significant gains in diagnostic accuracy can be achieved
using appearance features and RF classifiers.
Keywords: Osteoporosis, vertebral fracture, shape modelling.

1

Introduction

Osteoporosis is a common skeletal disorder characterised by a reduction in bonemineral density (BMD). This is commonly assessed using dual-energy X-ray absorptiometry (DXA); a T-score of < −2.5 (i.e. more than 2.5 standard deviations
below the mean in young adults) [1] is used as a criterion suggesting osteoporosis. It significantly increases the risk of fractures, most commonly occurring in
the hips, wrists or vertebrae. Approximately 40% of postmenopausal Caucasian
women are affected, increasing their lifetime risk of fragility fractures to as muchas 40% [1]. Osteoporosis therefore presents a significant public health problem
for an ageing population. However, between 30−60% of vertebral fractures (VFs)may be asymptomatic and only about one third of those present on images cometo clinical attention; they are frequently not reported by radiologists. Many ofthese cases involve images acquired for purposes other than VF identification,
so identification may be opportunistic. Computed tomography (CT) is arguably
B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proc. 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. 128−140, 2017.

Classification of Osteoporotic Vertebral Fractures

(a)

(b)

129

(c)

Fig. 1. Example vertebrae visualized on GE Luna iDXA images. (a) Mild lateral curvature of the lumbar spine may cause tilting of the vertebrae relative to the beam
direction, leading to the appearance of end-plate curvature although no fractures arepresent. (b) Osteoporotic vertebral fractures lead to loss of vertebral height and changes
in texture due to the presence of micro-fractures; the upper end plate is not symmetrical with the lower and appears blurred. (c) Spondylosis also results in changes invertebral shape and texture, due to the growth of osteophytes on the anterior portionof the end-plates and sclerosis i.e. the high-intensity region within the vertebral body,
below the anterior portion of the upper end-plate.

the ideal modality for opportunistic identification, due to the large number ofprocedures (4.3 million per year within the UK National Health Service [2]) andhigh image quality. However, a recent audit at the Manchester Royal Infirmary(MRI) revealed that only 13% of VFs visible on CT images were identified [3],similar to rates reported in the literature [4]. Proposed reasons for such low
rates [4] include the difficulty of identifying vertebral height reduction on axial
images. Routine coronal and/or sagittal reformatting has been proposed, andis being adopted, but reporting rates remain low [4]. The potential utility ofcomputer-aided VF assessment (VFA) systems is therefore considerable.
Several authors have investigated the use of methods based on statisticalshape models (SSMs) [5] to segment vertebrae in both radiographs (e.g. [6])and DXA images (e.g. [7, 8]). In particular, the random-forest regression voting
constrained local model (RFRV-CLM) has been used for both semi- and fully
automatic vertebral body segmentation in both CT [9] and DXA [10,11] images,
providing superior segmentation accuracy on more severely fractured vertebrae
compared to previous work using active appearance models (AAMs) [12]. However, all of these approaches share the common aim of providing a high-resolution
segmentation of the vertebrae, typically as landmarks annotated on the vertebral
body outline.

130

Bromiley et al.

Several procedures for manual VFA have been described in the literature;see [13] for a recent review. Most attempt to remove the subjectivity of qualitative assessment [14] by defining fractures in terms of height reduction at theposterior, middle and anterior parts of the vertebral bodies (e.g. [15]). However,
non-fracture deformities can affect vertebral shape, as shown in Fig. 1. In particular, tilting of the vertebrae can mimic the appearance of depressed end-plates.
VFA therefore requires a subjective assessment of whether any apparent shapechange is due to osteoporotic fracture or some other cause. The algorithm-based
quantitative (ABQ) method for VFA [16] defined a heuristic for this process. The
result is a complex procedure that involves consideration of multiple, interacting
factors, including the apparent shape of each vertebra and the spine as a whole,which is difficult to translate into a machine vision algorithm.
Whilst machine vision based VFA methods suffer from difficulties in terms ofreplicating such complex, heuristic approaches, they have a potential advantage
in that techniques based on SSMs can provide a precise mathematical description of the entire shape of a vertebra, and quantitatively compare this between
vertebrae. We investigate the interaction of these two effects by constructing VF
classifiers based on comparing the parameters of shape and appearance models of vertebrae using both k-nearest neighbours (kNNs) and random forests(RFs) [17]. These methods are compared to simple, height-based classifiers on
two data sets; 320 DXA VFA images and spinal mid-line sagittal images projected from 402 CT volumes. When RFs were used to classify appearance model
parameters, significant reductions in false positive rate (FPR) of ≈ 30% and≈ 60% were achieved at 80% sensitivity for VF identification from automaticand manual landmark annotations, respectively, on both image types.

2

Method2.1

Data Collection and Manual Annotation

The picture archiving and communication server (PACS; Centricity Universal
Viewer, GE Healthcare, Little Chalford, Buckinghamshire, UK) at the MRI was
queried to produce a list of CT scans acquired during May and June 2014 andJanuary to September 2015. Scans from non-trauma patients that included any
part of the thoracic or lumbar spine and were of patients over 18 years of age,
were selected. This gave a list of 868 patients’ scans. The PACS was also queried
for non-trauma CT scans during January to April and July to December 2014in patients over 60 years of age that contained osteoporotic VFs, producing asecond list of 132 patients. The sagittal reformatted volumes from both lists were
downloaded in DICOM format. 402 volumes were selected to form a training set,
including the 132 fracture-rich images to ensure high fracture prevalence. The
remaining images were reserved for future validation purposes. The 402 image
list was divided into quarters for leave-1/4-out training and testing, with thefracture-rich images distributed evenly. Each volume was up-sampled to give
isotropic voxel dimensions using tri-cubic interpolation.

Classification of Osteoporotic Vertebral Fractures

(a)

131

(b)

(d)

(c)

(e)

Fig. 2. (a) An example coronal maximum intensity projection of a computed tomography volume. (b) Manual annotations of the neural arch. (c) Extrapolated piecewiselinear curve and the ±5 mm range (dashed line) over which sagittal rasters weresummed to produce the sagittal projection (d). (e) Manual landmark annotation.

To avoid the difficulties of performing a high-resolution annotation of landmarks on vertebral bodies in 3D, analysis was limited to a single, two-dimensional(2D) image produced from each volume using the procedure described in [9]. The
orientation of the subject within the CT scanner was highly constrained, allowing production of a maximum intensity projection showing an approximately
anteroposterior view without registration. Landmarks were manually annotatedon the maximum intenisty projection images at the distinctive, U-shaped structure on each vertebra where the laminae join to form the spinous process ofthe neural arch (Fig. 2(a),(b)). A piecewise-linear curve was defined through thepoints and extrapolated vertically to the boundary of the volume (Fig. 2(c)).
For each axial slice from the original volume, all anteroposterior raster lines (i.e.
rasters of sagittal slices) that passed within Dt of this curve were averaged togive a single raster line of a sagittal image. Repeating for all axial images gavea single, thick-slice, 2D sagittal image that showed the midplane of each vertebra (Fig. 2(d)). The thickness Dt = ±5 mm was chosen by manual inspection to
optimise endplate visualisation.

132

Bromiley et al.

The images derived from CT volumes were projected onto the spinal midline, and so were unaffected by issues such as mild scoliosis. DXA images areprojections through the full body, and so will show the tilting of the vertebrae
encountered in this condition, making accurate diagnosis based on vertebral
shape more difficult. Therefore, a second data set of 320 DXA VFA images
scanned on various Hologic (Bedford MA, USA) scanners was also used. This
comprised: 44 patients from a previous study [18]; 80 female subjects in an
epidemiological study of a UK cohort born in 1946; 196 females attending a
local clinic for DXA BMD measurement, for whom the referring physician had
requested VFA (approved by the local ethics committee).
Manual annotation of 33 landmarks on each visualized vertebra from T7 to
L4, for the DXA images, and T4 to L4, for the CT midline images (Fig. 2(e)),
was performed by a trained radiographer. The vertebrae were also classified by
an expert radiologist into five groups: normal; deformed but not fractured; andmild (grade 1), moderate (grade 2) and severe (grade 3) fractures using the
Genant definitions [15]).
2.2

Height-Based Fracture Classification

A baseline for VF classification accuracy was derived by applying a simple classifier, based on six-point morphometry, as described in [10]. The anterior ha ,middle hm and posterior hp heights of each detected vertebra were calculated
from the relevant landmarks, together with a predicted posterior height hp0 , calculated as the maximum of the posterior heights of the four closest vertebrae.
The wedge rw = ha /hp , biconcavity rb = hm /hp , and crush rc = hp /hp0 ratios
were derived, and the data were whitened by subtracting the medians of eachratio and dividing by the square-root of the covariance matrix, calculated using the median standard deviation. The data contained far more normal thandeformed or fractured vertebrae, so this process whitened to the distribution of
the normal class. A simple fracture/non-fracture classification was performed by2; deformed vertebrae were counted
applying a threshold tclass to rc2 + rb2 + rw
correct when classified into either class.
2.3

Shape and Appearance Model Based Classifiers

SSMs provide a linear model of the distribution of a set of landmarks in an image.
The training data consists of a set of images I with manual annotations xl of a setof N points l = 1, . . . , N on each. The images are first aligned into a standardised
reference frame using a similarity registration, giving a transformation T withparameters θ. The concatenated, reference-frame coordinates of the points ineach training image define its shape. The SSM is generated by applying principal
component analysis (PCA) to the training shapes [12], generating a linear model
where the position of point l is given byxl = Tθ (x̄l + Psl bs ),(1)

Classification of Osteoporotic Vertebral Fractures

133

(d)

(e)

(g)

(a)

(b)

(c)

(j)

(f)

(h)

(i)

(k)

(l)

Fig. 3. Point sampling strategies, using (a) a single vertebra, (b) a triplet and (c) avertebra plus its neighbouring end-plates (nEP). Texture sampling strategies in the
single vertebra case. (d)-(f) Sampling using Delaunay triangulation with increasing
wf rame . (g)-(i) Sampling using Delaunay triangulation with increasing wborder . (j)-(l)Patch-based sampling with increasing wpatch .

where x̄l is the mean point position in the reference frame, Ps is a matrix ofmodes of variation, Psl is the sub-matrix of Ps relevant to point l, and bs
encodes the shape model parameters. The matrix Ps is orthogonal and so
bs = PTsl (Tθ−1 (xl ) − x̄l ).

(2)

A compact description of the shape in a query image can therefore be derived byannotating the landmarks, performing a similarity registration into the reference
frame of the model, and applying (2) to generate the vector bs .
The SSM considers only the distribution of landmarks on a shape. However,with reference to osteoporosis and potentially confounding pathologies, information is also present in the pixel intensities. Osteoporotic VF proceeds as a cascadeof micro-fractures in the vertebral end-plates leading to a blurred appearance(Fig. 1) [13]. Appearance models (APMs) such as those used by AAMs [12] adopt
the same PCA-based linear modelling approach as the SSM to characterize bothshape and intensity information. Each training image is resampled into the reference frame by applying Ir (m, n) = I(Tθ−1 (m, n)), where (m, n) specify pixel
coordinates. The reference frame width wf rame acts as a free parameter controlling the resolution. The intensities of each pixel within an image patch covering
the points are then concatenated into a vector g, and PCA applied as before to
generate a linear model based on modes of variation Pg
g = Tφ (ḡ + Pg bg )and bg = PTg (Tφ−1 (g) − ḡ),

(3)

134

Bromiley et al.

where Tφ represents an intensity normalisation. Correlations may exist between
the shape bs and intensity bg parameters, and so the models are concatenatedand a further PCA performed to extract the independent modes of variation ofboth shape and intensity Pc , referred to as appearance modes


Ws bs
b = Pc c where b =.
(4)
bg
The weights Ws scale the relative magnitude of the shape and intensity parameters, and are derived by sampling the change in g per unit change in bs in the
training images.
Two approaches for sampling the intensities contributing to g were tested(see Fig. 3). The first used a Delaunay triangulation of the landmarks to define a region of interest, with an optional, additional border of width wborderto ensure the whole edge was included. However, most relevant intensity information was expected to be located close to the end-plates i.e. the site of thefractures. Therefore, an alternative strategy that involved sampling a square
patch of width wpatch around each landmark was also implemented. Multiple
approaches for defining the landmarks used were also tested (see Fig. 3), including sampling from a single vertebra, a triplet of neighbouring vertebrae, anda vertebra plus the closest end-plates of its neighbours (nEP sampling). The
latter were intended to aid in identification of tilted vertebra, since these are
distinguished from fractures, in clinical practice, by the symmetry of adjacentend-plates (see Fig. 1).
The SSM or APM extract all significant shape and intensity informationfrom an image as a compact vector of features. A variety of classifiers could
then be applied to compare the features of a query image to those of annotated and diagnosed training images. Two were studied here. First, kNN was
applied, measuring Euclidean distance in the feature space of b or bs to identify
NkN N neighbours. However, kNN has the drawback that all features are considered; some will not be relevant to fracture status and so will potentially act
as confounding information. To determine whether this effect was significant,classification was also performed using RFs. Since each split node considers asingle feature, an RF has the capability to identify only those features relevantto the target. In both cases, the problem was treated as a regression task. The
gold-standard diagnosis was translated into a numerical score with 0 = normal,
1 = deformed but not fractured and 2, 3, and 4 = mild, moderate and severe fracture. The output of the kNN was the mean of this score across the identified
neighbours, and RFs were trained as regressors to predict the score. This createda potential problem as it assumed that deformed vertebrae are intermediate inshape between normal and fractured ones, which may not be the case. However,it more accurately represented fracture status as a position within a continuum,
rather than discrete classes. The alternative, treating the problem as an explicit
classification task, was investigated using a multi-class RF classifier. However,this led to significantly worse accuracy and full results are not reported here. As
with the height-based classifier, a simple fracture/non-fracture classification was
performed by thresholding the kNN or RF output.

Classification of Osteoporotic Vertebral Fractures

3

135

Evaluation

Throughout the evaluation, classifiers were trained and tested in a leave-1/4out procedure using the data from all vertebral levels. During SSM and APM
training, the number of modes of variation was constrained to model 98% of thevariation in the data. Free parameters were empirically optimised on the CT
midline images and manual annotations, such that the DXA images served as
an independent evaluation set. Initially, the shape and appearance model parameters were optimised in combination with a kNN classifier. Since the latterhad only one free parameter, NkN N , this greatly reduced the dimensionality ofthe parameter space compared to using a RF. Receiver operator characteristic(ROC) curves showing sensitivity against false positive rate (FPR) were generated by varying tclass , and the parameters leading to the highest value at which
sensitivity equalled FPR were selected, giving wf rame = 80 pixels, wborder = 30
pixels for triangulated intensity sampling, wpatch = 24 pixels for patch-based
sampling, and NkN N = 10. This was repeated for all combinations of point andintensity sampling as described in Sec. 2.3, and the optimae were consistent. In
general, dependence on the parameters was weak for all except wf rame . A second round of parameter optimisation focused on the RF, using the optimisedparameters for SSM/APM described above. The same ROC-curve based pattern search procedure was used to optimise the number of trees ntree in the RF,
the maximum depth of each tree Dmax , and the minimum number of training
samples nmin allowed at a split node, leading to ntree = 200, Dmax = 30 and
nmin = 1.
Finally, semi-automatic annotation of the vertebrae in both the DXA andCT midline images, the latter projected using the manual neural arch annotations, was performed using a RFRV-CLM, initialised using manual annotationsof vertebral centre points, following the procedure described in [11] and [9]. The
classifiers were then applied to the automatic annotations using the optimisedparameters. Classification accuracies for the optimised procedure with all combinations of SSM/APM, classifier, patch and intensity sampling procedure andmanual or automatic annotation were then compared.
Fig. 4 and 5 show the evaluation of various sampling, feature extraction andclassification procedures for the CT and DXA images, respectively, compared
to baselines established by the six-point morphometry approach. A universal
trend was noted across all experiments; accuracy for triplet sampling was always significantly worse than the alternatives (full results are not shown for thisreason). Triplet sampling results in more modes of variation and so more features in b and bs , and adding the neighbouring vertebrae spreads the training
data across the feature space depending on the fracture status of the neighbours.
Both reduce data density. In contrast, whilst 6-point morphometry benefits fromcomparison of posterior vertebral heights between neighbours to identify crush
fractures, where the height is reduced throughout a vertebral body, shape models
can extract equivalent information from a single vertebra through quantificationof its aspect ratio. Triplet sampling therefore adds little information to the fea-

Bromiley et al.

1.0

1.0

0.9

0.9

0.8

0.8

0.7
6-point morphometry
Tri app RF
Tri app kNN
Tri shape RF
Tri shape kNN

0.6
0.5
0.4
0.00

0.05

0.10

False positive rate

0.15

Sensitivity

Sensitivity

136

0.7
6-point morphometry
Patch app RF
Patch app kNN
Patch shape RF
Patch shape kNN

0.6
0.5
0.4
0.00

0.20

0.05

1.0

1.0

0.9

0.9

0.8

0.8

0.7
6-point morphometry
Tri app RF
Tri app kNN
Tri shape RF
Tri shape kNN

0.6
0.5
0.4
0.00

0.05

0.10

False positive rate

(c)

0.15

0.20

(b)

0.15

0.20

Sensitivity

Sensitivity

(a)

0.10

False positive rate

0.7
6-point morphometry
Patch app RF
Patch app kNN
Patch shape RF
Patch shape kNN

0.6
0.5
0.4
0.00

0.05

0.10

False positive rate

0.15

0.20

(d)

Fig. 4. Receiver operator characteristic curves for vertebral classification on computed
tomography midline images. (a),(b) Point sampling from single vertebrae. (c),(d) Point
sampling from vertebrae plus neighbouring end-plates. “Tri” refers to intensity sampling from Delaunay triangulated regions, “patch” to sampling from patches aroundeach landmark point, “shape” to shape and “app” to appearance features.

ture space, but makes it more difficult for a classifier to extract the information
present.
Differences between the remaining sampling/modelling/classification procedures were usually small and frequently not significant, but the optimal procedure was always significantly better than 6-point morphometry. Several trends
emerged from the results. On CT images (Fig. 4), there was little evidence of additional information in appearance compared to shape features. The kNN classifier showed a marked reduction in performance when triangulated intensity
sampling was used, adding large numbers of uninformative appearance features(Fig. 4(a),(c)). The RF classifier also showed some evidence of this effect, withthe combination of triangulated nEP sampling and appearance feature classification resulting in performance no better than the baseline (Fig. 4c). However,the RF, in general, resulted in better accuracy than the kNN classifier and did
not lose accuracy when nEP sampling was used, or when appearance features
were used if the number of features was controlled using patch-based sampling(Fig. 4(b),(d)).

1.0

1.0

0.9

0.9

0.8

0.8

0.7
6-point morphometry
Tri app RF
Tri app kNN
Tri shape RF
Tri shape kNN

0.6
0.5
0.4
0.00

0.05

0.10

False positive rate

0.15

Sensitivity

Sensitivity

Classification of Osteoporotic Vertebral Fractures

137

0.7
6-point morphometry
Patch app RF
Patch app kNN
Patch shape RF
Patch shape kNN

0.6
0.5
0.4
0.00

0.20

0.05

1.0

1.0

0.9

0.9

0.8

0.8

0.7
6-point morphometry
Tri app RF
Tri app kNN
Tri shape RF
Tri shape kNN

0.6
0.5
0.4
0.00

0.05

0.10

False positive rate

(c)

0.15

0.20

(b)

0.15

0.20

Sensitivity

Sensitivity

(a)

0.10

False positive rate

0.7
6-point morphometry
Patch app RF
Patch app kNN
Patch shape RF
Patch shape kNN

0.6
0.5
0.4
0.00

0.05

0.10

False positive rate

0.15

0.20

(d)

Fig. 5. Receiver operator characteristic curves for vertebral classification on dualenergy X-ray absorptiometry images. (a),(b) Point sampling from single vertebrae.
(c),(d) Point sampling from vertebrae plus neighbouring end-plates. “Tri” refers to intensity sampling from Delaunay triangulated regions, “patch” to sampling from patches
around each landmark point, “shape” to shape and “app” to appearance features.

DXA images (Fig. 5) represented a more challenging task since, being projections of the full vertebral body, the shape as visualised in the images wasmore complex, and the image quality was lower. Increased noise on individual features resulted in kNN outperforming RF when single-vertebra sampling
was used (Fig. 5(a),(b)), with accuracy gains resulting from using appearance aslong as the length of the feature vector was controlled by using patch-based sampling (Fig. 5(b)). However, using shape information from neighbouring vertebrae
through nEP sampling, to deal with cases where the vertebrae were tilted relative
to the beam direction, allowed the RF to achieve equal or better performanceon shape alone (Fig. 5(d)).
Fig. 6 shows results from the optimised procedure, using patch-based sampling and RF classification, for both manual and automatic annotations on both
image sets. Each experiment was repeated five times, using the stochastic natureof RF training to support error estimation, and the figures show the mean and(where shown) standard deviation of the repeats. As described above, nEP sampling resulted in increased accuracy for both image types when classifying frommanual annotations (Fig. 6(a),(c)). However, this requires accurate annotations

Bromiley et al.

1.0

1.0

0.9

0.9

0.8

0.8

0.7
6-point morphometry
Patch app RF nEP
Patch app RF vert
Patch shape RF nEP
Patch shape RF vert

0.6
0.5
0.4
0.00

0.05

0.10

False positive rate

0.15

Sensitivity

Sensitivity

138

0.7
6-point morphometry
Patch app RF nEP
Patch app RF vert
Patch shape RF nEP
Patch shape RF vert

0.6
0.5
0.4
0.00

0.20

0.05

1.0

1.0

0.9

0.9

0.8

0.8

0.7
6-point morphometry
Patch app RF nEP
Patch app RF vert
Patch shape RF nEP
Patch shape RF vert

0.6
0.5
0.4
0.00

0.05

0.10

False positive rate

(c)

0.15

0.20

(b)

0.15

0.20

Sensitivity

Sensitivity

(a)

0.10

False positive rate

0.7
6-point morphometry
Patch app RF nEP
Patch app RF vert
Patch shape RF nEP
Patch shape RF vert

0.6
0.5
0.4
0.00

0.05

0.10

False positive rate

0.15

0.20

(d)

Fig. 6. Receiver operator characteristic curves for random forest classification withpatch sampling in computed tomography midline (a),(b) and dual-energy X-ray absorptiometry (c),(d) images, from manual (a),(c) and RFRV-CLM (b),(d) annotations,compared to 6-point morphometry. “Shape” refers to shape and “app” to appearance
features.

of all vertebrae in the triplet, increasing the risk of a fit failure being present
in the automatic annotations. Single-vertebra sampling was therefore more accurate on RFRV-CLM annotations of CT images and appearance provided no
additional information over shape (Fig. 6(b)). However, the difficulties of theclassification task in DXA images, described above, resulted in higher accuracies when nEP sampling and appearance parameters were used, to disambiguate
tilted vertebra and confounding pathologies (Fig. 6(d)). At 80% sensitivity, theoptimal classifier reduced the FPR from 4.4% to 1.7% for manual and from 12.7%
to 8.8% for automatic DXA annotations. For CT images, FPR was reduced from
8.9% to 3.3% for manual and 9.7% to 7.0% for automatic annotations. Theseequate to a significant (p < 0.01) reduction in FPR of ≈ 30% for automatic, and≈ 60% for manual annotations.

4

Conclusion

We have evaluated several different methods for osteoporotic VF classification in
DXA VFA and CT midline images. In this context, the 6-point morphometry ap-

Classification of Osteoporotic Vertebral Fractures

139

proach can be viewed as the construction of a low-parameter shape model basedon hand-crafted modes of variation. Since these reflect the clinical definitions
used in fracture grading [15] they are guaranteed to be informative. A SSM or
APM can extract all statistically significant shape and appearance informationfrom the images as a compact feature vector, but not all of these features areguaranteed to be informative in terms of fracture classification. The way in which
the subsequent classifier handles non-informative dimensions of the feature space
therefore influences accuracy. Straightforward application of kNN considers allfeatures equally, whilst a RF considers only the most informative at each split
node, and so is more robust to non-informative features.
The results showed that statistically significant gains in classification accuracy can be achieved by applying kNN or RF classifiers to shape or appearance
model features. The optimal procedure across both CT and DXA was to apply
an RF classifier to features sampled from a vertebra and the closest end-plates
of its neighbours with manual landmark annotations, but fitting errors in automatic annotations resulted in single vertebra sampling being more accurate
for CT images. Evidence for improved performance when using appearance, as
opposed to shape features alone, was generally weak and inconsistent. Significant accuracy gains compared to 6-point morphometry were achieved for bothmanual and automatic annotations on both DXA VFA and CT images.
This work used a single model/classifier for all vertebral levels between T4(for CT) or T7 (for DXA VFA) and L4, inclusive. Vertebral shape varies gradually across the spine, and so further improvements in accuracy might be gained
through using multiple, level-specific classifiers. Roberts et al. [8] achieved higher
classification accuracies on DXA VFA images using this approach, and so we intend to investigate this in future work. However, the use of level-specific classifiers
requires a reasonably accurate method of level detection if combined with an automatic vertebral segmentation method, which may prove challenging given thesimilarity between vertebra.
Acknowledgements. This publication presents independent research supported by
the Health Innovation Challenge Fund (grant no. HICF-R7-414/WT100936), a parallelfunding partnership between the Department of Health and Wellcome Trust, and by
the NIHR Invention for Innovation (i4i) programme (grant no. II-LB 0216-20009). The
views expressed are those of the authors and not necessarily those of the NHS, NIHR,
the Department of Health or Wellcome Trust. The authors acknowledge the invaluable assistance of Mrs Chrissie Alsop, Mr Stephen Capener, Mrs Imelda Hodgkinson,
Mr Michael Machin, and Mrs Sue Roberts, who performed the manual annotations.

References
1. Rachner, T., Khosla, S., Hofbauer, L.: Osteoporosis: now and the future. Lancet
377(9773), 1276–1287 (2011)
2. Operational Information for Commissioning: Diagnostic imaging dataset statisticalrelease. Tech. rep., NHS, UK (2016), http://www.england.nhs.uk/statistics/wpcontent/uploads/sites/2/2015/08/Provisional-Monthly-Diagnostic-ImagingDataset-Statistics-2016-05-19.pdf

140Bromiley et al.

3. Williams, A., A Al-Busaidi, A., Sparrow, P., Adams, J., Whitehouse, R.: Underreporting of osteoporotic vertebral fractures on computed tomography. Eur. J.
Radiol. 69(1), 179–183 (2009)
4. Adams, J.: Opportunistic identification of vertebral fractures. J. Clin. Densitom.
19(1), 54–62 (2016)
5. Cootes, T., Taylor, C., Cooper, D., Graham, J.: Active shape models - their training
and application. Comput. Vis. Image Understand. 61(1), 38–59 (1995)
6. Brett, A., Miller, C., Hayes, C., Krasnow, J., Ozanian, T., Abrams, K., Block, J.,
van Kuijk, C.: Development of a clinical workflow tool to enhance the detection ofvertebral fractures. Spine 34(22), 2437–2443 (2009)
7. Roberts, M., Cootes, T., Adams, J.: Vertebral morphometry: semiautomatic determination of detailed shape from dual-energy X-ray absorptiometry images using
active appearance models. Invest. Radiol. 41(12), 849–859 (2006)
8. Roberts, M., Cootes, T., Adams, J.: Automatic location of vertebrae on DXA
images using random forest regression. In: Ayache, N., et al. (eds.) Proc. 15th
International Conference on Medical Image Computing and Computer-Assisted
Intervention – MICCAI 2012, LNCS, vol. 7512, pp. 361–368. Springer (2012)
9. Bromiley, P., Kariki, E., Adams, J., Cootes, T.: Fully automatic localisation ofvertebrae in CT images using random forest regression voting. In: Yao, J., et al.
(eds.) Proc. 4th MICCAI Workshop & Challenge on Computational Methods andClinical Applications for Spine Imaging – CSI 2016, LNCS, vol. 10182, pp. 51–63.
Springer (2016)10. Bromiley, P., Adams, J., Cootes, T.: Automatic localisation of vertebrae in DXA
images using random forest regression voting. In: Vrtovec, T., et al. (eds.) Proc.
3rd MICCAI Workshop & Challenge on Computational Methods and Clinical Applications for Spine Imaging – CSI 2015, LNCS, vol. 9402, pp. 38–51. Springer(2016)
11. Bromiley, P., Adams, J., Cootes, T.: Localisation of vertebrae on DXA images
using constrained local models with random forest regression voting. In: Yao, J.,et al. (eds.) Proc. 2nd MICCAI Workshop on Computational Methods and ClinicalApplications for Spine Imaging – CSI 2014, LNCVB, vol. 20, pp. 159–171. Springer(2015)
12. Cootes, T., Edwards, G., Taylor, C.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)
13. Griffith, J.: Identifying osteoporotic vertebral fracture. Quant. Imaging Med. Surg.
5(4), 592–602 (2015)
14. Jensen, G., McNair, P., Boesen, J., Hegedus, V.: Validity in diagnosing osteoporosis. Observer variation in interpreting spinal radiographs. Eur. J. Radiol. 4(1), 1–3(1984)
15. Genant, H., Wu, C., Kuijk, C., Nevitt, M.: Vertebral fracture assessment using asemi-quantitative technique. J. Bone Miner. Res. 8(9), 1137–1148 (1993)
16. Jiang, G.: Diagnosis of vertebral fracture using an ABQ method. Osteoporos. Rev.
18(3), 14–18 (2010)17. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
18. McCloskey, E., Selby, P., de Takats, D., Bernard, J., Davies, M., Robinson, J.,Francis, R., Adams, J., Pande, K., Beneton, M., Jalava, T., Loyttyniemi, E., Kanis,J.: Effects of clodronate on vertebral fracture risk in osteoporosis: a 1-year interim
analysis. Bone 28(3), 310–315 (2001)

DSMS-FCN: A Deeply Supervised Multi-Scale
Fully Convolutional Network for Automatic
Segmentation of Intervertebral Discin 3D MR Images
Guodong Zeng and Guoyan Zheng()

Institute for Surgical Technology and Biomechanics, University of Bern, Switzerland
guoyan.zheng@istb.unibe.ch

Abstract. This paper addresses the challenging problem of segmentation of intervertebral discs (IVDs) in three-dimensional (3D) T2-weighted
magnetic resonance (MR) images. We propose a deeply supervised multiscale fully convolutional network for segmentation of IVDs in 3D MR
images. After training, our network can directly map a whole volumetricdata to its volume-wise labels. Multi-scale deep supervision is designed
to alleviate the potential gradient vanishing problem during training. It
is also used together with partial transfer learning to boost the training
efficiency when only small set of labeled training data are available. The
present method was validated on the MICCAI 2015 IVD segmentation
challenge datasets. Our method achieved a mean Dice overlap coefficientof 92.0% and a mean average symmetric surface distance of 0.41 mm.
The results achieved by our method are better than those achieved by
the state-of-the-art methods.
Keywords: Intervertebral disc, MRI, segmentation, deep learning, fully
convolutional network.

1

Introduction

Intervertebral disc (IVD) degeneration is a major cause for chronic back pain
and function incapacity [1]. In clinical practice, spine magnetic resonance (MR)imaging (MRI) is the preferred modality in diagnosis and treatment planningof various spinal pathologies such as disc herniation, slipped vertebra and so
on, not only because MRI is non-invasive and does not use ionizing radiation,
but more importantly because it offers good soft tissue contrast that allows forvisualization of disc’s internal structure [2].
Accurate IVD segmentation from spine MR image is therefore very importantfor correct diagnosis and treatment planning [1, 3]. Traditionally, most quantitative studies on IVD degeneration have been done by manually segmenting thedata, which is tedious, time-consuming and error-prone. On the other hand, afully-automatic system for IVD identification will significantly reduce the timeof the diagnosis. An automatic system might also help reduce errors caused by
B. Glocker, J. Yao, T. Vrtovec, A.F. Frangi, G. Zheng (Eds.),
Proc. 5th MICCAI Workshop on Computational Methods and Clinical Applicationsin Musculoskeletal Imaging – MICCAI-MSKI2017, pp. 141−152, 2017.

142

Zeng and Zheng

subjective factors and improve the consistency of diagnosis standards. In thisway, it can immediately benefit clinical applications and spinal biomechanics
research.
In the literature, different methods have been proposed of IVD segmentation [4–8]. There exist methods based on watershed algorithm [4], atlas registration [5], graph cuts with geometric priors from neighboring discs [6], templatematching and statistical shape model [7], or anisotropic oriented flux detection [8]. Most of these methods work only on two-dimensional sagittal images
and only a few methods [7] address the challenging three-dimensional (3D) IVD
segmentation problem. See [9] for a comprehensive review of existing IVD segmentation methods.
Recently, machine learning-based methods have gained more and more interest. For example, Zhan et al. [10] presented a hierarchical strategy and local articulated model to detect vertebrae and discs 3D MR images and Kelmet al. [11] proposed to use iterated marginal space learning for spine detection
in computed tomography (CT) and MR images. A unified data-driven regression and classification framework was suggested by Chen et al. [12] to tackle theproblem of localization and segmentation of IVDs from T2-weighted MR data,
and Wang et al. [13] proposed to address the segmentation of multiple anatomic
structures in multiple anatomical planes from multiple imaging modalities via asparse kernel machines-based regression.
The more recent development on deep neural networks, and in particular on
convolutional neural networks (CNN), suggests another course of methods tosolve the challenging IVD segmentation problem [14–22]. Contrary to conventional shallow learning methods where feature design is crucial, deep learning
methods automatically learn hierarchies of relevant features directly from thetraining data [15]. More recently, 3D volume-to-volume segmentation networks
were introduced, including 3D U-Net [20], 3D V-Net [21] and a 3D deeply supervised network [22].
In this paper, we propose a deeply supervised multi-scale fully convolutional
network (FCN) called “DSMS-FCN” for fully automatic IVD segmentation in
3D T2-weighted MR images. After training, our network can directly map awhole volumetric data to its volume-wise label. Multi-scale deep supervision isdesigned to alleviate the potential gradient vanishing problem during training. It
is also used together with partial transfer learning to boost the training efficiency
when only small set of labeled training data are available.
The paper is organized as follows. In Sec. 2, we will describe the proposedarchitecture and algorithm. The application to the MICCAI 2015 IVD segmentation challenge dataset will be presented in Sec. 3, and we conclude with a
discussion in Sec. 4.

2

MethodFig. 1 illustrates the architecture of our proposed neural network for the automatic IVD segmentation in 3D T2-weighted MR images. Our network employs aDeeply Supervised Multi-Scale FCN

143

Fig. 1. A schematic illustration of our proposed network architecture. For each block,the digits above indicate the number of feature stack while the numbers below represent
the data size.

deeply supervised multi-scale fully convolutional network. In this section, firstly
the detailed architecture of our proposed model is elaborated, and then we will introduce the multi-scale deep supervision. Finally, partial transfer learning, which
is designed to boost the training efficiency, will be described.
2.1

3D FCN with Skip Connection

Our proposed network is inspired by 3DFCN [23] but with significant differences. Similar to 3DFCN, our network is also a 3D fully convolutional networkand consists of two parts, i.e., the encoder part (contracting path) and the decoder part (expansive path). The encoder part focuses on analysis and feature
representation learning from the input data while the decoder part generates
segmentation results, relying on the learned features from the encoder part. Our
network can take arbitrary-sized volumetric data as input and outputs voxel-wisesegmentation probability map in the same size as the input.
Different from 3DFCN, long and short skip connections, which help recover
spatial context lost in the contracting encoder, are used in our network as shownin Fig. 1. The importance of skip connection in biomedical image segmentation
has been demonstrated by previous studies [24]. Skip connections have been
widely used in many different convolutional neural networks including Resi-Net[25] and 3D U-Net [20].
In 3DFCN [23], big kernel sizes (e.g. 5×7×7) are utilized in the convolutional
layers. However, previous studies have shown that small kernel size are more
helpful for training of deep neural network [26]. For this reason, in our network,
kernel size of 3 × 3 × 3 and strides of 1 are utilized for all convolutional layers,

144

Zeng and Zheng

and kernel size of 2 × 2 × 2 is used in all max pooling layers. Batch normalization(BN) [27] and rectified linear unit (ReLU) are adopted to speed up the training
and to enhance the gradient back propagation.
2.2

Multi-Scale Deep Supervision

Training a deep neural network is challenging. As the matter of gradient vanishing, final loss cannot be efficiently back propagated to shallow layers, which
is more difficult for 3D cases when only a small set of annotated data is available. To address this issue, we inject two down-scaled branch classifiers into our
network in addition to the classifier of the main network, which is another difference between our network and 3DFCN [23]. By doing this, segmentation is
performed at multiple output layers. For the classifier at the coarse scale which
is closer to the encoder part, it generates segmentation results with the coarsest resolution, while the classifiers at the middle and the fine scales generatesegmentation results with the intermediate and the finest resolutions, respectively. As a result, classifiers in different scales can take advantage of multi-scale
context, which has been demonstrated in previous work on segmentation of 3D
liver CT and 3D heart MR images [22]. Furthermore, with the loss calculated by
the prediction from classifiers from different scales, more effective gradient backpropagation can be achieved by direct supervision on the hidden layers.
Specifically, let W be the weights of main network and w= {w0 , w1 , ...
M −1w} be the weights of classifiers at different scales, where M is the numberof classifier branches. For the training samples S = (X, Y ), where X represents
training sub-volume patches and Y represents the class labels while Y ∈ {0, 1}.
Lcls (X, Y ; W, w) =

M
−1
X

X

m=0 (xi ,yi )∈S m

αm lm (xi , yi |W, wm ),(1)

where S = {S 0 , S 1 , . . . , S M −1 }; S 0 is a sub-volume patch directly sampled froma training image while S m contains the examples (xi , yi ) at scale of m > 0, which
is obtained by downsampling S 0 by a factor of 2m along each dimension; wm is
the weights of the classifier at scale of m; αm is the weight of lm , which is theloss calculated by a training sample xi , yi at scale of m.
lm (xi , yi |W, wm ) = − log p(yi = t(xi )|xi ; W, wm )),(2)

where p(yi = t(xi )|xi ; W, wm ) is the probability of predicted class label t(xi )corresponding to sample xi ∈ S m .
The total loss of our multi-scaled deeply supervised model will be :X
Ltotal (X, Y ; W, w) = Lcls (X, Y ; W, w) + λ(ψ(W ) +ψ(wm )),(3)m

where ψ() is the regularization term (L2 norm in our experiment) with hyper
parameter λ.

Deeply Supervised Multi-Scale FCN

2.3

145

Partial Transfer Learning

It is difficult to train a deep neural network from scratch because of limited annotated data. Training deep neural network requires large amount of annotateddata, which are not always available, although data augmentation can partiallyaddress the problem. Furthermore, randomly initialized parameters make it moredifficult to search for an optimal solution in high dimensional space. Transfer
learning from an existing network, which has been trained on a large set of data,
is a common way to alleviate the difficulty. Previous studies [28] demonstrated
that transferring features from another pre-trained model can boost the generalization, and that the effect of transfer learning was related to the similarity
between the task of the pre-trained model and the target task. Furthermore,
the same study also demonstrated that weights of shallow layers in deep neuralnetwork were generic while those of deep layers were more related to specifictasks.
To best utilize the advantage of transfer learning, we need to transfer froma model trained on a related task. In this paper, a pre-trained model in ourprevious work was adopted [29], which is designed for the task of segmentationof the proximal femur from 3D T1-weighted MR Images. More specifically, theweights of the main network are initialized from our previous model [29], while
the weights of all branch classifiers are initialized from a Gaussian distribution(µ = 0, σ = 0.01).
2.4

Implementation Details

The proposed network was implemented in python using TensorFlow frameworkand trained on a desktop with a 3.6 GHz Intel(R) i7 CPU and a GTX 1080 Ti
graphics card with 11 GB GPU memory.

3

Experiments and Results

3.1

Data Description

The training data provided by the MICCAI 2015 IVD challenge organizers consist of 15 3D T2-weighted turbo spin echo MR images and the associated groundtruth segmentation [9]. These 15 3D T2-weighted MR images were acquired from
fifteen patients in two different studies. Each patient was scanned with 1.5 Tesla
MRI scanner of Siemens (Siemens Healthcare, Erlangen, Germany). The pixel
spacings of all the images are sampled to 2 × 1.25 × 1.25 mm3 . There are 7 IVDs
T11-L5 to be segmented from each image. Thus, in each image these IVD regions
have been manually identified and segmented.
The MICCAI 2015 IVD challenge organizers also released two test datasets.
Each test dataset consists of five 3D T2-weighted turbo spin echo MR images.
Thus, in this paper, our network was trained on the fifteen 3D training data
first, and are then evaluated on the ten test data.

146

Zeng and Zheng

3.2

Training Patches Preparation

In order to enlarge the training samples, data augmentation was utilized. Specifically, each training data was rotated (90, 180, 270) degrees around the y axis of
the image and flipped horizontally (taking the z axis as the vertical direction).
After that, we got in total 120 images for training our network.
Our network takes a fixed-sized sub-volume as input, and employs end-to-end
learning and voxel-wise inference. During training, sub-volume patches with thesize of 16 × 256 × 128 was randomly cropped from 120 training examples whose
size are about 40 × 300 × 300. In each epoch of training, 120 training images
were randomly shuffled and then sub-volume patches was randomly cropped withbatch-size of 2 by 5 times from each volumetric training image. Before fed into
the network for training, each sub-volume patch was normalized by zero mean
and unit variance. In total, for each epoch of training, we trained the network
using 1200 (120 × 5 × 2) sub-volume patches.
3.3

Training

We trained our network for 10,000 iterations after partial transfer learning.
All weights were updated by the stochastic gradient descent(SGD) algorithm(momentum = 0.9, weight decay = 0.005). Learning rate was initialized as 1 ×
10−3 and halved by every 3,000 times. In our experiment, we used three branchclassifiers at three different scales. The loss weights of three classifiers α0 , α1and α2 are 1.0, 0.67 and 0.33, respectively. The hyper parameter λ was chosen
to be 0.005.
3.4

Testing

Our trained models can estimate labels of an arbitrary-sized volumetric image.
Given a test volumetric image, we extracted overlapped sub-volume patches withthe size of 16 × 256 × 128, and fed them to the trained network to get prediction
probability maps. For the overlapped voxels, the final probability maps would
be the average of the probability maps of the overlapped patches, which werethen used to derive the final segmentation results. After that, we conductedmorphological operations to remove isolated small volumes and internal hole.
3.5

valuation

The segmented results were compared with the associated ground truth segmentation. For each test image, we evaluated both the surface distance as well as
the volume overlap measurements of results obtained by different segmentation.
In [9],to compute the average absolute distance (ASD) between the groundtruth IVD surface and the automatically segmented surface, surface meshesfrom binary IVD segmentation were generated first using the Matlab toolbox
Iso2mesh [30]. In contrast, in this study, we adopted the average symmetric surface distance (ASSD) as introduced in [31] to measure the surface distance. More

Deeply Supervised Multi-Scale FCN

147

Table 1. Results on the Test1 dataset from the MICCAI 2015 intervertebral discsegmentation challenge.
Parameters

Mean ± STD
91.4 ± 0.5
84.2 ± 1.0
94.2 ± 2.5
88.9 ± 2.3
0.44 ± 0.055

Dice overlap coefficients (%)
Jaccard (%)
Precision (%)
Recall (%)
ASSD (mm)STD - standard deviation
ASSD - average symmetric surface distancespecifically, ASSD is given in millimeters and based on the surface voxels (instead of surface meshes as in [9]) of two segmentation A and B. Surface voxels
are defined by having at least one on-object voxel within their 26-neighborhood.
For each surface voxel of A, the Euclidean distance to the closest surface voxelof B is calculated using the approximate nearest neighbor technique [32] andstored. The same process is then applied to surface voxels of B to A in orderto symmetry. The ASSD is then defined as the average of all stored distances,which is zero for a perfect segmentation.
Given two binary segmentations of a test image, we compute following volume
overlap measurements including Dice overlap coefficient [33], Jaccard coefficient[33], precision and recall.
3.6

Results

Tab. 1 shows the results of our method when evaluated on the Test1 dataset of
the MICCAI 2015 IVD segmentation challenge and Tab. 2 shows the results ofour method when evaluated on the Test2 dataset of the MICCAI 2015 IVD segmentation challenge. A mean Dice overlap coefficient of 92.0% and a mean ASSD
of 0.41 mm were achieved by our method. Furthermore, slightly better results
were obtained when our method was evaluated on the Test2 dataset than when
Table 2. Results on the Test2 dataset from the MICCAI 2015 intervertebral discsegmentation challenge.
Parameters

Mean ± STD

Dice overlap coefficients (%)
Jaccard (%)
Precision (%)
Recall (%)
ASSD (mm)STD - standard deviation
ASSD - average symmetric surface distance

92.6 ± 1.1
86.4 ± 2.0
93.8 ± 1.5
91.4 ± 1.6
0.38 ± 0.045

148Zeng and Zheng

Fig. 2. Qualitative comparison of the results achieved by our method on the Test1
dataset (top two rows) and ground truth segmentation (bottom two rows). For eachcase, two slices are shown.

our method was evaluated on Test1 dataset. Without using any time-consuming
registration step or incorporating any advanced shape prior, our method achieved
results that were better than those achieved by the state-of-the-art methods [9].
For example, the best segmentation method in the MICCAI 2015 IVD segmentation challenge was the on submitted by Korez et al. [34] where a mean Diceoverlap coefficient of 91.8% was reported. Fig. 2 shows examples of automatic
segmentation achieved by our method on the Test1 dataset and Fig. 3 shows
examples of segmentation achieved by our method on the Test2 dataset.
Implemented with Python using TensorFlow framework, our network took
about 40 seconds to test one volumetric MR image with size of 40 × 300 × 300.

4Conclusion

In this paper, we proposed to use a deeply supervised multi-scale fully convolutional network to solve the challenging IVD segmentation problem. The presentDeeply Supervised Multi-Scale FCN

149

Fig. 3. Qualitative comparison of the results achieved by our method on the Test2
dataset (top two rows) and ground truth segmentation (bottom two rows). For eachcase, two slices are shown.

method was evaluated on the MICCAI 2015 IVD segmentation challenge datasetsand the results achieved by the present method were better than those achievedby the state-of-the-art methods.
In comparison with 3DFCN as introduced by Chen et al. [23], where they incorporate neither skip connection nor multi-scale deep supervision, our method
achieved much better segmentation results. More specifically, evaluated on theTest1 and Test2 datasets of the MICCAI 2015 IVD segmentation challenge, their
method achieved a mean Dice overlap coefficient of 88.4% and 89.0%, respectively. In contrast, evaluated on the same two datasets, our method achieveda mean Dice overlap coefficient of 91.4% and 92.6%, respectively. The results
demonstrated that the incorporation of skip connections and the multi-scale
deep supervision, when combined with partial transfer learning, did improve theperformance of a 3D FCN.

150

Zeng and Zheng

References
1. Modic, M., Ross, J.: Lumbar degenerative disk disease. Radiology 245(1), 43–61(2007)
2. Parizel, P., Van Goethem, J., Van den Hauwe, Voormolen, L.: Degenerative discdisease. In: Van Goethem, J., et al. (eds.) Spinal Imaging, pp. 127–156. Medical
Radiology, Springer (2007)
3. An, H., Anderson, P., Haughton, V., Iatridis, J., Kang, J., Lotz, J., Natarajan, R.,
Oegema, T.J., Roughley, P., Setton, L., Urban, J., Videman, T., Andersson, G.,
Weinstein, J.: Introduction: disc degeneration: summary. Spine 29(23), 2677–2678(2004)
4. Chevrefils, C., Cheriet, F., Aubin, C., Grimard, G.: Texture analysis for automatic
segmentation of intervertebral disks of scoliotic spines from MR images. IEEE
Trans. Inf. Technol. Biomed. 13(4), 608–620 (2009)
5. Michopoulou, S., Costaridou, L., Panagiotopoulos, E., Speller, R., Panayiotakis,
G., Todd-Pokropek, A.: Atlas-based segmentation of degenerated lumbar intervertebral discs from MR images of the spine. IEEE Trans. Biomed. Eng. 56(9),2225–2231 (2009)
6. Ayed, I., Punithakumar, K., Garvin, G., Romano, W., Li, S.: Graph cuts withinvariant object-interaction priors: application to intervertebral disc segmentation.
In: Székely, G., Hahn, H. (eds.) Proc. 22nd International Conference on Information Processing in Medical Imaging – IPMI 2011, LNCS, vol. 6801, pp. 221–232.
Springer (2011)
7. Neubert, A., Fripp, J., Engstrom, C., Schwarz, R., Lauer, L., Salvado, O., Crozier,
S.: Automated detection, 3D segmentation and analysis of high resolution spine
MR images using statistical shape models. Phys. Med. Biol. 57(24), 8457–8376(2012)
8. Law, M., Tay, K., Leung, A., Garvin, G., Li, S.: Intervertebral disc segmentation in
MR images using anisotropic oriented flux. Med. Image Anal. 17(1), 43–61 (2013)
9. Zheng, G., Chu, C., Belavý, D., Ibragimov, B., Korez, R., Vrtovec, T., Hutt, H.,
Everson, R., Meakin, J., Andrade, I., Glocker, B., Chen, H., Dou, Q., Heng, P.,
Wang, C., Forsberg, D., Neubert, A., Fripp, J., Urschler, M., Stern, D., Wimmer,
M., Novikov, A., Cheng, H., Armbrecht, G., Felsenberg, D., Li, S.: Evaluation andcomparison of 3D intervertebral disc localization and segmentation methods for3D T2 MR data: a grand challenge. Med. Image Anal. 35, 327–344 (2017)
10. Zhan, Y., Maneesh, D., Harder, M., Zhou, X.: Robust MR spine detection using hierarchical learning and local articulated model. In: Ayache, N., et al. (eds.)
Proc. 15th International Conference on Medical Image Computing and ComputerAssisted Intervention – MICCAI 2012, LNCS, vol. 7510, pp. 141–148. Springer(2012)
11. Kelm, M., Wels, M., Zhou, S., Seifert, S., Suehling, M., Zheng, Y., Comaniciu, D.:
Spine detection in CT and MR using iterated marginal space learning. Med. Image
Anal. 17(8), 1283–1292 (2013)
12. Chen, C., Belavy, D., Yu, W., Chu, C., Armbrecht, G., Bansmann, M., Felsenberg,D., Zheng, G.: Localization and segmentation of 3D intervertebral discs in MR
images by data driven estimation. IEEE Trans. Med. Imaging 34(8), 1719–1729(2015)
13. Wang, Z., Zhen, X., Tay, K., Osman, S., Romano, W., Li, S.: Regression segmentation for M3 spinal images. IEEE Trans. Med. Imaging 34(8), 1640–1648 (2015)

Deeply Supervised Multi-Scale FCN

151

14. Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1),
1–127 (2009)
15. Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: Pereira, F., et al. (eds.) Proc. Neural Information
Processing Systems – NIPS 2012, vol. 25, pp. 1097–1105. NIPS (2012)
16. Prasoon, A., Petersen, K., Igel, C., Lauze, F., Dam, E., Nielsen, M.: Deep featurelearning for knee cartilage segmentation using a triplanar convolutional neuralnetwork. In: Mori, K., et al. (eds.) Proc. 16th International Conference on Medical
Image Computing and Computer-Assisted Intervention – MICCAI 2013, LNCS,
vol. 8150, pp. 246–253. Springer (2013)
17. Roth, H., Lu, L., Seff, A., Cherry, K., Hoffman, J., Wang, S., Liu, J., Turkbey, E.,
Summers, R.: A new 2.5D representation for lymph node detection using random
sets of deep convolutional neural network observations. In: Golland, P., et al. (eds.)
Proc. 17th International Conference on Medical Image Computing and ComputerAssisted Intervention – MICCAI 2014, LNCS, vol. 8673, pp. 520–527. Springer(2014)
18. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic
segmentation. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition – CVPR 2015. pp. 3431–3440 (2015)
19. Roth, H., Yao, J., Lu, L., Stieger, J., Burns, J., Summers, R.: Detection of sclerotic spine metastases via random aggregation of deep convolutional neural networkclassifications. In: Yao, J., et al. (eds.) Proc. 2nd MICCAI Workshop on Computational Methods and Clinical Applications for Spine Imaging – CSI 2014, LNCVB,
vol. 20, pp. 3–12. Springer (2015)
20. Çiçek, O., Abdulkadir, A., Lienkamp, S., Brox, T., Ronneberger, O.: 3D U-Net:
learning dense volumetric segmentation from sparse annotation. In: Ourselin, S.,et al. (eds.) Proc. 19th International Conference on Medical Image Computing andComputer-Assisted Intervention – MICCAI 2016, LNCS, vol. 9901, pp. 424–432.
Springer (2016)
21. Milletari, F., Navab, N., Ahmadi, S.A.: V-Net: fully convolutional neural networks
for volumetric medical image segmentation. In: Proc. 4th International Conference
on 3D Vision – 3DV 2016. pp. 565–571. IEEE (2016)
22. Dou, Q., Yu, L., Chen, H., Jin, Y., Yang, X., Qin, J., Heng, P.: 3D deeply supervised
network for automated segmentation of volumetric medical images. Med. Image
Anal. (2017)23. Chen, H., Dou, Q., Wang, X., Qin, J., Cheng, J., Heng, P.A.: 3D fully convolutional
networks for intervertebral disc localization and segmentation. In: Zheng, G., et al.
(eds.) Proc. 7th International Conference on Medical Imaging and Augmented
Reality – MIAR 2016, LNCS, vol. 9805, pp. 375–382. Springer (2016)24. Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., Pal, C.: The importanceof skip connections in biomedical image segmentation. In: Carneiro, G., et al. (eds.)
Proc. 1st MICCAI International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis – LABELS 2016 and 2nd MICCAI International Workshop on Deep Learning in Medical Image Analysis – DLMIA 2016,
LNCS, vol. 10008, pp. 179–187. Springer (2016)25. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.
In: Proc. IEEE Conference on Computer Vision and Pattern Recognition – CVPR
2016. pp. 770–778. IEEE (2016)26. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale
image recognition. arXiv p. arXiv:1409.1556 (2014)

152

Zeng and Zheng

27. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by
reducing. In: Proc. 32nd International Conference on Machine Learning – ICML
2015, vol. 37, pp. 448–456. PLMR (2015)28. Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features indeep neural networks? In: Ghahramani, Z., et al. (eds.) Proc. Advances in Neural
Information Processing Systems – NIPS 2014, pp. 3320–3328. MIT Press (2014)
29. Zeng, G., Yang, X., Li, J., Yu, L., Heng, P., Zheng, G.: 3D U-Net with multi-level
deep supervision:fully automatic segmentation of proximal femur in 3D MR images.
In: Submitted to 8th MICCAI International Workshop on Machine Learning in
Medical Imaging – MLMI 2017 (2017)30. Fang, Q., Boas, D.: Tetrahedral mesh generation from volumetric binary and grayscale images. In: Proc. 6th IEEE International Symposium on Biomedical Imaging– ISBI 2009. pp. 1142–1145. IEEE (2009)
31. Heimann, T., van Ginneken, B., Styner, M., Arzhaeva, Y., Aurich, V., Bauer, C.,
Beck, A., Becker, C., Beichel, R., Bekes, G., Bello, F., Binnig, G., Bischof, H.,
Bornik, A., Cashman, P., Chi, Y., Cordova, A., Dawant, B., Fidrich, M., Furst,
J., Furukawa, D., Grenacher, L., Hornegger, J., Kainmüller, D., Kitney, R., Kobatake, H., Lamecker, H., Lange, T., Lee, J., Lennon, B., Li, R., Li, S., Meinzer,
H., Nemeth, G., Raicu, D., Rau, A., van Rikxoort, E., Rousson, M., Rusko, L.,
Saddi, K., Schmidt, G., Seghers, D., Shimizu, A., Slagmolen, P., Sorantin, E., Soza,
G., Susomboon, R., Waite, J., Wimmer, A., Wolf, I.: Comparison and evaluationof methods for liver segmentation from CT datasets. IEEE Trans. Med. Imaging
28(8), 1251–1265 (2009)
32. Arya, S., Mount, D., Netanyahu, N., Silverman, R., Wu, A.: An optimal algorithm
for approximate nearest neighbor searching. J. ACM 45(6), 891–923 (1998)
33. Karasawa, K., Oda, M., Kitasaka, T., Misawa, K., Fujiwara, M., Chu, C., Zheng,G., Rueckert, D., Mori, K.: Multi-atlas pancreas segmentation: atlas selection basedon vessel structure. Med. Image Anal. 39, 18–28 (2017)
34. Korez, R., Ibragimov, B., Likar, B., Pernuš, F., Vrtovec, T.: Deformable modelbased segmentation of intervertebral discs from MR spine images by using the SSC
descriptor. In: Vrtovec, T., et al. (eds.) Proc. 3rd MICCAI International Workshop
and Challenge on Computational Methods and Clinical Applications for Spine
Imaging – CSI 2015, LNCS, vol. 9402, pp. 117–124. Springer (2016)

Author Index

Al Arif, S. M. Masudur Rahman, 12

Abramson, Richard G., 25
Adams, Judith E., 128

Bagci, Ulas
Program Committee, vii
Besler, Bryce A., 116
Blum, Alain, 35
Boyd, Steven K., 116
Bromiley, Paul A., 128
Program Committee, vii

Cai, Weidong
Program Committee, vii
Cai, Yunliang
Program Committee, vii
Cameron, John, 35
Cardoso, M. Jorge, 69
Carr, J. Jeffrey, 25
Chowdhury, Ananda S.
Program Committee, vii
Chung, Beom Sun, 82
Conway, Chris, 35
Cootes, Timothy F., 128

Damopoulos, Dimitrios, 56

Ebner, Michael, 69
Elcock, Dave, 35

Forkert, Nils D., 116
Forsberg, Daniel
Program Committee, vii
Fotiadou, Anastasia, 69
Frangi, Alejandro F.
Organizing Committee, vii
Proceedings Editors, viii
Fukuda, Norio, 82

Glocker, Ben, 56
Organizing Committee, vii
Proceedings Editors, viii
Hacihaliloglu, Ilker, 1Hart, Alister, 69
Hartley, Katherine G., 25
He, Huiguang
Program Committee, vii
Henckel, Johann, 69
Hu, Peijun, 25
Huo, Yuankai, 25
Ibragimov, Bulat
Program Committee, vii
Kadoury, Samuel, 46
Kariki, Eleni P., 128
Kirschke, Jan S., 104
Knapp, Karen, 12
Kong, Dexing, 25
Korez, Robert, 46, 91
Kukačka, Jan, 104
Landman, Bennett A., 25
Likar, Boštjan, 91
Liu, Jianfei
Program Committee, vii
Lorenz, Cristian
Program Committee, vii
Menze, Bjoern H., 104
Michalski, Andrew S., 116
Miyamoto, Kohei, 82
Modat, Marc, 69
Murray, Andrew, 35
O’Donnell, Lauren J., 82
Ollivier, Axel, 82
Otake, Yoshito, 82
Ourselin, Sébastien, 69

154
Parent, Stefan, 46
Park, Jin Seo, 82
Pernuš, Franjo, 91
Petković, Uroš, 46Pezold, Simon
Program Committee, vii
Plakas, Costas, 35
Ranzini, Marta B.M., 69
Reynolds, Steven, 35
Sato, Yoshinobu, 82
Sekuboyina, Anjany, 104
Slabaugh, Greg, 12Program Committee, vii
Stroud, Tyler, 35
Sugano, Nobuhiko, 82
Štern, Darko
Program Committee, vii
Takao, Masaki, 82
Tan, Sovira
Program Committee, vii
Teixeira, Pedro Augusto Gondim, 35
Tobon-Gomez, Catalina, 35

Ungi, Tamas
Program Committee, vii

Valentinitsch, Alexander, 104
Vercauteren, Tom, 69
Vrtovec, Tomaž, 46, 91
Organizing Committee, vii
Proceedings Editors, viii

Westin, Carl-Fredrik, 82
Wyeth, Daniel, 35

Yao, Jianhua
Organizing Committee, vii
Proceedings Editors, viii
Yokota, Futoshi, 82

Zeng, Guodong, 141
Zhan, Yiqiang
Program Committee, vii
Zheng, Guoyan, 56, 141
Organizing Committee, vii
Proceedings Editors, viii

c 2017
5th MICCAI Workshop on Computational Methods andClinical Applications in Musculoskeletal Imaging
MICCAI-MSKI2017
https://mski2017.wordpress.com