Computer Assisted Retrieval of Biomedical Image Features from Spine
X-rays: Progress and Prospects

L. Rodney Long, G. R. Thoma

National Library of Medicine, Bethesda, MD 20894

http://archive.nlm.nih.gov/staff/long.php

**Abstract**

Image indexing for biomedical content is a prohibitively expensive task
if done manually. This leads to the demand for effective automated or
computer assisted indexing methods. We are doing research on this
problem for a set of digitized x-ray films from the second National
Health and Nutrition Examination Survey (NHANES II). The 10,000 cervical
spine and 7,000 lumbar spine images from this survey are retrievable
through a resource we have developed, the Web-based Medical Information
Retrieval System (WebMIRS). Using WebMIRS, the user may retrieve x-rays
based on text data, such as demographics, health questionnaire
responses, or physican's examination results. Two National Institutes of
Health workshops have identified visual features of the images
specifically related to osteoarthritis, but the images have never been
indexed for these features, which include anterior osteophytes, disc
space narrowing, and subluxation. We are investigating the methods by
which the indexing and retrieval of the images using these particular
features, may be achieved, in a validated manner acceptable to the
biomedical community. In this paper we review our work to date, plus the
work of collaborating researchers, and present some possible outcomes
that appear to be realistic

**1. Introduction**

## The image data for our research consists of 10,000 cervical spine and 7,000 lumbar spine images digitized from films collected in the second National Health and Nutrition Examination Survey (NHANES II). The capability of retrieving x-ray images by important biomedical features has been a long-standing research goal within the biomedical image processing community. No comprehensive approach has yet been applied to a large image collection of x-rays in any practical system, to the best of our knowledge. In this paper we present a brief review of the relevant work that we and other researchers have done, and outline the remaining problems and potential solutions.

The creation of a database of x-ray images that are retrievable by image
content requires the development of practical methods for both indexing
and retrieval. To date we have concentrated on indexing. As part of our
work, we have developed the Web-based Medical Information Retrieval
System (WebMIRS) \[1-2\], a database system that allows retrieval of the
images by associated text, such as demographics, health questionnaire
responses, or physician's examination data. The images are viewable
through WebMIRS at one-fourth their original spatial resolution. The
full resolution images are available through a separate FTP site also
operated by our branch of the National Library of Medicine \[3\]. In
addition, we have developed a Digital Atlas of the Cervical and Lumbar
Spine \[4\] that contains validated examples of the biomedical image
features that could be used as indices to the images.

2.  **Specific goals for the x-ray database**

The question of what biomedical image information is available and
significant in our images was addressed by two workshops \[5\] held
under the sponsorship of the National Institute of Arthritis and
Musculoskeletal and Skin Diseases (NIAMS). These workshops concluded
that the following features could be determined from the images in
repeatable fashion by human observers: for cervical spine images,
anterior osteophytes, disc space narrowing, and subluxation; for lumbar
spine images, anterior osteophytes, disc space narrowing, and
spondylolisthesis. We have taken these recommendations as defining the
basic features to be indexed in our image set. Along with these basic
biomedical features, we have the goal of indexing and retrieving the
images by basic geometric features, such as vertebra dimensions and
spacing distances. Such measures are used in vertebral morphometry,
where the ratio of anterior to posterior vertebral height may be used as
a measure of fracture \[6\].

3.  **Accomplishments and published research in the field**

    1.  **Creation of reference data for vertebral structures**

![](media/image1.png){width="2.4895833333333335in"
height="1.8645833333333333in"}

We have acquired a set of coordinate data for the cervical and lumbar
spine vertebra that includes the key boundary points collected in
vertebral morphometry. The collected points are illustrated in Figure 1.
The data collected includes the standard six points of vertebral
morphometry (points 1-6), plus the anterior midpoint, and points marking
the extremities of any anterior osteophytes that are present. Data was
also collected denoting the presence of fused vertebrae in the lumbar
spine. The data collection was carried out under supervision of a
board-certified radiologist with expertise in bone radiology, on 550 of
the cervical and lumbar spine images. The data contains a large number
of vertebrae showing the presence of anterior osteophytes, one of the
conditions of interest for indexing, and is therefore a potential source
of training data for algorithms to classify vertebrae according to the
presence or absence of this biomedical feature. The significance of this
data is that it provides a reference for segmentation algorithm
development and validation, and for training of classification
algorithms, at least for the anterior osteophyte classification. We have
made the collected data available for research use through the WebMIRS
system and on our FTP site.

2.  **High-level analysis and basic orientation in the images**

Since, as we note in Section 3.3, the segmentation algorithms operate in
a local region of the image assumed a priori to contain the structures
of interest, it is necessary to determine this local region of interest
(ROI) either interactively or with a preliminary high-level analysis of
the image. Recent work toward this goal was reported by Zamora \[7\],
who used a method based on line integrals of image grayscale to
determine approximate spine axis locations on a test set of 40 cervical
spine images. Using the reference data of Section 3.1 to independently
calculate a "truth" spine axis, Zamora reported an orientation error in
his algorithm of less than 15 degrees, for 34 of the cases. Zamora
extended his method to estimate a rectangle conservatively containing
the spine ROI, and used the Section 3.1 reference data to evaluate
performance. For vertebrae C2-C6, the Zamora rectangle was found to
contain the vertebral boundaries in over 90% of the cases.

We have previously shown \[8-9\] that basic landmark information appears
to be preserved in the cervical spine images after heavy smearing and
subsampling. In a test case of 48 cervical spine images, we successfully
derived landmarks for skull, shoulder, and back-of-skull background in
46 cases. For the 46 successful cases, we further used the
skull/shoulder landmarks to find the spine ROI by detecting
characteristic curves assumed to lie in the spine ROI. (One of these is
the curve of brightest grayscale along the length of the spine.) We
estimated this curve for the 46 cases, then used our curve estimate in
turn to estimate the spine orientation. We compared the orientations
computed by this algorithm to the spine orientations computed using the
data of Section 3.1 and found that in 45 of the cases, the orientation
error was less than 15 degrees.

We have recently done work in direct search for spine region anatomy
without initial low resolution analysis, by direct search of two of the
most consistently prominent cervical spine image features: the
back-of-skull contour, and the spinous process on the C1 vertebra
(SPC1). Implementation of simple raster scan methods for skull boundary
detection, followed by curve analysis of the resulting boundary, have
yielded good initial results: in a test of 136 images, a conservative
bounding box for SPC1 was found in 88% of the cases.

Global analysis of such radiological images for localizing regions of
interest does not seem to be commonly reported in the literature,
although it does exist. For example, Chwialkowki \[10\] reported
"reasonably accurate and reliable" results in analyzing lateral MRI
images of the lower spine for localizing such anatomy as the spinal
canal. He used 26 MR studies with an average of 14 images per study.
Interactive methods of ROI localization appear to predominate, however

3.  **Segmentation of the vertebrae with local segmentation algorithms**

In recent years researchers have published results using various forms
of deformable template approaches to segment vertebrae. Without
exception, these methods rely on user interaction for initialization, or
the researchers report that lack of automated initialization methods is
a signficant performance problem. A fundamental and comprehensive
treatment of the whole field of Active Shape Modeling (ASM) has been
provided by Cootes \[11\]. A semi-automated implementation of this
approach has been achieved for the segmentation of lumbar spine dual
x-ray absorptiometry (DEXA) images \[12\]; here, the user manually
identifies two "anchor points" for placing a template; the template then
deforms by ASM, maintaining invariance of the anchor points, which are
placed at the top and bottom of a column of vertebrae. Gardner \[13-14\]
developed a semi-automated system, based on active contour (snake)
modeling of the vertebrae, which operated on digitized lumbar spine
x-ray film. In this system, points on the vertebral boundaries are
specified by the user, with assistance from the system in point
placement. The selected boundary points then become constraints on the
active contour that is automatically fit to the vertebra boundary. This
process is carried out vertebra by vertebra. The same author has pointed
out \[15\] the potential problems in taking dimensional measurements
from vertebrae on x-rays, due to the projective nature of the imaging
modality, which results in overlapping edges and concealed boundaries.

In \[16\] we reported work using ASM models of cervical spine vertebrae
to segment C2/C3 vertebrae in a test set of 40 images. The segmentation
performance was evaluated against a set of "truth" boundaries derived
manually from the images, and mean point-to-point boundary errors were
calculated. A number of erroneous results were observed, including
convergence to tissue boundaries near the vertebrae, and convergence to
lines on boundary regions interior to vertebrae. For 16 of the cases,
the results showed a mean point-to-point error of less than 1/10 inch.
Strong sensitivity of convergence to initial template positioning was
observed in some cases. Work done independently by Sari-Sarraf \[17\]
using similar techniques and tools also appeared to give successful
results to a first-order level in a significant number of cases.

4.  **Identification (classification) of anatomy in the images**

To date the only anatomical labeling we have investigated is gross level
labeling of the shoulder and skull, although the spine curves we have
calculated \[9\] admit the possibility of labeling the general spine
region. Labeling of the individual vertebrae will follow vertebral
segmentation and is expected to use skull anatomy (because of its
relative ease and robustness of detection) as a beginning reference
point.

5.  **Indexing (classification) by biomedical feature**

The final indexing and classification step for our images requires
taking the segmented, labeled images along with expert training data,
and dividing the labeled structures into classes of normal or abnormal
for the biomedical conditions of interest. It also generates all of the
geometry measures desired from the data. Work toward this step that
applies radius of curvature criteria to segmented vertebrae boundaries
has been reported by Stanley \[18\].

4.  **Problems**

The remaining technical problems in building a practical system for the
retrieval of data by image content from this set of x-ray images include
the need to solve the segmentation problem with methods that are robust,
of fine granularity, and biomedically validated. In spite of the
progress made, the test sets used to date are too small relative to the
image collection. In addition, most of the work has focussed on the
basic task of getting algorithms to produce boundaries that appear
reasonable, in a first order sense, with little attention paid to the
resolution quality of the segmentation achieved. This issue is closely
related to the lack of critical review of the results by biomedical
experts in the area. Each of these issues must be addressed for such
systems to become accepted tools in the biomedical research community.

It should be noted that both in our work and the work of Sari-Sarraf a
number of problems were observed, and technical issues that require
resolution were raised. Among the most outstanding of these are the need
for a good method for initializing ASM to segment the vertebrae , the
need to investigate the possiblity of improved modeling of the image
grayscale distribution in the neighborhood of vertebrae boundary points
(perhaps with a mixed Gaussian pdf), and the need to understand
nonconvergence of the algorithm for certain images even when the
template initial conditions are set near known truth. A final
significant outstanding problem is automated failure detection for the
cases of nonconvergence or convergence to erroneous solutions.

5.  **Prospects and conclusion**

As judged by the work of Gardner and Smyth, the prospects for achieving
an interactive system for basic-level segmenting of our images under
user guidance appear quite good, although it is not certain that the
quality of segmentation achievable will allow classification of the
biomedical features of interest with an accuracy comparable to a human
expert. This remains to be determined experimentally. The possibility of
fully automating the segmentation and indexing remains a more
problematic and distant goal; some of the imposing problems appear
tractable with time and effort, such as automatic detection of "problem"
images by histogram analysis, or initialization of template positioning
by grayscale line integral analysis or low resolution image analysis;
but other problems, such as automatic failure detection, appear more
fundamental, and no clear approach has been proposed.

**6. References**

1.  Long LR, Pillemer SR, Lawrence RC, Goh G-H, Neve L, Thoma GR. World
    Wide Web platform-independent access to biomedical text/image
    databases. *Proceedings of SPIE Medical Imaging 1998: PACS Design
    and Evaluation: Engineering and Clinical Issues,* SPIE Vol. 3339,
    San Diego, CA, February 21-26, 1998, pp. 52-63.

2.  Long LR, Pillemer SR, Lawrence RC, Goh G-H, Neve L, Thoma GR.
    WebMIRS: Web-based Medical Information Retrieval System.
    *Proceedings of SPIE Storage and Retrieval for Image and Video
    Databases VI*, SPIE vol. 3312, San Jose, CA, January 24-30, 1998,
    pp. 392-403.

3.  See the Web site of the NLM Communications Engineering Branch:
    [http://archive.nlm.nih.gov](http://archive.nlm.nih.gov/)

4.  Long LR, Pillemer S, Goh G-H, Berman LE, Neve L,Thoma GR, Premkumar
    A, Ostchega Y, Lawrence R, Altman RD, Lane NE, Scott WW, Jr. A
    digital atlas for spinal x-rays. *Proceedings of SPIE Medical
    Imaging 1997: PACS Design and Evaluation: Engineering and Clinical
    Issues*, SPIE vol. 3035, Newport Beach, CA, February 22-28, 1997,
    pp. 586-594.

5.  Digitized Radiographic Images: Challenges and Opportunities. Chaired
    by: Stanley R. Pillemer, M.D. June 2-3, 1993, Bethesda, MD,
    sponsored by the National Institute of Arthristis and
    Musculoskeletal and Skin Diseases (NIAMS), the National Center for
    Health Statistics, and the National Library of Medicine. Digitized
    Radiographic Images. Chaired by Stanley R. Pillemer, M.D., Reva C.
    Lawrence, M.P.H. May 9, 1995, Bethesda, MD, sponsored by NIAMS.

6.  Hedlund LR, Gallagher JC. Vertebral morphometry in diagnosis of
    spinal fractures. *Bone and Mineral*, Vol. 5, 1988, pp. 59-67.

7.  Zamora G, Sari-Sarraf H, Mitra S. Estimation of orientation of
    cervical vertebrae for segmentation with active shape models.
    *Proceedings of SPIE Medical Imaging 2001: Image Processing*, San
    Diego, CA, February 17-23, 2001 (forthcoming).

8.  Long LR, Thoma GR. Feature indexing in a database of digitized
    x-rays. *Proceedings of SPIE Electronic Imaging 2001: Storage and
    Retrieval for Media Databases*. Vol. 4315, San Jose, CA, January
    21-26, 2001, pp. 393-403.

9.  Long LR, Thoma GR. Identification and classification of spine
    vertebrae by automated methods. *Proceedings of SPIE Medical Imaging
    2001: Image Processing*. Vol. 4322, San Diego, CA, February 17-22,
    2001.

10. Chwialkowski MP, Shile PE, Pfeifer D, Parkey RW, Peshock RM.
    Automated localization and identification of lower spinal anatomy in
    magnetic resonance images. *Computers and Biomedical Research*, vol.
    24, 1991, pp. 99-117.

11. Cootes TF, Taylor CJ. Statistical models of appearance for computer
    vision. Wolfson Image Analysis Unit, Imaging Science and Biomedical
    Engineering, University of Manchester, September 13, 1999. Draft
    report, available at
    [http://www.wiau.man.ac.uk](http://www.wiau.man.ac.uk/).

12. Smyth PP, Taylor CJ, Adams JE. Vertebral shape: automatic
    measurements with active shape models. *Radiology*, May 1999, vol.
    211, no. 2, pp. 571-578.

13. Gardner JC, Heyano SL, Yaffe LG, von Ingersleben G, Chesnut CH III.
    A semi-automated computerized system for fracture assessment of
    spinal x-ray films. *Proceedings of SPIE Medical Imaging 1996: Image
    Processing*, SPIE vol. 2710, February 12-15, 1996, Newport Beach,
    CA, pp. 996-1008.

14. Gardner JC, von Ingersleben G, Heyano SL, Chesnut III, CH. An
    interactive tutorial-based training technique for vertebral
    morphometry. *Osteoporosis International*, vol. 12, 2001, pp. 63-70.

15. Gardner JC, Yaffe LG, Johansen JM, von Ingersleben G, Chesnut
    CH III. Problems with six point vertebral morphometry. *Proceedings
    of SPIE Medical Imaging 1998: Image Processing*, SPIE vol. 3338, San
    Diego, CA, February 23-26, 1998, pp. 107-117.

16. Long LR, Thoma GR. Use of shape models to search digitized spine
    x-rays. *Proceedings of IEEE Computer Based Medical Systems 2000*,
    Houston, TX, June 23-24, 2000, pp. 255-260.

17. Sari-Sarraf H, Mitra S, Zamora G, Tezmol A. Customized active shape
    models for segmentation of cervical and lumbar spine vertebrae,
    Texas Tech University College of Electrical Engineering technical
    report, available at <http://www.cvial.ttu.edu/~sarraf>

18. Stanley RJ, Long R. A radius of curvature approach to cervical spine
    vertebra image analysis*. Proceedings of the 38^th^ Annual Rocky
    Mountain Bioengineering Symposium*, Copper Mountain, Colorado, April
    20-22, 2001, pp. 385-90.