The role of image modality and visual characteristics in archiving biomedical images

Imaging in biomedicine has seen an explosive growth in recent decades. Clinicians can offer better diagnosis, and scientists and the lay public often better understand complex biomedical concepts through visual means. Typically, patient identifiers are used for archiving and indexing images’ metadata in the clinical setting, and bibliographic citation data are used in library collections, such as the open access biomedical research articles from the U.S. National Library of Medicine’s (NLM) PubMed Central® (PMC) repository. Automatically detected image modality and visual image characteristics can offer valuable addition to these traditional textual metadata for archiving and indexing visual material. Example image modalities include computerized tomography (CT), X-ray, magnetic resonance imaging (MRI), ultrasound, photographs, illustrations, charts, graphs, and sketches. The open-access biomedical literature in PMC is a source for OpenI (pronounced Open “eye”), a multimodal biomedical information retrieval system developed at the NLM that enables users to search for and retrieve relevant images and text. Our methods were evaluated in an international benchmarking forum in which we achieve a classification accuracy exceeding 90% at the highest level of a hierarchically organized image modality taxonomy, and 63.2% at the leaf level. Introduction The use of visual material in research publications has seen an explosive growth. Visual material is valuable not only in succinctly summarizing scientific and statistical results, but also expressing with high clarity what may take pages to describe. In particular in biomedicine, clinicians can offer better diagnosis, and scientists and the lay public are often able to better understand complex biomedical concepts through visual means. A study showed that figures are often among the first parts of articles that are reviewed by readers [1]. Therefore, it is of great significance if retrieval systems can provide the capability of searching for figures effectively. We recently conducted an analysis of the available open access biomedical research articles from the U.S. National Library of Medicine’s (NLM) PubMed Central® (PMC) repository and found that currently 1.8 million images are available as a part of the freely downloadable data subset at the rate of about 3.5 figures per article. This is but a fraction of all images in the published biomedical literature. Analysis of the collection also reveals a sampling of the types (modalities) of images. These include those 1 http://www.ncbi. nlm.nih. gov/pmc/ acquired through transmissive imaging techniques such as computerized tomography (CT), X-ray; nuclear imaging techniques, such as magnetic resonance imaging (MRI), positron emission tomography (PET); reflective imaging techniques, such as ultrasound and photography; and for scientific articles in particular, illustrative images, which includes charts, graphs, sketches, and so on. In order to archive and index biomedical images, a typical method has been to use available metadata such as the patient identifier in the clinical setting, or, bibliographic citation information such as title, authors, abstract, keywords, journal title, publication data, etc., in a library or archive. Recently some indexing techniques have also started including text from figure captions as metadata. Examples include Yale Image Finder and The American Roentgen Ray Society’s (ARRS) Goldminer® system [2]. While these techniques are valuable they do not capture the visual characteristics of the image or its modality, which can be key distinguishing characteristics. Image modality and visual image characteristics can offer valuable addition to traditional metadata for archiving and indexing visual material (along with text material) in the biomedical literature. In previous work we showed that visual content of the images combined with the automatically computed image modality, figure caption text, and other traditional metadata can significantly improve indexing and retrieval of biomedical material over text retrieval alone [3]. Our methods for automatic multimodal image modality detection were evaluated in an international benchmarking forum called ImageCLEF and achieved a high image modality classification accuracy and ranked within the submissions from the top three groups from among over 20 groups consisting of teams from academia and industry. In particular, we participated in ImageCLEFmed, the medical retrieval track of the benchmarking forum [4]. ImageCLEF is part of CLEF (the Cross Language Evaluation Forum) and has been organized yearly since 2003. It focuses on medical image annotation and retrieval and aims to provide support and resources for the evaluation of visual information retrieval systems. ImageCLEFmed 2012 consisted of three subtasks: modality classification, ad hoc image retrieval, and case-based article retrieval [5]. The modality classification task aims to evaluate the state of the art in figure classification. The remainder of the article is organized as follows. We describe the modality hierarchy and classification strategy in 2 Goldminer: http://goldminer.arrs. org 3 ImageCLEF: http://www.imageclef. org Methods. In Evaluation we describe the performance of our techniques. In Discussion we underscore the value of using the image modality as metadata in archiving and indexing. Finally, we conclude and provide next steps in Conclusions. Methods This section describes the image modality hierarchy that is critical in organizing the classification strategy and computing the metadata for archiving the images. It also describes the data set and image features and has other relevant discussion on the topic. Modality The modality of an image can be computed for its appearance, and / or its function. In the broadest terms, image modality is the type of the image. However, when the images are organized, the taxonomy can take different forms. Through our continued participation in ImageCLEFmed, and our analysis of the open access biomedical literature in PMC, we use a modified hierarchical taxonomy based on one proposed for ImageCLEFmed 2012 [6]. The original taxonomy was modified after noting limitations in the benchmark hierarchy. The published taxonomy had a greater focus on image function that made using visual features for describing image content more challenging. Our modifications, instead, focus on image archiving, indexing, and retrieval. Our new hierarchy enables improved automatic image modality detection. Figure 1. Modified image modality taxonomy Our new modality taxonomy, shown in Figure 1, organizes images with a combination of appearance and function. At the highest level the images are separated into Illustration (appearance) and General (diagnostic function) images. Illustration images include statistical figures, such as charts and graphs, diagrams, schematics, and sketches. The sole diagnostic images that have been included in this set are signal and wave images, e.g. electrocardiograms, electroencephalographic images. General images include diagnostic ones that are further subdivided into radiological images, microscopy images, 3D reconstructions, and photos. This classification also recognizes that images appearing in the biomedical literature can be organized in single panel or multipanel figures. Compound or multi-panel figures are detected within each category, and processed to separate the panels into individual images for meaningful classification [7]. These and other single panel images are then classified according to a mix of appearance, function, and acquisition. Features With the availability of the full text of the article including the figure captions, we took a multimodal classification approach that represents each image in the collection using text, visual, and combined features. Textual features are extracted from figure captions, discussion of the figure elsewhere in the article, and also citations listed as references. Visual features were computed from images and other figures. In our classification method, textual and visual features are extracted separately and then combined to achieve better classification performance. Textual feature extraction We represent each image as a structured document of imagerelated text, including the figure caption, discussion of the figure in the full text of the article, and other traditional bibliographic citation information. The structured documents may be indexed and searched with a traditional search engine, such as Lucene, or the underlying term vectors may be retrieved and added to a mixed image feature vector (multimodal, textual, and visual). To extract textual features for images, the terms in a structured document are commonly represented as a `bag of words'. Visual feature extraction From visual content we extract 15 features [8], including. color features such as the Color Layout Descriptor (CLD) of MPEG-7 [9]; edge features such as the Edge Histogram Descriptor (EHD); texture features such as moment-based features, wavelet, and Tamura feature; and features from combining two different types of features such as Color Edge Direction Descriptor (CEDD) and Fuzzy Color Texture Histogram (FCTH) available from the Lucene image retrieval engine (LIRE) library [10]. Each feature is extracted individually from the images and several features are combined into a single feature vector to represent an image. Illustration class specific feature extraction Illustration images in biomedical research articles significantly outnumber General images, as organized in Figure 1. Our analysis of the open access articles in PMC found that over 80% of the figures are illustrations. To increase the accuracy in classifying them, we implemented figure-type specific rule-based visual feature extraction methods. Illustrations frequently contain polygonal structures and overlaid graphic text strings on white background, e.g. flo