Promise and Potential Pitfalls: Re-creating Images or Generating New Images for AI Modeling

HomeRadiology: Artificial IntelligenceVol. 3, No. 4 PreviousNext CommentaryFree AccessPromise and Potential Pitfalls: Re-creating Images or Generating New Images for AI ModelingHeang-Ping Chan Heang-Ping Chan Author AffiliationsFrom the Department of Radiology, University of Michigan, 1500 E Medical Center Dr, Med Inn Building C477, Ann Arbor, MI 48109-5842.Address correspondence to the author (e-mail: [email protected]).Heang-Ping Chan Published Online:Jun 23 2021https://doi.org/10.1148/ryai.2021210102MoreSectionsPDF ToolsImage ViewerAdd to favoritesCiteTrack CitationsPermissionsReprints ShareShare onFacebookTwitterLinked In See also the article by Shu et al in this issue.Heang-Ping Chan, PhD, FAAPM, FInstP, is the Paul L. Carson Collegiate Professor and the director of the CAD-AI Research Laboratory in the department of radiology at the University of Michigan. Her research focuses on computer-aided diagnosis and medical image analysis using machine learning and artificial intelligence (AI) methods to provide clinical decision support and improve patient care. She is also interested in issues associated with implementing computer-aided diagnosis or AI in clinical use.Download as PowerPointOpen in Image Viewer Deep learning (DL) is the state-of-the-art machine learning approach for medical image analysis and computer-aided diagnosis (CAD), following its success in text and speech recognition, face recognition, autonomous vehicles, and games such as chess and Go. DL is a machine learning method that uses a multilayer neural network to automatically learn representation from data and extract pertinent information to make assessments or predict outcomes of a targeted task. In medical imaging, a deep convolutional neural network (DCNN) with supervised learning is the most commonly used DL approach. A DCNN may be considered a highly complex mathematical model with millions of parameters that are adjusted iteratively to extract relevant features from a large number of training samples, as guided by the known outcomes (or labels) of the retrospective clinical cases. The trained parameters carry the learned “knowledge” of how to extract relevant information and predict the outcome. If properly validated, the trained DCNN can be useful as clinical decision support or a second opinion when it is applied to new, unseen cases. Similarly, DL can be trained to model a transformation from an input image to an output image by learning from a large set of input and target output images; for example, to transform an input CT image to an MRI-like image or vice versa, to segment an organ or lesion from the surrounding background, to reduce the noise or artifact from a noisy input image and output a low-noise or artifact-free image, or even to generate new images after learning from a large number of the desired type of images (1,2). The ability of DCNN to model complex processes has stimulated tremendous research and development efforts to exploit its potential in medical imaging applications (3).Mammography has been one of the first areas to adopt DL technology for applications such as lesion detection, diagnosis, and risk prediction (4). Full-field digital mammography (FFDM), alone or with breast tomosynthesis, is currently the main imaging modality for breast cancer screening in the United States. Digital imaging facilitates image processing, including automated breast density quantification and CAD. All FFDMs displayed on the review workstations for radiologists’ interpretation have been processed by computer software to optimize the image quality and are therefore called “for-presentation” or “processed” images. The original digital data directly acquired by the FFDM system, called “for-processing” or “raw” images, are discarded after image processing by most breast imaging facilities to conserve data storage spaces. The image processing methods are proprietary and designed by the individual vendors for their FFDMs. The processed images cannot be easily converted back to raw images. The processed images have limitations for use in many image analysis applications. For example, image analysis software such as breast density quantification that makes use of imaging physics principles is applicable only to the raw image. Image analysis methods or predictive models developed with processed images may not be adaptable to other types of FFDM images or applicable to older generations of processed images from the same vendor because the processed image features are vendor dependent, and the processing algorithms often evolve over time even from the same FFDM vendor. Furthermore, to take advantage of fast-moving artificial intelligence (AI) technology, big retrospective imaging data are needed to train updated DL models. The loss of the original raw images creates an additional barrier in developing advanced CAD-AI models for assisting in breast cancer diagnosis and treatment. If a robust method can be developed to re-create a raw image from the processed image, it can potentially mitigate some of these problems.In their study, Shu et al developed a DL-based approach to re-create a raw image from a processed image (5). They trained a DCNN structure called U-Net with a processed image as input and the corresponding original raw image as the target output to obtain a model that could transform a processed image into an estimated raw image. The performance of the model was validated by comparison of the original and the estimated raw images in a set of test cases in terms of four image similarity measures, breast density quantification by a commercial software program, and 29 texture features. The DL approach was found to be very promising for breast density estimation as indicated by the high similarity measures, high Pearson correlation, and almost perfect Cohen κ agreement between the volumetric breast density estimation using the original and the estimated raw images.On the other hand, the study showed that the correlation coefficients between the texture features extracted from the estimated raw images and those from the original raw images spread over a wide range. Among the texture features analyzed, some were highly linearly correlated, some were moderately linearly correlated, and some were poorly correlated (Table 4 and Fig E2 [supplement]). For four of the seven run-length texture features, the correlations between the processed and the original raw images were much higher (> 0.8 vs ∼ −0.4) than the correlations between the estimated raw and the original raw images and in opposite directions. In addition, the correlations of seven of the 16 features varied substantially with the spatial distance between the pixel pairs used for construction of the gray level matrices from which the texture features were calculated. For several of the features, the correlations between the estimated raw and the original raw image switched from lower to higher than the correlations between the processed and the original raw images as the spatial distance increased. The complicated variations of the correlations between the texture features extracted from the estimated raw and the original raw images revealed that the detailed characteristics of the original raw images were not reproduced in the estimated raw images by the proposed U-Net model although the breast density achieved high similarity. A possible cause is that the spatial resolution of the paired input-output target image was reduced by a factor of about seven for the DL training, as discussed by Shu et al. However, it will be of interest to investigate whether the varied trends of the correlations may have resulted from other factors in the DCNN operations that may affect different texture features in different ways (6).Although the study evaluated a specific DL approach in mammography, it serves as a good example to illustrate the capability and potential limitations of DL in transforming or simulating images. DL has been shown to be able to generate images that have high structural similarity to the target images. However, the current study showed that the detailed characteristics in a DL-simulated image may not be similar to those in a real image despite the similarity in their gray-level intensities. Whether the simulated images may help or mislead the training of an AI model for an intended task will depend on the image characteristics that the model has to learn from the images. For instance, for lesion or breast density segmentation and size measurement, structural similarity may be adequate. In radiomics, texture features have been found to correlate with tumor heterogeneity and used in predictive models for malignancy or pathologic types. Texture features have also been shown to correlate with genomic factors in radiogenomic analysis. Mammographic parenchymal pattern characterized by texture features has been shown to improve breast cancer risk prediction. DL-simulated images intended for these applications will need to possess texture characteristics that have similar representation of the pathologic or phenotypic state to be consistent with that of the real images. For MRI, images are acquired with certain pulse sequences or contrast-enhancement techniques not only to obtain anatomic information but also the specific functional or physiologic properties of the tissue or tumors that may be characterized by the dynamic or other quantitative parameters. DL-simulated MR images will have to meet the more stringent requirement that the images can provide the clinical information consistent with that obtained from real images in the intended applications.DL is a versatile machine learning technology and holds great promise to provide CAD-AI support for various medical imaging applications. The need of big data for DL model development has prompted investigators to explore the possibility of regenerating lost image data from patient archives or creating new images of the same modality or another modality by utilizing DCNN’s modeling and generative ability (1). While these images are potentially very valuable for either CAD-AI development or even patient care, the study by Shu et al serves as a cautionary reminder that the high correlation of one type of features from the DL-simulated images with that from the real images may not translate to another type of features or even the same feature extracted with different parameters. All detailed characteristics of the DL-simulated images relevant to a given intended use should be thoroughly evaluated in addition to the structural appearance. Most importantly, as with any CAD-AI applications, AI models that involve the use of simulated images should undergo rigorous validation with real-world patient data before deployment to ensure effectiveness and generalizability in clinical settings.Disclosures of Conflicts of Interest: H.P.C. Activities related to the present article: author is an associate editor of Radiology: Artificial Intelligence. Activities not related to the present article: author’s institution has a grant from GE Healthcare for collaboration in research, which ended July 2020. Other relationships: disclosed no relevant relationships.Author declared no funding for this work.References1. Yi X, Walia E, Babyn P. Generative adversarial network in medical imaging: A review. Med Image Anal 2019;58101552. Crossref, Medline, Google Scholar2. Pesteie M, Abolmaesumi P, Rohling RN. Adaptive Augmentation of Medical Data Using Independently Conditional Variational Auto-Encoders. IEEE Trans Med Imaging 2019;38(12):2807–2820. Crossref, Medline, Google Scholar3. Sahiner B, Pezeshk A, Hadjiiski LM, et al. Deep learning in medical imaging and radiation therapy. Med Phys 2019;46(1):e1–e36. Crossref, Medline, Google Scholar4. Chan HP, Samala RK, Hadjiiski LM. CAD and AI for breast cancer-recent development and challenges. Br J Radiol 2020;93(1108):20190580. Crossref, Medline, Google Scholar5. Shu H, Chiang T, Wei P, et al. A Deep Learning Approach to Re-creating Raw Full-Field Digital Mammograms for Breast Density and Texture Analysis. Radiol Artif Intell 2021;3(4):e200097. Link, Google Scholar6. Hussain Z, Gimenez F, Yi D, Rubin D. Differential Data Augmentation Techniques for Medical Imaging Classification Tasks. AMIA Annu Symp Proc 2018;2017(979):984. Google ScholarArticle HistoryReceived: Apr 11 2021Revision requested: Apr 15 2021Revision received: Apr 16 2021Accepted: Apr 19 2021Published online: June 23 2021 FiguresReferencesRelatedDetailsAccompanying This ArticleA Deep Learning Approach to Re-create Raw Full-Field Digital Mammograms for Breast Density and Texture AnalysisApr 14 2021Radiology: Artificial IntelligenceRecommended Articles Artificial Intelligence for Mammography and Digital Breast Tomosynthesis: Current Concepts and Future PerspectivesRadiology2019Volume: 293Issue: 2pp. 246-259Breast Cancer Risk Prediction Using Deep LearningRadiology2021Volume: 301Issue: 3pp. 559-560Deep Learning: A Primer for RadiologistsRadioGraphics2017Volume: 37Issue: 7pp. 2113-2131Current Applications and Future Impact of Machine Learning in RadiologyRadiology2018Volume: 288Issue: 2pp. 318-328Deep Learning Predicts Interval and Screening-detected Cancer from Screening Mammograms: A Case-Case-Control Study in 6369 WomenRadiology2021Volume: 301Issue: 3pp. 550-558See More RSNA Education Exhibits Artificial Intelligence in Breast Imaging: Past, Present, and FutureDigital Posters2020Artificial Intelligence Applications in Breast Imaging and Challenges for Digital Breast TomosynthesisDigital Posters2020The Word Soup of Artificial Intelligence (AI): Basics of AI for the Breast ImagerDigital Posters2019 RSNA Case Collection Male invasive Ductal Carcinoma RSNA Case Collection2022Lactating BreastRSNA Case Collection2022Locally Advanced Breast CancerRSNA Case Collection2021 Vol. 3, No. 4 Metrics Downloaded 341 times Altmetric Score PDF download