CP‐GAN: Meet the high requirements of diagnose report to medical image by content preservation

IET Image ProcessingVolume 16, Issue 1 p. 29-38 ORIGINAL RESEARCH PAPEROpen Access CP-GAN: Meet the high requirements of diagnose report to medical image by content preservation Xianhua Zeng, Corresponding Author Xianhua Zeng zengxh@cqupt.edu.cn orcid.org/0000-0002-4280-2525 Chongqing Key Laboratory of Image Cognition, College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, China Correspondence Xianhua Zeng, Chongqing Key Laboratory of Image Cognition, College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China. zengxh@cqupt.edu.cnSearch for more papers by this authorZhengyi Huang, Zhengyi Huang hzy3024803591@gmail.com Chongqing Key Laboratory of Image Cognition, College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, ChinaSearch for more papers by this authorLiming Xu, Liming Xu xulimmail@gmail.com Chongqing Key Laboratory of Image Cognition, College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, ChinaSearch for more papers by this authorYicai Xie, Yicai Xie xieyicai@gnnu.edu.cn Chongqing Key Laboratory of Image Cognition, College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, ChinaSearch for more papers by this author Xianhua Zeng, Corresponding Author Xianhua Zeng zengxh@cqupt.edu.cn orcid.org/0000-0002-4280-2525 Chongqing Key Laboratory of Image Cognition, College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, China Correspondence Xianhua Zeng, Chongqing Key Laboratory of Image Cognition, College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China. zengxh@cqupt.edu.cnSearch for more papers by this authorZhengyi Huang, Zhengyi Huang hzy3024803591@gmail.com Chongqing Key Laboratory of Image Cognition, College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, ChinaSearch for more papers by this authorLiming Xu, Liming Xu xulimmail@gmail.com Chongqing Key Laboratory of Image Cognition, College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, ChinaSearch for more papers by this authorYicai Xie, Yicai Xie xieyicai@gnnu.edu.cn Chongqing Key Laboratory of Image Cognition, College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, ChinaSearch for more papers by this author First published: 20 October 2021 https://doi.org/10.1049/ipr2.12145AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Medical image generation from diagnostic report has important research significance for medical-aided diagnosis. This research can improve the diagnosis speed and accuracy of doctors and effectively save the storage resources of hospitals. The research has made significant progress in the field of natural images, but rarely used in medical images. Medical images have higher requirements for image quality. In this paper, a method based on attention mechanism and content preservation loss to improve image quality is proposed. The model consists of three stages, and each stage generates feature map of different scale combined with attention features as the input of the next stage to optimise semantic consistency between image and text. The last stage will generate an image, whose size is 256 × 256. And the content preservation loss can optimise the similarity between the generated image and the real image by low-level and high-level features to meet the high requirements of medical image for texture details. The content preservation loss consists of MSE loss, VGG loss and TV loss. The experiments on two datasets prove that the method can achieve excellent results. 1 INTRODUCTION Text-to-image refers to generating an image with the same content as the description from the text. The research has a wide range of application fields, such as medical aided diagnosis and image storage, but it is also challenging. In recent years, research on text-to-image has become more active. It also further promotes the study of multi-modal learning across language and vision. Currently, research on text-to-image are mainly aimed at natural images. And the study of text-to-image on medical images have a great significance. Most doctors make experiential judgments and analysis when they diagnose. Generate medical images with the same or similar diseases from the diagnosis report as a doctor's diagnosis reference can effectively improve the efficiency and accuracy of diagnosis. It can also solve the problem of difficulty in querying past cases caused by the hospital regularly deleting data. The hospital can only store the diagnosis report, whenever it is necessary to query past cases, a corresponding medical image can be generated from the diagnosis report. Therefore, it can effectively save the storage resource of hospital. At present, the research on text-to- image has made significant progress in the field of natural images. The recently proposed text-to-image methods are mainly based on generative adversarial networks (GAN) [1]. Those methods use text as the input condition. [2-5] are one stage models. [6-9] generate images of different scales through multiple stages. These methods rarely consider the preservation of the content and texture details of the generated image, and the texture structure of image is easy to deform. In the field of medical image, the requirements for image texture details and pathological information are demanding. Even small changes in texture details can have a huge impact on the disease information. Therefore, keeping the texture details of the image generated from diagnostic report is very important. Different from natural images, medical images have higher requirements for content and texture details. And there are two main challenge to achieve that goal: (1) maintain semantic consistency between image and diagnostic report; (2) generate richer and clearer textures. [8] has been proved the attention mechanism can effectively maintain semantic consistency between image and text. MSE loss can optimise the texture details of the generated image from the pixel level, and VGG loss [10] can make up for the problem MSE have poor perceptual quality. Therefore, motivated by mentioned analysis and discussion, we adopt several measures to achieve high quality medical image generations from corresponding diagnose report. Specially, we firstly use mean squared error loss (MSE loss) to optimise the generated image from low-level features. Secondly following [10], we use VGG loss to optimise the generated image from high-level features. Thirdly, we use total variation loss (TV loss) [11] to improve visual quality by smoothing the image. Finally, following [8] we use attention mechanism to keep the semantic consistency of generated images and diagnostic report. The main contribution of this paper can be summarized as follows: (1) A novel text-to-image generation model based on medical images is proposed to generate corresponding medical images from diagnostic reports, which is the first attempt in the community of text-to-image generation. (2) Content preservation loss is used to optimise image generation from the low-level and high-level features, which can keep the structural similarity between the generated image and the real image to meet the high requirements of medical image for texture structure. (3) A comprehensive experiment on two datasets is conducted to evaluate the model proposed by experience. The experiment shows that our proposed method achieves excellent results subjectively and objectively. The structure of this paper is as follows: In Section 2, we reviews related work about text-to-image; In Section 3, we introduce our proposed method of generating medical images from diagnostic reports. In Section 4, the experimental results are shown. In Section 5, we give the summarize of our method. 2 RELATED WORK Text-to-image has a broad application base, and significant progress has been made in the field of natural images. Currently, research on text-to-image is mainly based on generative adversarial networks. A brief overview of related methods is summarized as follows. Mansimov et al. [2] based on DRAW [12] proposed align-DRAW model, the decoder iteratively reconstructs the image by RNNs through the accumulated information of each layer. The align-DRAW model consists of two RNNs [13], which are used to encode images and reconstruct images, respectively. Unlike other generators, DRAW does not generate images at once, but reconstructs images iteratively through the accumulated information of each layer of the decoder. The align-DRAW model takes the text as the input of the original DRAW model and introduces the attention mechanism to make the generated image more relevant to the text description. Reed et al. [3] proposed a model based on visual distinction to generate real image. Because many image examples can correctly express the text description, and the change of the text has a great influence on the large pixel area of the image, which will make image lack of relevance. Therefore, the author proposes GAN-INT-CLS model base on visual distinction to extract text features, where the training texts include the extraneous texts. The extraneous texts generated by interpolation to increase the discriminability of discriminator. The discriminator not only discriminates the true and false of the image, but also discriminates matching degree of image and text. To generate higher resolution image, Reed et al. [4] proposed GAWWN model, which use bounding box based model and key points based model. In the generation model, which based on the bounding box, the embedded text features are spatially converted into a specific feature map, then the feature map is spatially twisted to correspond to the normalized bounding box. After averaging the feature maps of multiple texts which are corresponding to same image, the feature maps are convolved into a one-dimensional vector and concatenated with noise. And use the resulting vector as input of the generator and discriminator. In the generation model based on the key point, the key point codes are mapped to a feature vector. Then input the vector of key points to each stage of the network. Zhang et al. [5] introduced accompanying hierarchical-nested adversarial objectives. The generator generates different size images in different stage, and matches different discriminators for different size images. This method can standardize the representation of the middle layer, so that the generator can capture complex image information. And the multi-purpose adversarial loss is two tasks of the discriminator: discriminating the authenticity of the image and discriminating the matching degree of image and text. To solve the problem the CGAN [14] model cannot generate high-resolution images, Zhang et al. [6] proposed a multi-stage model StackGAN. StackGAN generates a low-resolution image in first stage, and then obtain a high-resolution image by repairing the low-resolution image in second stage. The first stage generates low-resolution images by text. When the training of first stage is completed, the low resolution image is used as the input of the second stage to train the second stage. The second stage is similar to the process of image super-resolution. Zhang et al. [7] proposed a new end-to-end model StackGAN++ based on StackGAN [6], which add more generation stages to improve the convergence speed of the model and improve the quality of generated image. And StackGAN++ no longer need to train each stage separately. Xu et al. [8] introduced an attention mechanism based on StackGAN++ [7]. In the text embedding stage, the text encoder is pre-trained by matching loss to optimise the text encoder and the image encoder. Word features are input to each stage through the attention mechanism to generate more relevant images. And the model optimises the generated image in the final stage through the matching loss between the image and the text. Qiao et al. [9] introduced a global attention mechanism, which also use sentence-level attention feature combined with feature map of each stage, and proposed a text-image-text model structure based on AttnGAN [8]. The model uses the semantic consistency of the text reconstructed by the generated image and the real text to optimise the generated image in the final stage. Zhu et al. [15] introduce a dynamic memory model to better associate words with different content of the generated image. This model mainly includes the three modules: (1) Memory writing gate: calculate the attention between the feature map and the word embedding vector. (2) Key value memories: Calculate the weight of the attention of each word as the similarity between the word and the image. (3) Response Gate: fuse the weight sum with the feature map to output a new image feature. Yin et al. [16] introduces a Siamese mechanism in the discriminator to learn consistent high-level semantics. It minimizing the distance of the fake images generated from two descriptions of the same real image and maximizing the distance of different real images. And semantic-conditioned batch normalization which use sentence feature and word feature as the condition of Conditional Batch Norm proposed by [17]. Bhise et al. [18] inspire by [19] introduce a Mode-seeking loss to optimise mode collapsing, which maximise the quotient of the distance between images generated by different random noises and the distance between corresponding random noises. Tao et al. [20] introduce regularization method called Matching-Aware zero-centred Gradient Penalty(MA-GP) and Deep Text-Image Fusion Block(DFBlock) to reduce the time complexity and space complexity of model. MA-GP is a regularization strategy on discriminator. By pushing the matching loss of image and text to the place where the gradients of the discriminator loss. The DFBlock is composed of a series of Affine Transformations, ReLU layers, and convolution layers. And the model only use Affine Transformation to manipulates visual feature maps. Affine Transformations is a linear function, where the sentence features are invariants and image features are variables. Text-to-image methods are mainly for natural image, and there are few methods for medical images, but some methods based on natural image can also achieve this goal. Although above text-to-image generation models have made preliminary progress in the field of natural images. But these methods do not directly consider the preservation of the content and texture details of the generated image, so the texture structure of image is easy to deform. The quality of the images generated by the above methods needs to be further improved, because the medical field has more stringent requirements on the content and texture of images. MSE loss, VGG loss and TV loss have achieved good results in style transfer [21, 22] and image super-resolution [10, 23-25]. MSE loss and VGG loss can well keep the structural similarity between the generated image and the real image and improve the visual effect of the generated image and TV loss can effectively smooth the image to enhance the visual effect of the image. Because MSE loss struggle to handle the uncertainty inherent in recovering lost high-frequency details, MSE have poor perceptual quality [10, 23-25]. And the combination of MSE loss and VGG loss can effectively solve this problem from low-level features and high-level features. And the attention mechanism has also achieved good results in image caption [26, 27], image question answering [27, 28], and machine translation [29, 30]. In [8], it has been proved the attention mechanism can effectively improve the quality of image generation for text-to-image. Therefore, we MSE loss, VGG loss, TV loss and attention mechanism to improve the quality of the generated image. AttnGAN [8] has achieved good results on natural images. It based on the generation model of StackGAN++ [7], and different scale images generated from multiple stages are used to optimise the generation model. In order to keep the semantic consistency of the generated image and text, AttnGAN [8] proposed match loss between text and image. It optimise the generated image by calculating the similarity between text and image. At the same time, combine the feature map of each stage with attention feature as input to the next stage to optimise the generative model. The loss of his generator is as follow: L G = L G D + λ L DAMSM , L G D = ∑ x = 0 X − 1 L G x D , (1) L G x D = − 1 2 E I ̂ x ∼ G x ( s ¯ , w ) [ log ( D ( I ̂ ) ) ] − 1 2 E I ̂ x ∼ G x ( s ¯ , w ) [ log ( D ( I ̂ , s ca ) ) ] , (2)where x is the number of generate stages. And I ̂ G ( s ¯ , w ) represents the fake image Î, which generate from generator G. And s ¯ is the feature obtained by conditional augmentation sentence feature s ca concatenated with the random noise z. w is a word feature. L DAMSM = L 1 w + L 2 w + L 1 s + L 2 s , (3) L 1 w = − ∑ i = 1 T I log P ( I i | S i ) , L 2 w = − ∑ i = 1 TI log P ( S i | I i ) , (4)where w is a word feature, and s is a sentence feature. And P ( I i | S i ) is the posterior probability that sentence S i is matched with its corresponding image I i . TI is the number of text-image pairs. L 1 s and L 2 s can be calculated in the same way. We followed AttnGAN [8] base on the StackGAN++ [7] generation model and combine StackGAN++ [7] with the attention mechanism. And we replace the matching loss through content preservation loss to keep the texture similarity between the generated image and the real image. In order to optimise the whole network with the content preservation loss, we only optimise the generation model through generated image in last stage. 3 DIAGNOSTIC REPORT-TO-MEDICAL IMAGE MODEL Our model is based on the generation adversarial network, because of its powerful image generation capabilities. In order to improve the quality of the images, we use the content preservation loss to optimise the generation model. In order to make the disease category of the generated image more consistent with the expression of the diagnostic report, following [8], we combine stackGAN++ [7] and attention mechanism. In the generation stage, the encoded word vector is used as the attention condition. We combine word vector and the feature map of each stage, then we can get a multimodal context vector as the input of the next stage. As shown in Figure 1, our proposed network model consists of three modules: text encoding module, image generation module, and content preservation module. We will introduction each of them in the rest of this section. FIGURE 1Open in figure viewerPowerPoint Our proposed text-to-image generation adversarial network model. Each stage attention model automatically generates attention information as a condition to improve the semantic consistency of the generated image and text. Finally, improve the quality of image by content preservation loss and adversarial loss 3.1 Objective function Objective function of our method is as follow: min G max D V ( D , G ) = λ 1 2 E I ∼ I real [ log ( D ( I ) ) ] + λ 1 2 E I ∼ I real [ 1 − log ( D ( G ( s ¯ , w ) ) ) ] + λ 1 2 E I ∼ I real [ log ( D ( I , s ca ) ) ] + λ 1 2 E I ∼ I real [ 1 − log ( D ( G ( s ¯ , w ) , s ca ) ) ] + λ 2 E I ∼ I real [ MSE ( I , G ( s ¯ , w ) ) ] + λ 3 E I ∼ I real [ MSE ( VGG ( I ) , VGG ( G ( s ¯ , w ) ) ) ] + λ 4 E [ TV ( G ( s ¯ , w ) ] , (5)where I ∼ I real represents image I distributed from real data. s ¯ is a feature obtained by conditional augmentation sentence feature s c a concatenated with the random noise z. w is a word feature, which is the output of LSTM. s is a sentence feature, which is the hidden state of the last layer of LSTM. 3.2 Text encoding module In this section, we will introduce the text encoding module, which will extract word-level feature and sentence-level feature from diagnose report. As shown in the left part of Figure 1, because of the better feature extraction capabilities for discrete text data, a long short-term memory network (LSTM) [31] is used to encode the diagnose report. The word encoding feature is w, and the sentence encoding feature is s. s , w = LSTM ( S ) , (6)where S is diagnose report, S = { S n | n = 0 , … , N − 1 } , N is the length of the sentence. w = { w n | n = 0 , … , N − 1 } ∈ R M × N is the concatenation of hidden state w n of each word. s ∈ R M is the hidden state of the last layer of LSTM. M is the dimension of w n and s. We use the conditional augmentation [12] to increase the robustness of text flow against small fluctuations. s ca = F ca ( s ) , (7)where F ca represents the conditional augmentation function. s ca ∈ R M ′ is the sentence feature after conditional augmentation. M ′ is the dimension of sentence feature after conditional augmentation. 3.3 Image generation module We stack three visual feature transformers in sequence, and each visual feature transformer generates feature maps of different scales. We use the attention model proposed by [8] to combine feature maps of various scales and word feature vectors into the input of the next visual feature transformer. Our model has multiple stages, and one stage consists of one visual feature transformer. { H 0 , H 1 , … , H x − 1 } represents x visual feature transformers. The visual feature f i at each stage and generated image I ̂ can be expressed as: s ¯ = concat ( z , s ca ) , f 0 = H 0 ( s ¯ ) , f i = H i ( f i − 1 , F i att ( f i − 1 , w ) ) , i ∈ { i , 2 , … , x − 1 } , I ̂ = H G ( f x − 1 ) , (8)where f i ∈ R C i × D i , I ̂ ∈ R P × Q . C i and D i is the dimension of f i . P and Q is the dimension of generated image. And z ∼ N ( 0 , 1 ) is a random noises. s ¯ denote sentence features get from concatenating z and s ca . F i att represents the attention model proposed in [8]. At each stage, word feature w and visual feature f i − 1 are input. Word features are first transformed by the perception layer A i − 1 ∈ R C i − 1 × M , i ∈ { 1 , 2 , … , x − 1 } into the same semantic space as the visual features. w i − 1 ′ = A i − 1 w , i ∈ { 1 , 2 , … , x − 1 } represents the transformed word features. We divide an image into j sub-regions. Then calculate the attention scores between word feature w i − 1 ′ and the sub-regions of visual feature f i − 1 . The attention scores e i − 1 j for each sub-region f i − 1 j is as follows: e i − 1 j = ∑ k = 0 N − 1 β j , k w i − 1 ′ k , β j , k = exp ( r i − 1 j , k ) ∑ l = 0 N − 1 exp ( r i − 1 j , l ) , (9)where r i − 1 j , k = f i − 1 j w i − 1 ′ k , w i − 1 ′ k represents vector of the kth word, then we can get word-context matrix w i att by w i − 1 att = F i att ( f i − 1 , w ) , where F i att ( f i − 1 , w ) = ( e i − 1 0 , e i − 1 1 , … , e i − 1 D i − 1 − 1 ) ∈ R C i − 1 × D i − 1 . And combine visual feature f i − 1 and word-context matrix w i att input to the next stage. 3.4 Content preservation module In the training phase, generator G and discriminator D alternately iterate to achieve the entire training process. 3.4.1 Generation stage The loss of the generator during training is as follows: L G = λ 1 L G D + λ 2 L MSE + λ 3 L VGG + λ 4 L TV . (10) The generator loss is mainly composed of two parts, the adversarial loss λ 1 L G D get from the discriminator and the content preservation loss L P = λ 2 L MSE + λ 3 L VGG + λ 4 L TV get from the content preservation model. λ i , i ∈ { 1 , 2 , 3 , 4 } is the weight of each loss. And the discriminant loss of the generator is as follows: L G D = − 1 2 E I ̂ ∼ G ( s ¯ , w ) [ log ( D ( I ̂ ) ) ] − 1 2 E I ̂ ∼ G ( s ¯ , w ) [ log ( D ( I ̂ , s ca ) ) ] , (11)where I ̂ ∼ G ( s ¯ , w ) represents the fake image Î , which generate from generator G. The adversarial loss of our generator is the same as that of AttnGAN [8] in the final stage, but the difference is that in order to improve the quality of the generated image, we do not use the matching loss between text and image to optimise image generation. Instead, we use the content preservation loss Lp to optimise the generated image. The content preservation loss keeps the structural similarity between the generated image and the real image from two aspects of high-level feature and low-level feature by MSE loss, VGG loss and TV loss. First, we use MSE loss to keep the structural similarity between the real image and generated image by calculating the mean square error between the pixels of the real image I ∈ R P × Q and the generated image I ̂ ∈ R P × Q . P and Q is the dimension of images I and I ̂ . L MSE = 1 P Q ∑ p = 0 P − 1 ∑ q = 0 Q − 1 ( y p , q − y ̂ p , q ) 2 , (12)where y p , q ∼ I , y ̂ p , q ∼ I ̂ . y p , q and y ̂ p , q represent a pixel of real image I and generated image I ̂ . Second we calculate the similarity between the deep features of the real image I and the generated image Î by VGG loss to improve image quality from high-level feature. We use VGG net [32] to extract the deep features V and I ̂ of images I and I ̂ , respectively, then calculate the mean square error between V and V ̂ . L VGG = 1 U 1 U 2 ∑ p = 0 U 1 − 1 ∑ q = 0 U 2 − 1 ( v p , q − v ̂ p , q ) 2 , (13)where v p , q ∼ V , v ̂ p , q ∼ V ̂ and V ∈ R U 1 × U 2 , V ̂ ∈ R U 1 × U 2 , U1 and U2 is the dimension of deep feature extracted by VGG net. v p , q and v ̂ p , q represent a feature point of deep feature extracted by VGG net. Final, we add TV loss to make the pixel changes of the generated image smoother. TV loss can limit the change of adjacent pixels by the mean square error between adjacent pixels. [33] has proven TV loss can effectively suppress image noise and smooth the image. L TV = ∑ p = 0 P − 1 ∑ q = 0 Q − 1 ( ( y ̂ p , q + 1 − y ̂ p , q ) 2 + ( y ̂ p + 1 , q − y ̂ p , q ) 2 ) , (14)where y ̂ p , q represents a pixel of generated image, P and Q is the dimension of generated image. Therefore, we Optimise the texture details of the generated image from low-level feature by L MSE and L TV , and from high-level feature by L VGG . 3.4.2 Discrimination stage The loss of the discriminator as follow: L D = − 1 2 E I ∼ I real [ log ( D ( I ) ] − 1 2 E I ̂ ∼ G ( s ¯ , w ) [ log ( 1 − ( D ( I ̂ ) ) ) ] − 1 2 E I ∼ I real [ log ( D ( I , s ca ) ) ] − 1 2 E I ̂ ∼ G ( s ¯ , w ) [ log ( 1 − ( D ( I ̂ , s ca ) ) ) ] . (15)We followed [34], the discriminator has two discriminating tasks: one is to discriminate whether the image is visually true; the other is to discriminate whether the visual content of the image matches the text description. 4 EXPERIMENTS In this section, we will evaluate our proposed model through corresponding experiments. We will compare our method with GAN-INT-CLS [3], StackGAN [6], StackGAN++ [7], AttnGAN [8], MirrorGAN [9] and DM-GAN [15]. Then we conduct a weights analysis on our method. At the same time, we also show an ablation analysis, the feature map of VGG net and explored the possibility of text-to-image on medical image storage in the appendix document. The implementation is available at https://github.com/huang742/demo-for-CP-GAN/tree/master 4.1 Experiment setup Some detail of parameter settings, evaluation indicators and datasets information of our method on experiment will show in this section. 4.1.1 Datasets We evaluated our model on an ultrasound dataset and a X-ray dataset. Open-i: The Open-i dataset is an open public medical image dataset, which consists of over 7418 chest X-ray images and 4000 corresponding diagnose reports. The image includes three parts: heart, spine and lung . The diagnosis report mainly describes heart, spine, and lung diseases of the chest X-ray image. And the diagnose report described in English. There are many