# CMMCSegNet: Cross-Modality Multicascade Indirect LGE Segmentation on Multimodal Cardiac MR

Authors: Yu Wang; Jianping Zhang
Journal: Computational and Mathematical Methods in Medicine
 (2021)

---

## Abstract

Since Late-Gadolinium Enhancement (LGE) of cardiac magnetic resonance (CMR) visualizes myocardial infarction, and the balanced-Steady State Free Precession (bSSFP) cine sequence can capture cardiac motions and present clear boundaries; multimodal CMR segmentation has played an important role in the assessment of myocardial viability and clinical diagnosis, while automatic and accurate CMR segmentation still remains challenging due to a very small amount of labeled LGE data and the relatively low contrasts of LGE. The main purpose of our work is to learn the real/fake bSSFP modality with ground truths to indirectly segment the LGE modality of cardiac MR by using a proposed cross-modality multicascade framework: cross-modality translation network and automatic segmentation network, respectively. In the segmentation stage, a novel multicascade pix2pix network is designed to segment the fake bSSFP sequence obtained from a cross-modality translation network. Moreover, we propose perceptual loss measuring features between ground truth and prediction, which are extracted from the pretrained vgg network in the segmentation stage. We evaluate the performance of the proposed method on the multimodal CMR dataset and verify its superiority over other state-of-the-art approaches under different network structures and different types of adversarial losses in terms of dice accuracy in testing. Therefore, the proposed network is promising for Indirect Cardiac LGE Segmentation in clinical applications.


---

## Body

## 1. Introduction

Multimodal CMR imaging is an essential tool in clinics for the screening and diagnosis of cardiac diseases. Different imaging modalities contain different sorts of useful information for cardiac disease screening task; the combination of different imaging modalities can overcome the limitations of an individual modality. The contrast agent for the LGE MR imaging is injected for 10-20 minutes; LGE images with distinctive locally brightness compared with the healthy tissues can enhance myocardial necrosis or scarring, which is a standard practice to evaluate cardiac structure, cardiac function, myocardial perfusion, and myocardial activity. Different from LGE images, the bSSFP can highlight the high signal area of the fluid but appear a uniform signal for other tissues; e.g., the large blood vessels and coronary arteries can be observed clearly in bSSFP because of more obvious contrast in the heart muscle and blood pool. T2-weighted MRI is effective in reducing false-positive results. Considering different MRI modalities is thus important for the acquisition of accurate cardiac information [1].Segmentation of multimodal CMR images is a critical step in the process for the following diagnosis and surgical planning. However, it takes 20 minutes/case for an experienced doctor to manually segment the LGE images, it is extremely time-consuming to manually identify and delineate the corresponding structure in cardiac, and the result depends on the professional ability of doctors and varies from person to person. Therefore, the development of automatic and reliable LGE image segmentation algorithms is of high clinical values for patients suffering from myocardial infarction.Tao and Der Geest proposed a method for segmenting the LGE images using myocardial morphological information [2]. Popescu et al. used a mask SLIC clustering method and Otsu threshold to segment LGE images [3]. In recent years, deep learning has achieved remarkable success in computer vision. More and more image processing methods are based on the CNN model [4, 5]. Chen et al. [6] proposed to use the domain adaption to fuse the features of unlabeled LGE images and then use the fused features to train the segmentation network. In addition, many approaches based on attention mechanisms [7, 8] and multiview methods [9] have been developed recently for segmenting medical images. Yang et al. combined multiview and attention mechanism to segment cardiac LGE images [10]. An automatic cardiac LGE segmentation algorithm based on the CNN is far more efficient and robust, and commonly more accurate than traditional methods [11, 12], so it is necessary to automatically segment the LGE images.However, automatic LGE CMR segmentation is still arduous. Besides the great variations of the location and geometry of the heart region across different patients, Zhuang [1] pointed three major challenges related to the intensity distributions of the LGE CMR modality: (i) the intensity range of myocardium in LGE imaging leads to indistinguishable boundaries from its adjacent organs; (ii) the pathologies result in heterogeneous intensity of the myocardium, making the assumption of a simple distribution such as the single component Gaussian density invalid; and (iii) the preprocessing enhancement for the LGE CMR modality can be complex. So it is more difficult to segment directly LGE modality, especially in case of a small amount of labeled LGE data.GAN was first proposed by Goodfellow et al. [13] for image synthesis, which uses a generator network and discriminator network, to pit one against the other (thus the “adversarial”) in order to generate fake synthetic instance that can pass for real data. Here, the generator generates a fake image by random noise, the discriminator judges whether the input data is true (data comes from real labels) or false (the data comes from the output of the generator). The aim of GANs is to learn the underlying distribution of training data in order to generate data that the discriminator cannot distinguish. At the same time, the game between the generator and the discriminator reaches the Nash equilibrium, i.e., the generated data distribution pg is equal to real data distribution pd. With the development of GANs [14], such models are widely used in image processing, including image and video generation [15], image segmentation [16], image synthesis [17], and image super resolution [18].In this work, we propose a novel cross-modality multicascade framework for indirect LGE segmentation (CMMCSegNet), which is trained on multimodal cardiac MR data with a very small amount of LGE labels (for the LGE modality in Multisequence Cardiac MR Segmentation Challenge 2019 datasets [1], only five patients are labeled). The main contributions of this work are clarified as follows:
(1)
We develop a novel indirect LGE segmentation framework based on multimodal images; one of the primary components is to translate the LGE modality that needs to be segmented but only has very small amount of labeled data, into the bSSFP modality that is easy to be segmented by our proposed method(2)
We propose a multicascade pix2pix network for image segmentation; that is, the generator is formed by cascading multiple subnetworks. In the segmentation network, we regard segmentation as the translation process from the original image to the segmentation target(3)
We employ the perceptual loss that uses a pretrained VGG19 network to compare the feature differences between the labels and generation during the proposed multicascade pix2pix network trainingThe rest of this work is organized as follows. We first give some preliminaries in Section2. We describe our CMMCSegNet in details in Section 3. We give experimental results in Section 4. Finally, we conclude this work in Section 5.


## 2. Related Works

Tissue or organ segmentation plays an important role in the field of medical image processing. Medical image segmentation has been explored extensively; however, challenges in generality, robustness, and efficiency still remain. For brevity, we only focus below on the most closely related works.


### 2.1. Cascade Structure

A cascading network is to connect multiple subnetworks together to form a multilevel network. The cascading method has been effectively used in many vision applications like classification [19], image translation [20], detection [21], super resolution [22], and semantic segmentation [23]. For example, Cui et al. proposed a deep cascade network for image super resolution [22]. Cai and Vasconcelos proposed the use of cascade structure for object detection [21]. Zhao et al. proposed the recursive cascaded networks for medical image registration [24]. Armanious et al. proposed the use of cascaded generator network for image translation [20]. Havaei et al. proposed a new cascade architecture for brain tumor segmentation [23]. Li et al. [25] proposed to classify easy regions in a shallow network and train deeper networks to deal with hard regions. Lin et al. [26] proposed a top-down architecture with lateral connections to propagate deep semantic features to shallow layers.Different from previous cascade networks, the multicascade pix2pix network proposed in this paper is a multiple U-net cascade structure for image segmentation, which allows an innovative way to supervise each generator individually for pix2pix GANs. To our best knowledge, this is an early and original attempt to adopt a cascade architecture in pix2pix GAN-based medical image segmentation. We will introduce more details in Section3.


### 2.2. Multimodal Cardiac MR Image Segmentation

Recent literature suggests two main approaches to complete multimodal CMR image segmentation. One popular approach is about the GAN strategy based on cross-modality image translation that refers to the translation of images with modalityX into images with modality Y, which plays an increasingly important role in computer vision. Isola et al. [18] proposed the use of conditional GAN to implement a paired image-to-image translation. Ben-Cohen et al. used CT images to synthesize PET images based on the pix2pix network [27]. Cycle-GAN [28] was proposed for unpaired image-to-image translation. BiCycle-GAN [29] solved the translation process from single image to multicategory image. In addition, some GAN networks including DualGAN [30] and UNIT [31] were also proposed for unpaired image-to-image translation.In CMR datasets [1], MR images of the different modalities are not strictly matched, so the classical unpaired image-to-image translation [32] can be applied to cross-modality CMR segmentation. Chen et al. [33] proposed to use UNIT to translate bSSFP images into LGE images and then train the segmentation network where the LGE images are provided by the translation architecture. Campello et al. also proposed to use Cycle-GAN to translate bSSFP images into LGE images but train the U-net network [34] for LGE segmentation. Tao et al. [35] proposed to integrate the translation network (Cycle-GAN) with the segmentation network to achieve LGE image segmentation.Another promising approach is about the strategy on image registration. Roth et al. proposed to register LGE images with ground truths into LGE images without ground truths, after multiatlas label fusion by majority voting; they obtained a noisy LGE label and then trained a LGE segmentation network [36]. Liu et al. proposed a registration method for histogram matching to achieve augmentation of the LGE images [37].


## 2.1. Cascade Structure

A cascading network is to connect multiple subnetworks together to form a multilevel network. The cascading method has been effectively used in many vision applications like classification [19], image translation [20], detection [21], super resolution [22], and semantic segmentation [23]. For example, Cui et al. proposed a deep cascade network for image super resolution [22]. Cai and Vasconcelos proposed the use of cascade structure for object detection [21]. Zhao et al. proposed the recursive cascaded networks for medical image registration [24]. Armanious et al. proposed the use of cascaded generator network for image translation [20]. Havaei et al. proposed a new cascade architecture for brain tumor segmentation [23]. Li et al. [25] proposed to classify easy regions in a shallow network and train deeper networks to deal with hard regions. Lin et al. [26] proposed a top-down architecture with lateral connections to propagate deep semantic features to shallow layers.Different from previous cascade networks, the multicascade pix2pix network proposed in this paper is a multiple U-net cascade structure for image segmentation, which allows an innovative way to supervise each generator individually for pix2pix GANs. To our best knowledge, this is an early and original attempt to adopt a cascade architecture in pix2pix GAN-based medical image segmentation. We will introduce more details in Section3.


## 2.2. Multimodal Cardiac MR Image Segmentation

Recent literature suggests two main approaches to complete multimodal CMR image segmentation. One popular approach is about the GAN strategy based on cross-modality image translation that refers to the translation of images with modalityX into images with modality Y, which plays an increasingly important role in computer vision. Isola et al. [18] proposed the use of conditional GAN to implement a paired image-to-image translation. Ben-Cohen et al. used CT images to synthesize PET images based on the pix2pix network [27]. Cycle-GAN [28] was proposed for unpaired image-to-image translation. BiCycle-GAN [29] solved the translation process from single image to multicategory image. In addition, some GAN networks including DualGAN [30] and UNIT [31] were also proposed for unpaired image-to-image translation.In CMR datasets [1], MR images of the different modalities are not strictly matched, so the classical unpaired image-to-image translation [32] can be applied to cross-modality CMR segmentation. Chen et al. [33] proposed to use UNIT to translate bSSFP images into LGE images and then train the segmentation network where the LGE images are provided by the translation architecture. Campello et al. also proposed to use Cycle-GAN to translate bSSFP images into LGE images but train the U-net network [34] for LGE segmentation. Tao et al. [35] proposed to integrate the translation network (Cycle-GAN) with the segmentation network to achieve LGE image segmentation.Another promising approach is about the strategy on image registration. Roth et al. proposed to register LGE images with ground truths into LGE images without ground truths, after multiatlas label fusion by majority voting; they obtained a noisy LGE label and then trained a LGE segmentation network [36]. Liu et al. proposed a registration method for histogram matching to achieve augmentation of the LGE images [37].


## 3. Proposed Cross-Modality SegNet

The goal of this work is to achieve cardiac segmentation for LGE modality where a small amount of samples are labeled. Our CMMCSegNet (https://github.com/wangyu719/CmmcSegNet) framework is designed to facilitate indirect segmentation for the multimodal CMR images. The total framework is shown in Figure 1, including a training architecture and a testing architecture.Figure 1
The proposed cross-modality multicascade SegNet flowchart of indirect segmentation for LGE modality. bSSFP images are important in distinguishing the cardiac structure from all enhanced areas. Accordingly, the two imaging modalities are treated differently in the proposed method. Cross-modality translation from LGE to bSSFP is chosen as Stage I of the proposed architecture having a greater impact on the LGE segmentation results. bSSFP is regarded as the assistant modality completing the LGE segmentation.
(a)
The proposed CMMCSegNet framework(b)
The Cycle-GAN architecture(c)
The multicascade pix2pix architectureOur datasets are from Multisequence Cardiac MR Segmentation Challenge 2019 datasets (MS-CMRSeg 2019) [1]. In this work, we use LGE modality with 45 patients and bSSFP modality with 35 annotated patients (see Figure 2 for more details). Only five ground truth annotations are available in LGE modality of MS-CMRSeg 2019 datasets; hence, it is difficult to directly segment LGE modality using deep CNN-based methods. Figure 2 shows the differences between LGE and bSSFP images from the same patient. Furthermore, it is found that the bSSFP modality has a more obvious contrast than LGE modality, so we believe that the bSSFP is easier to be segmented. Besides, the bSSFP modality has a large number of images (35 patients) with ground truth annotations, so it is not difficult to train the bSSFP modality using the deep learning-based method.Figure 2
Different modalities of CMR imaging from the same patient (the two images are unpaired). LGE can enhance myocardial necrosis or scarring, which can evaluate effectively cardiac structure, cardiac function, myocardial perfusion, and myocardial activity, while bSSFP can highlight clearly the large blood vessels and coronary arteries because of more obvious contrast in the heart muscle and blood pool. To better adapt to cardiac structure segmentation, we will build a cross-modality translation network based on Cycle-GAN (Figure1(b)).
(a)
LGE(b)
bSSFP


### 3.1. Cross-Modality Image Translation

One of the primary components in training architecture is a cross-modality translation network, which can be trained end-to-end with unpaired modalities. Before segmenting the bSSFP images to achieve indirect segmentation of LGE images, we first present a Cycle-GAN architecture of translating LGE into bSSFP images.Inspired by the knowledge distillation between unpaired image-to-image translation networks [32], we employ Cycle-GAN to achieve cross-modality image translation for CMR datasets. Let X,Y be two image domains that represent the LGE and bSSFP modalities, respectively. GAt:X⟶Y and GBt:Y⟶X are two generators of the cross-modality translation network such that GAt and GBt are inverse mappings of each other; that is, GBtGAtIX≈IX, GAtGBtIY≈IY for any unpaired images IX∈X, IY∈Y. DAt and DBt are the discriminators of the cross-modality translation network, to distinguish that the input of discriminator is real or fake.The Cycle-GAN architecture implementing cross-modality image translation for unpaired LGE/bSSFP datasets consists of two cycles: LGE cycle and bSSFP cycle. In the LGE cycle, the first generatorGAt is trained to transform LGE modality into fake bSSFP modality, the second generator GBt is trained to transform the generated fake bSSFP modality back to the original LGE modality, and the discriminator DAt discriminates between real and synthesized bSSFP modalities. In fact, enlightened by the activation-based attention transfer strategies, the discriminator DAt is designed to extract the supervision information that modulates the learning of the generator GAt. In the bSSFP cycle, real bSSFP was transformed to fake LGE by using the generator GBt, the generator GAt transforms the generated LGE to the original bSSFP, and the discriminator DBt discriminates between real and fake bSSFP modality. Finally, the network framework is shown in Figure 1(b).The overall training loss of our translation network is defined as(1)GAt∗,GBt∗;DAt∗,DBt∗=argminmaxGAt,GBt,DAt,DBtLtGAt,GBt;DAt,DBt≕λ1LcycGAt,GBt+LganGAt,DAt,X,Y+LganGBt,DBt,X,Y,where LganGAt,DAt,X,Y and LganGBt,DBt,X,Y are two adversarial losses defined by
(2)LganGAt,DAt,X,Y=EIY~pdIYlogDAtIY+EIX~pdIXlog1−DAtGAtIX,LganGBt,DBt,X,Y=EIX~pdIXlogDBtIX+EIY~pdIYlog1−DBtGBtIY,and the generation similarity LcycGAt,GBt is defined by
(3)LcycGAt,GBt=EIX~pdIXGBtGAtIX−IX1+EIY~pdIYGAtGBtIY−IY1,and λ1 is the weight parameter for balancing the contributions of the generation loss LcycGAt,GBt and the two adversarial losses LganGAt,DAt,X,Y and LganGBt,DBt,X,Y.


### 3.2. Multicascade pix2pix Segmentation

Recently, the GAN-based framework is proposed to segment the retinal vessel [38]. We understand image segmentation as the translation from paired image to image (from an original image to a predicted segmentation results); hence, we propose a new image segmentation method using a multicascade technique and pix2pix structure, which we call a multicascade pix2pix network.


#### 3.2.1. Multicascade Network

Our multicascade pix2pix segmentation network shown in Figure1(c) is based on the GAN architecture, which consists of multiple generators Gks (k=1,⋯,n) and a shared discriminator Ds.The generatorG1s: Y⟶S translates IY to IS1, where the original input IY∈Y is 1×256×256 real or fake bSSFP image, and the first generation IS1∈S is a prediction for the corresponding label. The other generators Gks: S⟶Sk=2,⋯,n furtherly improve the previous predicted probability ISk−1 to obtain more optimal prediction
(4)ISk≔Gks∘Gk−1s∘⋯∘G1sIY,where IY and ISk have the same size. In this work, Gks is formed by the U-net [5] or ResNet [39] network for the purpose of more accurate segmentation. In experimental evaluation, we will compare the effects of different generator networks on the segmentation results. The purpose of this network is to obtain the final segmented result ISf of the original input IY, which also is the result of the LGE segmentation. Therefore, the generated prediction obtained from the multicascade pix2pix segmentation network can be denoted as
(5)ISf≔ISn=Gns∘Gn−1s∘⋯∘G1sIY.The discriminatorDs is a binary classifier based on pixels or patch-images which provides a network learning-based stopping criterion during generating. For the discriminator Ds in our multicascade pix2pix segmentation network, we employ a convolutional Patch-GAN [18] to distinguish real or fake between the prediction ISk and the ground truth IL, where ISk is divided into ℓ×ℓ patches with overlapping images, and each patch is discriminated with those of the ground truth IL, respectively; finally, a 2D probability map is obtained as the discriminator outputs.To train an optimal segmentation network, the measures betweenISk and target label IL can be estimated and minimized to update discriminator Ds that enforces to discriminate the generation and the ground truth. The segmentation network we propose is a conditional version of pix2pix GAN with the multicascade architecture, so the adversarial input in Ds is mainly composed of there components, where the first component is the source image IY used as the condition and the others are the generation ISk and the ground truth IL. At the same time, each generator Gks is also optimized to generate domain-invariant representations ISk that confuses the discriminator Ds.


#### 3.2.2. Loss Functions in Segmentation Stage

The dice score and Jaccard index are commonly used as metrics for the evaluation of image segmentation task. CNNs trained for image segmentation task are usually optimized by minimizing a weighted cross-entropy. In this work, we employ a specially designed loss functionLs to simultaneously measure the generation similarity and the adversarial error, which contains three types of loss functions: adversarial loss Lgan, L1 loss, and perceptual loss Lvgg.The original adversarial loss (Vanilla GAN loss) is given by the Kullback-Leibler (KL) divergence score as(6)LganG1s,⋯,Gns,Ds=∑k=1nωkLgEIY,IL~pdIY,ILlogDsIY,IL+EIY~pdIYlog1−DsIY,ISk,where ωkLgk=1,⋯,n is the given weight enforcing the trade-off between the n cascade cross-entropy losses and IY is a condition input of each convolutional Patch-GAN in our multicascade pix2pix segmentation network. Recently, the most commonly used adversarial losses are WGAN-GP [40] and LSGAN [41]. In the next section, we will compare the performances of three different adversarial losses in our experiments.L1 loss is a weighted sum of the absolute distance between the calculated output data ISk in the k-th cascade block and the ground truth IL, which can make the segmentation results closer to the real results [18], and is defined by
(7)L1G1s,⋯,Gns=∑k=1nωkL1EIY~pdIYIL−ISk1,where ωkL1k=1,⋯,n are weight constants. Without loss of generality, we will take ωkLg=ωkL1 for all k=1,⋯,n in our experiments.Besides, we also employ the perceptual loss in our multicascade pix2pix segmentation network, which is composed of a pretrained VGG19 network and is first proposed in image super resolution application [42]. The perceptual loss focuses on feature maps between the output data and the ground truth [43]. It can hence be computed by
(8)LvggIL,G1s,⋯,Gns=∑k,i,j=1n,M,Ni∑p,q=1wij,hijSpqij,kwijhij,where Spqij,k=Dφi,jILpq−φi,jISkpq and φi,j represents the feature map of the j-th feature channel of the i-th feature layer (after activation) [42], Ni is the number of feature channels in the i-th feature layer and M is the number of convolution layers, and wij and hij represent the size of the feature map in the VGG19 network. Here, D is the error measure of the vgg/ResNet feature maps between the ground truth IL and prediction ISk. The most widely used feature distances also contain the manhattan distance Dmanh and the cosine similarity Dcosine difined by
(9)DmanhX,Y=∑X−Y1,DcosineX,Y=1−X,YX2Y2,where X and Y are feature maps.The total proposed segmentation model is trained by jointly minimizing the total lossLs for the three parts as follows:
(10)G1s,⋯,Gns,Ds∗=argminmaxG1s,⋯,Gns,DsLsG1s,⋯,Gns,Ds≕λlL1G1s,⋯,Gns+LganG1s,⋯,Gns,Ds+λvggLvggIL,G1s,⋯,Gns,where the λl and λvgg are two given weight parameters.


## 3.1. Cross-Modality Image Translation

One of the primary components in training architecture is a cross-modality translation network, which can be trained end-to-end with unpaired modalities. Before segmenting the bSSFP images to achieve indirect segmentation of LGE images, we first present a Cycle-GAN architecture of translating LGE into bSSFP images.Inspired by the knowledge distillation between unpaired image-to-image translation networks [32], we employ Cycle-GAN to achieve cross-modality image translation for CMR datasets. Let X,Y be two image domains that represent the LGE and bSSFP modalities, respectively. GAt:X⟶Y and GBt:Y⟶X are two generators of the cross-modality translation network such that GAt and GBt are inverse mappings of each other; that is, GBtGAtIX≈IX, GAtGBtIY≈IY for any unpaired images IX∈X, IY∈Y. DAt and DBt are the discriminators of the cross-modality translation network, to distinguish that the input of discriminator is real or fake.The Cycle-GAN architecture implementing cross-modality image translation for unpaired LGE/bSSFP datasets consists of two cycles: LGE cycle and bSSFP cycle. In the LGE cycle, the first generatorGAt is trained to transform LGE modality into fake bSSFP modality, the second generator GBt is trained to transform the generated fake bSSFP modality back to the original LGE modality, and the discriminator DAt discriminates between real and synthesized bSSFP modalities. In fact, enlightened by the activation-based attention transfer strategies, the discriminator DAt is designed to extract the supervision information that modulates the learning of the generator GAt. In the bSSFP cycle, real bSSFP was transformed to fake LGE by using the generator GBt, the generator GAt transforms the generated LGE to the original bSSFP, and the discriminator DBt discriminates between real and fake bSSFP modality. Finally, the network framework is shown in Figure 1(b).The overall training loss of our translation network is defined as(1)GAt∗,GBt∗;DAt∗,DBt∗=argminmaxGAt,GBt,DAt,DBtLtGAt,GBt;DAt,DBt≕λ1LcycGAt,GBt+LganGAt,DAt,X,Y+LganGBt,DBt,X,Y,where LganGAt,DAt,X,Y and LganGBt,DBt,X,Y are two adversarial losses defined by
(2)LganGAt,DAt,X,Y=EIY~pdIYlogDAtIY+EIX~pdIXlog1−DAtGAtIX,LganGBt,DBt,X,Y=EIX~pdIXlogDBtIX+EIY~pdIYlog1−DBtGBtIY,and the generation similarity LcycGAt,GBt is defined by
(3)LcycGAt,GBt=EIX~pdIXGBtGAtIX−IX1+EIY~pdIYGAtGBtIY−IY1,and λ1 is the weight parameter for balancing the contributions of the generation loss LcycGAt,GBt and the two adversarial losses LganGAt,DAt,X,Y and LganGBt,DBt,X,Y.


## 3.2. Multicascade pix2pix Segmentation

Recently, the GAN-based framework is proposed to segment the retinal vessel [38]. We understand image segmentation as the translation from paired image to image (from an original image to a predicted segmentation results); hence, we propose a new image segmentation method using a multicascade technique and pix2pix structure, which we call a multicascade pix2pix network.


### 3.2.1. Multicascade Network

Our multicascade pix2pix segmentation network shown in Figure1(c) is based on the GAN architecture, which consists of multiple generators Gks (k=1,⋯,n) and a shared discriminator Ds.The generatorG1s: Y⟶S translates IY to IS1, where the original input IY∈Y is 1×256×256 real or fake bSSFP image, and the first generation IS1∈S is a prediction for the corresponding label. The other generators Gks: S⟶Sk=2,⋯,n furtherly improve the previous predicted probability ISk−1 to obtain more optimal prediction
(4)ISk≔Gks∘Gk−1s∘⋯∘G1sIY,where IY and ISk have the same size. In this work, Gks is formed by the U-net [5] or ResNet [39] network for the purpose of more accurate segmentation. In experimental evaluation, we will compare the effects of different generator networks on the segmentation results. The purpose of this network is to obtain the final segmented result ISf of the original input IY, which also is the result of the LGE segmentation. Therefore, the generated prediction obtained from the multicascade pix2pix segmentation network can be denoted as
(5)ISf≔ISn=Gns∘Gn−1s∘⋯∘G1sIY.The discriminatorDs is a binary classifier based on pixels or patch-images which provides a network learning-based stopping criterion during generating. For the discriminator Ds in our multicascade pix2pix segmentation network, we employ a convolutional Patch-GAN [18] to distinguish real or fake between the prediction ISk and the ground truth IL, where ISk is divided into ℓ×ℓ patches with overlapping images, and each patch is discriminated with those of the ground truth IL, respectively; finally, a 2D probability map is obtained as the discriminator outputs.To train an optimal segmentation network, the measures betweenISk and target label IL can be estimated and minimized to update discriminator Ds that enforces to discriminate the generation and the ground truth. The segmentation network we propose is a conditional version of pix2pix GAN with the multicascade architecture, so the adversarial input in Ds is mainly composed of there components, where the first component is the source image IY used as the condition and the others are the generation ISk and the ground truth IL. At the same time, each generator Gks is also optimized to generate domain-invariant representations ISk that confuses the discriminator Ds.


### 3.2.2. Loss Functions in Segmentation Stage

The dice score and Jaccard index are commonly used as metrics for the evaluation of image segmentation task. CNNs trained for image segmentation task are usually optimized by minimizing a weighted cross-entropy. In this work, we employ a specially designed loss functionLs to simultaneously measure the generation similarity and the adversarial error, which contains three types of loss functions: adversarial loss Lgan, L1 loss, and perceptual loss Lvgg.The original adversarial loss (Vanilla GAN loss) is given by the Kullback-Leibler (KL) divergence score as(6)LganG1s,⋯,Gns,Ds=∑k=1nωkLgEIY,IL~pdIY,ILlogDsIY,IL+EIY~pdIYlog1−DsIY,ISk,where ωkLgk=1,⋯,n is the given weight enforcing the trade-off between the n cascade cross-entropy losses and IY is a condition input of each convolutional Patch-GAN in our multicascade pix2pix segmentation network. Recently, the most commonly used adversarial losses are WGAN-GP [40] and LSGAN [41]. In the next section, we will compare the performances of three different adversarial losses in our experiments.L1 loss is a weighted sum of the absolute distance between the calculated output data ISk in the k-th cascade block and the ground truth IL, which can make the segmentation results closer to the real results [18], and is defined by
(7)L1G1s,⋯,Gns=∑k=1nωkL1EIY~pdIYIL−ISk1,where ωkL1k=1,⋯,n are weight constants. Without loss of generality, we will take ωkLg=ωkL1 for all k=1,⋯,n in our experiments.Besides, we also employ the perceptual loss in our multicascade pix2pix segmentation network, which is composed of a pretrained VGG19 network and is first proposed in image super resolution application [42]. The perceptual loss focuses on feature maps between the output data and the ground truth [43]. It can hence be computed by
(8)LvggIL,G1s,⋯,Gns=∑k,i,j=1n,M,Ni∑p,q=1wij,hijSpqij,kwijhij,where Spqij,k=Dφi,jILpq−φi,jISkpq and φi,j represents the feature map of the j-th feature channel of the i-th feature layer (after activation) [42], Ni is the number of feature channels in the i-th feature layer and M is the number of convolution layers, and wij and hij represent the size of the feature map in the VGG19 network. Here, D is the error measure of the vgg/ResNet feature maps between the ground truth IL and prediction ISk. The most widely used feature distances also contain the manhattan distance Dmanh and the cosine similarity Dcosine difined by
(9)DmanhX,Y=∑X−Y1,DcosineX,Y=1−X,YX2Y2,where X and Y are feature maps.The total proposed segmentation model is trained by jointly minimizing the total lossLs for the three parts as follows:
(10)G1s,⋯,Gns,Ds∗=argminmaxG1s,⋯,Gns,DsLsG1s,⋯,Gns,Ds≕λlL1G1s,⋯,Gns+LganG1s,⋯,Gns,Ds+λvggLvggIL,G1s,⋯,Gns,where the λl and λvgg are two given weight parameters.


## 3.2.1. Multicascade Network

Our multicascade pix2pix segmentation network shown in Figure1(c) is based on the GAN architecture, which consists of multiple generators Gks (k=1,⋯,n) and a shared discriminator Ds.The generatorG1s: Y⟶S translates IY to IS1, where the original input IY∈Y is 1×256×256 real or fake bSSFP image, and the first generation IS1∈S is a prediction for the corresponding label. The other generators Gks: S⟶Sk=2,⋯,n furtherly improve the previous predicted probability ISk−1 to obtain more optimal prediction
(4)ISk≔Gks∘Gk−1s∘⋯∘G1sIY,where IY and ISk have the same size. In this work, Gks is formed by the U-net [5] or ResNet [39] network for the purpose of more accurate segmentation. In experimental evaluation, we will compare the effects of different generator networks on the segmentation results. The purpose of this network is to obtain the final segmented result ISf of the original input IY, which also is the result of the LGE segmentation. Therefore, the generated prediction obtained from the multicascade pix2pix segmentation network can be denoted as
(5)ISf≔ISn=Gns∘Gn−1s∘⋯∘G1sIY.The discriminatorDs is a binary classifier based on pixels or patch-images which provides a network learning-based stopping criterion during generating. For the discriminator Ds in our multicascade pix2pix segmentation network, we employ a convolutional Patch-GAN [18] to distinguish real or fake between the prediction ISk and the ground truth IL, where ISk is divided into ℓ×ℓ patches with overlapping images, and each patch is discriminated with those of the ground truth IL, respectively; finally, a 2D probability map is obtained as the discriminator outputs.To train an optimal segmentation network, the measures betweenISk and target label IL can be estimated and minimized to update discriminator Ds that enforces to discriminate the generation and the ground truth. The segmentation network we propose is a conditional version of pix2pix GAN with the multicascade architecture, so the adversarial input in Ds is mainly composed of there components, where the first component is the source image IY used as the condition and the others are the generation ISk and the ground truth IL. At the same time, each generator Gks is also optimized to generate domain-invariant representations ISk that confuses the discriminator Ds.


## 3.2.2. Loss Functions in Segmentation Stage

The dice score and Jaccard index are commonly used as metrics for the evaluation of image segmentation task. CNNs trained for image segmentation task are usually optimized by minimizing a weighted cross-entropy. In this work, we employ a specially designed loss functionLs to simultaneously measure the generation similarity and the adversarial error, which contains three types of loss functions: adversarial loss Lgan, L1 loss, and perceptual loss Lvgg.The original adversarial loss (Vanilla GAN loss) is given by the Kullback-Leibler (KL) divergence score as(6)LganG1s,⋯,Gns,Ds=∑k=1nωkLgEIY,IL~pdIY,ILlogDsIY,IL+EIY~pdIYlog1−DsIY,ISk,where ωkLgk=1,⋯,n is the given weight enforcing the trade-off between the n cascade cross-entropy losses and IY is a condition input of each convolutional Patch-GAN in our multicascade pix2pix segmentation network. Recently, the most commonly used adversarial losses are WGAN-GP [40] and LSGAN [41]. In the next section, we will compare the performances of three different adversarial losses in our experiments.L1 loss is a weighted sum of the absolute distance between the calculated output data ISk in the k-th cascade block and the ground truth IL, which can make the segmentation results closer to the real results [18], and is defined by
(7)L1G1s,⋯,Gns=∑k=1nωkL1EIY~pdIYIL−ISk1,where ωkL1k=1,⋯,n are weight constants. Without loss of generality, we will take ωkLg=ωkL1 for all k=1,⋯,n in our experiments.Besides, we also employ the perceptual loss in our multicascade pix2pix segmentation network, which is composed of a pretrained VGG19 network and is first proposed in image super resolution application [42]. The perceptual loss focuses on feature maps between the output data and the ground truth [43]. It can hence be computed by
(8)LvggIL,G1s,⋯,Gns=∑k,i,j=1n,M,Ni∑p,q=1wij,hijSpqij,kwijhij,where Spqij,k=Dφi,jILpq−φi,jISkpq and φi,j represents the feature map of the j-th feature channel of the i-th feature layer (after activation) [42], Ni is the number of feature channels in the i-th feature layer and M is the number of convolution layers, and wij and hij represent the size of the feature map in the VGG19 network. Here, D is the error measure of the vgg/ResNet feature maps between the ground truth IL and prediction ISk. The most widely used feature distances also contain the manhattan distance Dmanh and the cosine similarity Dcosine difined by
(9)DmanhX,Y=∑X−Y1,DcosineX,Y=1−X,YX2Y2,where X and Y are feature maps.The total proposed segmentation model is trained by jointly minimizing the total lossLs for the three parts as follows:
(10)G1s,⋯,Gns,Ds∗=argminmaxG1s,⋯,Gns,DsLsG1s,⋯,Gns,Ds≕λlL1G1s,⋯,Gns+LganG1s,⋯,Gns,Ds+λvggLvggIL,G1s,⋯,Gns,where the λl and λvgg are two given weight parameters.


## 4. Results and Discussion

The proposed CMMCSegNet framework is implemented using PyTorch. The experiments are conducted on a single GeForce RTX 2080Ti GPU with 11 GB RAM. To identify the model design, we performed several ablation experiments. They are described as follows.


### 4.1. Dataset and Experimental Setting

To demonstrate our CMMCSegNet framework, we use MS-CMRSeg 2019 datasets [1], which contain three different modalities: LGE with 45 patients but only 5 patients being labeled and bSSFP with 35 annotated patients and T2-weighted. The goal of CMR segmentation challenge is to achieve LGE image segmentation. Since there are fewer T2-weighted slices for each patient in the dataset (about 3-7 slices for each patient), we only use bSSFP modality and LGE modality in our experiments.The cross-modality translation network is trained for200 epochs, and the model that performs best on the validation set was selected for translation from LGE to bSSFP in the proposed CMMCSegNet framework. The dataset training the segmentation network contains two parts, most of them are from real annotated bSSFP images (slices from the 25 patients), and a small amount of fake bSSFP images are translated from the annotated LGE images (slices from about two patients) by the Cycle-GAN translation network.We also train200 epochs for the segmentation network. The both models are trained using Adam optimization with a minibatch size of 1, a decayed learning rate with an initial value 1.0e−2, the size nD=70 of patch D in the discriminator based on Patch-GAN, and the weight hyperparameters λ1=10, λgan=1, λl=100, and λvgg=1.


### 4.2. Performance of Cross-Modality Translation

We first use Cycle-GAN to achieve translation between LGE and bSSFP modalities; we also employ three evaluation metrics, including Structural Similarity (SSIM), Peak Signal To Noise Ratio (PSNR), and Mutual Information (MI), to evaluate the performance of Cycle-GAN translation network, which is tested on the whole LGE and bSSFP images. Many randomly chosen results from the translated (fake) LGE or bSSFP modalities are shown in Figure3. In Table 1, our translation model also leads to a comparable synthesis quality between LGE and bSSFP modalities for the whole datasets, where Ar, Af, Br, and Bf denote real LGE, fake LGE, real bSSFP, and fake bSSFP, respectively.Figure 3
Performance evaluations of Cycle-GAN cross-modality translation.
(a)
From top to bottom: original LGE, fake bSSFP, and reconstructed LGE(b)
From top to Bottom: original bSSFP, fake LGE, and reconstructed bSSFPTable 1
Performance evaluations of Cycle-GAN cross-modality translation.
TranslationEvaluationSSIMPSNRMIAr⟶Bf⟶AfAr,Af0.94429.5570.484Br⟶Af⟶BfBr,Bf0.96228.9110.493


### 4.3. Comparisons for Different Choices of Adversarial Loss and Perceptual Loss

After the cross-modality translation, two fake bSSFP patients with annotated masks (obtained from the cross-modality translation of two LGE patients with the ground truth) and fully real labeled bSSFP patients (35 patients) are used to train our proposed segmentation network. Next, we did several different comparison experiments for segmentation evaluations of fake bSSFP without annotated data (obtained from the cross-modality translation).Table2 shows the dice score of cardiac LGE segmentation in using different adversarial losses (Vanilla GAN, LSGAN, and WGAN-GP) and different CMMCSegNet generator blocks (U-net and ResNet) and with/without perceptual loss (Lmanh or Lcosine). We can also see that the overall segmentation performance of the U-net generator is slightly better than that of the ResNet generator using 6 different losses in terms of the LV (left ventricle), MYO (myocardium), and RV (right ventricle). For the U-net generator, the model using LSGAN loss yields better diagnostic performance than those of both Vanilla GAN and WGAN-GP losses. Besides, the Lmanh perceptual loss or Lcosine perceptual loss added for kernel feature comparisons can guarantee that the network learn relevant high feature levels and content features, which will improve the segmentation results for Vanilla GAN and LSGAN. However, the dice score of LV and RV segmentation slightly decreases when WGAN-GP with the Lmannh perceptual loss is used, while in the ResNet generation network, the models with the perceptual loss (Lmannh or Lcosine) achieve higher segmentation performance in all three terms and outperform those without the perceptual loss.Table 2
Indirect segmentation performance comparisons between CMMCSegNet models based on U-net and ResNet generator blocks using different training losses, where only one cascade generation block is used, “P-Lcosine” means to add cosine similarity perceptual loss into training loss, and “P-Lmanh” means to add Lmanh perceptual loss into training loss.
BlockLossLVMYORVU-netVanilla GAN loss0.8038±0.20680.7463±0.05340.8208±0.0304Vanilla GANloss+P‐Lcosine0.8575±0.11530.7580±0.05730.8459±0.0306Vanilla GANloss+P‐Lmanh0.8815±0.08330.7412±0.06300.8600±0.0279WGAN-GP loss0.8420±0.10680.7169±0.08140.8448±0.0584WGAN-GPloss+P‐Lcosine0.8162±0.11060.7485±0.04860.8563±0.0349WGAN-GPloss+P‐Lmanh0.8363±0.13930.7436±0.12040.8422±0.0779LSGAN loss0.8938±0.07750.7542±0.06010.8541±0.0267LSGANloss+P‐Lcosine0.9038±0.06190.7513±0.07390.8693±0.0287LSGANloss+P‐Lmanh0.9018±0.06740.7606±0.06450.8631±0.0362ResNetVanilla GAN loss0.8326±0.10630.7486±0.10190.8306±0.0771Vanilla GANloss+P‐Lcosine0.8762±0.08120.7612±0.08660.8524±0.0246Vanilla GANloss+P‐Lmanh0.8787±0.09770.7742±0.06870.8568±0.0210WGAN-GP loss0.7665±0.22100.7729±0.04980.8357±0.0299WGAN-GPloss+P‐Lcosine0.8158±0.13550.7504±0.08050.8710±0.0236WGAN-GPloss+P‐Lmanh0.8589±0.11370.7804±0.047810.8574±0.0210LSGAN loss0.8663±0.12650.7406±0.09820.8558±0.0888LSGANloss+P‐Lcosine0.8831±0.08990.7533±0.07750.8737±0.0289LSGANloss+P‐Lmanh0.8814±0.11090.7566±0.10650.8534±0.0966


### 4.4. Comparisons of the Cascade Generators

High-level semantic features in each branch contain sufficient localization information of corresponding region. To make full use of the features, we propose the multicascade architecture to extract implicitly geometrical and textural information that guides the cardiac segmentation. In order to enhance the competitiveness of the proposed architecture, we evaluate the performances by running a pix2pix segmentation on the training dataset (real/fake bSSFP images with ground truths). Final results are achieved with an ensemble of 1-4 cascades using corresponding LSGAN’s adversarial loss and perceptual loss. Comparisons with different number of cascades are shown in Table3; we can see that the number of cascades is increased from one to four and the dice values of some terms dropped slightly for the model with/without perceptual loss. The reason for this may be that the increase in the number of cascades may cause a lot of edge information to be lost in the original fake bSSFP images. As we can see from Figure 1, when the first segmentation network G1s obtains the segmentation result of the input fake bSSFP images, if original fake bSSFP image IY is not used as a conditional input in the later Gk+1s, modifying the previous result ISk, Gk+1s extracts fewer features comparing with the G1s. To optimize the computational costs, starting from the second generator, we reduce the number of upsampling/downsampling layers in the middle part of the U-net generators from (8,8,8) to (2,4,5) for generators (G2s,G3s,G4s), respectively. From Table 3, we observe that the proposed network with the simplified U-net versions can improve the segmentation results. Figure 4 shows the original LGE images, the translated bSSFP images, the corresponding ground truths, and the prediction results with varying the numbers of cascades.Table 3
Performance comparisons for the number of cascade generators on the multicascade pix2pix segmentation network, where “P-Lmanh” means to use Lmanh perceptual loss and “simple” means to use cascade generator with the simplified U-net version (where the number of upsampling/downsampling layers in the middle part of the U-net generators is reduced from (8, 8, 8) to (2, 4, 5) for generators (G2s,G3s, G4s), respectively).
Number of cascadesLVMYORV10.8938±0.07750.7542±0.06010.8541±0.0267+P-Lmanh0.9018±0.06740.7606±0.06450.8631±0.036220.8824±0.06320.7608±0.06470.8631±0.0273+P-Lmanh0.8929±0.08530.7690±0.06620.8831±0.0310+P-Lmanh+simple0.8966±0.08080.7732±0.07680.8841±0.029530.8527±0.06590.7202±0.09110.8248±0.0284+P-Lmanh0.8762±0.07690.7901±0.05380.8874±0.0175+P-Lmanh+simple0.8931±0.05570.7664±0.05200.8739±0.029940.8778±0.09050.7899±0.05450.8764±0.0177+P-Lmanh0.8944±0.07700.7561±0.07350.8815±0.0252+P-Lmanh+simple0.9019±0.06340.7690±0.07060.8911±0.0233Figure 4
Qualitative comparisons of our CMMCSegNet for different number of cascade blocks on fake bSSFP translated from LGE modality. From left to right: LGE, bSSFP translated by Cycle-GAN, ground truth with zoom-in views, and prediction results with zoom-in views using 1-4 cascade blocks.


### 4.5. Comparisons of the WeightsωkLg and ωkL1 of Multicascade Blocks

The performance of the multicascade architecture may be directly limited by the loss weight parameter of each cascade generatorGks. We compare the choice of the weights ωkLg and ωkL1, and ISk represents the output of the k-th generator Gks. From Table 4, the model with LSGAN adversarial loss and vgg perceptual loss is optimized solely using loss weights ω1L1,ω2L1,ω3L1=ω1Lg,ω2Lg,ω3Lg=1/3,1/2,1/6 and achieves the better results on the evaluation dice of IS2. Due to the efficiency of the multicascade technique, the proposed segmentation network automatically improves image multilevel features that benefits the segmentation performance. Figure 5 shows the results of different generators in a multicascade pix2pix network with different weights; G2s can further modify the details of IS1 making the output result closer to ground truth.Table 4
Performance comparisons of indirect segmentation predictionISi with different generation loss weights and U-net block, where the LSGAN adversarial loss and vgg Lmanh perceptual loss are employed.
Loss weightISiLVMYORV1/3,1/3,1/3IS10.8792±0.08460.7855±0.06010.8580±0.0279IS20.8818±0.07790.7960±0.05740.8830+0.0215IS30.8762±0.07690.7901±0.05390.8874±0.01761/6,1/3,1/2IS10.8860±0.07640.7569±0.08490.8532±0.0384IS20.8881±0.06830.7668±0.07910.8460±0.0326IS30.8631±0.08460.7661±0.08950.8535±0.037712,13,16IS10.9001±0.07140.7332±0.09320.8481±0.0369IS20.9039±0.06610.7472±0.08790.8669±0.0344IS30.8973±0.06860.7308±0.08470.8640±0.034613,12,16IS10.9037±0.06330.7410±0.07990.8579±0.0283IS20.9061±0.05930.7459±0.08530.8733±0.0273IS30.8726±0.05850.7204±0.08880.8513±0.0214Figure 5
Comparisons of the loss weights of different cascade predictionISk in proposed CMMCSegNet, where the LSGAN adversarial loss and vgg Lmanh perceptual loss are employed and the loss weight parameters ω1L1,ω2L1,ω3L1=ω1Lg,ω2Lg,ω3Lg=1/3,1/2,1/6 are manually given. (a) the LGE image and the corresponding ground truth; (b) from left to right: the predicted results ISk under different loss weights 1/3,1/3,1/3, 1/6,1/3,1/2, 1/2,1/3,1/6, and 1/3,1/2,1/6, from top to bottom: IS1,IS2,IS3.
(a)
LGE, fake bssFP, and ground truth(b)
CMMCSegNet predictionsISk in different loss weights ωkLg=ωkL1k=1,2,3


### 4.6. Comparison to Conventional Methods

Table5 benchmarks the performance of the proposed framework against the direct and indirect LGE segmentation networks. First, we compare the performance of the four direct segmentation methods, including FCNs [4], U-net [5], U-net++ [44], and Attention U-net [45] networks by directly training a segmentation network from a small number of annotated LGE images. As reported in Table 5, although U-net performs better than others, it produces low dice value. Figure 6(a) visualizes the segmentation results by direct methods. We also compare the performance of the five indirect segmentation methods, including FCNs, U-net, U-net++, and Attention U-net networks and the proposed CMMCSegNet by indirectly training networks from a small number of annotated fake bSSFP images and fully real bSSFP annotated images. As shown in Table 5, the proposed technique provides the highest dice score of LV and MYO and the fair value in RV. This means that our proposed CMMCSegNet outperforms the other techniques. Figure 6(b) further illustrates a more detailed comparison between the proposed and other techniques; our proposed CMMCSegNet has obvious advantages that it is easier to learn the location information of the target area.Table 5
Performance comparisons between direct and indirect segmentation of LGE modality using different techniques, where the LSGAN adversarial loss and vggLmanh perceptual loss are employed and the loss weight parameters ω1L1,ω2L1,ω3L1=ω1Lg,ω2Lg,ω3Lg=1/3,1/2,1/6 are manually given.
StrategyNetworkLVMYORVDirectFCNs [4]0.4223±0.21240.4696±0.21740.6020±0.2908U-net [5]0.5746±0.30620.4475±0.25760.6876±0.2291U-net++ [44]0.5534±0.34670.4294±0.35410.6208±0.3098Attention U-net [45]0.6022±0.15960.4544±0.23760.6698±0.2328IndirectFCNs [4]0.7595±0.13210.6113±0.16890.8731±0.0677U-net [5]0.8505±0.09910.7708±0.07650.9151±0.0441U-net++ [44]0.8459±0.10600.7470±0.10300.9163±0.0409Attention U-net [45]0.8438±0.10600.7593±0.09530.9133±0.0355CMMCSegNet0.8762±0.07690.7901±0.053860.8874±0.0176Figure 6
Qualitative comparisons of our CMMCSegNet with the other four state-of-the-art CNN-based segmentation methods, where the LSGAN adversarial loss and vggLmanh perceptual loss are employed and the loss weight parameters ω1L1,ω2L1,ω3L1=ω1Lg,ω2Lg,ω3Lg=1/3,1/2,1/6 is manually given. (a) Direct segmentation, from left to right: LGE, ground truths with zoom-in views and prediction results with zoom-in views using FCNs, U-net, U-net++, and Attention U-net for segmentation on real LGE modality; (b) indirect segmentation, from left to right: LGE, fake bSSFP, ground truths with zoom-in views, and prediction results with zoom-in views using FCNs, U-net, U-net++, Attention U-net, and our CMMCSegNet for segmentation on fake bSSFP modality translated from LGE modality.
(a)
Direct segmentation for the LGE modality(b)
Indirect segmentation for fake bSSFP translated from LGE modality


## 4.1. Dataset and Experimental Setting

To demonstrate our CMMCSegNet framework, we use MS-CMRSeg 2019 datasets [1], which contain three different modalities: LGE with 45 patients but only 5 patients being labeled and bSSFP with 35 annotated patients and T2-weighted. The goal of CMR segmentation challenge is to achieve LGE image segmentation. Since there are fewer T2-weighted slices for each patient in the dataset (about 3-7 slices for each patient), we only use bSSFP modality and LGE modality in our experiments.The cross-modality translation network is trained for200 epochs, and the model that performs best on the validation set was selected for translation from LGE to bSSFP in the proposed CMMCSegNet framework. The dataset training the segmentation network contains two parts, most of them are from real annotated bSSFP images (slices from the 25 patients), and a small amount of fake bSSFP images are translated from the annotated LGE images (slices from about two patients) by the Cycle-GAN translation network.We also train200 epochs for the segmentation network. The both models are trained using Adam optimization with a minibatch size of 1, a decayed learning rate with an initial value 1.0e−2, the size nD=70 of patch D in the discriminator based on Patch-GAN, and the weight hyperparameters λ1=10, λgan=1, λl=100, and λvgg=1.


## 4.2. Performance of Cross-Modality Translation

We first use Cycle-GAN to achieve translation between LGE and bSSFP modalities; we also employ three evaluation metrics, including Structural Similarity (SSIM), Peak Signal To Noise Ratio (PSNR), and Mutual Information (MI), to evaluate the performance of Cycle-GAN translation network, which is tested on the whole LGE and bSSFP images. Many randomly chosen results from the translated (fake) LGE or bSSFP modalities are shown in Figure3. In Table 1, our translation model also leads to a comparable synthesis quality between LGE and bSSFP modalities for the whole datasets, where Ar, Af, Br, and Bf denote real LGE, fake LGE, real bSSFP, and fake bSSFP, respectively.Figure 3
Performance evaluations of Cycle-GAN cross-modality translation.
(a)
From top to bottom: original LGE, fake bSSFP, and reconstructed LGE(b)
From top to Bottom: original bSSFP, fake LGE, and reconstructed bSSFPTable 1
Performance evaluations of Cycle-GAN cross-modality translation.
TranslationEvaluationSSIMPSNRMIAr⟶Bf⟶AfAr,Af0.94429.5570.484Br⟶Af⟶BfBr,Bf0.96228.9110.493


## 4.3. Comparisons for Different Choices of Adversarial Loss and Perceptual Loss

After the cross-modality translation, two fake bSSFP patients with annotated masks (obtained from the cross-modality translation of two LGE patients with the ground truth) and fully real labeled bSSFP patients (35 patients) are used to train our proposed segmentation network. Next, we did several different comparison experiments for segmentation evaluations of fake bSSFP without annotated data (obtained from the cross-modality translation).Table2 shows the dice score of cardiac LGE segmentation in using different adversarial losses (Vanilla GAN, LSGAN, and WGAN-GP) and different CMMCSegNet generator blocks (U-net and ResNet) and with/without perceptual loss (Lmanh or Lcosine). We can also see that the overall segmentation performance of the U-net generator is slightly better than that of the ResNet generator using 6 different losses in terms of the LV (left ventricle), MYO (myocardium), and RV (right ventricle). For the U-net generator, the model using LSGAN loss yields better diagnostic performance than those of both Vanilla GAN and WGAN-GP losses. Besides, the Lmanh perceptual loss or Lcosine perceptual loss added for kernel feature comparisons can guarantee that the network learn relevant high feature levels and content features, which will improve the segmentation results for Vanilla GAN and LSGAN. However, the dice score of LV and RV segmentation slightly decreases when WGAN-GP with the Lmannh perceptual loss is used, while in the ResNet generation network, the models with the perceptual loss (Lmannh or Lcosine) achieve higher segmentation performance in all three terms and outperform those without the perceptual loss.Table 2
Indirect segmentation performance comparisons between CMMCSegNet models based on U-net and ResNet generator blocks using different training losses, where only one cascade generation block is used, “P-Lcosine” means to add cosine similarity perceptual loss into training loss, and “P-Lmanh” means to add Lmanh perceptual loss into training loss.
BlockLossLVMYORVU-netVanilla GAN loss0.8038±0.20680.7463±0.05340.8208±0.0304Vanilla GANloss+P‐Lcosine0.8575±0.11530.7580±0.05730.8459±0.0306Vanilla GANloss+P‐Lmanh0.8815±0.08330.7412±0.06300.8600±0.0279WGAN-GP loss0.8420±0.10680.7169±0.08140.8448±0.0584WGAN-GPloss+P‐Lcosine0.8162±0.11060.7485±0.04860.8563±0.0349WGAN-GPloss+P‐Lmanh0.8363±0.13930.7436±0.12040.8422±0.0779LSGAN loss0.8938±0.07750.7542±0.06010.8541±0.0267LSGANloss+P‐Lcosine0.9038±0.06190.7513±0.07390.8693±0.0287LSGANloss+P‐Lmanh0.9018±0.06740.7606±0.06450.8631±0.0362ResNetVanilla GAN loss0.8326±0.10630.7486±0.10190.8306±0.0771Vanilla GANloss+P‐Lcosine0.8762±0.08120.7612±0.08660.8524±0.0246Vanilla GANloss+P‐Lmanh0.8787±0.09770.7742±0.06870.8568±0.0210WGAN-GP loss0.7665±0.22100.7729±0.04980.8357±0.0299WGAN-GPloss+P‐Lcosine0.8158±0.13550.7504±0.08050.8710±0.0236WGAN-GPloss+P‐Lmanh0.8589±0.11370.7804±0.047810.8574±0.0210LSGAN loss0.8663±0.12650.7406±0.09820.8558±0.0888LSGANloss+P‐Lcosine0.8831±0.08990.7533±0.07750.8737±0.0289LSGANloss+P‐Lmanh0.8814±0.11090.7566±0.10650.8534±0.0966


## 4.4. Comparisons of the Cascade Generators

High-level semantic features in each branch contain sufficient localization information of corresponding region. To make full use of the features, we propose the multicascade architecture to extract implicitly geometrical and textural information that guides the cardiac segmentation. In order to enhance the competitiveness of the proposed architecture, we evaluate the performances by running a pix2pix segmentation on the training dataset (real/fake bSSFP images with ground truths). Final results are achieved with an ensemble of 1-4 cascades using corresponding LSGAN’s adversarial loss and perceptual loss. Comparisons with different number of cascades are shown in Table3; we can see that the number of cascades is increased from one to four and the dice values of some terms dropped slightly for the model with/without perceptual loss. The reason for this may be that the increase in the number of cascades may cause a lot of edge information to be lost in the original fake bSSFP images. As we can see from Figure 1, when the first segmentation network G1s obtains the segmentation result of the input fake bSSFP images, if original fake bSSFP image IY is not used as a conditional input in the later Gk+1s, modifying the previous result ISk, Gk+1s extracts fewer features comparing with the G1s. To optimize the computational costs, starting from the second generator, we reduce the number of upsampling/downsampling layers in the middle part of the U-net generators from (8,8,8) to (2,4,5) for generators (G2s,G3s,G4s), respectively. From Table 3, we observe that the proposed network with the simplified U-net versions can improve the segmentation results. Figure 4 shows the original LGE images, the translated bSSFP images, the corresponding ground truths, and the prediction results with varying the numbers of cascades.Table 3
Performance comparisons for the number of cascade generators on the multicascade pix2pix segmentation network, where “P-Lmanh” means to use Lmanh perceptual loss and “simple” means to use cascade generator with the simplified U-net version (where the number of upsampling/downsampling layers in the middle part of the U-net generators is reduced from (8, 8, 8) to (2, 4, 5) for generators (G2s,G3s, G4s), respectively).
Number of cascadesLVMYORV10.8938±0.07750.7542±0.06010.8541±0.0267+P-Lmanh0.9018±0.06740.7606±0.06450.8631±0.036220.8824±0.06320.7608±0.06470.8631±0.0273+P-Lmanh0.8929±0.08530.7690±0.06620.8831±0.0310+P-Lmanh+simple0.8966±0.08080.7732±0.07680.8841±0.029530.8527±0.06590.7202±0.09110.8248±0.0284+P-Lmanh0.8762±0.07690.7901±0.05380.8874±0.0175+P-Lmanh+simple0.8931±0.05570.7664±0.05200.8739±0.029940.8778±0.09050.7899±0.05450.8764±0.0177+P-Lmanh0.8944±0.07700.7561±0.07350.8815±0.0252+P-Lmanh+simple0.9019±0.06340.7690±0.07060.8911±0.0233Figure 4
Qualitative comparisons of our CMMCSegNet for different number of cascade blocks on fake bSSFP translated from LGE modality. From left to right: LGE, bSSFP translated by Cycle-GAN, ground truth with zoom-in views, and prediction results with zoom-in views using 1-4 cascade blocks.


## 4.5. Comparisons of the WeightsωkLg and ωkL1 of Multicascade Blocks

The performance of the multicascade architecture may be directly limited by the loss weight parameter of each cascade generatorGks. We compare the choice of the weights ωkLg and ωkL1, and ISk represents the output of the k-th generator Gks. From Table 4, the model with LSGAN adversarial loss and vgg perceptual loss is optimized solely using loss weights ω1L1,ω2L1,ω3L1=ω1Lg,ω2Lg,ω3Lg=1/3,1/2,1/6 and achieves the better results on the evaluation dice of IS2. Due to the efficiency of the multicascade technique, the proposed segmentation network automatically improves image multilevel features that benefits the segmentation performance. Figure 5 shows the results of different generators in a multicascade pix2pix network with different weights; G2s can further modify the details of IS1 making the output result closer to ground truth.Table 4
Performance comparisons of indirect segmentation predictionISi with different generation loss weights and U-net block, where the LSGAN adversarial loss and vgg Lmanh perceptual loss are employed.
Loss weightISiLVMYORV1/3,1/3,1/3IS10.8792±0.08460.7855±0.06010.8580±0.0279IS20.8818±0.07790.7960±0.05740.8830+0.0215IS30.8762±0.07690.7901±0.05390.8874±0.01761/6,1/3,1/2IS10.8860±0.07640.7569±0.08490.8532±0.0384IS20.8881±0.06830.7668±0.07910.8460±0.0326IS30.8631±0.08460.7661±0.08950.8535±0.037712,13,16IS10.9001±0.07140.7332±0.09320.8481±0.0369IS20.9039±0.06610.7472±0.08790.8669±0.0344IS30.8973±0.06860.7308±0.08470.8640±0.034613,12,16IS10.9037±0.06330.7410±0.07990.8579±0.0283IS20.9061±0.05930.7459±0.08530.8733±0.0273IS30.8726±0.05850.7204±0.08880.8513±0.0214Figure 5
Comparisons of the loss weights of different cascade predictionISk in proposed CMMCSegNet, where the LSGAN adversarial loss and vgg Lmanh perceptual loss are employed and the loss weight parameters ω1L1,ω2L1,ω3L1=ω1Lg,ω2Lg,ω3Lg=1/3,1/2,1/6 are manually given. (a) the LGE image and the corresponding ground truth; (b) from left to right: the predicted results ISk under different loss weights 1/3,1/3,1/3, 1/6,1/3,1/2, 1/2,1/3,1/6, and 1/3,1/2,1/6, from top to bottom: IS1,IS2,IS3.
(a)
LGE, fake bssFP, and ground truth(b)
CMMCSegNet predictionsISk in different loss weights ωkLg=ωkL1k=1,2,3


## 4.6. Comparison to Conventional Methods

Table5 benchmarks the performance of the proposed framework against the direct and indirect LGE segmentation networks. First, we compare the performance of the four direct segmentation methods, including FCNs [4], U-net [5], U-net++ [44], and Attention U-net [45] networks by directly training a segmentation network from a small number of annotated LGE images. As reported in Table 5, although U-net performs better than others, it produces low dice value. Figure 6(a) visualizes the segmentation results by direct methods. We also compare the performance of the five indirect segmentation methods, including FCNs, U-net, U-net++, and Attention U-net networks and the proposed CMMCSegNet by indirectly training networks from a small number of annotated fake bSSFP images and fully real bSSFP annotated images. As shown in Table 5, the proposed technique provides the highest dice score of LV and MYO and the fair value in RV. This means that our proposed CMMCSegNet outperforms the other techniques. Figure 6(b) further illustrates a more detailed comparison between the proposed and other techniques; our proposed CMMCSegNet has obvious advantages that it is easier to learn the location information of the target area.Table 5
Performance comparisons between direct and indirect segmentation of LGE modality using different techniques, where the LSGAN adversarial loss and vggLmanh perceptual loss are employed and the loss weight parameters ω1L1,ω2L1,ω3L1=ω1Lg,ω2Lg,ω3Lg=1/3,1/2,1/6 are manually given.
StrategyNetworkLVMYORVDirectFCNs [4]0.4223±0.21240.4696±0.21740.6020±0.2908U-net [5]0.5746±0.30620.4475±0.25760.6876±0.2291U-net++ [44]0.5534±0.34670.4294±0.35410.6208±0.3098Attention U-net [45]0.6022±0.15960.4544±0.23760.6698±0.2328IndirectFCNs [4]0.7595±0.13210.6113±0.16890.8731±0.0677U-net [5]0.8505±0.09910.7708±0.07650.9151±0.0441U-net++ [44]0.8459±0.10600.7470±0.10300.9163±0.0409Attention U-net [45]0.8438±0.10600.7593±0.09530.9133±0.0355CMMCSegNet0.8762±0.07690.7901±0.053860.8874±0.0176Figure 6
Qualitative comparisons of our CMMCSegNet with the other four state-of-the-art CNN-based segmentation methods, where the LSGAN adversarial loss and vggLmanh perceptual loss are employed and the loss weight parameters ω1L1,ω2L1,ω3L1=ω1Lg,ω2Lg,ω3Lg=1/3,1/2,1/6 is manually given. (a) Direct segmentation, from left to right: LGE, ground truths with zoom-in views and prediction results with zoom-in views using FCNs, U-net, U-net++, and Attention U-net for segmentation on real LGE modality; (b) indirect segmentation, from left to right: LGE, fake bSSFP, ground truths with zoom-in views, and prediction results with zoom-in views using FCNs, U-net, U-net++, Attention U-net, and our CMMCSegNet for segmentation on fake bSSFP modality translated from LGE modality.
(a)
Direct segmentation for the LGE modality(b)
Indirect segmentation for fake bSSFP translated from LGE modality


## 5. Conclusion

In this work, we proposed a CMMCSegNet framework based on multimodal cardiac MR images for indirect LGE segmentation. Firstly, we utilized Cycle-GAN to translate LGE modality into bSSFP modality and then segmented the translated (fake) bSSFP images to achieve indirect segmentation of LGE images. The advantage of this method is that only a small number of annotated LGE images can be required to achieve accurate segmentation of LGE by employing many annotated bSSFP images. This indirection also solved the problem of LGE images itself having a low contrast. Compared with the direct segmentation of LGE images, the indirect segmentation method has better segmentation performance.For the multicascade pix2pix network, we regard the segmentation as a translation from image to ground truth; the purpose of multicascade architecture is to better improve the previous prediction through several generators. We also compared the use of different adversarial losses, the experimental results show LSGAN loss is better than the Vanilla GAN and WGAN-GP, and WGAN-GP loss is not significantly better than the Vanilla GAN loss. To improve the training effect of the model, the perceptual losses based onLmanh and Lcosine measures are also used to optimize the features of each feature layer. In addition, we investigated the influence of the weights of the generation loss of multicascade structures, where the optimal weight coefficient is set to (1/3,1/2,1/6) for 3 cascade generation networks.We also demonstrated the effectiveness of the proposed CMMCSegNet by comparing with FCNs, U-net, U-net++, and Attention U-net. In the future, we will consider the end-to-end segmentation method to segment the multimodal cardiac MR, combining the translation and segmentation together.



---

*Source: 9942149-2021-06-07.xml*