# Synthesize and Segment: Towards Improved Catheter Segmentation via Adversarial Augmentation
## 3 Methods
### Synthesize: GAN Based X-ray Translation  
In this study, we achieve the task of synthesizing realistic X-ray from catheter masks using GANs. This can be considered as an image-to-image translation problem from real to synthetic using a generator \(G\) and a discriminator network \(D\) in an unsupervised setting. Figure 1: Overall framework of the proposed synthesis and segmentation of catheters in X-ray. The catheter sythesis is represented with red dotted line, while the catheter segmentation is represented with a green dotted line. Herein, \(G\) is trained to map random vectors \(z\in\mathcal{R}^{z}\) to a synthetic vector \(a=G(z)\), with \(D\) distinguishing real from synthetic samples. To achieve this, we employ a CycleGAN [15] architecture and augment the existing loss functions for realistic generation. Figure 1 (top) shows the framework. Following the formulation presented in [17], we consider \(X\) as the domain of X-ray images composited with catheter masks \(x_{i}\in X^{n}\) and \(Y\) as the domain of the original X-ray images \(y_{i}\in Y^{n}\) with no existing catheter annotations i.e., in a single video, we select frames that do not show an inserted catheters. In a similar fashion, the composite is obtained via \(x_{i}=\hat{c}_{i}\oplus y_{i}\), where \(\hat{c}_{i}\) is the binary mask highlighting the pixel location of the catheter. Following, we use \(x_{i}\) as input in \(G\). In the adversarial framework, mappings between \(G:X\to Y\) and \(F:Y\to X\) are learned for arbitrary unpaired samples i.e., both in forward and backward cycles. A residual network (ResNet) [39] with skip connections is used as the network \(G\) and a network [31] consisting of five convolutional layers followed by batch normalization and leaky ReLU is used as \(D\), respectively. The objective function of the forward cycle \(G:X\to Y\) and \(D_{Y}\) is formally expressed as:  
\[min_{G}max_{D_{Y}}\mathcal{L}_{adv}(G,D_{Y},X,Y)=\mathbb{E}[logD_{Y}(y)]+ \mathbb{E}[log(1-D_{Y}(G(x))]. \tag{1}\]  
On the other hand, for the backward cycle \(F:Y\to X\), the objective is \(min_{F}max_{D_{X}}\mathcal{L}_{adv}(F,D_{X},Y,X)\). In order to achieve cycle-consistency between the generated samples in both cycles, a consistency loss is employed following:  
\[\mathcal{L}_{cyc}(G,F)=\mathbb{E}[||F(G(x))-x||_{1}]+\mathbb{E}[||G(F(y)-y)|| _{1}], \tag{2}\]  
collectively, the loss is defined as  
\[\mathcal{L}(G,F,D_{X},D_{Y})=\mathcal{L}_{adv}(G,D_{Y},X,Y)+\mathcal{L}_{adv} (F,D_{X},Y,X)+\lambda\mathcal{L}_{cyc}(G,F). \tag{3}\]  
We further propose to include additional terms based on the structural similarity (SSIM) [40] and perceptual losses [41] in order to enforce consistency in semantic quality in both the later and earlier levels of the networks. Formally, SSIM for a pixel \(p\) is defined as  
\[SSIM(p)=\frac{2\mu_{x}\mu_{y}+C_{1}}{\mu_{x}^{2}+\mu_{y}^{2}+C_{1}}.\frac{2 \sigma_{xy}+C_{2}}{\sigma_{x}^{2}+\sigma_{y}^{2}+C_{2}},\mathcal{L}_{ssim}(P)= \frac{1}{N}\sum_{p\in P}1-SSIM(p), \tag{4}\]  
where \(\mu\) and \(\sigma\) represent the mean and standard deviation of the inputs that are computed using a Gaussian filter. Moreover, SSIM has been shown to enforce visual consistency in generated images. On the other hand, a perceptual loss enables the use of earlier level model features to improve learning. The loss is defined as the Euclidean distance between the feature maps of the original image \(X\) and the reconstructed image \(\hat{X}\). This loss is formulated as:  
\[\mathcal{L}_{prep}\theta^{ij}(X,\hat{X})=\frac{1}{H_{ij}W_{ij}}\sum_{x,y=1}^{ H_{ij}W_{ij}}\theta,\text{with }\theta=(\theta_{ij}(X)_{xy}-(\hat{X})_{xy})^{2}, \tag{5}\]  
where \(H_{ij}\) and \(W_{ij}\) represent the size of the feature map \(\theta\) for a particular layer in the pre-trained ResNet network. Herein, the final loss \(\mathcal{L}\) includes the GAN loss as well as the additional objectives that are computed between the forward and backward cycles.