# ODEM-GAN: An Object Deformation Enhancement Model Based on Generative Adversarial Networks
## 3 Approach
### Overview  
To facilitate the demonstration, we use the \(512\times 512\) resolution of the image as input, and only \(PF_{3}\) (the third of feature pyramid) of CBAF [22] is introduced; the other levels are the same. The resolution of \(PF_{3}\) is \(64\times 64\) and the channel is \(c=256\). Therefore, the shape of \(PF_{3}\) is \([b,64,64,256]\) (\(b\) is the batch size). As shown in the black-dashed region of Figure 3, the feature map division layer takes the feature map of \(PF_{3}\) as input and divides the feature map of \(PF_{3}\) into \(N_{fp}\) feature patches of size \(n*n\) according to the space coordinates. We set \(n=8\). Therefore, the shape of a feature patch is \([b,n,n,c]=[b,8,8,256]\), the number of feature patches for the \(PF_{3}\) level is \(N_{fp}=\frac{h_{3}}{n}\times\frac{w_{3}}{n}=\frac{64}{8}\times\frac{64}{8}=64\), and the output of the feature map division layer is \([b,N_{fp},n,n,c]=[b,64,8,8,256]\). After obtaining the output of the feature map division layer, the feature patches are used as input for the contextual merger layer to concatenate with the surrounding feature patches, as shown in the red-dashed region of Figure 3. We used a \(m*m=9\) (we set \(m=3\)) sliding window to obtain the contextual information of each feature patch; second, we concatenated the \(m*m\) feature patches obtained in the sliding window along their channel dimensions. The shape of the merged feature patch is \([b,n,n,c\times m*m]\); finally, the merged feature pieces are passed to a \(3\times 3\) convolutional layer with \(c_{f}=512\) filters. The output shape of the contextual merger layer is \([b\times N_{fp},n,n,c_{f}]=[b\times 64,8,8,512]\).  
A GAN is composed of a generator network and a discriminator network, and its structural diagram is shown in Figure 5[27]. The generator network (abbreviated as \(G\)) uses a given noise vector \(z\) (generally subject to uniform distribution or normal distribution) to generate data. The discriminator network (abbreviated as \(D\)) is used to discriminate whether it is generated data or real data. The former tries to generate more real data, while the latter tries to perfectly distinguish between real data and generated data. The two learn from each other in the adversarial process and make progress together until the data generated by \(G\) become perfect so that \(D\) cannot distinguish. We can consider \(D\) as a binary classifier, so the basic model of GAN can be represented by cross entropy. The formula is shown in Equation (1):  
\[\underset{G}{min}\underset{D}{max}\,V(D,G)=E_{x\sim P_{data}(x)}[logD(x)]+E_{ z\sim p_{z}(z)}[log(1-D(G(z)))], \tag{1}\]  
where \(probability of \(D\) on the generated data. Through the max-min game, \(G\) and \(D\) are trained alternately until \(D\) cannot distinguish between the real and fake data.  
Similar to traditional GAN, we propose that ODEM-GAN also consists of \(G\) and \(D\), which compete by adversarial loss and alternating training. However, the difference is that this paper adopts conditional GAN (conditional GAN, referred to as C-GAN); that is, the input of \(G\) not only has a noise vector \(z\) but also a given condition. Adding conditions to \(G\) cannot only generate samples under the conditions (specify the desired object that hard deformation samples are generated) but also reduce the training difficulty of GAN. For the overall structure of ODEM-GAN, this paper adopts a structure similar to that of BE-GAN [32]. The structural feature of BE-GAN is that \(D\) adopts the structure of the autoencoder, which is different from the data distribution of the learning samples of other GAN models. BE-GAN learns the distribution of the loss of the autoencoder through the Wasserstein adversarial loss.  
As shown in the blue-dashed region of Figure 6, the ODEM-GAN module includes a CBAF module, a GAN branch, and a classification and regression module for each level of the feature pyramid.