# Internet Video Delivery Improved by Super-Resolution with GAN
## 4 Video Super-Resolution with GAN
### VSRGAN+ Architecture  
The adversarial network architecture includes a generator network \(G(f^{LR})\) and a discriminator network \(D(G(f^{LR}))\), which compete against each other during training. \(G(. )\) learns how to generate frames, \(f^{SR}\) that are indistinguishable from the ground truth frames, expecting to go undetected by \(D(.)\). \(D(. )\) learns how to distinguish the generated frames from the ground-truth frames. In other words, adversarial training works to balance these two dynamics. In [51], the authors classified adversarial networks as a min-max problem. In this work, we use this classification to define an adversarial network for video super-resolution as follows:  
\[ \begin{split}\min_{\theta_{G}}\ \max_{\theta_{D}}\ V(D_{\theta_{D}},G_{ \theta_{G}})=\\ &\begin{split} f^{HR}\sim p_{train}(f^{HR})\big{[}\log D _{\theta_{D}}(f^{HR})\big{]}+\\ f^{LR}\sim p_{G}(f^{LR})\big{[}\log\big{(}1-D_{\theta_{D}}(G_{ \theta_{G}}(f^{LR}))\big{)}\big{]},\end{split}\end{split}  \tag{2}\]  
where we train the discriminator network \(D_{\theta_{D}}\) to maximize the probability of its outcomes correctly classifying both frames: the ground-truth and the super-resolved frames. The generator network \(G_{\theta_{G}}\) learns how to generate more realistic frames \(f^{SR}\) by tuning the parameters \(\theta_{G}\) to minimize \(\log\big{(}1-D_{\theta_{D}}(G_{\theta_{G}}(f^{LR}))\big{)}\), where \(\theta_{G}=\{W_{1:L};b_{1:L}\}\). \(W\) are the weights, and \(b\) is the bias of the \(L\)-layer neural network that is optimized by a loss function \(L_{G}\) during training. During the generator network training sessions, we use a set of frames, \(f_{i}^{IR}\), encoded at a low bitrate, and their counterparts \(f_{i}^{HR}\), subject to the minimization of the following loss function. \[\theta_{G}=\arg\min_{\theta_{G}}\frac{1}{N}\sum_{i=1}^{N}L_{G}\Big{(}G_{\theta_ {G}}(f_{i}^{LR}),f_{i}^{HR}\Big{)} \tag{3}\]  
where \(L_{G}\) is the generic function that computes the loss between the super-resolved image and the ground truth image; in Section 4.2, this loss function is presented in detail. The generator network has a pre-residual block (part 1) containing a convolution layer (k9n64s1)--namely, 64 filters sized \(9\times 9\) and one stride, and the PreLu activation function [52]. The core of the generator network (part 2) has \(B\) residual-in-residual dense blocks (RRDBs) [22] with dense skip connections. Each RRDB has five convolution layers, with this k3n64s1 setup followed by the activation LeakyReLU [53]. At the output of each RRDB, there is a residual scaling \(\beta\)[54]. After the RRDBs, there is a residual scaling value \(\beta\), a skip connection from the pre-residual block, and a convolution layer set as k3n64s1, followed by another pre-residual skip connection. For upscaling resolution, \(G(. )\) has \(\log_{2}(r)\) blocks (part 3), where \(r\) is the scaling factor, which is a multiple of 2. Each block has a convolution layer set as k3n256s1, followed by a SubpixelConv2D and PReLU [32]. Finally, the last part of \(G(. )\) has two convolution layers, set as k9n64s1 and k9n3s1. The discriminator has two parts, \(C\) (part 4) and RaD (part 5). \(C\) consists of convolution k3n64s1, followed by an activation LeakyReLU with \(\alpha=0.2\). The core of \(C\) has seven blocks composed of a convolution layer, a BN layer, and LeakyReLU activation with \(\alpha=0.2\). The convolution layer of the first block consists of k3n64s2. The other blocks conduct convolution with three sizes of filters; the number of filters varies for each pair of blocks in 128, 256, and 512, with the first pair having one stride and the second having two strides. The final part of \(C\) includes a dense layer with 1024 neurons, LeakyReLU activation, dropout of 40%, and a dense layer with only one neuron. Generally, a standard discriminator, \(D\), is used to calculate the probability of a realistic frame.