# Use of Generative Adversarial Networks (GAN) for
## 2 Methods and Samples
### Phase 1: Parameter Selection and Model Protocol
It is supposed to work better in deep architectures (>40 layers) because of the problems in correctly activating deep layers. Here, we will compare the performance of both functions for BSM image analysis. \"Mish\" is another recent function innovation. The formula is a little more complex:  
\(f(x)=x\cdot tanh(c(x))\) where, \(c(x)=\ln(1+e^{x}x)\). This latter is the softplus activation function [23]. \"Mish\" is not limited in its upper slope values, which avoids saturation and deals efficiently with zero-gradient problems. In contrast, the slope is bounded at the bottom, which enables better performance at avoiding overfitting during training. It has been argued that this function outperforms \"relu\" and \"swish\" [23]. We tried alternative functions, such as \"tanh\" and \"LeakyReLU\" and the results were not as good as with \"relu\", \"swish\" and \"mish\". The second parameter that we will analyze and compare is the optimizer type. Optimizers try to minimize the loss function by tweaking weight parameters. In the case of the optimizers, the responsibility of the analyst is bigger than in the selection of the activation functions, because optimizers need to be tuned. For example, convergence to an optimum depends on how fast or slow gradients operate. This is why selection of the learning rate is important. Regularization can also be implemented to avoid overfitting. Other parameters may also be selected. Gradients (which are partial derivatives connecting the loss function to the weights) show any minor or major impact that any modification of parameters does to the weights. The goal is to reduce the loss function through gradient descent. One of the most widely used optimizers is stochastic gradient descent (SGD). This optimizer uses only batches or random selection of cases in the original training sample, instead of all the cases contained therein. It is used in combination with another technique called \"momentum\", which consists of accumulating the gradient values of past steps along the descent trajectory in order to determine directionality. It is initiated at 0.5 and it increasingly must reach 0.9 over subsequent iterations. Here, we will use SGD as the baseline optimizer. The default parameters were: Learning rate = 1 \(\times\) 10\({}^{-3}\); momentum = 0.9. Adam (adaptive moment estimation) has been proposed as another of the most efficient optimizers [24]. It takes another optimizer (Adagrad, adaptive gradient algorithm) as a baseline and modifies it by scaling the learning rate instead of averaging it. It maintains an exponentially decreasing average of previous gradients. The algorithm implements an exponential modification of the average of the moving gradients and squares the gradient. We selected the VGG16 and Jason 2 models and tested them with alternative optimizers: SGD, Adam, Adagrad, Adadelta, and RMSProp. Given that the best results were obtained with SGD and Adam, we decided to use these two optimizers as competitors in combination with different activation functions. This created a network of combinationsincluding: four different DL architectures, three activations functions and two optimizer types. This resulted in 24 different combinations or models. Model evaluation was based on the joint consideration of accuracy, loss and F1-scores (Table 1). The latter was determinant given that it provides a good indication on how good balanced accuracy is. We considered the two best combinations for each model. The first one was labeled \"best\" and the second one as \"good\", with the remainder being labeled as \"normal\". This was applied to the six function-optimizer combinations of each model. To properly assess the importance of each function and optimizer, a Bayesian network was used.