## 3 Methodology
### Methodology and Autoencoder Formulation
#### 3.3.2 Hidden Layers  
As mentioned, CNNs are a particular network type used on two-dimensional image databases. The critical feature of convolutional neural networks and hence of convolutional autoencoders is the convolutional layer that gives the network its name. Convolution is the simple filter applied to an input that results in activation. Convolution is a linear operation that involves multiplying a set of weights with the input data using a two-dimensional array of weights called a filter or kernel. The filter is smaller than the input data, and the multiplication is performed using a dot product between the filter-sized patch of input and the filter. This systematic application of the same filter across an image allows the filter to detect a specific type of feature in the input, allowing it to discover that feature anywhere in the image. When a filter is multiplied with the input array, it produces a single value. A two-dimensional array of output values is obtained by repeatedly applying the same filter to different parts of the input, known as a feature map. The feature map represents a filtered version of the input [26]. The feature map implicitly depends on the learning model class used and on the input space where the data lies. Feature maps are produced using feature detectors or filters on either the input image or the feature map generated by the previous layers. These feature maps can provide useful information about the internal representations of the input for each Convolutional layer in the model. Visualising these feature maps can help gain insight into these representations. Convolutional layers also have a parameter which is called stride. The stride is the number of pixels the filter moves over the input array. When the step equals one, the filters are shifted by 1 pixel at a time. When we build a neural network, we need an activation function that takes the linear neuron output as input and generates a non-linear output based on it. The activation function can be a step transfer function, a linear transfer function, a non-linear transfer function or a stochastic transfer function. ReLU is one of the most widely used activation functions in neural networks today. It is usually added to some layers in neural networks to add nonlinearity, which is required to handle'today's complex and non-linear datasets. ReLU is more well-known than older activate functions, such as Sigmoid or Tanh, because it can be computed without a considerable cost, although it faces various problems when we use it. Its output is ReLU (x) = max (0, x). First, ReLU is not continuously differentiable. The gradient cannot be computed at x = 0, the breaking point between x and 0. Being unable to compute the gradient is not a big problem, but it can very lightly impact training performance. Second and graver, ReLU set all values < 0 to zero. This is beneficial regarding sparsity, as the network will adapt to ensure that the most critical neurons have values of >0. However, this is also a problem since the gradient of 0 is 0. Hence neurons arriving at large negative values cannot recover from being stuck at 0 [27]. What if we cause a small but significant leak of information to the left part of ReLU, i.e., where the output is always stuck to 0? The answer is the Leaky ReLU (rectified linear activation function), widely used in many machine learning applications. Specifically, it is an improvement of the traditional ReLU, and we recommend it be used more often. So, the activation function that we use is Leaky ReLU and is mathematically defined as f(x) = {0.01x if x < 0 or x otherwise}. Deep learning neural networks will likely quickly overfit a training dataset with few examples. This phenomenon happens when the model fits very well in the training dataset. Therefore, it becomes difficult for the model to adapt to new examples that do not belong to the training dataset. To make it more understandable, our model can recognise specific images from the training dataset, not general patterns.