## 3 Methodologies and Research Framework
### Deep Neural Networks and Fully Connected Network
One of the basic cost functions for this neural network is the gradient decent algorithm which always converges to the global minimum [34]. The main limitation is that, although it is optimized for linear regression, it has a difficulty to linearly separate an exclusive OR (XOR) gate [35]. However, this limitation could be solved by the concept of backpropagation [36; 37], and this algorithm became the origin of a DNN. A DNN in this research refers to the number of layers in the network that is derived from a simple neural network; in other words, the more layers between the input and the output layers, the deeper the network [38]. The basic DNN architecture is the FCN consisting of an input layer, output layer, and a hidden layer. There are two things to keep in mind to use this model, i.e. preventing the issues of vanishing gradient and overfitting. For this reason, finding and setting the best parameters is very important. Parameters here refer to the number of inputs, hidden layers, learning rate, epoch, etc. For our research, we decided to employ the FCN model with Adam optimizer and ReLus activation instead of gradient descent for improving training accuracy. The main computational advantage of our model based on HOG and FCN is that both HOG and FCN are simple matrix calculation-based, thus they are easily paralleled, and the GPU-based embedded toolkit with limited specifications and computations is easy to use. All the detailed mathematical concepts related to the employed FCN model are described as follows:  
* Input: \(X\in\mathbb{R}^{m\times 1,188}\)
* Output: \(Y\in[0,1]^{m\times 2}\)  
* ReLu activation function: \(\Omega_{i}=max(0,W_{i}x+h_{i})\)
* \(Y=\Theta\left(\Omega_{1}(...\Omega_{i}(X))\right)\)
* Softmax output layer: \(\Theta(\Omega_{i}^{j})=\frac{e^{X_{j}}}{\sum_{k=1}^{k}e^{X_{k}}}\), where \(i=1,...,3\) and is defined as the number of layers, and \(j=1,...,K\) and is defined as output classes
* Categorical cross-entropy objective function: \(\mathcal{H}(Y,Y^{\prime})=-\sum_{m}Y_{m}log(Y^{\prime}_{m})\), where \(m\) is the batch size, \(Y\) is an actual output, and \(Y^{\prime}\) is a predicted output
* Empirical loss function: \(\mathcal{L}=\frac{1}{m}\mathcal{H}(Y,Y^{\prime})\)Our goal was to achieve more than 95% training and testing accuracy with the FCN model. If this goal could not be achieved by the FCN model, Plan B was to employ any of the deep learning models such as CNNs, RNNs, etc. To see whether the FCN model achieved more than 95% accuracy, we started to train the model, which was implemented by open source software on the cloud platform. Hardware specifications for training the model are as follows:  
* Machine Name: n1-standard-4
* Virtual CPUs: 4
* Memory: 15 GB
* Hard Disk Drive = 200 GB  
In addition, the model was trained with hyperparameters as follows:  
* the number of inputs (features): 1188 images
* the number of hidden layers: 2
* the number of units in hidden layers: 1188 in the first hidden layer and 594 in the second hidden layer
* learning rate = 1 \(\times\) 10\({}^{-6}\)
* epoch = 10  
According to training and testing results shown in Table 2, the FCN model was employed for further steps as the training and testing accuracies achieved were 99.4% and 99.0%, respectively.