## 2 Materials and Methods
### Convolutional Neural Networks  
Convolutional neural networks is one of the most successful deep neural network models, which are specifically designed to consider local information in the input data [PERSON] et al. (2016). Through their successful application, CNN models have been demonstrated capable of producing state-of-the-art results for data with either temporal or spatial relational structure [PERSON] et al. (2014). The implementation of a CNN model leads to the design choices of convolutional layers, the hidden layers, pooling layers, choice of activation functions and finally fully connected layers. In this work, we proposed two convolutional neural network models for RUS prediction as shown in Figures 7 and 8. Figure 7 shows the first CNN model. The first set of components in our CNN models are the convolutional layers themselves, which discover the inputs' encoded patterns and then produce a resulting feature map through a series of convolution operations. The mathematical operation of a convolutional filter is illustrated in Figure 9 below with respect to a one-dimensional input. After each convolution layer, a Rectified Linear Unit (ReLU) activation function is inserted, which maps the output from an input layer to a non-linearly transformed output. ReLU, standing for the rectified linear unit, can be defined as a function mapping input values to zero if negative or to the values themselves otherwise, as described in Equation (2). \[f(z)=max(0,z) \tag{2}\]  
Following the convolution layers, pooling layers were added. Through their non-linear down-sampling operations, pooling layers reduced the dimensions of the convolution layer outputs by either taking the average or the maximum values over various local regions within the provided input. Lastly, one or more fully connected layers composed of sequentially arranged neurons produced the predicted RUS output through multiple non-linear transformations. The first CNN model consists of three convolutional layers and two dense layers. The first convolutional layer used 32 filters of size 9, the second used 64 of size 7 and the last one used 256 of size 5. The number of neurons used by the two fully connected layers were 2304 and 512. Batch-normalization then ReLU activation functions followed each one of the convolutional layers. A drop-out with a rate of 0.3 was applied only to the third convolutional layer. The general training schema of our CNN models is represented in Figure 10. The loss function used in our CNN models is the root mean square error (_RMSE_), which is described in Equation (3):  
Figure 8: Architecture of the SqueezeNet based CNN model. Figure 7: Architecture of the basic convolutional neural network (CNN) model. Figure 9: Illustration of one-dimensional convolution filter operation. \[RMSE(y,\hat{y})=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}} \tag{3}\]  
where \(y\) defines the experimental RUS and \(\hat{y}\) the predicted RUS. In our second CNN model (Figure 8), we adapted the Squeezenet network [25], which is a deep neural network developed in 2016 with demonstrated success in many computer vision applications. Essentially, the Squeezenet network is composed of sequentially arranged convolutional layers, pooling operations, and blocks of convolutional layers. Each block, named a fire-block, consists of a total of 3 convolutional layers, which efficiently capture the local features of their input. As done in the first version, batch normalization and ReLU activation functions follow each of the convolutional layers within this network. Compared to the standard CNN model in Figure 7, the Squeezenet model has the benefits of a deep architecture for potentially increased accuracy power with significantly less parameters. Additionally, our choice for the Squeeze-Net architecture is also based on its relatively lower computation and parameter storage costs when compared to other more common deep neural architectures.