## 5 Machine Learning Framework for Dimension Optimization
### Preliminary: The General DNN Architecture  
Our proposed machine learning-based dimension optimization utilizes a DNN [27], so, in this subsection, we briefly explain a general DNN model. A DNN is one of the most popular algorithms in machine learning, which is considered as a multiple layer perceptron (MLP) [20; 21; 22]. The DNN is comprised of one input layer, \(L-2\) hidden layers, and one output layer. The DNN with many hidden layers enhances its learning and mapping abilities, so it is capable of handling complicated nonlinear problems [20; 21; 22; 25; 26]. Let \(\textbf{w}\in\mathbb{R}^{N_{0}}\) be the input vector of the input layer and \(\textbf{o}\in\mathbb{R}^{N_{(L-1)}}\) be the output vector of the output layer. Then, the mapping between them can be mathematically represented as follows:  
\[\textbf{o}=f\left(\textbf{w};\boldsymbol{\theta}\right)=f^{(L-1)}(f^{(L-2)}( \cdots f^{(2)}(f^{(1)}(\textbf{w};\boldsymbol{\theta}_{l})))), \tag{31}\]  
where \(f^{(l)}(\textbf{w};\boldsymbol{\theta}_{l})\) for \(l\in\left\{1,2,...,L-1\right\}\) is an activation function of the \(l\)-th layer, which maps the input and output vectors of the \(l\)-th layer, and \(\boldsymbol{\theta}=\left\{\boldsymbol{\theta}_{1},...,\boldsymbol{\theta}_{( L-1)}\right\}\) is a set of parameters such as weights and biases to calculate the weighted sum of nodes, which can be adjusted during the training procedure [20; 21; 22]. Activation functions in (31) are prerequisite to tackle nonlinear problems in the DNN model. In particular, multiple nodes in each layer operate with the activate functions to map its input vector to the output vector with nonlinear operations. Various activation functions are listed in Table 1 and illustrated in Figure 3.  
A set of parameters in (31), i.e., \(\mathbf{\theta}\), is adjusted in the training procedure to minimize the loss function [20; 21; 22]. In supervised learning, each training sample is labeled by the desired output \(\mathbf{\theta}\) (i.e., the correct answer), and hence the loss function between the DNN-based model output \(\mathbf{\mathsf{o}}\) and the desired output \(\mathbf{\mathsf{\bar{o}}}\) becomes  
\[loss(\mathbf{\theta})=\frac{1}{\bar{n}}\sum_{j=1}^{n}L_{f}\left(\mathbf{\mathsf{\bar{o }}}_{j},\mathbf{\mathsf{o}}_{j}\right), \tag{32}\]  
where \(\mathbf{\mathsf{\bar{o}}}_{j}\) and \(\mathbf{\mathsf{o}}_{j}\) are the desired output and the predicted output from the \(j\)-th training sample, respectively. In addition, \(\bar{n}\) is the batch-size of training samples, where the batch-size means the total number of training samples in a mini batch. There are mainly two types of loss functions for supervised learning in a DNN [20; 21; 22] as shown in Table 2.  
For supervised learning, the ultimate goal of DNN design is to map \(\mathbf{\mathsf{w}}\) to \(\mathbf{\mathsf{\bar{o}}}\) from (31) based on training samples. Thus, training is the process of adjusting \(\mathbf{\theta}\) to minimize the loss function in (32). The widely used algorithm for training is a stochastic gradient descent (SGD) method [20; 21; 22; 24], which updates its gradient for each layer to minimize the loss function at each step, i.e., mini batch. In addition, there are many modified ones such as SGD with momentum (SGDM), adaptive gradient (AdaGrad), and root mean square propagation (RMSProp) [20; 21; 22; 30; 31; 32]. Adaptive moment estimation (Adam) is also widely used for DNN training. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Name** & **Sigmoid** & **tanh** & **ReLU** & **SSaLU** & **Softmax** \\ \hline \(f(w)\) & \(\frac{1}{1+e^{-w}}\) & \(\frac{1-e^{-w}}{1+e^{-w}}\) & \(\max(0,w)\) & \(\begin{cases}\max(-1,w)&w<0\\ \min(1,w)&w\geq 0\end{cases}\) & \(\frac{e^{w}}{\sum_{j}e^{w}}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Mathmatical expressions for activation functions.