## 5 Machine Learning Framework for Dimension Optimization
### The Proposed DNN Framework
#### 5.2.4 DNN Training  
For the precise classification in the training to map the input of (33) into the desired output of (34), we need enough training samples that reflects many situations. Thus, we generate the training samples varying SNR, the feedback size of each user, and eigenvalues of the covariance matrices. In particular, we focus on randomly generating covariance matrices varying angular spread in (3) to learn the feature information of eigenvalues, i.e., channel statistics of the system model. With the fixed azimuth center angle in (3), the number of eigenvalues of covariance matrix increases as the angular spread increases. It also leads to variation of magnitude of the eigenvalues. Note that the number of groups and the number of users in each group are the system model parameters, so they are fixed in the DNN training.  
Then, we obtain the optimal dimensions of the problem \(\mathbf{P1}\) by numerical search for every channel realization and label the samples according to (34) with perspective of the given SNR, feedback size, and covariance matrices of the groups. For the proposed DNN training, we adopt cross entropy as the loss function in (32), which is widely used for multi-class classification of DNN architecture [20; 21; 22]. Thus, the loss function of the \(g\)-th group DNN model is given by  
\[loss(\mathbf{\theta}_{g}) =\frac{1}{\bar{n}}\sum_{j=1}^{n}L_{f}\left(\mathbf{\delta}_{j}^{g}, \mathbf{\sigma}_{j}^{g}\right)\] \[=-\frac{1}{\bar{n}}\sum_{j=1}^{n}\sum_{i=1}^{M}\left(\bar{\sigma} _{j,i}^{g}\log\sigma_{j,i}^{g}+(1-\bar{\sigma}_{j,i}^{g})\log(1-\sigma_{j,i}^{ g})\right), \tag{35}\]  
where \(\bar{n}\) is the batch-size of the training samples, and \(j\) denotes the \(j\)-th training sample; \(\bar{\sigma}_{j}^{g}\) and \(\mathbf{\sigma}_{j}^{g}\) are the desired output vector determined by \(s_{g}^{\star}\) and (34), and the predicted output in the output layer obtained by the Softmax activation function of the \(g\)-th group; \(\bar{\sigma}_{j,i}^{g}\) and \(\sigma_{j,i}^{g}\) denote the \(i\)-th elements of \(\mathbf{\theta}_{j}^{g}\) and \(\mathbf{\sigma}_{j}^{g}\), respectively.  
Figure 4: The proposed DNN framework.  
For our DNN model training, we divided the samples into the training and the validation sets with the ratio of 0.85 and 0.15, respectively. We use the Adam algorithm [32; 33] for training, which is the combined and updated version of AdaGrad [30] and RMSProp [31]. To prevent under-fitting and over-fitting, we set the maximum number of epochs to 3000, the batch-size to 1000, and initial learning rate to 0.001. The learning rate is adjusted by the drop factor of 0.5 for every 1000 epoch during the training. In addition, we used the \(L_{2}\)-regularization factor of 0.01, gradient decay factor of 0.98, and squared gradient decay factor of 0.99. To prevent the over-fitting, we also adopt an early stopping strategy, which terminates the training when the maximum number of epochs or validation patience criterion is reached. The validation patience is the number of times that the loss on the validation set is larger than or equal to the previously obtained smallest loss [33]. We adopt the validation check with every five epochs and validation patience criterion of eight.