# Intelligent Neural Network Schemes for Multi-Class Classification
## 2 Proposed Network Schemes
### Construction and Learning of RBF Networks  
Next, we describe how the M-RBF network is constructed. In a neural network application, one has to try many possible values of hyper-parameters and select the best configuration of hyper-parameters. Typically, hyper-parameters include the number of hidden layers, the number of nodes in each hidden layer, the activation functions involved, the learning algorithms used, etc. Several methods can be used to tune hyper-parameters, such as manual hyper-parameter tuning, grid search, random search, and Bayesian optimization. For MLPs and CNNs, many different activation functions can be used, e.g., symmetrical hard limit, linear, saturating linear, log-sigmoid, hyperbolic tangent sigmoid, positive linear, competitive, etc. The networks can be trained by many different learning algorithms. Also, the number of hidden layers and the number of hidden nodes can vary in a wide range. This may lead to a huge search space, and thus finding the best configuration of hyper-parameters from it is a very inefficient and tedious tuning process. Determining the hyper-parameters is comparatively easier for RBF networks. An RBF is a two-layer network, i.e., containing only one hidden layer. Clustering techniques, e.g., a self-constructing clustering algorithm, can be applied to determine the number of hidden neurons. There are several types of activation function that can be used, but the Gaussian function is the one most commonly used. RBF networks can be trained by the same learning techniques used in MLPs. However, they are commonly trained by a more efficient two-stage learning algorithm. In the first stage, the centers and deviations in the first layer are found by clustering. In the second stage, the weights and biases associated with the second layer are calculated by least squares. In summary, regarding the configuration of hyper-parameters, (1) the number of layers, i.e., two layers, the Gaussian basis function, the least squares method, and the activation functions associated with the output layer are the decisions taken, and (2) the number of neurons in the hidden layer is the customized parameter and is determined by clustering. Here, we describe how the multi-class RBF network, M-RBF, is constructed and trained. Firstly, the training instances are divided, by the iterative self-constructing clustering algorithm (SCC) [32] briefly summarized in the Appendix A, into \(J\) clusters each having center \(\mathbf{c}_{j}=\big{(}c_{1,j},c_{2,j},\ldots,c_{n,j}\big{)}\) and  
Figure 4: Diagram of multi-label (ML)-Scheme for multi-label classification. deviation \(\mathbf{v}_{j}=\left(v_{1,j},v_{2,j},\ldots,v_{n,j}\right)\), \(1\leq j\leq J\). Then, the hidden layer in M-RBF\({}_{i}\) has \(J\) hidden nodes. The basis function \(g_{j}(\mathbf{x})\) of node \(j\) in the hidden layer of M-RBF is the Gaussian function  
\[g_{j}(\mathbf{x})\ =\ \prod_{k=1}^{n}\exp\left[-\left(\frac{x_{k}-c_{k,j}}{v_{k,j} }\right)^{2}\right] \tag{6}\]  
for \(1\leq j\leq J\). Note that several different types of basis function can be used [29], but Gaussian is the one most commonly used in the neural network community. The settings for \(w_{k,0}\), \(w_{k,1}\),..., \(w_{k,J}\), \(1\leq k\leq m\), are optimally derived as follows. For training instance \(\left\{x_{i},\mathbf{y}_{i}\right\}\), \(1\leq i\leq N\), let  
\[y_{1,i} =\ w_{1,0}+o_{1}^{1}(\mathbf{x}_{i})w_{1,1}+\ldots+\ o_{j}^{1}( \mathbf{x}_{i})w_{1,J}, \tag{7}\] \[y_{2,i} =\ w_{2,0}+o_{1}^{1}(\mathbf{x}_{i})w_{2,1}+\ldots+\ o_{j}^{1}( \mathbf{x}_{i})w_{2,J},\] (8) \[\vdots\vdots;\] (9) \[y_{m,i} =\ w_{m,0}+o_{1}^{1}(\mathbf{x}_{i})w_{m,1}+\ldots+\ o_{j}^{1}( \mathbf{x}_{i})w_{m,J}. \tag{10}\]  
In this way, for each \(i\), \(1\leq i\leq m\), we have \(N\) equations, which are expressed as  
\[\mathbf{O}\mathbf{w}_{i}=\mathbf{d}_{i} \tag{11}\]  
with  
\[\mathbf{O}=\left[