## 3 Proposed Methodology
### Details of the Neural Network  
In this study, the ACD problem is transformed into a multi-label multi-class sentence classification problem to test the performance of the representations proposed in Section 3.1. The goal is to assign single or multiple categories to the given input sentence vector \(\overrightarrow{x}\). The output of the classification algorithm is a vector \(\overrightarrow{y}\) with size equal to the number of the predefined categories. \(\overrightarrow{y}\) as one or more classes enabled for the given sentence. A rectified linear unit (ReLU) feed forward multi-layer neural network [33; 34] is used as a classifier. The input layer of the neural network takes a sentence feature vector as an input. The number of neurons that represent this layer must be equal to the length of the input vector. For this study, two hidden layers were used. The purpose of using two layers is to extract enough features to learn the weights from training examples. The hidden layer passes input features through non-linearities to perform linear transformations on the input feature vectors. ReLU is used as an activation function. ReLU is preferred because (1) it has low computational cost as compared to sigmoid or tanh functions as it does not require expensive operations like calculating exponential; and (2) ReLU has a fast convergence rate on stochastic gradient descent as compared to sigmoid and tanh functions. The Softmax regression is used as a cost function and is shown in Equation (11). \[L_{i}=-log\Bigg{(}\frac{e^{f_{y_{i}}}}{\sum_{j}e^{f_{j}}}\Bigg{)} \tag{11}\]  
After simplifying the above equation, we get:  
\[L_{i}=-f_{y_{i}}+log\Bigg{(}\sum_{j}e^{f_{j}}\Bigg{)} \tag{12}\]  
Softmax regression (or multinomial linear regression) is a generalization of logistic regression. In logistic regression, only binary labels: \(y_{i}\in\{0,1\}\) are allowed while Softmax regression allows more than two classes, \(y_{i}\in\{1,\ldots,k\}\), k is the number of classes and \(y\) is the output vector. To prevent neural network from over-fitting, \(L2\)-regularization is used. \(L2\)-regularization is one way to control over-fitting in neural networks. It penalizes the squared magnitude of all the neural network parameters except bias inputs, and then adds it in to the objective function, as shown in Equation (13):  
\[R(W)=\frac{1}{2}\lambda\left|w\right|^{2} \tag{13}\]  
\(\lambda\) is a regularization controlling factor to penalize weight. The final objective function became \(L_{i}+R(W)\) by using regularization. The weights of the network are randomly initialized by Gaussian distribution with standard deviation of \(\sqrt{2/n}\), where \(n\) is the number of inputs to the neuron layer. A stochastic method, Adam optimizer [35] is used to optimize the network weights. Mathematically, the optimizer can be written as:  
\[\left(m_{t}\right)_{i}=\beta_{1}(m_{t-1})+(1-\beta_{1})(\Delta L(W_{t}))_{i} \tag{14}\]  
\[\left(v_{t}\right)_{i}=\beta_{2}(v_{t-1})_{i}+(1-\beta_{2})(\Delta L(W_{t}))_{ i}^{2} \tag{15}\]  
\[\left(W_{t+1}\right)_{i}=\left(W_{t}\right)_{i}-\alpha\frac{\sqrt{1-(\beta_{2 })_{i}^{t}}}{1-(\beta_{1})_{i}^{t}}\frac{\left(m_{t}\right)_{j}}{\sqrt{\left(v _{t}\right)_{i}+\epsilon}} \tag{16}\]  
A hyper parameter called the learning rate (\(\alpha\)) is used to control the step size during each update of weights. An exponential decay method is used to update the net weights. This parameter automatically slows down the learning rate with the increase in the size of the epochs. A score function is used that takes three parameters that are the result vector \(x_{i}\in R^{D}\), weight matrix \(W\in[K\times D]\) and bias \(b\in[1\times K]\) as an input and return scores for all classes \(y_{i}\in[1\times K]\) as shown in Equations (17) and (18). \[\sigma(x)=1/\left(1+e^{(-x)}\right) \tag{17}\]  
\[f(x_{i},W,b)=\sigma(Wx_{i}+b) \tag{18}\]  
where example \(x_{i}\) varies, \(i=\{1,\ldots,n\}\), \(y_{i}\in\{1,\ldots,K\}\), \(D\) is the dimensions of the input vector and \(K\) is the number of classes.