# A Deep Belief Network Classification Approach for Automatic Diacritization of Arabic Text
## 5 Methodology
### Training with DBN
#### 5.4.1 Automatic Diacritization Using DBN  
The function of any classification algorithm is to transfer the input data \(X_{t}\) to a certain output \(Y_{t}\). For our DBN, the input layer \(v\) is the text samples, and the output of \(h_{n}\) is the feature learning results produced by DBN. Given that automatic diacritization of Arabic text is a sequence-to-sequence problem, the classification RBM at the top layer holds the diacritic marks (i.e., labels) thus it generates a sequence holding the corresponding label for each input data \(X_{t}\).  
As demonstrated in Figure 5, the binary representation of input text is received by the first RBM in the bottom layer (i.e., RBM1). RBM1 works on the received input text and learns hidden features. Next, RBM1 propagates what it has learned to the subsequent RBM on the next hidden layer, that is RBM2. RBM2 receives the output of RBM1 and treats it as its input. Now RBM2 is ready to learn hidden features and propagates data to the next layer. At each RBM layer, learned features of the previous layer is used as the input of the subsequent RBM layer with each higher layer representing higher-level abstraction of the input data. This greedy layer-wise training is conducted until reaching the highest hidden layer which is RBM3.  
Next, the second training stage begins, fine-tuning, which consists of fine-tuning the weights and supervised learning at the top layer. In this stage, a new layer is added on top of the DBN, specifically, after RBM3 and removes the links in the top to down direction. Labeled data are used to train the new layer, which acts as a classifier. As shown in Figure 5, the labels are provided on top and include the diacritic marks listed in Table 3. The backpropogation algorithm is used to learn the network weights and biases based on labeled training examples. The trained weights and biases that are fine-tuned in each layer correspond to features at different layers. Here, the learning goal is to minimize the classification error given the labeled examples. Since the features obtained in the unsupervised learning stage captures the properties of the data, they can be saved for future classifications. After the supervised learning and backpropagation to train a DBN for our classification problem, it is used to predict the diacritic mark for each character.  
Many sequence transcription problems, including diacritization, require the exploitation of future context. Future context can be exploited by maintaining long-term dependencies, however, not all deep learning models are capable to keep long-term dependencies and memorize them. Taking into account the potentials of DBNs to reconstruct input through layering of RBMs, each RBM transfers to the next layer what it has learned from the complete input and taking it to the next level. At each level, the RBM will dig deeper once it receives its input from the previous layer and reprocess it again to generate its own, consequently, full context will be covered once it reaches the top.  
Figure 5: A schematic representation of the designed DBN model for the automatic diacritization of Arabic text.