# A Deep Belief Network Classification Approach for Automatic Diacritization of Arabic Text
## 3 Preliminaries
### Artificial Neural Networks and Deep Architectures
The training algorithm consists of two phases [PERSON] and [PERSON] (1943); [PERSON] and [PERSON] (1943). In the first phase, known as the forward phase, where weights are randomly initialized, and the input data are propagated forward through the network, layer-wise. The forward phase terminates after computing the output error [PERSON] and [PERSON] (1943). In the second phase, called the backward phase, the calculated error is propagated backward through the network. Then the network parameters are adjusted such that the error is minimized and the neural network learns the training data. Despite its popularity, this gradient-based training algorithm is highly subject to getting stuck into local minima [49], and is susceptible to parameter settings and to the network initial weights, biases and architecture [53]. Therefore, evolutionary algorithms were proposed by researchers as alternatives to gradient-based methods for training FFNNs. Evolutionary algorithms are proved to be more efficient in escaping from local optima for optimization problems. Evolutionary algorithms are appealing for ANN training since they maintain a population of solutions during search in a search space, enabling extreme exploration and massive parallelization. Evolutionary neural networks, or \"neuroevolution\", which utilize evolutionary algorithms to optimize neural networks have been used to train FFNNs [54]. One of the earliest methods for training FFNNs was using the Genetic Algorithm (GA) [55]. Results showed that GA excel the backpropagation algorithm when applied on a classification problem. This contribution of neuroevolution was followed by many other similar contributions that employ nature inspired algorithms. For example, in [56] [PERSON] et al. proposed a hybrid Particle Swarm Optimization (PSO) and Gravitational Search Algorithm (GSA) to train FFNNs. Ant Colony Optimization (ACO) [57], Multiverse Optimizer (MVO) [58], Cuckoo Search (CS) [59], Artificial Bee Colony (ABC) [60], Grey Wolf Optimizer (GWO) [61], and many more nature inspired algorithms were adopted for the sake of training FFNNs and overcoming the flaws of backpropagation. Just as the human brain processes information in multiple stages of transformation that reflect a deep architecture of the organization of the brain cortex, neural networks have developed to act similarly. This is done via training deep multi-layer neural networks. The architecture of such multi-layered neural networks consists of neural nets with many hidden layers. In the case of a multi-layer neural network, depth corresponds to the number of (hidden and output) layers. A neural network with deep architecture is known as Deep Neural Networks (DNN), and training them is called as deep learning. Deep learning is a branch of artificial intelligence that attempts to develop the techniques that will allow computers to handle complex tasks such as recognition and prediction at high performance [62]. Deep learning is able to learn more abstract features in the higher levels of the representation as well as generalizing expressive data representations [63]. As mentioned earlier, deep learning usually occurs in two phases: first, unsupervised layer-wise training and second, supervised training. In the unsupervised phase, each layer is added and trained greedy manner. Each layer uses the representation learned by the previous layer as input that it tries to model and transform to a new and better representation.