# A Deep Belief Network Classification Approach for Automatic Diacritization of Arabic Text
## 3 Preliminaries
### Deep Belief Network (DBN)
#### 3.2.1 Restricted Boltzmann Machine (RBM)  
RBMs are probabilistic graphical models that can be interpreted as stochastic neural networks that can learn a probability distribution over its set of inputs. RBMs are a variant of Boltzmann machines, with the restriction that their neurons must form a bipartite graph. A bipartite graph is a graph whose vertices (V), can be divided into two independent sets, V\({}_{1}\) (visible units) and V\({}_{2}\) (hidden units), and every edge of the graph connects one vertex in V\({}_{1}\) to one vertex in V\({}_{2}\). These two sets may have a symmetric connection between them, and there are no connections between nodes within the same group [64] as shown in Figure 1.  
An ordinary RBM accepts binary-values for the visible and hidden units. This type of RBM is known as Bernoulli-Bernoulli RBM. The Bernoulli distribution is a discrete distribution having two possible outcomes labelled by \(n=0\) and \(n=1\). If \(n=1\) it means that the true value occurs with probability p and when n = 0 it means the false case occurs with probability \(q=1-p\), where \(0<p<1\)[65]. RBM is an energy-based model, it consists of n visible units and m hidden units, vector \(v\) and \(h\) represent the state of visible and hidden units respectively. Given a set of state (\(v\), \(h\)), the energy of an RBM system is defined as [66]:  
\[E(v,h)=-\sum_{i=1}^{n}a_{i}v_{i}-\sum_{j=1}^{m}b_{j}h_{j}-\sum_{i=1}^{n}\sum_{ j=1}^{m}v_{i}W_{ij}h_{j}, \tag{1}\]  
where \(v_{i}\) represents the state of the \(i\)th visible unit, and \(h_{j}\) represents the state of the \(j\)th hidden unit. \(W_{ij}\) represents connection weights of visible and hidden units. There are also bias weights (offsets) \(a_{i}\) for the visible units and \(b_{j}\) for the hidden units. When the parameters are defined, we can find the joint probability distribution of (\(v\), \(h\)) in terms of the energy function as:  
\[P(v,h)=\frac{1}{Z}e^{-E(v,h)} \tag{2}\]  
\[Z=\sum_{v,h}e^{-E(v,h)}, \tag{3}\]  
where \(Z\) is a normalization constant. When the state of a visible unit is given, the activation states of each of the hidden units are conditionally independent. Thereupon, the probability of activation of the \(j\)th hidden unit is:  
\[P(h_{j}=1|v)=\sigma\bigg{(}b_{j}+\sum_{iv_{j}}W_{ij}\bigg{)}, \tag{4}\]  
Figure 1: The structure of RBM. where \(\sigma(x)=1/(1+\mathrm{e}^{(-x)}\) is a logistics sigmoid activation function. Similarly, given a specific hidden state, h, the activation states of each visible unit are also conditionally independent and the probability of the \(i\)th visible units of v given h is obtained by:  
\[P(v_{i}=1|h)=\sigma\bigg{(}a_{i}+\sum_{j\mu_{j}}W_{ij}\bigg{)}. \tag{5}\]  
Differentiating a log-likelihood of training data with respect to W is computed as follows:  
\[\frac{\partial logp(v)}{\partial W_{ij}}=\bigg{\langle}v_{i\mu_{j}}\bigg{\rangle} _{data}-\bigg{\langle}v_{i\mu_{j}}\bigg{\rangle}_{model}, \tag{6}\]  
where \(\big{\langle}\cdot~{}\big{\rangle}_{data}\) and\(\big{\langle}\cdot~{}\big{\rangle}_{model}\) indicate expected values in the data or model distribution. The learning rules for weights of the network in the log-likelihood-based training data can be obtained as:  
\[\Delta W_{ij}=\epsilon\bigg{(}\bigg{\langle}v_{i\mu_{j}}\bigg{\rangle}_{data }-\bigg{\langle}v_{i\mu_{j}}\bigg{\rangle}_{model}\bigg{)}, \tag{7}\]  
where \(\epsilon\) is the learning rate. Since there are no direct connections in the hidden layer of an RBM, we can get an unbiased sample of \(\big{\langle}\mathrm{v}_{i}h_{j}~{}\big{\rangle}_{data}\) easily. Unfortunately, it is difficult to compute an unbiased sample of \(\big{\langle}\mathrm{v}_{i}h_{j}~{}\big{\rangle}_{model}\) since it requires exponential time. To avoid this problem, a fast learning algorithm, called Contrastive Divergence (CD) [67], is proposed by [PERSON] [68]. CD sets visible variables as training data. Then the binary states of hidden units are all computed in parallel using Equation (4).