# Modified Conditional Restricted Boltzmann Machines for Query Recommendation in Digital Archives
## 3 RBMs and CRBMs  
### Restricted Boltzmann Machines  
An RBM is a two-layer undirected graphical model constructed with a binary visible layer and a binary hidden layer, as shown in Figure 1. RBMs are a kind of energy-based model (EBM) [27] that captures the dependencies between variables by associating a scalar energy to each configuration of the variables. The energy function of RBMs is given by  
\[E(\mathbf{v},\mathbf{h})=-\mathbf{h}^{\top}\mathbf{W}\mathbf{v}-\mathbf{b}_{ \mathbf{v}}{}^{\top}\mathbf{v}-\mathbf{b}_{\mathbf{h}}{}^{\top}\mathbf{h}, \tag{1}\]  
where \(\mathbf{v}\) is the visible layer variables, \(\mathbf{h}\) is the hidden layer variables, \(\mathbf{b}_{\mathbf{v}}\) is the bias of the visible layer, \(\mathbf{b}_{\mathbf{h}}\) is the bias of the hidden layer, and \(\mathbf{W}\) is the weight between \(\mathbf{v}\) and \(\mathbf{h}\). Every joint configuration \((\mathbf{v},\mathbf{h})\) has a probability of the form  
\[p(\mathbf{v},\mathbf{h})=\frac{1}{Z}\exp(-E(\mathbf{v},\mathbf{h})), \tag{2}\]  
where \(Z\) is the partition function given by enumerating over all possible configurations of \((\mathbf{v},\mathbf{h})\). This makes the energy function become the format of probability. \(Z\) is defined by the following formula:  
\[Z=\sum_{\mathbf{v},\mathbf{h}}\exp(-E(\mathbf{v},\mathbf{h})), \tag{3}\]The marginal probability \(p(\mathbf{v})\) could be calculated by summing all the hidden layer variables \(\mathbf{h}\) in \(p(\mathbf{v},\mathbf{h})\):  
\[p(\mathbf{v})=\frac{1}{Z}\sum_{h}\exp(-E(\mathbf{v},\mathbf{h})), \tag{4}\]  
Because hidden units are conditionally independent in RBMs, we have  
\[p(\mathbf{h}\mid\mathbf{v})=\prod_{i}p(h_{i}\mid\mathbf{v}), \tag{5}\]  
where \(h_{i}\) is the binary state of each unit. Then, by utilizing the joint probability in Equation (2), the marginal probability in Equation (4), and the independent rule in Equation (5), we could calculate the conditional probability of hidden variables given the visible variables:  
\[p(\mathbf{h}\mid\mathbf{v}) =p(\mathbf{h},\mathbf{v})/p(\mathbf{v})\] \[=\frac{\exp\left(\mathbf{h}^{\top}\mathbf{W}\mathbf{v}+\mathbf{b_ {v}}^{\top}\mathbf{v}+\mathbf{b_{h}}^{\top}\mathbf{h}\right)/Z}{\sum_{ \mathbf{h}}\exp\left(\mathbf{h}^{\top}\mathbf{W}\mathbf{v}+\mathbf{b_{v}}^{ \top}\mathbf{v}+\mathbf{b_{h}}^{\top}\mathbf{h}\right)/Z}\] \[=\frac{\prod_{j}\exp\left(h_{j}\mathbf{W}_{j}\mathbf{v}+b_{j}^{ \mathbf{h}}h_{j}\right)}{\prod_{j}\left(\sum_{h_{j}\in\{0,1\}}\exp\left(h_{j} \mathbf{W}_{j}\mathbf{v}+b_{j}^{\mathbf{h}}h_{j}\right)\right)}\] \[=\prod_{j}\frac{\exp\left(h_{j}\mathbf{W}_{j}\mathbf{v}+b_{j}^{ \mathbf{h}}h_{j}\right)}{1+\exp\left(b_{j}^{\mathbf{h}}+\mathbf{W}_{j}\mathbf{ v}\right)}, \tag{6}\]  
where \(b_{j}^{\mathbf{h}}\) is the bias of each hidden unit and \(b_{j}^{\mathbf{v}}\) is the bias of each visible unit. From Equation (6), we can see that in the forward inference phrase, to obtain the probability of activation of each unit in the hidden layer (which can also be considered the probability that one exists in each hidden unit), we just calculate  
\[p\left(h_{j}=1\mid\mathbf{v}\right) =\frac{\exp\left(b_{j}^{\mathbf{h}}+\mathbf{W}_{j}\mathbf{v} \right)}{1+\exp\left(b_{j}^{\mathbf{h}}+\mathbf{W}_{j}\mathbf{v}\right)}\] \[=sigmoid\left(b_{j}^{\mathbf{h}}+\mathbf{W}_{j}\mathbf{v}\right) \tag{7}\]  
Figure 1: A restricted Boltzmann machine with a binary visible layer and binary hidden layer.  
Since an RBM is symmetrical, we can calculate the conditional probability of visible variables given the hidden variables in the backward inference as follows:  
\[p(v_{k}=1\mid\mathbf{h})=sigmoid\Big{(}b_{k}^{\mathbf{v}}+\mathbf{h}^{\top} \mathbf{W}_{\cdot k}\Big{)} \tag{8}\]  
The conditional probabilities \(p\big{(}h_{j}=1\mid\mathbf{v}\big{)}\) and \(p(v_{k}=1\mid\mathbf{h})\) are used in the contrastive divergence (CD) training method of RBMs [28]. The details will be stated in Section 3.2.