# Multimodal Tucker Decomposition for Gated RBM Inference
## 3 Gated RBM  
Gated RBMs are a natural extension of RBMs, in which the gating idea is applied via the units of a hidden layer connecting/gating the neurons of two other layers (input and output). As in the RBM model, the gated RBM is governed by an energy function. [PERSON] and [PERSON] [10] propose using the following three-way energy function that captures all possible correlations among the components of the \(\mathbf{x}\) (input), \(\mathbf{y}\) (output), and \(\mathbf{h}\) (hidden) layers:  
\[-E(\mathbf{y},\mathbf{h};\mathbf{x})=\sum_{ijk}W_{ijk}x_{i}y_{j}h_{k} \tag{5}\]  
Figure 3: The CD-\(k\) learning procedure. To estimate \(\left\langle v_{i}h_{j}\right\rangle_{model}\), we initialize the visible units of the data and alternately sample the hidden and then the visible units. The observed value of \(v_{i}h_{j}\) at the \((N+1)\)st sample is used as the estimate. where \(i\), \(j\) and \(k\) index the units in the input, output and hidden layers, respectively; \(x_{i}\) is the binary state of input pixel \(i\); \(y_{j}\) is the binary state of output pixel \(j\); and \(h_{k}\) is the binary state of hidden unit \(k\). Figure 4 shows a fully connected gated RBM. The components \(W_{ijk}\) of its three-way interaction tensor connect units \(x_{i}\), \(y_{j}\) and \(h_{k}\) and learn to weight the importance of the possible correlations given some training data. This type of multiplicative interaction among the input, output, and hidden units leads to a type of higher-order Boltzmann machine that retains the computational benefits of RBMs, such as being amenable to contrastive divergence training and allowing for efficient inference schemes that use alternating Gibbs sampling [11] (p. 1474). In practice, to be able to model affine and not just linear dependencies, it is useful to add biases to the output and hidden units, which makes (5):  
\[-E(\mathbf{y},\mathbf{h};\mathbf{x})=\sum_{ijk}W_{ijk}x_{i}y_{j}h_{k}+\sum_{k} W_{k}^{h}h_{k}+\sum_{j}W_{j}^{y}y_{j} \tag{6}\]  
where the terms \(\sum_{k}W_{k}^{h}h_{k}\) and \(\sum_{j}W_{j}^{y}y_{j}\) are bias terms used to model the base rates of activity of the hidden and output units, respectively. In general, a higher-order Boltzmann machine can also contain bias terms for the input units, but following [11], we do not use these. The negative energy \(-E(\mathbf{y},\mathbf{h};\mathbf{x})\) captures the compatibility between the input, output and hidden units. As in the RBM model, we can use this energy function to define the joint distribution \(p(\mathbf{y},\mathbf{h}\mid\mathbf{x})\) over output and hidden variables by exponentiating and normalizing:  
\[p(\mathbf{y},\mathbf{h}\mid\mathbf{x})=\frac{1}{Z(\mathbf{x})}\exp(-E(\mathbf{ y},\mathbf{h};\mathbf{x})) \tag{7}\]  
where  
\[Z(\mathbf{x})=\sum_{\mathbf{h},\mathbf{y}}\exp(-E(\mathbf{y},\mathbf{h}; \mathbf{x})) \tag{8}\]  
is a normalizing constant, which depends on the input image \(\mathbf{x}\). To obtain the distribution over output images, given the input, we marginalize and get:  
\[p(\mathbf{y}\mid\mathbf{x})=\sum_{\mathbf{h}}p(\mathbf{y},\mathbf{h}\mid \mathbf{x}) \tag{9}\]  
This marginalization over the hidden units is known in the literature as _free energy_. Note that \(p(\mathbf{y},\mathbf{h}\mid\mathbf{x})\) or \(Z(\mathbf{x})\) cannot be computed exactly, since both contain sums over the exponentially large number of all possible instances of the hidden units and output units  
Figure 4: A fully connected multiplicative gated RBM. for \(Z(\mathbf{x})\). However, we do not actually need to compute any of these quantities to perform either inference or learning, as we shall see in the next section. It is important to note that the normalization step in (8) is performed over \(\mathbf{h}\) and \(\mathbf{y}\); thus, it defines the conditional distribution \(p(\mathbf{y},\mathbf{h}\mid\mathbf{x})\) rather than the joint \(p(\mathbf{y},\mathbf{h},\mathbf{x})\).