# Multimodal Tucker Decomposition for Gated RBM Inference
## 3 Gated RBM
This is done deliberately to free the model from many of the independence assumptions that a fully generative model would need to make, hence simplifying inference and learning [10]. Inference then consists of guessing the transformation, or equivalently its encoding \(\mathbf{h}\), from a given pair of observed images \(\mathbf{x}\) and \(\mathbf{y}\). Since the energy function does not contain interactions between any pairs of output units or pairs of hidden units, it is possible to derive a closed-form expression for \(p(h_{k}=1\mid\mathbf{x};\mathbf{y})\); this is the probability of a particular hidden unit when a input-output image pair is given:  
\[p(h_{k}=1\mid\mathbf{x};\mathbf{y})=\sigma\Big{(}\sum_{i}\sum_{j}W_{ijk}x_{i}y _{j}+w_{k}^{k}\Big{)} \tag{10}\]  
This is also true for the output units when input and hidden units are given:  
\[p(y_{j}=1\mid\mathbf{h};\mathbf{x})=\sigma\Big{(}\sum_{i}\sum_{k}W_{ijk}x_{i} h_{k}+w_{j}^{y}\Big{)} \tag{11}\]  
where \(\sigma(z)=1/(1+\exp(-z))\). Note that in practice, function \(\sigma(\cdot)\) can be changed for some other non-linear activation function. Regardless of the activation function, these models are called _bilinear_ because, if one input is held fixed, the output is linear in the other input. Consider now the task of predicting the hidden layer \(\mathbf{\tilde{h}}\) given the input \(\mathbf{x}\) and output \(\mathbf{y}\) units, in such multiplicative network, this consists in computing all the values \(\tilde{h}_{k}\) of \(\mathbf{\tilde{h}}\) using (10):  
\[\forall k,\tilde{h}_{k}=\sigma_{h}\Big{(}\sum_{i}\sum_{j}W_{ijk}x_{i}y_{j}+w_{ k}^{h}\Big{)} \tag{12}\]  
Alternatively, one may compute \(\mathbf{\tilde{y}}\) given the input \(\mathbf{x}\) and hidden \(\mathbf{h}\) units using (11):  
Figure 3:0  
[PERSON] and [PERSON] [10] point out that this type of three-way model can be interpreted as a mixture of experts. Note from (5) that in the way the energy is defined, the importance that each hidden unit \(h_{k}\) attributes to the correlatedness (or anti-correlatedness) of a particular pair \(x_{i},y_{j}\) is determined by \(W_{ijk}\). To train the probabilistic mode,l we can use the same principle from Contrastive Divergence in the RBM model: we maximize the average conditional log-likelihood \(L=\frac{1}{N}\sum_{a}\log p(\mathbf{y}^{a}\mid\mathbf{x}^{a})\) for a set of training pairs \(\{(\mathbf{x}^{a},\mathbf{y}^{a})\}\). The derivative of the (negative) log probability with respect to the weight parameter \(W_{ijk}\) is given by the difference of two expectations:  
Figure 3:1  
where \(\left\langle\cdot\right\rangle_{z}\) denotes the expectation with regard to variable \(z\). Note that the expectation in the first term in (14) is over the posterior distribution over hidden units, and it can be computed efficiently using (12). The expectation in the second term in (14) is over all possible output/hidden instantiations and is intractable. However, because of the conditional independences of \(\mathbf{h}\) given \(\mathbf{x}\) and \(\mathbf{y}\) and \(\mathbf{y}\) given \(\mathbf{x}\) and \(\mathbf{h}\), we can easily sample from the conditional distributions \(p(\mathbf{h}\mid\mathbf{x},\mathbf{y})\) and \(p(\mathbf{y}\mid\mathbf{x},\mathbf{h})\). Using Gibbs sampling with Equations (12) and (13), respectively, for the hidden and output layer, we can approximate the intractable term. Factorized Gated RBM  
[PERSON] and [PERSON] [11] propose a way of reducing the number of weights that consists of projecting the \(\mathbf{x}\), \(\mathbf{y}\), and \(\mathbf{h}\) layers onto smaller layers, noted, respectively, as \(\mathbf{f}^{x}\), \(\mathbf{f}^{y}\), and \(\mathbf{f}^{h}\) before performing the product between these smaller layers. Given their multiplicative role, these layers are called _factor_ layers.