# Multimodal Tucker Decomposition for Gated RBM Inference
## 5 Materials and Methods
### Training the Refactored Gated RBM  
To train the refactored gated RBM, we can maximize its average conditional log-likelihood  
\[L=\sum_{\alpha}\log p(\mathbf{y}^{\alpha}\mid\mathbf{x}^{\alpha})\]  
for a set of training pairs \(\{(\mathbf{x}^{\alpha},\mathbf{y}^{\alpha})\}\). By substituting Equations (7) and (9), the derivative of the negative log probability with regard to any element \(\theta\) of the parameter tensor is given by  
\[-\frac{\partial L}{\partial\theta}=\left\langle\frac{\partial E(\mathbf{y}^{ \alpha},\mathbf{h};\mathbf{x}^{\alpha})}{\partial\theta}\right\rangle_{ \mathbf{h}}-\left\langle\frac{\partial E(\mathbf{y}^{\alpha},\mathbf{h}; \mathbf{x}^{\alpha})}{\partial\theta}\right\rangle_{\mathbf{h},\mathbf{y}} \tag{23}\]  
where \(\left\langle\cdot\right\rangle_{\mathbf{z}}\) denotes the average with regard to variable \(\mathbf{z}\). By substituting the reparametrized energy function from (21) into (23), we get  
\[-\frac{\partial L}{\partial\theta}=\left\langle\frac{\partial\boldsymbol{ \mathscr{G}}\times_{P}\mathbf{x}^{T}\mathbf{A}\times_{Q}\mathbf{y}^{T} \mathbf{B}\times_{R}\mathbf{h}^{T}\mathbf{C}}{\partial\theta}\right\rangle_{ \mathbf{h}}-\left\langle\frac{\partial\boldsymbol{\mathscr{G}}\times_{P} \mathbf{x}^{T}\mathbf{A}\times_{Q}\mathbf{y}^{T}\mathbf{B}\times_{R}\mathbf{h }^{T}\mathbf{C}}{\partial\theta}\right\rangle_{\mathbf{h},\mathbf{y}} \tag{24}\]  
Equation (24) calculates the derivative of the (negative) log probability with respect to any parameter in the refactored weights tensor: core tensor \(\boldsymbol{\mathscr{G}}\) and the three factor matrices \(\mathbf{A}\), \(\mathbf{B}\), and \(\mathbf{C}\). Similar to the unfactored gated RBM model presented in Section 3, note that the derivative of the (negative) log probability with respect to any parameter of the Tucker refactored gated RBM is given by the difference of two expectations. The first expectation in (24) is over the posterior distribution over the hidden units. On the other hand, the second expectation in (24) is over all possible output/hidden instantiations and is intractable. Note that the first term in (24) amounts to inferring the transformation (encoding) \(\mathbf{h}\) from a given pair of observed inputs \(\mathbf{x}\) and \(\mathbf{y}\) as considered in Equation (12). It is possible to plug the Tucker refactored energy function from (22) into (12) (bias term dropped for clarity), which becomes:  
\[\tilde{\mathbf{h}}=\sigma_{h}\big{(}\mathcal{G}\times_{I}\hat{\mathbf{x}} \times_{I}\hat{\mathbf{y}}\times_{K}\mathbf{C}\big{)}. \tag{25}\]  
In an analogous way, we may consider the task to compute \(p(\tilde{\mathbf{y}}\mid\mathbf{h},\mathbf{x})\) from an input image \(\mathbf{x}\) and a given fixed transformation \(\mathbf{h}\) considered in Equation (13). By plugging the Tucker decomposed energy function from (22) into (13) (bias term dropped for clarity), when input and hidden units are given, we get  
\[\tilde{\mathbf{y}}=\sigma_{y}\big{(}\mathcal{G}\times_{I}\hat{\mathbf{x}} \times_{K}\hat{\mathbf{h}}\times_{I}\mathbf{B}\big{)}. \tag{26}\]  
Let us now focus again on Equation (24). Note that the second term, also known as the _model expectation_, is an expectation over all possible instances of the output/mapping units and is intractable. However, similar to the bipartite structure in an RBM, the tripartite structure of a gated RBM facilitates Gibbs-sampling. With this in mind, we also consider the task to compute \(p(\tilde{\mathbf{x}}\mid\mathbf{y},\mathbf{h})\). \[\tilde{\mathbf{x}}=\sigma_{x}\big{(}\mathcal{G}\times_{I}\hat{\mathbf{y}} \times_{K}\hat{\mathbf{h}}\times_{I}\mathbf{A}\big{)}. \tag{27}\]  
Then, Gibbs sampling suggests itself as a way to approximate the intractable term in (24).