# Multimodal Tucker Decomposition for Gated RBM Inference
## 1 Introduction  
Feature engineering is the process of transforming raw data into a suitable representation or feature vector that can be used to train a machine learning model for a prediction problem. For decades, the traditional approach to feature engineering was to manually build a feature extractor that required careful engineering and considerable domain expertise. Manual feature engineering required writing code that was problem-specific and that had to be adjusted for each new dataset. Deep learning allows computational models that are composed of multiple processing layers to learn representation of data with multiple levels of abstraction [1] (p. 436). Deep learning improves the process of feature engineering by automatically extracting useful and interpretable features. It also eliminates the need for domain expertise and hard core feature extraction by learning high-level features from the data in a hierarchical manner. The main building blocks in the deep learning literature are restricted Boltzmann machines [2, 3], autoencoders [4, 5], convolutional neural networks [6, 7], and recurrent neural networks [8]. Most of these architectures are used to learn a relationship between a single input source and the corresponding output. However, there are many domains where the representation to be learned is the correspondence between more than one source and one output [9]. For instance, many tasks in vision carry the relevant information in the encoding of the relationship between observations, not the content of a single observation. The above deep learning building blocks can be extended to contain gated connections and allow them to learn relationships between at least two sources of input and at leastone output. A defining feature of the gated networks is that they contain gating connections. Unlike other networks, whose layer-to-layer connections are linear, gated networks introduce higher order interactions. The connection between two neurons \(x\) and \(y\) is in fact modulated by the activity of a third neuron \(h\). Figure 1 illustrates two different approaches for the connection relationship between three neurons: to control the flow of information in the network or to model multiplicative interactions between several inputs. In the first type of connection, the neuron \(h\) is used as a switch or a gate that stops or does not stop the flow of information between \(x\) and \(y\). In the second type of connection, the connection implements a multiplicative relationship between \(x\) and \(h\), whose values are multiplied before being projected to the output \(y\) by the synaptic connection. In the multiplicative interaction, we can say that the neuron \(h\) modulates the signal between \(x\) and \(y\). Despite the growing interest, the literature about gated networks is still sparse [9] (p. 2). The focus of this paper is the specific family of neural networks implementing a multiplicative gating relationship that are built on an RBM architecture. This concept of a gated restricted Boltzmann machine was first introduced in [10]. The basic idea of the gated model is to use the binary hidden units to learn the conditional distribution of one image (the input) given another image (the output). In [11], the authors revisit the problem and present a factorization alternative to the gated RBM. A gated RBM can be also considered a higher-order Boltzmann machine. As cited in [10] (p. 1474), Boltzmann machines that contain multiplicative interactions between more than two units are known in general as higher-order Boltzmann machines [12]. The remainder of this paper is organized as follows: in the following section, the RBM model and its popular training method Contrastive Divergence (CD) are presented. In Section 3, the standard gated RBM model is presented and a mechanism to reduce the number of its weights by projecting onto factor layers is discussed. Next, we present in Section 4 a small overview on the subject of tensors and the factorization known as Tucker decomposition.