## 2 Materials and Methods
### Model  
Figure 2 summarizes the framework of our network. It only contains fully connected network with \(L\) layers each with \(N_{l}\) units, where \(h_{1,l}\) represents the hidden vector in layer \(l\) for the first twin,  
Figure 1: Selecting voxels with a classification network of five fully connected layers. In order to choose those voxels that are maximally correlated with the visual images during training. Figure 2: A fully connected architecture selected for reconstruction task. The twin networks join immediately after the 512 unit fully-connected layer where the L2 component-wise distance between vectors is computed. and \(h_{2,l}\) denotes the same for the second twin. We use exclusively rectified linear units (ReLU) in all layers. This network consists of twin networks that accept distinct inputs but they are joined by an energy function at the last but one layer and with shared weight matrices at each layer. Additionally, the output of the last layer is measured the similarity of the original images by L2 component-wise distance. In addition, the detailed network architectures are shown as in Table 1. Let FC (K) denote the fully connected network with K filters. **Loss Function. ** In this paper, we compute two losses. One is to compute L2 component-wise distance between high dimensional features as the Siamese loss, the other is to compute L2 component-wise distance between reconstructed images and original images as MSE loss. **Siamese Loss. ** From the last but one layer, the anchor fMRI data and target fMRI data are referenced from the distinct data batches. Subsequently, we calculate the embedding of them. The embedding is represented by \(f(x)\in R^{d}\). It embeds the voxel responses x into a \(d\)-dimensional Euclidean space. Here, we want to ensure that if the high dimension feature \(f(x_{i}^{a})\) (_anchor_) and \(f(x_{i}^{t})\) (_target_) of a specific voxels vector from same classes are closer than those from different classes. Figure 3 visualizes this. Let \(M\) represent the minibatch size, where \(i\) indexes the \(i\)th minibatch. Now, let \(y(x_{i}^{a},x_{i}^{t})\) be a length \(M\) vector, which contains the labels for the minibatch, where we assume \(y(x_{i}^{a},x_{i}^{t})=1\) whenever \(x^{a}\) and \(x_{t}\) are from the same character class and \(y(x_{i}^{a},x_{i}^{t})=0\) otherwise. The Siamese loss that is being minimized is then  
\[L_{s}(x_{i}^{a},x_{i}^{t})=y(x_{i}^{a},x_{i}^{t})\log p(x_{i}^{a},x_{i}^{t})+( 1-y(x_{i}^{a},x_{i}^{t}))\log p(1-y(x_{i}^{a},x_{i}^{t})) \tag{1}\]  
**MSE Loss. ** From the last layer, the anchor network outputs a batch of 784 dimensional vectors \(f(x_{i}^{a})\), which is compared with the original image \(y_{i}\), to make them more similar. The MSE loss where the L2 component-wise distance between vectors is computed that is being minimized is, then  
\[L_{m}=\sum_{i}^{N}\left[\left\|f(x_{i}^{a})-y_{i}\right\|_{2}^{2}\right] \tag{2}\] \begin{table}
\begin{tabular}{c} \hline \hline
**Anchor Network** \\ \hline FC(2048) + ReLU + Dropout \\ FC(1024) + ReLU + Dropout \\ FC(1024) + ReLU + Dropout \\ FC(512) + ReLU + Dropout \\ FC(784) + ReLU \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Best fully connected architecture selected for reconstruction task. Anchor data is fed to Anchor network. Siamese twin is not depicted, but joins immediately after the 512 unit fully-connected layer where the L2 component-wise distance between vectors is computed. Figure 3: The Siamese Loss minimizes the distance between an _anchor_ and a positive _target_, both of which coming from same class, and maximizes the distance between the an _anchor_ and a negative _target_ coming from the different classes. Subsequently, we combine the Siamess loss \(L_{s}\) and MSE loss \(L_{m}\) together, as follows. \(\alpha\) and \(\beta\) are the hyperparameters, that are the weighting factors for Siamese loss and MSE loss, respectively. \(w\) is the weight of network and \(\lambda\|w\|_{2}\) is the regularization term avoiding over fitting.