Learning Paired-associate Images with An Unsupervised Deep Learning Architecture
Authors: Ti Wang, Daniel L. Silver
ICLR 2014

Abstract: This paper presents an unsupervised multi-modal learning system that learns associative representation from two input modalities (channels) such that input on one channel will correctly generate the associated response at the other channel and vice versa. In this way, the system develops a kind of supervised classification model meant to simulate aspects of human associative memory. The system uses a deep learning architecture (DLA) composed of two input/output channels formed from stacked Restricted Boltzmann Machines (RBM) and an associative memory network that combines the two channels. The DLA is trained on pairs of MNIST handwritten digit images to develop hierarchical features and associative representations that are able to reconstruct one image given its paired-associate. Experiments show that the multi-modal learning system generates models that are as accurate as back-propagation networks but with the advantage of unsupervised learning from either paired or non-paired training examples.


Peer review:
[Review]:
Summary:

This paper presents a neural network that predicts a structured output. The authors proposed a new algorithm for training this structure. The new algorithm firstly pretrains the neural network as a stack of RBMs, and subsequently finetunes the model by untying the recognition and generation weights (which is as authors mention reminiscent of wake-sleep algorithm.)

Novelty:

Unfortunately, I don't see any novelty in the proposed approach. The exactly same neural network was proposed earlier by Ngiam et al. (2011). Also, it is very close to the multi-modal DBM proposed by Srivastava & Salakhutdinov (2012). The authors may argue that Ngiam et al. (2011) did not attempt to actually generate a missing modality, but precisely that was done by S & S (2012). Furthermore, I don't think the proposed wake-sleep-type algorithm should be better than finetuning the whole model (or each path) as an (denoising) autoencoder. 

Pros:

(1) Representation learning from multiple modalities is important.

Cons:

(1) Experiments are weak.
(2) Explanation of the methods could be done better.
(3) The relationship to the previous research should be made more explicit and clear.

Detailed Comments:

- Sec 2. 1st paragraph: 'back-prop ANNs are ... not as good for reconstructing, or recalling a pattern' => I don't really understand this sentence. Does it mean that the backpropagation (or SGD with backprop) is not able to find a solution of a neural net even if the input and target were constructed explicitly to make the neural net reconstruct corruption or noisy input? 

- Sec 2.1, the end of the 1st paragraph: I don't think the last sentence is correct. First of all, what is the energy state? Isn't it a simple scalar corresponding to the log of unnormalized probability? Then, if two orthogonal patterns were in data, shouldn't their probabilities (or energy states) be close to each other regardless of their orthogonality?

- 'Weight is updated until the global energy E reduces below a threshold' What is the global energy? Do you mean the average of all training samples' energies?

- Sec. 2.1., the end of the 2nd paragraph: 'w_ij is equal to the probability of feature h_j given input v_i'. Why is it so? Shouldn't the probability of h_j given v_i require marginalizing out all v_j (j 
eq i), which probably is computational intractable and without any analytical form? 

- Sec. 2.2, the end of the third paragraph: I believe many people consider convent as one of deep learning methods as well.

- Sec. 3.1, 2nd paragraph: 'RBM is unable to recall patterns when only half of the visible neurons are given correct pattern values' I completely disagree with this sentence. It may highly depend on data as well as the model size. On MNIST (which the authors used for their experiments), I believe reconstructing the missing half is not too difficult and can be done pretty well with a reasonably large and well-trained RBM. 

- Sec. 3: What is noticeably missing in Sec. 3 is how the authors actually reconstruct the missing modality given the other modality. Is it simply a single forward pass from one modality to the other using the recognition, then generation weights? Does it involve sampling at each layer in between? If so, it's likely that p(x2 | x1) has multiple modes (x2 missing modality, m1 observed modality). How do you resolve among multiple possible reconstructions? If there's no sampling involved, why does the proposed model work better than a conventional autoencoder (two-way) trained with SGD and backprop? Is it possible that the problem of the conventional NN is due to learning difficulty only?

- Sec. 3.1, the last paragraph: I believe S&S (2012) did not do supervised finetuning for all experiments. For instance, for image retrieval, multimodal DBM does not require any discriminative finetuning.

- Sec. 4: In general, I'm not sure why the authors had to use only that small dataset. And, due to this small size dataset and the pretraining strategy used during the experiments, each parameter of the BP-ANN gets unfairly smaller number updates.

- Sec. 4.1, the last paragraph: why do you suspect that? I think it's simply that BP-ANN wasn't trained enough.

- Sec. 4.2: you should state the type of noise you used.

- Sec. 4.2, the last paragraph: 'DLAs attempt to probabilistically differentiate features from noises' I think I understand what the authors are trying to say, but I'm not entirely sure if that's correct or I'm not misunderstanding. Anyway, this sentence sounds somewhat weird (if not wrong). 

- Sec. 4.2, the last paragraph: 'BP ANNs .. features ... are for the purpose of mapping and not reconstruction' I'm lost here. How does 'mapping from one modality to the other' differ from 'reconstructing the other modality from one modality'? 

- Sec. 4.3, Results and Discussion: 'using non-paired examples to better develop .. associate learning system' Either this sentence has been mistyped, or I'm misunderstanding the Fig. 11 completely. It seems to me that adding non-paired examples does not help at all. Also, the remaining of the paragraph after this sentence is extremely difficult to understand, and I'd suggest to rephrase it.