Learning Paired-associate Images with An Unsupervised Deep Learning Architecture
Authors: Ti Wang, Daniel L. Silver
ICLR 2014

Abstract: This paper presents an unsupervised multi-modal learning system that learns associative representation from two input modalities (channels) such that input on one channel will correctly generate the associated response at the other channel and vice versa. In this way, the system develops a kind of supervised classification model meant to simulate aspects of human associative memory. The system uses a deep learning architecture (DLA) composed of two input/output channels formed from stacked Restricted Boltzmann Machines (RBM) and an associative memory network that combines the two channels. The DLA is trained on pairs of MNIST handwritten digit images to develop hierarchical features and associative representations that are able to reconstruct one image given its paired-associate. Experiments show that the multi-modal learning system generates models that are as accurate as back-propagation networks but with the advantage of unsupervised learning from either paired or non-paired training examples.


Peer review:
[Review]:
The authors propose a an algorithm, in the setting of multimodal data, for learning to generate one modality given the other. The algorithm contains stack of RBM's for each modality, a join RBM on the top and then a predictor of joint probabilities from one modality. The experiments are too weak to demonstrate strength of the proposed approach.

Novelty: Small. A standard multimodal deep belief net with an addition of a relatively obvious idea, which is anyway quite similar to previous ideas.
Quality: Experiments are too weak. 

Details:
First, the main complaint - the weakness of the experiments. Whether feedforward network Figure 6 can learn the task depends on how you optimise it. I think if you do a good optimisation then it would do it. In particular you can take your pre trained deep network Figure 4 and, go up the left and then down the right as your feedforward net - which is essentially the idea in 'Multimodal deep learning' (http://ai.stanford.edu/~ang/papers/icml11-MultimodalDeepLearning.pdf).

The task is also too simple - may be it would be at least good to pair a random digit from one class with a random digit from another. But even better (really necessary) it would be to do some different sets of data as in the above mentioned paper. 

Other details: 
Page 1, paragraph 2 - I wouldn't say deep learning typically uses RBM's. (e.g. feedforward networks, auto encoders…)
Page 2, Background - I don't think we have an ability to recover *complete* information. Also 'many do not work in the same fashion as the human visual system' - and which do?…
Page 3, 4th paragraph - you should write the cost function between the two probabilities
Page 7 - What does it mean 'Features … for the purpose of mapping and not reconstruction'
Page 8 - Results and Discussion - First you say that the results are not significant (within error bars) and then you draw a conclusion as if they were.