Learning Paired-associate Images with An Unsupervised Deep Learning Architecture
Authors: Ti Wang, Daniel L. Silver
ICLR 2014

Abstract: This paper presents an unsupervised multi-modal learning system that learns associative representation from two input modalities (channels) such that input on one channel will correctly generate the associated response at the other channel and vice versa. In this way, the system develops a kind of supervised classification model meant to simulate aspects of human associative memory. The system uses a deep learning architecture (DLA) composed of two input/output channels formed from stacked Restricted Boltzmann Machines (RBM) and an associative memory network that combines the two channels. The DLA is trained on pairs of MNIST handwritten digit images to develop hierarchical features and associative representations that are able to reconstruct one image given its paired-associate. Experiments show that the multi-modal learning system generates models that are as accurate as back-propagation networks but with the advantage of unsupervised learning from either paired or non-paired training examples.


Peer review:
[Review]:
This paper is trying to employ DBNs at the traditional task of associative memory. This is an interesting problem as the human brain is thought to contain an “association cortex” dedicated to combining sensory modalities. However, while the paper is written reasonably well, it did not introduce any significantly new learning method. The experimental section is weak and leaves this reviewer unconvinced of their conclusions.

The framework of having dual modalities has been proposed by the original DBN paper [8] (the reference is missing a third author). Contrastive wake-sleep algorithm is proposed for unsupervised learning in [8], which seems more principled than what this paper proposes in the paragraph starting with “To fine-tune channel 1…”.

In the experimental section, the natural comparison is to an autoencoder (e.g. net in figure 6), which is also trained in an unsupervised manner. However, it is hard to believe that a network with 500-1000-500 hidden nodes can’t reconstruct better than what is shown in Fig. 7. last row. What kind of learning algorithm was used? CG or SGD, did the optimization converge?  The authors also ref “Hinton’s software” and it’s 1.15% error on MNIST. However, that is a totally different net with 10 1-of-k label units. It is unclear what the authors mean by using hinton’s software: was there no additional learning for your particular image pair association task been performed? If that is the case, then the results are believable but not a good baseline.

Several suggestions to improve the paper: compare with the contrastive wake-sleep algorithm and autoencoder trained with CG/SGD. Investigate more in depth on your proposed algorithm, is it approximating some objective? You should truly use multimodal data like images and speech, since it is so often mentioned in the introduction/background sections. It is a lot of work to combine audio and natural images are much higher dimensionality, but this would make the paper stronger.

The claim in section 2.2 that “The mammalian brain is..” is a huge claim which is not proven. It is the current mainstream theory, but you should not treat it as a fact. Note that models like [14] do not even do learning except the last SVM layer.

The claim that RBM can’t recall patterns when half of is corrupted is not convincing and maybe should be qualified to a particular task/learning algorithm. There are papers on rbm and noise+occlusion which suggest otherwise. It is true that how RBM performs is dependent on how it is trained (e.g. using fast pcd is critical).