Learning Paired-associate Images with An Unsupervised Deep Learning Architecture
Authors: Ti Wang, Daniel L. Silver
ICLR 2014

Abstract: This paper presents an unsupervised multi-modal learning system that learns associative representation from two input modalities (channels) such that input on one channel will correctly generate the associated response at the other channel and vice versa. In this way, the system develops a kind of supervised classification model meant to simulate aspects of human associative memory. The system uses a deep learning architecture (DLA) composed of two input/output channels formed from stacked Restricted Boltzmann Machines (RBM) and an associative memory network that combines the two channels. The DLA is trained on pairs of MNIST handwritten digit images to develop hierarchical features and associative representations that are able to reconstruct one image given its paired-associate. Experiments show that the multi-modal learning system generates models that are as accurate as back-propagation networks but with the advantage of unsupervised learning from either paired or non-paired training examples.


Peer review:
[Review]:
My thanks to the reviewers for their great comments.    My apologies for not providing feedback sooner.  New responsibilities had to be dealt with in Jan and Feb.  This having been said, we are now moving forward with the advancement of this initial research, most notably the use of multiple mixed modalities (ear - audio in, voice - audio out, eye - image in, drawing - image out).   One item I will clarify for the reviewers -- A method such as BP that can clearly do a great job fine-tuning the weights between one of the RBM stacks and another via the top layer, does not scale well to more than two modalities.  This is the reason we use the back-fitting fine-tuning method - it trains the recognition weights local to each RBM stack and the top layer - so it does scale to multiple modalities.