Deep Belief Networks for Image Denoising
Authors: Mohammad Ali Keyvanrad, mohammad pezeshki, Mohammad Mehdi Homayounpour
ICLR 2014

Abstract: Deep Belief Networks which are hierarchical generative models are effective tools for feature representation and extraction. Furthermore, DBNs can be used in numerous aspects of Machine Learning such as image denoising. In this paper, we propose a novel method for image denoising which relies on the DBNs' ability in feature representation. This work is based upon learning of the noise behavior. Generally, features which are extracted using DBNs are presented as the values of the last layer nodes. We train a DBN a way that the network totally distinguishes between nodes presenting noise and nodes presenting image content in the last later of DBN, i.e. the nodes in the last layer of trained DBN are divided into two distinct groups of nodes. After detecting the nodes which are presenting the noise, we are able to make the noise nodes inactive and reconstruct a noiseless image. In section 4 we explore the results of applying this method on the MNIST dataset of handwritten digits which is corrupted with additive white Gaussian noise (AWGN). A reduction of 65.9% in average mean square error (MSE) was achieved when the proposed method was used for the reconstruction of the noisy images.


Peer review:
[Review]:
This paper presents a simple method for denoising noisy mnist digits with a deep belief network. The method looks at the relative activities of the hidden units when the input is a normal vs a noisy image. After obtaining these statistics, at test time, noisy nodes or nodes which are affected by noise are removed, leading to better reconstructions.

The authors are recommended to reference the paper: Deep Networks for robust visual recognition, Tang&Eliasmith icml 2010, where the tasks are similar but with a more elaborate algorithm and experimental results.

While the task is interesting and potentially very important, the method proposed in this paper is extremely simple and are not shown to work for difficult noise/occlusion cases. For example, simple reconstruction with DBN is already very good for the mnist digits.

Possible improvements include trying more difficult noise and occlusions; look at how denoising can help reduce recognition error; and coming up with a more principled way of determining which nodes are affected. Since a DBN is a distributed network, it is likely that all hidden nodes would be affected somewhat, and each image would lead to a different activation for a particular hidden node, simply by looking at relative activations seems very ad hoc. There are several papers related to denoising using DBNs/DBMs that can be found with simple google search. The authors should compare/contrast, cite and test on similar experiments with those other papers.