Deep Belief Networks for Image Denoising
Authors: Mohammad Ali Keyvanrad, mohammad pezeshki, Mohammad Mehdi Homayounpour
ICLR 2014

Abstract: Deep Belief Networks which are hierarchical generative models are effective tools for feature representation and extraction. Furthermore, DBNs can be used in numerous aspects of Machine Learning such as image denoising. In this paper, we propose a novel method for image denoising which relies on the DBNs' ability in feature representation. This work is based upon learning of the noise behavior. Generally, features which are extracted using DBNs are presented as the values of the last layer nodes. We train a DBN a way that the network totally distinguishes between nodes presenting noise and nodes presenting image content in the last later of DBN, i.e. the nodes in the last layer of trained DBN are divided into two distinct groups of nodes. After detecting the nodes which are presenting the noise, we are able to make the noise nodes inactive and reconstruct a noiseless image. In section 4 we explore the results of applying this method on the MNIST dataset of handwritten digits which is corrupted with additive white Gaussian noise (AWGN). A reduction of 65.9% in average mean square error (MSE) was achieved when the proposed method was used for the reconstruction of the noisy images.


Peer review:
[Review]:
This work presents a method for denoising images using a DBN by identifying feature nodes that are associated with noise.  This is done by measuring the mean differences in activations when presented with noisy versus corresponding ground-truth clean images in the training set.  Test images are denoised by performing inference, reseting the 'noise' feature nodes to their average values across the clean images, and reconstructing from the resulting representation.  The method is tested on MNIST with additive Gaussian noise.

The method is simple and appealing; however, it is evaluated only on one very limited test case, and is under-analyzed.  How does this method perform for other types of input or noise?  Also, although the authors review some prior work on the subject, they do not explicitly compare their method against any other algorithm.

A larger question I have is whether it is necessary to require clean/noisy inputs be associated in pairs, or if this association could be weakened or removed.  This would be a major advantage of this method if it were the case.  That is, is it enough to have a pool of known clean images and a pool of known noisy images, with no elementwise correspondence between the two?  If the clean data underlying the noisy data generation is the same between these two populations, then the difference in mean activations should be unchanged.  But it seems to me that these means might also not change much if the two sets of underlying clean images are distinct but from the same general population, e.g. if half of training images are clean, and the other half is used to generate noisy ones.

The paper is pretty clearly written, though I think the overview of RBMs and DBNs takes up too much space (2 pages); this seems it could be condensed, and replaced with more experiments and details on the method presented.  I also would have liked to see more illustrating the method's internals.  Why was the threshold of 0.9 chosen for identifying a node as a 'noise' feature, for example?  Some plots/histograms of the activations and 'relative activity' measurement could have been useful here.


Pros:

- Natural and simple method with limited demonstrated effectiveness.

Cons:

- Applied to only one limited setting
- No comparisons to other methods
- Could have more measurements to illustrate the method's internals