Distinction between features extracted using deep belief networks
Authors: mohammad pezeshki, Sajjad Gholami, Ahmad Nickabadi
ICLR 2014

Abstract: Data representation is an important pre-processing step in many machine learning algorithms. There are a number of methods used for this task such as Deep Belief Networks (DBNs) and Discrete Fourier Transforms (DFTs). Since some of the features extracted using automated feature extraction methods may not always be related to a specific machine learning task, in this paper we propose two methods in order to make a distinction between extracted features based on their relevancy to the task. We applied these two methods to a Deep Belief Network trained for a face recognition task.


Peer review:
[Review]:
The submission investigates how to distinguish between relevant and irrelevant features for a given task, of a given trained model. The context seems to be that of face recognition. The proposed methods of analysis are not explained particularly well -- the “method of variances” and “method of relative activities” sections would benefit from some equations that explain the actual details on a more-than-intuitive level.

The experimental setup is rather unique (and strange) -- it looks like the authors are training on a dataset of faces corrupted by digit images, and then presenting either images or digits to measure the relative importance of the features using the two methods exposed in Section 3. The thresholds in 4.1 and 4.2 seem rather arbitrary -- any particular insights into how the results change as they change? Figures 2-a and 2-b, referenced in the same sections, seem non-existent, too, unless the authors refer to Figure 3 instead?

Figure 3 is confusing and not sure it adds much value to the paper. While Figure 4 has some interesting results, in and of itself it does not make a paper. With some more analysis as to why and how their methods work, this body of work could be interesting to the community, as a way to analyze a trained model. But as it stands, it’s unclear whether their results are simply an artefact of the actual training data being bimodal and thus the hidden units modeling that efficiently...