Title: Marginal Deep Architectures: Deep learning for Small and Middle Scale Applications
Authors: Yuchen Zheng
Guoqiang Zhong
Junyu Dong
Venue: ICLR 2017

Abstract: In recent years, many deep architectures have been proposed in different fields. However, to obtain good results, most of the previous deep models need a large number of training data. In this paper, for small and middle scale applications, we
propose a novel deep learning framework based on stacked feature learning models. Particularly, we stack marginal Fisher analysis (MFA) layer by layer for the initialization of the deep architecture and call it “Marginal Deep Architectures” (MDA). In the implementation of MDA, the weight matrices of MFA are first learned layer by layer, and then we exploit some deep learning techniques, such as back propagation, dropout and denoising to fine tune the network. To evaluate the effectiveness of MDA, we have compared it with some feature learning methods and deep learning models on 7 small and middle scale real-world applications, including handwritten digits recognition, speech recognition, historical document understanding, image classification, action recognition and so on. Extensive experiments demonstrate that MDA performs not only better than shallow feature learning models, but also state-of-the-art deep learning models in these applications.


Review discussion:
What is the A matrix in equation 3?
For classification on Cifar-10, wouldn't the simplest baseline be to compare MFA to state-of-the-art CNNs trained from random weights?
Many thanks for your comments!
1. In equation 3, "a^k" is the activation output of the k-th hidden layer.
By the way, do you mean the matrix "A" in equation 4? It's the similarity matrix between data. Sorry for the missing of its introduction.
2. Since the inputs of MDA are in the form of vectors, while the input of classic CNNs are gray or color images, the most suitable baseline of MDA should be deep neural networks that deal with vector inputs, like deep autoencoders or stacked denoising autoencoders. In this work, we focus on deep architectures with vector inputs is because there are many applications where data are represented in the form of vectors, such as that in the domain of finance.
Moreover, in many deep models, there are some fully connected layers.
Indeed, the data are represented in the form of vectors here, and need to be dealt with subsequently. For fair comparison with CNNs, we transform the color images of CIFAR-10 to gray images, and use 1024 dimensional vectors as inputs, which is just as same as MDA.
1. Could you clarify whether the weights learned by MFA are used as initialization and the whole network is then fine-tuned by back-propagation? Or if MFA is performed during training and only the last layer is then learned in a supervised fashion?
2. Could you clarify whether MFA is performed for each mini-batch or on the whole dataset?
3. Why do you think MFA should work better than PCA or DAE or other layerwise initialization strategy?
4. Why did you not also compare with standard random initialization + dropout training (i.e. no fancy layerwise initialization strategy)?
Thank you so much for your review!
1. The first one. The weights learned by MFA are used as initialization and the whole network is then fine-tuned by back-propagation.
2. In our experiments, we implemented MFA on the whole dataset. On large scale of data, one can use a subset of the data for the learning of MFA.
3. MFA is a supervised feature learning method, for the purpose of classification, which can learn more effective representations of data than PCA and random initialization. For the initialization of the whole nets, intuitively, stacked MFAs can deliver a starting point close to a better local minimum than what its counterparts can do. This is the reason why we think MFA can work better than PCA or DAE or other layerwise initialization strategy.
4. We have compared MDA with the standard random initialization + dropout training, whose results are similar to that without dropout.
Therefore, for the space limitation of the table, we didn't report them.
1. Can you give an example on how the matrix A in equation (4) is constructed for your experiments?
2. How important is the random projection step?
Thank you so much for your review!
1. For an example of matrix A, it can be a graph Laplacian, which is widely used in manifold learning and semi-supervised learning. For more details about MFA, please refer to [Shuicheng Yan et al., TPAMI 2007].
2. The random projection step is to map the learned new representations of data into a high-dimensional space. We can consider it as a trick in deep learning, as many deep learning models include such a layer to map the activation outputs of one layer into a higher dimensional space.
This paper proposes to initialize the weights of a deep neural network layer-wise with a marginal Fisher analysis model, making use of potentially the similarity metric.
Pros:
There are a lot of experiments, albeit small datasets, that the authors tested their proposed method on.
Cons:
lacking baseline such as discriminatively trained convolutional network on standard dataset such as CIFAR-10.
It is also unclear how costly in computation to compute the association matrix A in equation 4.
This is an OK paper, where a new idea is proposed, and combined with other existing ideas such as greedy-layerwise stacking, dropout, and denoising auto-encoders.
However, there have been many papers with similar ideas perhaps 3-5 years ago, e.g. SPCANet.
Therefore, the main novelty is the use of marginal Fisher Analysis as a new layer. This would be ok, but the baselines to demonstrate that this approach works better is missing.
In particular, I'd like to see a conv net or fully connected net trained from scratch with good initialization would do at these problems.
To improve the paper, the authors should try to demonstrate without doubt that initializing layers with MFA is better than just random weight matrices.
Many thanks for your review!
Firstly, please notice that we have compared the proposed method with related work on the CIFAR-10 data set in Section 4.4. The results are shown in Table 6, where deep autoencoders (DA), stacked denoising autoencoders (SDA) and other deep learning models, as baseline approaches, have been compared with. For sure, these compared approaches are trained using supervised back propagation.
Secondly, in this paper we propose a new deep learning algorithm that is initialized based on supervised layer-wise pre-training. The contribution is not focused on the marginal Fisher analysis (MFA) algorithm.
Moreover, the association matrix A is computed during model training. It's totally offline and can be approximated in the case of large scale applications.
Thirdly, yes, there are some similar work using stacked modules to build deep architectures, such as DA, SDA and SPCANet.
However, none of them is initialized using supervised layer-wise pre-training, which is a main contribution of our paper.
Fourthly, please notice that there is no a deep network which can solve any problems.
For example, conv net, which may perform well on large 2D images, will not perform well on the inputs of the vector form. For the applications with vector inputs, do we still need to compare the proposed method with conv net?
In contrast, we compared the proposed method with more related approaches, such as DA, SDA and others.
Fifthly, from the experimental results, we can see that MDA performs much better than DA, SDA and other deep networks, and it's well known from many deep learning literatures that these deep learning methods perform well compared to random initialized networks. Why should we compare MDA with random weight matrices?
The authors pointed out some limitations of existing deep architectures, in particular hard to optimize on small or mid size datasets, and proposed to stack marginal fisher analysis (MFA) to build deep models. The proposed method is tested on several small to mid size datasets and compared with several feature learning methods. The authors also applied some existing techniques in deep learning, such as backprop, denoising and dropout to improve performance.
The new contribution of the paper is limited. MFA has long been proposed. The authors fail to theoretically or empirically justify the stacking of MFAs. The authors did not include any deep architectures that requires backprop over multiple layers in the comparison, which the authors set out to address, instead all the methods compared were learned layer by layer. Will a randomly initialized deep model such as DBN or CNN perform poorly on these datasets? It is also not clear how the authors came up with each particular model architecture and hyper-parameters used in the different datasets. The writing of the paper needs to be significantly improved. A lot of details were omitted, for example, how is dropout applied in the MFA.
Many thanks for your comments!
Firstly, for sure, the compared approaches, deep autoencoder (DA), stacked denoising autoencoders (SDA) and others, have been fine-tuned by back propagation. It's obvious that, they are pre-trained layer by layer.
But, the fine-tuning is definitely by back propagation.
Secondly, random initialized deep models such as DBN (somewhat equivalent to DA) performed worse than the proposed method, MDA. And moreover, since we consider inputs of the vector form in this paper, CNN cannot perform well also.
Thirdly, like DA and SDA, to use MDA, one can define the number of layers and number of neurons in each layer. For the hyper-parameters, like that in other deep models, they can be pre-defined with cross validation or learned using back propagation.
Fourthly, dropout is widely in the deep learning area. It is used here in the common way of the deep learning models.
The proposed approach consists in a greedy layer wise initialization strategy for a deep MLP model, which is followed by global gradient-descent with dropout for fine-tuning. The initialization strategy uses a first randomly initialized sigmoid layer for dimensionality expansion followed by 2 sigmoid layers whose weights are initialized by Marginal Fisher Analysis (MFA) which learns a linear dimensionality reduction based on a neighborhood graph constructed using class label information (i.e. supervised dimensionality reduction). Output layer is a standard softmax layer.
The approach is thus to be added to a growing list of heuristic layer-wise initialization schemes.
The particular choice of initialization strategy, while reasonable, is not sufficiently well motivated in the paper relative to alternatives, and thus feels rather arbitrary.
The paper lacks clarity in the description of the approach:  MFA is poorly explained with undefined notations (in Eq. 4, what is A? It has not been properly defined); the precise use of alluded denoising in the model is also unclear (is there really training of an additional denoting objective, or just input corruption?).
The question of the (arguably mild) inconsistency of applying a linear dimensionality reduction algorithm, that is trained without any sigmoid, and then passing its learned representation through a sigmoid is not even raised. This, in addition to the fact that sigmoid hidden layers are no longer commonly used (why did you not also consider using RELUs?).
More importantly I suspect methodological problems with the experimental comparisons: the paper mentions using *default* values for learning-rate and momentum, and having (arbitrarily?) fixed epoch to 400 (no early stopping?) and L2 regularization to 1e-4 for some models.
*All* hyper parameters should always be properly hyper-optimized using a validation set (or cross-validation) including early-stopping, and this separately for each model under comparison (ideally also including layer sizes). This is all the more important since you are considering smallish datasets, so that the various initialization strategies act mainly as different indirect regularization schemes: they thus need to be carefully tuned. This casts serious doubts as to the amount of hyper-parameter tuning (close to none?) that went into training the alternative models used for comparison.
The Marginal Fisher Analysis dimensionality reduction initialization strategy may well offer advantages, but as it currently stands this paper doesn’t yet make a sufficiently convincing case for it, nor provide useful insights into the nature of the expected advantages.
I would also suggest, for image inputs such as CIFAR10, to use the qualitative tool of showing the filters (back projected to input space) learned by the different initialization schemes under consideration, as this could help visually gain insight as to what sets methods apart.
Many thanks for your review!
Firstly, thanks for pointing out the writing problems. We will revise this paper accordingly.
Secondly, if large scale of labeled data are available, it's intuitive that it's a good idea to used supervised layer-wise pre-training to learn the deep nets. This is a main difference between our work with other related approaches.
Thirdly, marginal Fisher analysis (MFA) is an algorithm published in TPAMI and it's well known in the area of dimensionality reduction.
Hence, we didn't introduce it in very detail. For the alluded denoising, as presented in the paper, it's just for input corruption.
Fourthly, ReLu may leads to sparse representations during the network training. For low dimensional data, it may not work well.
Hence, we use sigmoid (logistic) activation function.
Fifthly, for the parameters in the deep networks including the compared ones, it's not possible to tune them one by one, even we have a validation set.
However, using fixed parameters, the advantages and disadvantages are shared by all the compared methods.
Hence, in our work, we didn't try to greatly optimize the network, but used some default or fixed values for the parameters.
Sixthly, We think that we cannot only focus on large scale data. There are many applications with only small or middle size data. How do we deal with these problems? In this case, MDA shows its advantages as shown in the experiments.
Finally, thanks for your suggestions about visualization. We will add it in the next version of our paper.

Meta-review:
role: PC
data: comment: The reviewers unanimously recommend rejection.

Decision: Reject