Title: Exploring the Application of Deep Learning for Supervised Learning Problems
Authors: Jose Rozanec
Gilad Katz
Eui Chul Richard Shin
Dawn Song
Venue: ICLR 2017
Keywords: Deep learning
Supervised Learning

Abstract: One of the main difficulties in applying deep neural nets (DNNs) to new domains is the need to explore multiple architectures in order to discover ones that perform well. We analyze a large set of DNNs across multiple domains and derive insights regarding their effectiveness. We also analyze the characteristics of various DNNs and the general effect they may have on performance. Finally, we explore the application of meta-learning to the problem of architecture ranking. We demonstrate that by using topological features and modeling the changes in its weights, biases and activation functions layers of the initial training steps, we are able to rank architectures based on their predicted performance. We consider this work to be a first step in the important and challenging direction of exploring the space of different neural network architectures. 


TL;DR: We explore the multiple DNN architectures on a large set of general supervised datasets. We also propose a meta-learning approach for DNN performance prediciton and ranking

Review discussion:
For the datasets used, would be nice to understand if the proposed meta learning method actually achieves  good results compared to what's out there (e.g. other papers that evaluate on the same data)
We are not aware of any publications that evaluated DNNs on these datasets. As we explain in Section 1, one of the problems we are attempting to begin to address is the application of deep learning to domains for which there is little or no existing knowledge on the application of deep learning. We therefore chose a type of problems that is not commonly addressed.
In order to enable the reader to evaluate the performance of our meta-learning approach we do the following: 1) Compare the performance of the best architecture found by our approach (in the ranked top 10) to the best architecture overall; 2) To provide perspective on the performance compared to other classification methods, we present the results obtained by running the Rotation Forest algorithm, a popular ensemble algorithm that is often applied to these types of problems (we used the Weka ML framework with default parameters in order to ensure reproducibility). These results are presented in Table 1.
It should also be noted that the goal of the meta-learning (in the scope this work, at least) is not to outperform a specific baseline but rather to successfully identify the top-performing architectures in a given set.
For this reason we also compare the performance of our approach to random sampling (Table 4) and calculate the precision@K for various points (Table 3).
Does the paper target any kind of DNN including CNN, RNN, GAN, NTM, or any combinations of them? I think at this stage the paper addresses rather small-layered vanilla MLPs. Is this correct?
Thank you for the question. You are indeed correct in your understanding that we do not use convolusions or RNNs. In section 4 we state the components used in our generated architectures:  “fully-connected layers, softmax, batch normalization, dropout and the ReLU, sigmoid and tanh activation functions”.
Our reason for not including components such as convolusions in our architecture is that it does not “make sense” to apply them to the types of problems that we analyze. Convolusions are only relevant in domains such as image classification, where the grouping of features based on their proximity is logical. This is not the case for general classification problems on tabular datasets (such as determining credit risk, for example). We intentionally chose these domains because there is little available knowledge on designing DNNs for them.
As we state in Section 6.1 the maximal number of components in each of our generated architectures is 14.
While this depth is not close to that of state-of-the-art networks used in fields such as image classification, we have been able to obtain results that are close to (and often exceed) the performance of the well-known ensemble algorithms Random Forest (see Table 1).
Another reason for limiting the depth of the architectures is our exploration of wider architectures with parallel layers (see Figure 1) which significantly increased the number of possible architectures for each increase in depth.
Finally, we would like to stress again (as we have done in the paper) that this is a preliminary work whose goal is to explore the application of DL to relatively little explored domains and to begin to evaluate our proposed meta-learning approach. We intend to experiment with additional types of networks and larger depths in future work.
This paper presents an intriguing study of how one can pose architecture search as a meta learning problem. By collecting features from networks trained on various datasets and training a “ranking classifier” (the actual details of the classifier do not seem to be described in detail) one can potentially infer what a good architecture for a new problem could be by simply running the ranker on the extracted features for a new problem setup.
One notable comment from the paper is that the authors fix some important hyper-parameters for all the networks. I am of the opinion that optimizing the learning rate (and its decay schedule) is actually quite important. I hypothesize that a lot of the conclusions of this paper may change quite a bit if the authors did an actual search over the rates instead. I suspect that instead of training 11k nets, one can train 2k nets with 5 learning rates each and get a much better result that is actually compelling.
I am not convinced that the protocol for generating the various architectures is doing a good job at creating a diversity of architecture (simply because of the max depth of 8 layers and 14 components overall). I suspect that most of these generated architectures are actually almost identical performance-wise and that it’s a waste to train so many of them on so many tasks.
Unless the authors are already doing this, they should define a pruning mechanism that filters out nets that are too similar to already existing ones.
The batch normalization experiments in Table 2 seem odd and under-explained. It is also well-known that the optimal learning rates when using batch norm vs. not using batch norm can differ by an order of magnitude so given the fixed learning rate throughout all experiments, I take these results with some grain of salt.
I am not sure we got many insights into the kinds of architectures that ended up being at the top. Either visualizations, or trends (or both), would be great.
This work seems to conflate the study of parallel vs. serial architectures with the study of meta learning, which are somewhat distinct issues. I take issue with the table that compares parallel vs. serial performance (table 2) simply because the right way would be to filter the architectures by the same number of parameters / capacity.
Ultimately the conclusion seems to be that when applying deep nets in a new domain, it is difficult to come up with a good architecture in advance. In that sense, it is hard to see the paper as a constructive result, because it’s conclusions are that while the ranker may do a good job often-times, it’s not that reliable.
Thus I am not convinced that this particular result will be of practical use to folks who are intending to use deep nets for a new domain.
We thank the reviewer for his thoughtful comments. We would like to respond to some of the points raised in the review:
1) “the actual details of the [ranking] classifier do not seem to be described in detail”
- as explained in section 6.3, we use the Random Forest algorithm as the ranking classifier. In Section 5 we provide a description of the meta-features categories used by our algorithm and in Appendix A we provide a comprehensive list of all the meta-features used by our ranking classifier and the way they were generated.
2) “One notable comment from the paper is that the authors fix some important hyper-parameters for all the networks. I am of the opinion that optimizing the learning rate (and its decay schedule) is actually quite important. I hypothesize that a lot of the conclusions of this paper may change quite a bit if the authors did an actual search over the rates instead. I suspect that instead of training 11k nets, one can train 2k nets with 5 learning rates each and get a much better result that is actually compelling.”
- We agree with the reviewer that the absence of different parameter configurations is a shortcoming of the paper (and this is one of the main reasons for our description of this work as preliminary). Our reasoning in prioritizing a large set of architectures over multiple parameter settings is as follows:
a) Because of the relatively unexplored domain in which we run our experiments, we had little idea which type of architectures are likely to perform well. We therefore preferred to create a large and diverse set of architectures.
b) We were interested in creating a sufficiently diverse set of architectures to ensure robustness when we trained the meta-model ranking classifier.
c) One of the goals of this work was to compare the performance of serial and parallel architectures. For our results to be valid, we needed to evaluate a large number of both types.
Nonetheless, we acknowledge that the lack of multiple parameters is a highly important subject that we intend to address in future work.
3) “I am not convinced that the protocol for generating the various architectures is doing a good job at creating a diversity of architecture (simply because of the max depth of 8 layers and 14 components overall). I suspect that most of these generated architectures are actually almost identical performance-wise and that it’s a waste to train so many of them on so many tasks”
- As shown in Figure 2 and Appendix D, the variance in the performance of the various architectures changes significantly across datasets. These results demonstrate that the architectures are not similar performance-wise. It is also worth pointing out that this large variance in performance distribution demonstrates the usefulness of the meta-learning ranking approach: using the ranker, one needs only evaluate a small number of architectures in order to obtain a top-performing architectures.
In order to further investigate the hypothesis raised by the reviewer, we analyzed all parent-child architectures and their differences in performance. Our conclusion is that adding even a single component can induce significant changes in accuracy. Our results have been added to the paper - please see tables 1 and 2.
That being said, we fully acknowledge the limitations of our work mentioned but the reviewer. As we state in the submitted paper, we consider this work a first step in a relatively unexplored domain.
4) “The batch normalization experiments in Table 2 seem odd and under-explained. It is also well-known that the optimal learning rates when using batch norm vs. not using batch norm can differ by an order of magnitude so given the fixed learning rate throughout all experiments, I take these results with some grain of salt.”
- We were also surprised by these results. We updated the paper to indicate that we intend to explore this subject further in future work. (please note that table 2 is now table 4)
5) “I am not sure we got many insights into the kinds of architectures that ended up being at the top. Either visualizations, or trends (or both), would be great” -
Please note that in Appendix C we provide both an analysis of the percentage of architectures with parallel layers in the 100 top-performing architectures (it is clear that the number is significant for all datasets) and also a figure presenting one of these architectures. In order to further investigate the point raised by the reviewer we have added an additional analysis of the components composition of the 100 top-performing architectures in each dataset. The most interesting point found in this analysis is that the relatively shallow networks (~4 FC layers) seem to achieve the best performance across all datasets. For space considerations, this information has also been added to Appendix C. We have updates the text of Section 6.2 accordingly.
6) “This work seems to conflate the study of parallel vs. serial architectures with the study of meta learning, which are somewhat distinct issues.”
- We designed the meta-features to be applicable to both serial and parallel architectures, so that our meta-learning technique would remain agnostic to architectures structure. The comparison of serial vs. parallel architectures and the analysis of their performance is a separate subject studied in this paper.
7) “I take issue with the table that compares parallel vs. serial performance (table 2) simply because the right way would be to filter the architectures by the same number of parameters / capacity”
- The goal of the analysis in Table 2 was to evaluate the contribution of adding parallel layers to a given architecture. Given the fact that the ordering of the layers can be critical to the performance of almost every architectures, we believe the presented evaluation to be important. This assumption is supported by the results presented in Appendix C, where it is shown that parallel layers make up a large percentage of the 100 top-performing architectures for all datasets.
8) “Ultimately the conclusion seems to be that when applying deep nets in a new domain, it is difficult to come up with a good architecture in advance. In that sense, it is hard to see the paper as a constructive result, because it’s conclusions are that while the ranker may do a good job often-times, it’s not that reliable.
Thus I am not convinced that this particular result will be of practical use to folks who are intending to use deep nets for a new domain.”
- In the first version of the paper, the ranking approach achieved high performance for 7 out of 10 evaluated datasets. In the current version we have increased the number of datasets in our experiments from 10 to 13 and that led to an improvement in performance (the ranking classifier had more datasets to learn from). As shown in Table 4, we now outperform random sampling on ALL datasets, often by a large margin.
This paper aims at attacking the problem of preselecting deep learning model structures for new domains. It reported a series of experiments on various small tasks and feed-forward DNNs. It claims that some ranking algorithm can be learned based on these results to guide the selection of model structures for new domains.
Although the goal is interesting I found their conclusion is neither convincing nor useful in practice for several reasons:
1. They only explored really simple networks (feed-forward DNNs).
While this significantly limited the search space, it also limited the value of the experiments.
In fact, the best model architecture is highly task (domain) dependent and the type of model (DNN vs CNN vs LSTM) is often much more important than size of the network itself.
2. Their experiments were conduced with some important hyper parameters (e.g., learning rate schedule) fixed.
However, it is well known  that learning rate often is the most important hyper parameter during training. Without adjusting these important hyper parameters the conclusion on the best model architecture is not convincing.
3. Their experiments seem to indicate that the training data difference is not important.
However, this is unlikely to be true as you would definitely want to use larger models (total number of parameters) when your training set is magnitude larger (i.e., log(datasize) can be an important feature). This is likely because they did not run experiments on large datasets.
In addition, I think the title of the paper does not accurately reflect what the paper is about and should be modified.
Also, this paper cited Sainath et al. 2015 as the work that leads to breakthrough in speech recognition.
However, the breakthrough in ASR happened much earlier. The first paper with all three key components was published in 2010:
Yu, D., Deng, L. and Dahl, G., 2010, December. Roles of pre-training and fine-tuning in context-dependent DBN-HMMs for real-world speech recognition. In Proc. NIPS Workshop on Deep Learning and Unsupervised Feature Learning.
and the more detailed paper was published in 2012
Dahl, G.E., Yu, D., Deng, L. and Acero, A., 2012. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 20(1), pp.30-42.
As a conclusion, this paper presented some very preliminary result.
Although it's interesting it's not ready for publishing.
We thank the reviewer for his thoughtful comments. We would like to respond to some of the points raised in the review:
1. “They only explored really simple networks (feed-forward DNNs).
While this significantly limited the search space, it also limited the value of the experiments.
In fact, the best model architecture is highly task (domain) dependent and the type of model (DNN vs CNN vs LSTM) is often much more important than size of the network itself.”
- As we explain in Section 1, we explore the application of deep learning to tabular datasets (“classic” machine learning problems). In this setting CNNs make little sense because there is no meaningful way of grouping the features together (unlike in image classification, for example) and there is no definition of proximity. The same reasoning applies to the use of RNNs.
Because of this we focus our experiments on different activation functions and parallel architectures.
2. “Their experiments were conducted with some important hyper parameters (e.g., learning rate schedule) fixed.
However, it is well known  that learning rate often is the most important hyper parameter during training. Without adjusting these important hyper parameters the conclusion on the best model architecture is not convincing.”
- We agree with the reviewer that the absence of different parameter configurations is a shortcoming of the paper (and this is one of the main reasons for our description of this work as preliminary). Our reasoning in prioritizing a large set of architectures over multiple parameter settings is as follows:
a) Because of the relatively unexplored domain in which we run our experiments, we had little idea which type of architectures are likely to perform well. We therefore preferred to create a large and diverse set of architectures.
b) We were interested in creating a sufficiently diverse set of architectures to ensure robustness when we trained the meta-model ranking classifier.
c) One of the goals of this work was to compare the performance of serial and parallel architectures. For our results to be valid, we needed to evaluate a large number of both types.
Nonetheless, we acknowledge that the lack of multiple parameters is a highly important subject that we intend to address in future work.
3. "Their experiments seem to indicate that the training data difference is not important.
However, this is unlikely to be true as you would definitely want to use larger models (total number of parameters) when your training set is magnitude larger (i.e., log(datasize) can be an important feature). This is likely because they did not run experiments on large datasets."
- We agree with the reviewer that the some of the datasets are not very large and we plan to address this issue in future work (hence our definition of this work as preliminary).
Nonetheless, as shown in appendix B, the sizes of the datasets vary from 1,000 to over 45,000 data points. This diversity demonstrates the robustness of our meta learning-based ranking approach.
Moreover, as shown in Table 3 (Table 1 in the version read by the reviewer) the performance of the architectures vary significantly from one dataset to another. This is demonstrated by the column “Architecture with best average ranking overall”, where we show that simply choosing the architecture that performed best on average on other datasets yields inferior results.
4. “this paper cited Sainath et al. 2015 as the work that leads to breakthrough in speech recognition.
However, the breakthrough in ASR happened much earlier. The first paper with all three key components was published in 2010”
- We thank the reviewer for his suggestion. We have updated the paper accordingly.
The topic is very interesting, but the paper is not convincing.
Specifically, the experiment part is weak. The study should include datasets that are familiar to the community as well as the ones "that are not often addressed by deep learning". The comparison to other approaches is not comprehensive.

Meta-review:
role: PC
data: comment: The reviewers unanimously recommend rejection.

Decision: Reject