Feature Graph Architectures
Authors: Richard Davis, Sanjay Chawla, Philip Leong
ICLR 2014

Abstract: In this article we propose feature graph architectures (FGA), which are deep learning systems employing a structured initialisation and training method based on a feature graph which facilitates improved generalisation performance compared with a standard shallow architecture. The goal is to explore alternative perspectives on the problem of deep network training. We evaluate FGA performance for deep SVMs on some experimental datasets, and show how generalisation and stability results may be derived for these models. We describe the effect of permutations on the model accuracy, and give a criterion for the optimal permutation in terms of feature correlations. The experimental results show that the algorithm produces robust and significant test set improvements over a standard shallow SVM training method for a range of datasets. These gains are achieved with a moderate increase in time complexity.


Peer review:
[Review]:
A few comments in response to the above points:

1. We think the novel aspects are a) the use of a deep objective function, in which each node is separately optimized b) the additional objective that the global training error must improve to retain node modifications, c) the initialization of the deep SVM to the coefficients of a shallow SVM which guarantees the starting point is a good one, and d) for deep learning in regression, feature learning at a node level can be effectively done by training the node to the target.

We are not claiming that the novelty lies in stacking machine learning modules, which is a commonplace technique. Rather, the main idea is that the deep objective function can produce significantly better results even with a deep SVM using linear kernels (which as you correctly point out is just a linear function), which we feel is an interesting result. 

2. Learning a non-linear function such as (sum x_i)^2 using linear building blocks is non-trivial, and the method significantly outperformed the other competing methods, some of which were non-linear. We tested other functions with more terms and found similar results.

3. We plan to test the method on a much wider range of datasets but the ones we tested are regression datasets from a well-known repository, 
http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/. We used datasets of dimension ~25 to show that the technique can improve over a standard SVM even for datasets of relatively small dimension. 

5. The method did show some benefits on a few classification tests but we decided to focus on regression first. 

6. Finding correlated groups of features can be done quite easily using pairwise correlation tests. Its not necessary to find the optimal permutation, significant gains over the shallow SVM can be achieved even with a fairly good permutation.

7. overline denotes a mean over the node outputs. Each SVM is trained using the output of the previous layer, scaling the target so that it has the correct range using the output mean from the current node.

8. Adding normalized errors would help with the presentation and can be done easily.

9. We achieved significant gains in out-sample error performance using the technique.

If you have time we would be very grateful to hear your thoughts on these areas.