Feature Graph Architectures
Authors: Richard Davis, Sanjay Chawla, Philip Leong
ICLR 2014

Abstract: In this article we propose feature graph architectures (FGA), which are deep learning systems employing a structured initialisation and training method based on a feature graph which facilitates improved generalisation performance compared with a standard shallow architecture. The goal is to explore alternative perspectives on the problem of deep network training. We evaluate FGA performance for deep SVMs on some experimental datasets, and show how generalisation and stability results may be derived for these models. We describe the effect of permutations on the model accuracy, and give a criterion for the optimal permutation in terms of feature correlations. The experimental results show that the algorithm produces robust and significant test set improvements over a standard shallow SVM training method for a range of datasets. These gains are achieved with a moderate increase in time complexity.


Peer review:
[Review]:
In response to the above comments:
1. The main idea is a mechanism which forms a hierarchy of new features which are decorrelated successively from one layer to the next. This mechanism is the reason why the deep architecture is able to decorrelate features better than using PCA in combination with a simple SVM.

We thought the novel aspects were the following:
a) the use of a deep objective function, in which each node is separately optimized 
b) the additional objective that the global training error must improve to retain node modifications, 
c) the initialization of the deep SVM to the coefficients of a shallow SVM which guarantees the starting point is a good one, and 
d) for deep learning in regression, feature learning at a node level can be effectively done by training the node to the target.
 
We are not claiming that the novelty lies in stacking machine learning modules, which is a commonplace technique. Rather, the multi-layer objective function can produce significantly better results even using layers of linear SVMs (which overall is just a linear function), which we thought was an interesting result.

2. We tested regression datasets from a well-known repository, http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/. 

3. Learning a non-linear function such as (sum x_i)^2 using linear building blocks is non-trivial, and the method significantly outperformed the other competing methods, some of which were non-linear. We tested other functions with more terms and found similar results.  

4. We used datasets of dimension ~25 to show that the technique can improve over a standard SVM even for datasets of relatively small dimension.   

5. The method did show some benefits on a few classification tests but we decided to focus on regression first.   

6. Finding correlated groups of features can be done quite easily using pairwise correlation tests. Its not necessary to find the optimal permutation, significant gains over the shallow SVM can be achieved even with a fairly good permutation.   

7. overline denotes a mean over the node outputs. Each SVM is trained using the output of the previous layer, scaling the target so that it has the  correct range using the output mean from the current node.   

8. Adding normalized errors would help with the presentation and can be done easily.  

9. We achieved significant gains in out-sample error performance using the technique.

10. The gains from the method over a standard shallow SVM are dataset-dependent. If the data have completely uncorrelated features, the method will not improve over the shallow SVM. However if some features are correlated, as is usually the case, the method is able to improve over the shallow SVM using the architecture.

11. The method gives a way to move from a shallow to a deep model incrementally so that at each stage the model improves. It has multiple outputs per group of correlated features, since there are multiple layers in the model. This multilayer structure allows for more complex models to be built.