Feature Graph Architectures
Authors: Richard Davis, Sanjay Chawla, Philip Leong
ICLR 2014

Abstract: In this article we propose feature graph architectures (FGA), which are deep learning systems employing a structured initialisation and training method based on a feature graph which facilitates improved generalisation performance compared with a standard shallow architecture. The goal is to explore alternative perspectives on the problem of deep network training. We evaluate FGA performance for deep SVMs on some experimental datasets, and show how generalisation and stability results may be derived for these models. We describe the effect of permutations on the model accuracy, and give a criterion for the optimal permutation in terms of feature correlations. The experimental results show that the algorithm produces robust and significant test set improvements over a standard shallow SVM training method for a range of datasets. These gains are achieved with a moderate increase in time complexity.


Peer review:
[Review]:
The authors propose a hierarchical SVM approach for learning. The method uses the prediction of lower layer SVMs as input to the higher-layer SVMs. Adaptive methods are proposed where lower nodes are updated incrementally in order to improve the overall accuracy.

I could not find what the proposed algorithm exactly does and what is the exact function implemented by the feature graph architecture. I assume that the authors use linear SVMs for regression. Then, should not a combination of these linear SVMs also be linear? Or is a nonlinear kernel being used? The overall lack of specification of the proposed approach, makes it difficult to compare with existing alternatives (hierarchical kernels, boosting, neural networks).

The paper is rich content-wise, including generalization bounds and stability analysis. Authors derive a generalization bound for the feature graph architecture, showing that surplus error grows linearly with the number of modified nodes in the graph. The generalization bound does not seem to be very tight, as authors show empirically that generalization is on par with basic SVMs.

The authors propose maximizing feature correlation in the first layer as a heuristic to construct the feature hierarchy. This is generally a good idea, but I am wondering whether having only one output per group of correlated features is sufficient in order to learn complex models.