Feature Graph Architectures
Authors: Richard Davis, Sanjay Chawla, Philip Leong
ICLR 2014

Abstract: In this article we propose feature graph architectures (FGA), which are deep learning systems employing a structured initialisation and training method based on a feature graph which facilitates improved generalisation performance compared with a standard shallow architecture. The goal is to explore alternative perspectives on the problem of deep network training. We evaluate FGA performance for deep SVMs on some experimental datasets, and show how generalisation and stability results may be derived for these models. We describe the effect of permutations on the model accuracy, and give a criterion for the optimal permutation in terms of feature correlations. The experimental results show that the algorithm produces robust and significant test set improvements over a standard shallow SVM training method for a range of datasets. These gains are achieved with a moderate increase in time complexity.


Peer review:
[Review]:
This paper presents a tree-structured architecture whose leafs are
SVMs on subsets of attributes and whose internal nodes are SVM taking
as input predictions computed by the children.

The generality of the method is unclear: you seem to implicitly assume
regression problems but this is never explicitly mentioned. Could the
method work for classification?

The significance seems modest to me. First, it is unclear in what
sense the algorithm performs feature learning as the intermediate
layers contain in facts predictions. Additionally, all the experiments
use a linear kernel and thus from what I understand from Section 3 the
tree in Figure 1 computes a linear function of its input. Clearly I
must be missing something otherwise this would not be a deep learning
system at all. But the presentation should be improved to clarify
this. The notation is also confusing, for example does overline{y}
denote a mean? Over what quantities exactly (targets or predictions)?
Pseudo-code 2 seem to contradict Figure 1 as each SVM is trained on
inputs x (again probably a notational issue).

The experiments are only preliminary and based on small data sets. The
reported R and hat{R} seem to be unnormalized, why?