Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: From my understanding this work is very similar to the paper Deep Learning via Semi-Supervised Embedding done by Weston et al. (2012).  This paper claims three extensions on top of Weston et al. (2012):

(a) their approach can handle multiple graphs from multiple domains - but it is also trivial for the network in Weston et al. (2012) to handle such cases;
(b) they provide more experiments on "properly constructed" graphs and different kinds of neural networks - this seems to be a valid point but the novelty of this is questionable;
(c) they propose to use the graphs as inputs when there is no features to be used - the work by Weston et al. (2012) can also use such features to tackle similar problems.

It looks to me like the authors just took the Weston et al. (2012) model and rebranded it as "Neural Graph Machines", and then did some experiments and not even comparing with Weston et al. (2012).  If the authors do think this paper is significantly different from Weston et al. (2012), then I hope they can clarify what are the fundamental differences that Weston et al. (2012) model cannot do but the model in this paper can do.
review_title: This paper is very similar to the work done by Weston et al. (2012)

final_score: rating: 3: Clear rejection
review_content: The paper proposes a model that aims at learning to label nodes of graph in a semi-supervised setting. The idea of the model is based on the use of the graph structure to regularize the representations learned at the node levels. Experimental results are provided on different tasks

The underlying idea of this paper (graph regularization) has been already explored in different papers – e.g 'Learning latent representations of nodes for classifying in heterogeneous social networks' [Jacob et al. 2014],   [Weston et al 2012] where a real graph structure is used instead of a built one. The experiments lack of strong comparisons with other graph models (e.g Iterative Classification, 'Learning from labeled and unlabeled data on a directed graph', ...). So the novelty of the paper and the experimental protocol are not strong enough to accpet the paper.

Pros:
* Learning over graph is an important topic

Cons:
* Many existing approaches have already exploited the same types of ideas, resulting in very close models
* Lack of comparison w.r.t existing models

review_title: Review

final_score: rating: 3: Clear rejection
review_content: The authors introduce a semi-supervised method for neural networks, inspired from label propagation.

The method appears to be exactly the same than the one proposed in (Weston et al, 2008) (the authors cite the 2012 paper). The optimized objective function in eq (4) is exactly the same than eq (9) in (Weston et al, 2008).

As possible novelty, the authors propose to use the adjacency matrix as input to the neural network, when there are no other features, and show success on the BlogCatalog dataset.

Experiments on text classification use neighbors according to word2vec average embedding to build the adjacency matrix. Top reported accuracies are not convincing compared to (Zhang et al, 2015) reported performance. Last experiment is on semantic intent classification, which a custom dataset; neighbors are also found according to a word2vec metric.

In summary, the paper propose few applications to the original (Weston et al, 2008) paper. It rebrands the algorithm under a new name, and does not bring any scientific novelty, and the experimental section lacks existing baselines to be convincing.
review_title: Very similar to previous work, rebranded.

final_score: rating: null
review_content: Dear authors,

 The proposed model seems very close to existing works like:
Learning latent representations of nodes for classifying in heterogeneous social networks. 
Revisiting Semi-Supervised Learning with Graph Embeddings
...
(these articles also propose relevant references in the same family of approaches).

Could you please explain the differences with these approaches since the differences seem quite small. Moreover, you only compare with an ''unsupervised'' baseline (node2vec). Could you please provide stronger experimental results by comparing with other existing approaches? Could you also comment on the values of the different hyper-parameters (values of the alpha, size of the latent space, etc...) ? 

review_title: pre-review question

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes the Neural Graph Machine that adds in graph regularization on neural network hidden representations to improve network learning and take the graph structure into account.  The proposed model, however, is almost identical to that of Weston et al. 2012.

As the authors have clarified in the answers to the questions, there are a few new things that previous work did not do:

1. they showed that graph augmented training for a range of different types of networks, including FF, CNN, RNNs etc. and works on a range of problems.
2. graphs help to train better networks, e.g. 3 layer CNN with graphs does as well as than 9 layer CNNs
3. graph augmented training works on a variety of different kinds of graphs.

However, all these points mentioned above seems to simply be different applications of the graph augmented training idea, and observations made during the applications.  I think it is therefore not proper to call the proposed model a novel model with a new name Neural Graph Machine, but rather making it clear in the paper that this is an empirical study of the model proposed by Weston et al. 2012 to different problems would be more acceptable.
review_title: Very similar to previous work.

final_score: rating: null
review_content: In section 4.2, why is the number of training epochs limited to 20? Why not reporting other Zhang et al results as a baseline?

review_title: Comparison with Zhang et al.

Meta-review: The paper is an interesting contribution, primarily in its generalization of Weston's et al's work on semi-supervised embedding method. You have shown convincingly that it can work with multiple architectures, and with various forms of graph. And the PubMed results are good. To improve the paper in the future, I'd recommend 1) relating better to prior work, and 2) extending your exploration of its application to graphs without features.
Decision: Reject