Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Positive
1. The paper studies a universal policy for labeling nodes on graphs with multiple training graphs which can be transferred to new unseen graphs.
2. The paper focuses on minimizing human efforts in obtaining labeled data. The model is based on active learning and transfer learning. 

Negative
1. The proposed method combines existing models as the solution, which is heuristic and lacks persuasive theoretical proofs. 
2. In graphs, data (nodes) to be labeled are highly correlated. However, there is no method for solving this challenge.
3. In Section 3.2, ACTIVEL EARNING ON A SINGLE GRAPH, the authors formalize the problem, i.e., learning a policy for selecting a set of nodes for annotation, as a sequential decision process, and the reinforce algorithm is applied to optimize the objective function. However, explanation about the active learning is confusing. More details are needed to explain their respective goals and to explain how to integrate active learning and reinforcement learning. 
4. The authors claim that the details of heuristic features are represented in Appendix, please add these information.
5. The settings of active learning need more consideration. The total budget for active learning is set as $5\times N_{class}$. How to choose these nodes? Is it to select all samples at once or in batches during iterative epoch? If the samples are selected in batches, what is the specific experimental setting?
6. The experimental method about active learning. The paper focuses on minimizing human efforts in obtaining labeled data. Compared to the results of the model before selecting, how significant is the improvement after selecting all the node in the budget? More experiments are needed.
7. Minor format issue: the fonts in Table 2 and Table 3 are different. 

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Thank you for the author response.
Original review:

This paper presents a method for active learning on graphs, including a novel setting of transferring an active learning policy to unseen graphs.  The problems tackled here are important and the method is shown to improve over previous work in some cases.  On the down side, the evaluation may be missing one important method of comparison, the reasons for the proposed approach winning over previous work are not made explicit, and the empirical advantage of the approach is inconsistent (by my count, in the majority of cases the F1 advantage over previous work is within the standard error).  This paper has strengths but I feel it needs further refinement before publication.

The introduction of the paper claims that existing approaches for active learning on graphs are domain-specific and may not apply well to new domains.  But later, in the experiments, different reasons for the proposed approach's wins are given (in particular, on large graphs the proposed approach does relatively better vs. AGE which the paper suggests is due to the more complex nonlinear models used in the proposed approach).  In general, this paper’s approach tends to win over the primary baseline (the AGE method from Cai et al. 2017), but the wins are relatively small and inconsistent (esp. taking into account the standard error) whether the methods are evaluated in the single-graph setting or the transfer learning setting (of the homologous or heterogeneous variety).  If the limitation of previous work was domain-specificity, I would expect to see much larger wins on the transfer learning setting.  In general, an analysis that explains more what is driving the gains of this approach over the AGE approach would help us know how to build on this paper’s method in future work.

I was curious why the paper does not compare against the following work, which also presents an approach that wins over AGE:
“Active Discriminative Network Representation Learning,” Gao et al., IJCAI 2018

Lastly, the distillation-based approach, which learns graph-specific policies that are trained to fit their target graphs and to minimize their KL divergence from a single global shared policy, was interesting.  The fact that it doesn’t work much better than the joint policy is somewhat disappointing, but it’s still interesting.

Minor:
Sec 3.2: unclosed parenthesis in first paragraph
“Moreover, we also average the struc2vec features of all previously annotated nodes to capture the historical information” -- since the model has only node-level features, I didn’t understand how this average across multiple nodes was fed in as a node feature.  Is it used in all nodes?  Only annotated nodes?
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors proposed a new method for active learning on node classification with GCN. RL based framework is used. The labeled graph is treated as state and the action is labeling the nodes. Validation accuracy on the hold out set is used as reward. Further transfer learning framework is also proposed, where graph-specific policy and master policy are jointly learned. Experiments on benchmark dataset show the effectiveness of proposed method compared to several baselines.

The idea of applying RL on active learning with GCN seems to be new and it sounds natural and technically. Also the idea of transferring the learned policy to new graphs make sense for similar graphs. However, the empirical results are a bit weak and not convincing enough for me. Please find the detailed comments below.

1. Is the state defined on node level or graph level? Eq (1) is defined on node level, but I believe it should be a global policy on graph.
2. All the results have a rather high variance. To compare such results, the authors should make a significant test. Otherwise, one cannot say that the performance from one method is better than the other. Especially, for Table 3 and 4, DAG-distill performs not better than DAG-Joint.
3. What do "0-4, 5-9,..." mean in Table 3?
4. Can the authors show curve as in figure 2 for table 2 and 3. It is important to see the progress for active learning.
5. Why the results differ so much in Figure 2 for only 1 query? I believe the first one should be randomly picked. Thus all the methods should perform equally.
6. "Graphs encode the relations between different objects and are ubiquitous in real-world." Typo in first sentence.
7. homologous or homogenous?
review_title: Official Blind Review #3

Meta-review: Paper proposes a method for active learning on graphs. Reviewers found the presentation of the method confusing and somewhat lacking novelty in light of existing works (some of which were not compared to). After the rebuttal and revisions, reviewers minds were not changed from rejection. 
Decision: Reject