Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors present an algorithm for actively learning the nodes in a graph that should be sampled/labeled in order to improve classifier performance the most. The proposed techniques use both the graph structure, and the current classifier performance/accuracy into account while (actively) selecting the next node to be labeled.

There seem to be two main contributions in the paper. 1) The propose to sample nodes nodes based on "regional" uncertainty rather than node uncertainty 2) They use an variant of pagerank to determine nodes that are central, and hence most likely to affect subsequent classification in graph convolution classifiers. Both approaches seem to be interesting. There are experiments to show effectiveness of these techniques, and there are some interesting observations (for example, that the APR technique works better for smaller sample sizes, while the regional uncertainty methods do better for larger sampling fractions.).

While both techniques seem straightforward extensions of previous approaches (and are well explained in the paper),  the experiments indicate that they work better than prior approaches. It would have been nice if the authors had also discussed ways in which one or more of these techniques could be combined though, or discussed how we could pick the right approach (in a more empirical way, since it is not clear what the threshold for high sampling rate/low sampling rate distinction is, or if it varies from problem to problem)
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The paper studies active learning in graph representations. To decide which nodes in a graph to label during the active labeling, the paper proposes two approaches. First it argues that one should consider region-based measures rather than single nodes. Second, it proposes to adapt the page rank algorithm (APR) to determine which nodes are far away from labeled nodes.

Overall it remains unclear *how* to select the right strategy (before seeing the results for a dataset) i.e. which of the proposed approaches or variants should one select for a new dataset.

Strength:
-	One of the ideas of the paper, using region entropy over single node entropy makes sense to me.
-	The paper evaluates on 6 datasets and compares different variants as well to related work on 2 datasets.

Weaknesses:
1.	The paper contains several confusing and contradicting statements or claims which are not supported by the experimental results:
For example:
1.1.	“APR outperforms all other methods at low sampling fractions”.  This is supported neither in Table 1 nor Table 2, where APR is frequently not highest performing
1.2.	 “We have here shown that the accuracy of AL when uncertainty is computed regionally is much higher than when either local uncertainty or representative nodes are used”, this is not the case on CiteSeer in Table 1
1.2.1.	Also e.g. “Region Margin” is worse than random on 5% Email-EU; or “Region Margin AE” on 3% SubeljCora (Table 2) [It is unclear how to select with or without AE]
1.3.	“We outperform all existing methods in the Cora dataset, and get very similar results to the best accuracy obtained by Chang et al methods:”
1.3.1.	The difference to Cai et al. on Cora is very small (improvement by only 0.002), while on Citeseer the performance is comparatively bigger (Cai et al. is by 0.016 better)
1.3.2.	It should be “Cai et al”
2.	Clarity: I found the paper rather difficult to understand and follow:
Some specifics:
2.1.	The introduction could be more concisely discussing the motivation, the main idea of the paper, as well as contributions.
2.2.	Figure 1: according to the caption, APR should point to node 15, but in the figure it points to node 14. From the example it makes much more sense to label node 14 to me.
2.3.	Page 6 mentions twice the “ratio between APR and PR”, is this is this used/evaluated in the results?
2.4.	The decision what is bold and what is not is not consistent throughout the table 2.
2.5.	“Thus, hybrid techniques, combining several approaches, outperform using only one approach have been proposed.” It is not clear what this refers to and where the hybrid techniques have been evaluated.

Minor:
The paper contains many minor writing issues, e.g.
-	missing spaces, e.g. “distribution,and” (page 2)
-	Table 1: incomplete sentence: “∗∗ scores for smaller budget, since it was the”
-	Table 2: unclear: “accuracy without content”

The paper’s incorrect claims (weakness 1) are highly concerning and strongly suggest rejecting the paper. Furthermore, the clarity of the paper should be improved to follow the author arguments and make the paper easier to read.


review_title: Official Blind Review #1

Meta-review: The paper proposes a method for performing active learning on graph convolutional networks. In particular, instead of performing uncertainty-based sampling based on an individual node level, the authors propose to look at regional based uncertainty. They propose an efficient algorithm based on page rank. Empirically, they compare their method to several other leading methods, comparing favorably.  

Reviewers found the work poorly organized and difficult to read. The idea to use region based estimates is intuitive but feels like nothing more than just that. It's not clear if there is a mathematical basis to justify such a method (e.g. an analysis of sample complexity as has been accomplished in other graph active learning problems, Dasarathy, Nowak, Zhu 2015). 

The idea requires further study and justification, and the paper needs an improved exposition. Finally, the authors were not anonymized on the PDF. 
Decision: Reject