Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: While the completion of missing node attributes is an important problem, the Structure-Attribute Transformer (SAT) is recent work that tackles that problem with leveraging all the observed attributes equally. The authors challenge the part of using all the observed attributes in an equal manner, and propose the SAT learning strategy based on active node sampling. For the active sampling, authors propose the the representativeness and the uncertain measurements for nodes on a graph and use them for active sampling. When the proposed sampling strategy is combined with SAT, it shows better performance in node classification tasks compared to the other baseline models, including the original SAT.
strength_and_weaknesses: * Strengths
- The proposed node measurements are clear and reasonable to represent the importance for active sampling.
- The manuscript is easy to follow.
- The proposed method combined with SAT shows the better performance compared to baseline models

* Weakness
- The objective of the active sampling-based strategy would be great if it is clearly defined. The efficiency is described as the objective, but it needs to be further clarified and better-defined.
- Despite aiming the efficiency, the experiments or evaluations with respect to efficiency or training time are missing.
- The proposed active sampling is very tightened with SAT. It is not clear why the sampling should be coupled with SAT, not some general embedding-based learning. More generalization of the proposed method or clear explanation specific to SAT would be great to have. 
- Experiments could be more enforced. The ablation study with respect to learning speed would be great to have. For example, when the same learning strategy is applied (starting from 1% training and expanding the training size over epochs), how soon does the proposed method achieve a reasonable performance as opposed to the uniform sampling? Or, how different is the learning algorithm's convergence speed when gradually expanding the training node set as opposed to directly training on the entire dataset?

clarity,_quality,_novelty_and_reproducibility: - The manuscript is straightforward to follow and the proposed method and experiments are clearly present.
- The proposed method is incremental in the sense that some common measurements are used for active sampling and it is applied to only a certain kind of method.
summary_of_the_review: Despite the clearness and fair performance of the proposed method, the proposed method needs to be 1) more generalized, 2) evaluated on the matters of interests, and 3) more studied to the role of each strategy part to understand the impact of proposed methods better.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposed an active sampling method to improve an existing approach called SAT for graph node attribute completion. It claims that the observed nodes are not equally important, and they designed several metrics to measure the nodes’ representativeness and uncertainty and use them for active sampling. To better control the weighting scheme, it further proposed to use Beta distribution to adjust the metric weights. Experiments show that the proposed method achieve superior performance on node classification and profiling tasks. 
strength_and_weaknesses: There are some interesting points in the paper, e.g. the importance metric of the nodes. However, its overall quality is not good enough for ICLR. 
1.	Motivation. First of all, the scope of the paper is too small. Node attribute completion is not a standard task, and SAT is not a well-known base model. Only designing a specific sampling method to improve the SAT model looks very trivial.   
2.	I do not see the necessity of active sampling. As described in the related works, active sampling is originally used for sampling new data to label, but this paper mainly used it to adjust the training sample weights. I do not understand why they should start from a small portion of observations. They can just train a SAT using the full data, and then resample all the data to adjust the weights. 
3.	Datasets. All datasets are manually made but not from a real problem (the four datasets all have full attributes). That makes the “node attribute completion” task look like a toy problem. But we know that in real world there should be such tasks. If possible real data and tasks will be much better.
4.	 Evaluation. Using the node classification task for evaluation is not reasonable. Take a very simple example, for each label we just generate a one-hot vector and use it as the missing node feature, the node classification accuracy will be 100%, but that does not mean the feature completion is good.
5.	Ablation study. The node metric has three components. I think an ablation study which removes one or two components will be necessary to show the importance of each metric component.


clarity,_quality,_novelty_and_reproducibility: Other minor issue: 
Typo in Equation(2). The second z_p \sim p(z) should be z_a

summary_of_the_review: The solved problem is not significant, and the approach and evaluation have something unreasonable.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers the problem of imputing the missing attributes of nodes in a network.This is done by identifying the most informative nodes for renewing the training set after each training epoch. Some uncertainty and representativeness metrics are proposed to achieve the task of identification. Their method achieves improved performance on node classification and profiling tasks with 4 real datasets.


strength_and_weaknesses: Strengths: the paper tackles an important problem known to be of interest in practice (e.g. attributes of individuals are often missing due to confidentiality or privacy settings); improved performance compared to some state of the art techniques. 

Weaknesses: Specifically, what is the ` node attribute completion problem' - this is clear at a high level but the complexities concerning the exact problem are never discussed (type of missing attributes, proportion of missing attributes and how it links with properties of the network). The intuition and reasoning behind some of the key choices and assumptions is not explained and remains unclear. For example, why is a shared latent space assumption required for the node attribute completion problem (second para in Section 4.1)? What do you mean by 'information density'? Is this somehow linked to network edge density? Is `structural centrality' linked to network centrality measures commonly studied in this area?
There are many vague statements here, for example (second para in Section 4.1, line 3): `If there is a dense distribution of representation vectors in a local region of the latent space, the corresponding nodes will have more similar features ..'
What does this mean exactly and why? Some mathematical notation here could have helped in stating things more precisely. 
Under equation (4): 'The larger \phi_density is, the more representative the node is ....' : How does this follow from the formula above and what exactly is the node representative of and why? 
Why is the PageRank score specifically used as structural centrality?
 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written in some parts (e.g. the literature review and in high level description of the methodology), however, lacks clarity in the technical description. I feel that important mathematical details are missing. I am unable to comment on the originality of the work as I am not very familiar with literature on this problem. 
summary_of_the_review: The paper tackles an interesting problem and leads to improved performance, however, the assumptions and choices made for the implementation (e.g. why these are optimal/intuitive in any way) of the proposed method are not clear. Also links with existing literature on networks seems to have been ignored.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a technique to prioritize which samples to select next in an active learning setting for node attribute completion in graphs. The proposed method uses metrics related to the "density" of a data point as well as its centrality in the graph. The proposed approach is used as input to SAT, a previously-proposed model for node attribute learning.
strength_and_weaknesses: Strengths:
- The criteria designed by the authors to select the next point to train on is well-motivated
- The authors leverage on a state of the art model for node attribute detection

Weaknesses:
- Lack of novelty. The main contribution is a method to prioritize training points in an active learning setting. However, the model that actually learns node attributes (which is arguably the crux of the problem) has been proposed before.
- Incremental empirical results. It is not surprising that the proposed approach marginally improves performance over SAT and GAT. Any sensible heuristic to select data points would be better than not doing anything.
clarity,_quality,_novelty_and_reproducibility: The technical description was clear overall. I think the introduction would benefit from some additional editing. At the moment, it reads as if it were  written in a rush.
summary_of_the_review: I think that the proposed approach is rather incremental in the literature of learning node attributes in graphs. My main criticism of the paper is that the authors rely on an existing model for the bulk of the machine learning task, and their contribution can be summarized as a heuristic method to suggest new data points in an active learning framework. 

Since the work is empirical in nature, I also think that the authors could provide more evaluation on why their particular metric is of value over potential alternatives. While I agree with the authors that density and centrality are important criteria for this task, there are numerous metrics that one could come up with. For instance, instead of PageRank, we could consider node centrality or betweeness centrality.
The authors instead spend decide to report a comparison of competitors for SAT. In my opinion, this takes away from their contribution, since it makes SAT the focus of the paper. I would reframe the experiments to show how the proposed metric is superior to other density/centrality alternatives, all in the context of SAT (and potentially GAT).

Lastly, as I mentioned above, the empirical results are not surprising, since the authors are choosing points in an intelligent way. It is obvious that one should expect an improvement from this, and I don't see how the proposed work or results add much to the existing literature.

I would recommend this work to be published as a short paper, but I cannot recommend it to be featured as a full research paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: A version of a GNN is the Structure-attribute Transformer (SAT) framework which incorporates information about attributes.  Some of these attributes can be missing.  An active learning approach is used to developing a learning approach to tackle this problem.  Experiments are done on four popular datasets.  The paper is well written.

Note, the task of selecting which data to select from existing classified data is called curriculum learning.  The paper does appear to be doing curriculum learning and not active learning, as pointed out by Reviewer #P86Z (who didn't use the right term), although the methods are very related.

The results presented in Tables 1 & 2 appear good.  Reviewers would like to see additional studies, however.  One issue was the fact that the data used had complete attributes.  The test data didn't though, but for training, all attributes where complete.

Reviewers felt use of the SAT model was very specialised.  Authors point out their approach can be adapted to other GNN models.   So it should be.  As it stands the method looks incremental.
justification_for_why_not_higher_score: All agreed on Reject
Decision: Reject