Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a method called "Dimensional Reweighting" for graph convolutional networks. The method involves a reparametrization of GCNs (by adding an extra reweighting block in each layer), which the authors show theoretically can reduce variance. The authors supplement this with extensive empirical experiments showing slightly improved performance by adding their method to existing methods.

I would vote to weakly reject this paper for two key reasons - first, I think the writing can be improved and explanations can be clarified, especially for people less familiar with the field like myself. Second, I am not certain how significant the final experimental improvements are (other than on the Reddit dataset), as most of the numbers are not that far apart, and it seems that different methods in the literature already produce fairly different results.

Overall, I think the structure of the paper is fairly good. I feel that a few things should be modified for clarity.
- You claim a 40% improvement in error rate in the intro, which sounds enormous. I would say "relative error rate" to avoid overclaiming, because 40% improvement sounds like you are reducing (absolute) error from 60% to 20%, while in reality you are reducing error from 3.6% to 2.1%.
- In section 2.1, did not know if the projection matrix W was learned or predefined.
- I was not sure why you used sigma_g and sigma_s as opposed to just sigma in equation (5). Do you use different activation functions? Also, I did not find what activation function the authors end up using in their experiments.
- I may have misunderstood something, but the theory does not seem to match the proposed method exactly. The mean-field theory analysis has the activation function after the reweighting by S but before multiplying by W, while the framework in Section 2.2 has the activation after the reweighting by S and after the multiplying by W. I am not sure how much this difference makes, or if it is significant, but I think it should be explained by the authors.
- I also did not understand exactly what the "variance" the authors are reducing is. The authors talks about "reducing the learning variance brought by perturbations on the input," but when is the input ever perturbed for GCNs? Explaining this more clearly would improve the motivation for this work.
- I would appreciate a better intuitive explanation of the measure "K." I gather that it is related to the "variance" being reduced, but it is different from that.

The experimental results are good overall, as the proposed method tends to give the best results (by a small margin) across the board. I especially appreciate that the authors performed many experiments over many different datasets and repeated runs 20 times to try to get confident estimates of how well each method performs. I also appreciate that the authors cleaned up the Citeseer and Cora datasets, and I hope the cleaned datasets will be useful for the research community.
With that said, I do not know how significant the improvement is. I think something that would be helpful would be to measure the "variance" that the method is supposed to be reducing (since it sounds like it is not exactly the same thing as K), and showing that in a table as well. This would show experimentally that the method achieves its intended goal.

Minor comments
- I would recommend that the authors proofread for English grammar and style in updated versions of the paper. For example, in the first paragraph of the introduction, the authors use "is proposed" instead of "were proposed" and typo "broad" as "board."
- Just curious, why did you choose a 2 layer network with 2 activation functions for the Dr block? Why not just have 1 hidden layer?
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The paper is out of my research area. I could only provide little recommendation. I have tried to read this paper, but it was rather tedious with heavy notations. It would be more friendly to represent the models in visible way for example using diagrams as I can see that the model is a sequence matrix operators with non-linear transformations after that. The paper states that the proposed DrGCNs can improve the stability of GCN models via mean field theory. The experiments were conducted  on benchmark datasets and the proposed method was compared to several GCN variations. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a method, known as DrGCN, for reweighting the different dimensions of the node representations in graph convolutional networks (GCN). Specifically, the representation of every node is element-wise multiplied with a weight vector, which is parameterized as a function of the average input node representation, where the function is a two-layer neural network. 

At a conceptual level, this is similar to various existing normalization schemes, such as batch normalization and weight normalization. While it is claimed in Section 4.3 that the difference is that “[batch normalization] reduces variance between samples, while DrGCN reduces variance between dimensions”, I am not sure if this characterization is accurate. Batch normalization actually makes each dimension have unit (sample) variance and so does not make the variance of each dimension small. What it does is to make the sample variance of each dimension equal, which is also what DrGCN tries to do, i.e.: reduce variance across dimensions (since samples in the case of GCNs are the representations of different nodes). DrGCN is also similar to weight normalization because like weight normalization, DrGCN learns a transformation on top of the vanilla representation (in the case of weight normalization, the vanilla representation is the normalized weight vector; in the case of DrGCN, the vanilla representation is the node representation before reweighting). The conceptual contribution therefore seems incremental. 

Incremental conceptual contributions would be fine if (1) they result in a surprising theoretical result, or if (2) they result in a surprising improvement in empirical performance. Unfortunately, neither seems to be demonstrated in this paper. 

In the theoretical analysis, there are various occurrences of unjustified leaps of logic; as a result, what is claimed to be shown by the analysis is different from what is actually shown, and it is unclear what is actually shown is substantially related to the proposed method. For example, in Section 3.1, the paper says that “GCNs are different from fully-connected networks only in \tilde{A}, and degrade to fully-connected networks when \tilde{A} = I. So, our analysis can be somehow be generalized to DrGCNs.” The first sentence is true; in other words, it says that fully-connected networks are a special case of GCNs when \tilde{A} = I. However, the second sentence does not follow - just showing a special case (which is what the subsequent analysis does) does not say much about the general case. Relatedly, the architecture that is analyzed is of the form H^l = W^l \phi(S^l H^{l-1}) + b^l for l = 1, …, k, whereas the architecture that is proposed is H^l = \phi(W^l S^l H^{l-1} \tilde{A}^{l} + b^{l}) for l = 1, …, k. The latter cannot be cast as the former unless \tilde{A} is diagonal. Also, the caveat of the mean field approximation are not stated - whatever result that is shown is only valid at the infinite width limit, which is different from what is claimed in the abstract, which says that "We prove that DrGCNs can reduce the variance of the node representations by connecting our problem to the theory of the mean field.” Additionally, the analysis is done in the case where S is directly parameterized, whereas the proposed method parameterizes S as the output of a two-layer neural network. I presume the reason why latter was done in practice because it worked better empirically. Because this is not explained by the analysis, the theory is incomplete, and so this caveat should be clearly stated in the abstract. 

In the experimental results, the performance improvement of DrGCN over layer normalization is not statistically significant. Also, DrGCN is only compared to other normalization schemes (batch normalization and layer normalization) on one dataset, and so there is insufficient evidence to conclude that DrGCN generally works better than existing methods empirically, 

Additionally, in section 2.1, it is claimed that “sampling-based GCNs still lie within the framework of equation (2), as we can set all unsampled edges to 0 in \tilde{A}”. This does not seem to be accurate, because in sample-based GCNs, different edges are sampled in every minibatch, and so there is no fixed choice of \tilde{A} that makes these GCNs equivalent to the formulation in eq. (2). 

Also, if the goal is to reduce variance across dimensions (as the paper claims in Section 4.3), why was the average node representation fed into the two-layer neural network rather than its reciprocal?
review_title: Official Blind Review #2

Meta-review: As Reviewer 2 pointed out in his/her response to the authors' rebuttal, this paper (at least in current state) has significant shortcomings that need to be addressed before this paper merits acceptance.
Decision: Reject