Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Short summary
---------------------
The authors propose a graph convolutional network (GCN) approach to perform the diagnosis and prognosis of COVID-19 from chest CT scans. They propose a novel pooling that takes into account the edges weights compared to recent methods. They validate their method on one dataset and compare the results to one recent baseline.

Strengths
--------------
The paper is very well written and easy to follow. The figures and tables are clear and pleasing. The work is sound.

Weaknesses
-------------------
While the motivation is clearly depicted, I lacked the clinical relevance or justification to tackle this problem. In addition, case analyses or the usefulness of the proposed prognosis tool are not discussed, and it is difficult to estimate the complexity of the task without this information (e.g. how clinicians evaluate severity, is it obvious from the pulmonary damage?).
On the other hand, the technical contribution is underwhelming: the proposed method is only marginally better than its recent counterpart, no confidence intervals are provided and the effect of the preprocessing is much larger. It is also unclear what the motivation was for some choices, how these relate to the literature or how they were practically implemented (see detailed comments).

Novelty
-------------
I did not find the technical contribution of significance, especially given the results comparing DAP to ASAP. While the problem of COVID-19 diagnosis and prognosis is timely, it lacked clinical insights.

Clarity
----------- 
Very clear, very well written. A couple of details are missing (see detailed comments), but overall this is an enjoyable read. I wished there was a discussion of the technique, the results and the limitations.

Rigor
--------
The methods seemed sound. I wished confidence intervals were provided.

Detailed comments
---------------------------
- node clustering: can parts of the graph be “dropped” during this step?
- If the adjacency is already defined as the cosine distance between nodes, isn’t message passing already doing/reinforcing some kind of clustering? How does the clustering ranking (based on A) relate to comparing nodes in “input” space compared to the layer at stake? Given the proposed next layer connectivity, isn’t the next layer A an exponent version of the first layer A for the terms in top k?
- How to define h_d and k? What is their effect on model performance?
- Is there a justification behind the aggregation? Is there a reference for this? Doesn’t that strategy assume that each feature in 1...D’ across nodes N’ represent the same dimension?
- How would this compare to using a “master node” (Gilmer et al., 2017), where all nodes could write and read from a separate node, to allow for long-distance communication?
- From my understanding, one-drop localization is similar to occluding a cluster at a time. Is there a reference for this method? 
- How is k_s chosen? Why is a fixed number chosen compared to e.g. a “% of prediction variance explained” type of approach?
- Any clinical input in this work?
- Is the number of slices acquired or selected through sampling mentioned in the text? This would help in illustrating the size of each graph.
- What are the reasons behind the choice of the 2 preprocessing techniques? They seem arbitrary without referencing.
- On such a low number of patients, confidence intervals are strongly recommended. I am not confident about the significance of DAP compared to ASAP, especially given the large performance gap between feature extractors.
- Comparison with CNNs would be interesting. This is especially important given the impact of the feature representation. I am wondering whether GNNs are not “too sensitive” to the choice of initial node embedding (which also defines A) and whether this should be investigated in more depth, or maybe CNNs display similar performance without needing this additional step.
- Won’t IoU on slices heavily depend on the choice of k_s?
- Selecting slices “included” in the set of slices highlighted by one-drop seems like an arbitrary choice to me. The fact that slices are sequential is specific to this type of data, and this choice seems like an “a posteriori” choice that better aligns with the ground truth.
- IoU is not reported across all patients, but rather on cherry-picked examples  in Table 2.
- There is no discussion

Minor
-------
- One-drop: I am confused on the wording, where the authors mention the score on the “target class”. Do they refer to the predicted class of the model?
- The prognosis task is undefined


review_title: Interesting application but lacking novelty

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary
The manuscript proposes  a distance aware pooling method to use in  graph convolutional neural for predicting whether a subject is infected with Covid-19 (diagnosis) and progression of the disease (prognosis). Experiments were conducted on CT images from three groups: Covid-19 group, common pneumonia group, and heathy group with about 900 samples in each group. The proposed model achieved 94.7% accuracy.    

Pros:

The manuscript proposes  a distance aware pooling method which, with the help of features generated through inception v3, could achieve an error rate of a little bit over 5%


Cons:

In terms of the potential impact of the work, my concerns are as follows:
1.	Although the model trained on the dataset achieved a relatively low error rate (~ 5.3% ),  there is no evidence that this error rate is low enough for the model developed to be really useful, e,g. could actions taken based on a diagnosis method of this level of error rate potentially create a local epidemic?
2.	The model is not tested on an independent dataset that consists of other type of lung diseases other than common pneumonia. Is it possible that the model could actually categorize CT from a patient with another type of lung disease as Covid-19?  Thus, this 94.7% accuracy probably needs to be taken with a grain of salt when the model is to be used to test patients clinically.

In terms of the technical soundness of the work and writing of the manuscript, my concerns are as follows.

1.	Although the manuscript claims the distance aware pooling method as its main contribution, the description of the method is no more than one figure. It is unclear how this pooling function is computed exactly. Nor is it clearly stated how does it fit in overall model of the graph convolutional neural network as well. Even in the figure, some of the terms is used without giving clear definition (e.g. adjacency matrix centered at a specific node ). Further, There is no discussion of strengths and limitations of the proposed pooling method and the method it is trying to compare against ASAP, specifically in the context of CT imagining. 
2.	Almost the same can be said about the section of “improved receptive field”.  What is the motivation of improving receptive field? Why is it relevant in the case of graph formed of CT imaging. (incidentally, it seems that how authors do not mention in the manuscript how the edges are added to the graph, but I might have missed this). The impact of the radius “h_d” on the model performance is  not discussed, either.  
3.	How many time have the authors randomly split training and testing set? the variance from different runs are not reported. It is a standard practice to report both the mean of the performance from different runs as well as the variance. 
4.	Since the problem that the manuscript is trying to solve is a graph classification and main contribution of the manuscript is a pooling method, the authors should consider comparing their pooling methods with different pooling methods and using different graph classification methods as baseline as well. 
See
Zhang et al   Hierarchical Graph Pooling with Structure Learning      https://arxiv.org/abs/1911.05954  
Li et al   Graph Matching Networks for Learning the Similarity of Graph Structured Objects      ICML 2019

Overall, the manuscript could be improving by stating more clearly the motivation of their pooling method in the context of CT imaging,  giving a more complete and rigorous definition of the pooling function,   comparing their method against different pooling methods  using different classification methods,  and having a more detailed discussion on the strengths and limitations of different pooling methods. 

review_title: Recommendation to Reject

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper is an application of GCN with good features on chest CT scan images for Covid-19 diagnosis and prognosis. First of all this is a relevant and appreciated effort when the world is fighting the pandemic. Hence some bonus points is directed towards that. As a whole, to the representation learning community, it adds limited research values apart from being an application of GCN which is aligned to the application track of ICLR. The paper claims that with less than 1% number of total parameters in the baseline 3D ResNet model, their method achieves 94.7% accuracy for diagnosis, which is marginally better than state of art - however whether the model was over-fitted is not clear. Prognosis information is an added claim, though automation part is not integrated.
It will be better if approach is compared with another concurrent work - "Covid-19 Classification by FGCNet with Deep Feature Fusion from Graph Convolutional Network and Convolutional Neural Network" - https://www.sciencedirect.com/science/article/pii/S1566253520303705.

I found too many repeating verbatim and technical descriptions - most can be delegated to references / citations or section numbers.

Sections 2.2 and 2.3 is not needed as such - if any special benefit wrt to current problem is given it will be helpful - more text from appendix can be added to paper than spending on explanations of easily searchable material.

Fig. 1 is too simple - go deep - can get influenced by similar works in state of art papers - how they present.

"Empirically, it is shown to be more robust for densely connected graphs" - this claim needs more substance.

"may be bootless" - use inefficient.

In Node clustering, how is 'k' determined?

As a general comment, too many to point individually - break the sentences around mathematical terminology into parts and also check the flow of grammar.

"Haar wavelet function with resolution 3" - more details on this.

I am not sure about the ablation studies and following of Train-Eval-Test based experiments (refer Andrew Ng ML Yearning).

Conclusion - write with quantitative numbers and a few sentences on author's learning / observations from overall experiments / method. And to conclude from my side, the paper is easy to read, needs some polishing and quantitative explained presentation; but overall is a relevant application paper.
review_title: Relevant to Covid-19 the current need, relevant to conference application track (GCN) - the paper is easy to read but core novelty is a bit low wrt ICLR's standards.

Meta-review: The paper presents a GCN-based solution with a distance aware pooling method for diagnosis and prognosis of COVID-19 based on CT-scan. It aims to address an important and timely problem. The proposed solution is reasonable. 

The paper receives mixed ratings, and therefore we had extensive discussions. It is agreed by all of us that  

(1) the novel contribution of the proposed method is relatively low compared with standard ICLR papers; 

(2) the evaluation is interesting, but could be improved with state-of-art baselines on CT-scan (not limited to GCN-based method); 

(3) the authors have improved the writing of the paper significantly, which convinces two reviewers to elevate their scores. 

The paper addresses a timely topic, but there is still room for improvement in methodology and evaluation. We hope that the reviews can help the authors prepare a strong publication in the future. 

Decision: Reject