Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary

This paper proposes a retrieval-augmented method for generating code summarization. The model encodes the input code based on its graph structure (Code Property Graph) with a hybrid GNN architecture. The model augments the initial graph representation of the input code based on the representation of the top-1 retrieval result. It also augments the final graph encoding with the BiLSTM encoding of the retrieved summary.

The proposed model is evaluated on a newly curated C code summarization data set and shows state of the art performance compared against previous systems.

Strengths
- Releases a new C code summarization data, which will be beneficial for the community.
- This paper reports human evaluation results.
- Ablation study shows the retrieval augmentation and the new hybrid GNN architecture is helpful.

Weaknesses
- The model is not evaluated on any existing code summarization benchmarks. Showing that the proposed architecture is generally applicable by getting good results on more benchmarks will make the story a lot more convincing.
- Could include more analysis (see below for details)

Other questions/comments
- Would be nice to provide the mathematical formulation of “Step 3: Retrieved Summary-based Augmentation”
- Would it be possible to ablate code-based summarization vs. summary-based augmentation separately? It would be interesting to see their relative impact.
- Suggestion: it would be helpful to provide an input-output example earlier in the paper.
- Suggestion: would be nice to include real examples of retrieval results in the analysis section.
- Have you considered using top-k retrieval results instead of top-1?

review_title: Retrieval-augmented code summarization model with state-of-the-art results on a newly curated dataset

final_score: rating: 7: Good paper, accept
review_content: Summary:

This paper leverages similar code-summary pairs from existing data to assist code summary generation. The model first retrieves a similar code snippet from the existing database. Then, the author applied GNN over the code property graphs (CPGs).  A challenge is that CPGs are typically deep therefore it is difficult to capture long dependencies. The author proposed an attention mechanism to capture global information between nodes, and then a hybrid GNN layer encodes the retrieve-augmented graph.  Finally, a generator takes both GNN's output and the retrieved text summary and predict outputs. Experimental results over a new C code indicates that the proposed method outperforms both IR and neural generation methods.

########################################

Reason for score: 

Overall, I vote for accepting. Both the idea of leveraging existing code and also the adaptive layer to capture long dependencies are interesting and the experiments look solid. Although I would still like to see the results from previous existing datasets.

########################################
Some comments about the experiments:

a. As an application study, it is still necessary to compare the model over previous benchmarks, even though there are some issues with those datasets. 

b. A pair of missing ablation studies are: a generator still takes the text summary of retrieved code, but not use the augmented graph; and vice versa, the generator only takes the graph information but not the retrieved text summary. This can further indicate which part of the retrieved information is more useful. 
review_title: This paper leverages similar codes to help generate code summarization, and an attention-based dynamic graph model is introduced to further capture the global graph information.

final_score: rating: 7: Good paper, accept
review_content: 
Overview:

The authors tackle the code summarization problem. They retrieve the most similar code and use it as additional input features. They also using GNN to get features from static and dynamic graphs. They evaluate their results on their collected C projects (C-Code-Summarization Benchmark) with 1-2% improvement on automatic evaluation.


Reasons to accept:
* They collect and release a new challenging C benchmark for code summarization.
* They propose a hybrid-GNN solution to capture global graph information.

Reasons to reject:
* No evaluation on any publicly available datasets. Even though there are some issues about duplication, I will still expect to see such a comparison with other baselines.

* Attention-based dynamic message passing of graph model is not proposed in this paper, and I don't think it is necessary to have this "hybrid" design, maybe only dynamic one is enough. As shown in Table 1, the dynamic is more important than static (Although we do see an overall performance, except the out-of-domain meteor score, using both are still better, but it is marginal).


Questions & Suggestions:
* Can you provide more dataset information? For example, the average lines of the code, the average length of the natural language summarization. It seems to me from the examples that each example only has few lines of code and the summarization is very short, it is more like a topic modeling task. 

* To my understanding, this work is not the first work combining retrieval solution with generation model for code summarization (e.g., Retrieval-based Neural Source Code Summarization) so please modify some of the claims in the paper.

* Can you provide more details about the human evaluation? What is the agreement value? 

* "z is the similarity score, which is introduced to weaken the negative impact of c′ on the original training data c", how do you find the best c?

* When you run baselines on your dataset, did you do a hyper-parameter search or just use their default setting (especially the Rencos model)?

* There are some retrieved-augment language models in the NLP field that the authors may want to take a look and compare with, for example, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks".

* Did you run any baselines that are based on pre-trained language model, such as BERT, BART, T5, or even more code related like CodeBERT: A Pre-Trained Model for Programming and Natural Languages?
review_title: Good Results and More Evaluation Needed

Meta-review: This paper proposes an interesting method for combining retrieval-based models and graph neural networks for source code summarization. Finding new ways of bringing in additional context for graph-based models is an important research direction in this space, and the paper presents a novel and effective approach. The initial submission was missing experiments on existing benchmarks, but new experiments presented in the discussion phase are enough to resolve that concern. Reviewers are unanimously in support of acceptance. 
Decision: Accept (Spotlight)