Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The proposed idea presented in this work is based on the observation that skilled programmers write and read source codes in a hierarchical way and pay close attention to the conceptual entities like statements, tokens, sub-tokens, and the mapping relations between them. Driven by this observation, the authors argue that a multi-granularities formulation incorporating the entities in different granularities may benefit the code summarization task. To this end, the paper proposes constructing a pyramid-shaped input representation based on a pyramid attention mechanism for efficient feature aggregation and distribution across different hierarchies. The proposed framework is evaluated on two source code summarization benchmarks where it surpasses the prior works and reaches new state-of-the-art results.
strengths_and_weaknesses: **Strengths**
Generally, the paper did a good job of presenting the motivation, describing the proposed method, and demonstrating the effectiveness of the proposed approach based on experiment and evaluation.

**Weaknesses**

- The paper argues the existing structure-aware code summarization methods are inconsistent with the multi-grained nature of source code. Then the paper linked human behavior to the multi-granularity formulation of source code which I am skeptical to accept. Consequently, it is not clear why the multi-granularity nature of source code would encourage programmers to write summaries accordingly. The paper should verify the main claim of the work.
- Evaluation is only in one language.
questions: - Are the experiments done with multiple seeds? Are the improvements achieved by PA-former is significant? I just want to make sure that the improvements are not due to some randomness in the experiments.

- Can we apply the proposed framework for other languages, e.g., python, go, PHP? Performing experiments on the CodeXGlue summarization benchmark may add value to the work.

- Is the proposed idea scalable? Can we apply the idea to fine-tune a large language model?

- Figure 1 shows that PA-former generates a better summary by producing additional useful information. But a comparison with ground truth, especially based on BLEU score-like metric, should not be able to demonstrate the efficacy of the proposed approach. What do the authors think about it?

- The paper shows improvements on the target task. But does the paper show whether the proposed model truly learns a pyramid-shaped input representation? Is there any way the authors can validate the claim?


limitations: My concern is around the authors' observation that skilled programmers write and read source codes in a hierarchical way and pay close attention to the conceptual entities like statements, tokens, sub-tokens, and the mapping relations between them. I am skeptical about accepting this argument. As Figure 1 shows, the PA-former generates a summary with more information, but the reference (human-written summary) does not have the extra information "if the string can be parsed as an integer". Therefore, I felt the observation of the authors is not well supported. Performing a human study to demonstrate that the proposed model-generated summaries are of higher quality would be a plus for this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces pyramid attention for source code summarization and
integrates it into a transformer architecture (they call the resulting
transformer PA-former). The pyramid attention combines three levels of looking
at the source code (statement, token, and subtoken level). The levels are
combined bottom up from subtoken to statement level, and also with a
cross-granularity interaction across levels where each level can query every
other level (including its own).

strengths_and_weaknesses: ## Strengths

I think that the presented method is elegant, logical, and not overcomplicated.
It also improves state-of-the-art results.

## Weaknesses

I think the writing could be improved. Overall the paper is written well and is
easy to follow, but there are numerous typos and errors. I found Section 3.1 a
little bit hard to understand at first reading, especially the "Semi-grained
sequence" part, the writing could be much improved there. Correcting the typos
and the plurals would help, and maybe a figure about the AST like Figure 5 in
the Appendix and concrete examples if there is space for that.

There is also a typo in Eq. (2), the disjunction should go over k instead of j.

Some examples of typos and errors:
- line 19: as => because of
- line 124 and 125:  order=> ordered, sequenc => sequence, denotes => denote
- lines 131 and 132: corresponds => corresponding, reveals logic side of program => reveals the logic side of the program.A
- line 172: lenght => length
- line 290: We => we
- line 304: The => the



questions: Table 5 shows that using just the fine-grained representation for decoding
slightly outperforms using all three representations. I find this
counterintuitive. Could you explain this finding?

In Table 6, the difference between PA-former and w/o grammar is very low. Could
it mean that the input design (e.g., the token grammar types) is less important,
and the good results are obtained because of the pyramid attention and the
structure the input provides for it ($M_{G \rightarrow T}$ and $M_{L \rightarrow
G}$?

limitations: I cannot think of any limitations which are not addressed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a multi-granularity method for the code summarization task. Specially, one code snippet is first hierarchically broken into three granularities: statements, tokens, and sub-tokens levels. Then a new mechanism named pyramid attention mechanism is proposed for the aggregation of multi-granularity features. The authors also conducted extensive experiments to demonstrate the performance of their model and the effectiveness of the model components.

strengths_and_weaknesses: This paper targets automatic generation of natural language descriptions for source code, which could be useful in a variety of source code related tasks.  The paper illustrates and explains the idea well. The paper is generally well organized and clearly describes the architecture of the model. The proposed multi-granularity method for code representation is novel to me.

The improvement of the proposed method over related work is rather small. Although the paper says that “the proposed methods obtain 5% improvement over the strong NeuralCodeSum on a middle-size dataset and achieve new state-of-the-art results”, actually the results achieved by the proposed method are very similar to the results of CAST and TPTrans, as shown in Table 2. The differences are around or below 1.0 in almost all metrics. It is not clear if those improvements are indeed significant.

Also, there is lack of human evaluation to show the effectiveness of the proposed method in practice. Automatic evaluation (using BLEU metric, etc) may not be sufficient. Some papers (e.g. [a] and [b] below) on code summarization show that automatic metrics mainly calculate the textual similarity between the reference and the generated summaries, rather than the semantic similarity. Thus, human evaluation is needed to confirm the effectiveness of a code summarization method in practice.

[a] Li, J., Li, Y., Li, G., Hu, X., Xia, X., & Jin, Z., EDITSUM : A Retrieve-and-Edit Framework for Source Code Summarization.  Proc. ASE2021.

[b]J. Zhang, X. Wang, H. Zhang, H. Sun, and X. Liu, “Retrieval-based Neural Source Code Summarization.” Proc. ICSE2020.

Furthermore, the below reference [c] shows that the BLEU metric, which is widely used in existing code summarization work, actually has many variants. Ignoring the differences among these variants could greatly affect the validity of the claimed results. Please check the BLEU scores in Tables 2-6 and make sure that they are calculated in the same way.

[c] Shi, E., et al., On the Evaluation of Neural Code Summarization, in Proc.  ICSE2022.

Minor:

. The reference [26] is wrong. CAST is published at EMNLP2021 instead of ICPC2021.

. Line 259. Bleu -> BLEU. Please keep the term consistent in the paper.

questions: 1. Are the improvements achieved by the proposed method over existing work significant?

2. How did you calculate the BLEU scores of your approach and the baselines in Tables 2-6? 

limitations: No, the authors did not adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents the method to tackle the source code summarization task by leveraging multi-scale features that appear in it. Such a human-like multi-scale comprehension is modeled relatively easily using the Transformer architecture enhanced with the information at each granularity level and cross-granularity interactions.
strengths_and_weaknesses: *Pros*:
1. I like how the paper is structured and how illustrations benefit the narrative. A reader gets a nice overview before digging into details (Section 3.1 provides details on something that was previously elegantly explained on a high level, making it easy to process)
2. Design choices are motivated well (e.g., use of the vocabulary, graining of the sequence, cross-granular interactions). Important ablations are performed; in general, paper naturally answers most questions that a reader wants to ask.
3. The proposed method is elegant, does not overcomplicate the process, and seems easily reproducible. Moreover, the results suggest it provides stable gains on both used datasets.

*Cons*:
1. Since page 6, the quality of grammar and writing radically decreases, suggesting it needs proofreading, as, e.g., "All model is trained using..." is one among other trivial mistakes.
2. A bit of obscured equations (4)-(9) suddenly makes processing this part much harder than the rest of the paper. Interestingly, this gracious note made me understand the steps easily: "we just need concat the three sequences and the relative position maps, then directly adopt a relative position encoded multi-head self-attention layer to implement the cross-granularity interaction module." I recommend reworking the "Cross-granularity interaction" section a bit and, again, allowing someone to proofread that.
questions: 1. How much longer are inputs being passed to the multi-head cross-attention?  
1a. How much did the computational cost increase due to this? 
2. I wish the paper provided the results with error bars. Is it possible to add this? 
3. How was the segmentation on each level performed? Were there any additional tools used for such a segmentation? Is the description of "Data preprocessing" section enough?
limitations: Authors do not provide limitation section nor analyze the results qualitatively. I believe one important factor that would be fair to disclose is the computational expense of the model, relative to the Vanilla Transformer baseline.
review_title: null

Meta-review: metareview: The paper presents a multi-granularity input represenaion and a pyramid attention mechanism for code summarizaiton tasks. After extensive discussion, the reviewers still cannot agree on accepting or rejecting this paper. The key discussion points and my opinion are summarized in below. 


1. Performance improvement -- a few reviewers point out the performance improvement is relatively small (about 1%) compared with baseline. With the additional error bar provided by the authors, it seems to me the improvement is statistically significant. The authors also provide sufficient ablation study to justify the improvement. Although it's arguable if the proposed appracoh is substantial, the progress of AI is often driven by incremental improvement in terms of performance. Therefore, I'm less concerned by this issue. 

2. Comparison only on 1 language and 2 datasets. I partially agree with the authors and reviwers Xtyo that the paper already conducted extensive experiments and the merits of the proposed approach are justified. However, I disagree with the attritue that the comparison on 1 language is sufficient given the recent progress of code summariziton. As the proposed approach is mainly justified by empirical comparison, conducting results on a limited dataset raises the concern whether the proposed approach is generalizble to other languages and datasets. It also makes future work harder to compare with this work. I especially disagree the point that some earlier papers only compared on limited datasets. These papers are published earlier than the benchmark CodeXGlue has been released. As most recent baselines are compared on CodeXGlue, there is a need to justify the missing of results on this dataset. Besides, the argument of the dataet is noisy the performance is low do not seem rigorous and reasonable to me.

3. Human evaluation. The authors provide a preliminary study of human evaluation on the generated outputs. On one hand, it shows the proposed approach indeed improve the quality of the summary, but on the other hand, the study requires more rigorous design. I would suggest including the human evaluation on the main text rather than in appendix. 

4. Presentation. The paper is mostly well-written and provide nice intuition behind the propose method. However, I also agree with Q2Zz that some statements might overclaim and require justification. The later part of the paper has significant number of typos and require a careful proofread. It's pity that the authors do not take the opportunity during the rebuttal period to revise the paper.

Overall, I think the paper has sufficient merits but still have room to improve. 
confidence: Less certain
Decision: Accept