Probing API Name Knowledge in Pre-trained Code Models
Authors: Terry Yue Zhuo, Xiaoning Du, Zhenchang Xing, Jiamou Sun, Haowei Quan, Li Li, Liming Zhu


Reviewer paper summaries: The paper investigates the ability of pre-trained code models to understand and use API names correctly through a novel evaluation framework.
Summary of strengths: The findings show the importance of integrating API structure during pre-training.
Summary of weaknesses: 1. The methodology relies heavily on cloze-style tests for probing API knowledge. While this approach is useful, the paper does not sufficiently justify why this method is the best or how it improves upon existing probing techniques. Moreover, there might be concerns regarding the robustness and generalizability of the cloze-style tests.
2.  The evaluation is primarily based on the CSNet dataset, which, as mentioned in limitations, may not cover the full range of API usage and might be outdated. This limitation could impact the generalizability of the findings. The paper does not address how newer APIs or other datasets could affect the results.
3. The paper lacks comprehensive comparisons with state-of-the-art models (after 2021, larger than 500M, like codet5xxlarge) beyond the selected few. The performance metrics provided (e.g., P@k scores) show that the models still perform suboptimally on API name predictions, raising questions about the effectiveness and practical significance of the proposed probing framework.
4. The paper does not convincingly demonstrate the practical impact of its findings on improving pre-trained code models.
Comments, suggestions and typos: Missing reference

[1]  PLBART in paper baseline    Unified Pre-training for Program Understanding and Generation

[2] CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation

[3] CAT-probing: A Metric-based Approach to Interpret How Pre-trained Models for Programming Language Attend Code Structure
Overall assessment scores: 2.5