Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a system to predict types in untyped or partially typed code. The authors propose to treat type prediction as a code completion problem, which can be solved using transformer models trained for code. However, just applying a transformer model "out of the box" would fail because there can be caller-callee dependencies in the code, sometimes across files, and these dependencies are essential to correctly predict types.

To solve this problem, the authors use static analysis to form a usage graph of these dependencies, and encode these as strings to provide additional context for the model while making predictions. They traverse the usage graph according to a topological ordering, and make predictions for each function sequentially. When the model makes a prediction for types in a function, these are carried forward to be used while predicting types for the next function in the sequence. Thus, the model is "conditioned" upon its previous type predictions.

They train their model using the "ground truth" type predictions, and evaluate it on a dataset of Python code without any type annotations. They show that their model achieves a large improvement in prediction accuracy (from ~55% to ~80%) as compared to other state of the art models.
strength_and_weaknesses: **Strengths**

1. Very precisely written. Excellent paper structuring that makes it easy to understand the approach. The motivating example is insightful and clearly illustrates both the problem and the benefits of the proposed solution.

1. I appreciate the thorough dataset setup as described in 4.1. I think it's very important to ensure that train and test data are properly separated. Interesting to see that Type4Py's performance falls by so much.

1. Very large performance increases. 55% to 80% is quite impressive.

**Weaknesses**

* Why do you evaluate only on public APIs, especially when you mention that that is a major reason for the drop in performance of Type4Py (Appendix A.5)?

* I agree with the point about not including "simple" labels inferable from Pyre. But just to be thorough, it would be nice to include the results for the exact same setting as Type4Py. It would be okay even if Type4Py performed better than TypeT5 in this setting (no free lunch - every model has an inductive bias, but not all biases are bad!)

EDIT - adding one more crucial point that slipped my attention in the initial review.

* Do you re-train the baseline models on your custom (filtered) dataset? As in, when you present the results of Type4Py etc, do you retrain/fine-tune them on your collected dataset or do you just use their provided weights? It is vital to retrain. I am now wondering whether some of the extremely large performance improvement is due to these models not being trained on your dataset.

**Questions for the authors**

* Why not evaluate the coherence error (Table 4 and 5) on the baseline models Typilus and Type4Py?

* UserToUsee performs worse than independent. Why is it a "bad ordering"? As shown in your motivating example, aren't there cases where you need User info before Usee, and vice-versa?


clarity,_quality,_novelty_and_reproducibility: **Clarity/Quality**

Overall the paper is very well written, however I have a few comments/questions:

* In Figure 4, why is the line from `eval_on_dataset` to `predict` dotted? Isn't there an explicit call (eval.py, line 12)?

* Page 5 under "Usees" : I'm a little unclear about why $\textbf{s}_\text{usee}$ is constructed like this. If a function `foo` calls two functions `bar` and `abc`, then `foo` is in users(`bar`), and `abc` is in usees(`foo`). So $\textbf{s}_\text{usee}(\texttt{bar})$ would contain `abc` also? Why would that be beneficial?

* In the two-pass system (forward and backward), do you retain type predictions from the forward pass while doing the backward pass? What if they contradict?

* In Table 2, 4 and 5, it would be nice if the best model in each column could be made bold.

**Novelty**

This approach is conceptually simple - static analysis, encoding, and a seq2seq model, along with some miscellaneous details (2 pass, etc). However, the performance gains are very impressive, and the evaluation is thorough. Therefore, this has considerable *empirical* novelty.

**Reproducibility**

As far as I can see, the tool has not yet been made publicly available. The description of the approach is reasonably thorough, and although there are a few unclear details in the approach, I think it should be possible to replicate (possibly with some assistance from the authors).
summary_of_the_review: This paper presents an approach to automatically annotate types in Python code. The approach is conceptually simple and I have some concerns about the construction of the dataset, but a) the paper is well written and well structured, b) the evaluation is very thorough, and c) the performance gains are impressive. Given these factors, I would recommend acceptance.

I am slightly less than confident in my evaluation because this is a very active research problem (predicting missing type annotations), and I am not familiar with all the related work. It is possible that I have overlooked a very similar approach or a relevant baseline that should have been compared against.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Type inference is a challenging task for the Python language given its dynamic nature. Scholars have recently proposed machine learning (ML)-based techniques to infer types for Python. Previous techniques infer types based on seen examples during training, i.e, they cannot synthesize types, which hinders their ability to infer project-specific types. This paper presents TypeT5, a CodeT5-based model, whose main idea is to create a usage graph to capture usee-user relationships with static analysis, which provides global information to the CodeT5 model about code elements that need to be typed. Also, the proposed model has a two-pass sequential decoding that is conditioned based on previous type predictions. This allows bidirectional information flow from usees to users given a usage graph. Overall, the experimental results show that TypeT5 outperforms the state-of-the-art type inference models, namely, Type4Py and Typilus.
strength_and_weaknesses: ### Pros
- A novel type inference technique based on CodeT5, which leverages global information in code and performs two-pass sequential decoding for inference.
- The paper is well-written and has a good presentation. The proposed approach is explained with sufficient detail to reproduce the work.

### Cons
- The evaluation of the proposed approach is not fully sound and rigorous. The evaluation has some (major) shortcomings that need to be addressed. I have explained the shortcomings in “feedback for the authors”.

clarity,_quality,_novelty_and_reproducibility: ## Feedback for the authors
### Major comments

- I would appreciate seeing a comparison between TypeT5 and HiTyper [2] in the evaluation section. HiTyper [2] is based on the combination of static analysis and deep learning. It also outperforms Type4Py in the prediction of rare and user-defined types. Also, HiTyper should have been mentioned in related work.
- For the evaluation, developer-provided type annotations are used, which are not always sound or coherent [3]. This might be a potential threat to the validity of the obtained results. To address this, in Type4Py’s paper, the authors used a type-checked dataset for both training and evaluation. Also, using a type-checked dataset might change the produced type errors by TypeT5. Hence, I highly suggest to type-check ground truth in the dataset using a type checker, e.g., Mypy.
- As mentioned in the text, Type4Py’s performance is pretty low compared to its original paper. I believe this is because Type4Py was trained on a different dataset and also different type normalization rules were used for BetterTypes4Py. This is also true for Typilus. I assume BT4Py’s projects were cloned in 2022 whereas MT4Py’s projects were gathered in Sep. 2020. For a fair comparison, I highly recommended to re-train both Type4Py and Typilus on BT4Py and evaluating them on the test set of BT4Py and IT4Py.
- I highly suggest including Typilus in Table 3, “Accuracy comparison on all types (common + rare).” to have a consistent and rigorous comparison in the evaluation section.
- It should be pointed out that in both Type4Py and Typilus papers, the depth of the parametric types is reduced to 2. For example, the type annotation List[List[Tuple[int]]] is converted to List[List[Any]]]. This conversion is not performed for the BetterTypes4Py dataset, which makes it even harder for Type4Py and Typilus to predict some complex types with deep nested levels. This is another reason why I believe that both Type4Py and Typilus should be re-trained on BT4Py with the same type normalization rules.

### Minor comments
- In the ablation study, TypeT5 produces quite a number of type errors, i.e., ~5K. Given this, a critical reader might question the usefulness and practicality of TypeT5 if used by developers. I would suggest showing a percentage of type errors considering all the predictions made for code elements in the test set, similar to what was done in Typilus’ paper regarding type-checking the model’s predictions.
- Looking at Table 5, the two-pass decoding scheme produces negligible accuracy improvement over the other strategies (e.g., Random). One might ask is it worth using considering its additional overhead at inference time? Also, with TwoPass, the number of type errors still seems high. Based on the reported results in Table 5, the two-pass decoding scheme does not seem convincing to use. Maybe adding inference time to Table 5 might better justify using the TwoPass strategy.
- In the text, it is unclear what are exactly the code elements used to construct the usage graph? Only two examples were provided in the text for functions and variables. Please clarify this in the text and, if needed, provide a comprehensive list of code elements in the Appendix, which are extracted to build a usage graph. This helps to reproduce the results of the paper.

## Questions for the authors
- Is the BetterTypes4Py dataset de-duplicated? Code duplication [1] has been shown to have adverse effects on the performance of ML models and it would blur the results. It is essential to perform this step.
- What dataset is exactly used in subsection 4.3 for the ablation study of TypeT5? I could not find it in the text. Please clarify this in the text. I assume it is BetterTypes4Py by comparing the accuracy of TypeT5 in Tables 3 and 4.


## References
[1] Allamanis, M. (2019, October). The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (pp. 143-153).

[2] Peng, Y., Gao, C., Li, Z., Gao, B., Lo, D., Zhang, Q., & Lyu, M. (2022, May). Static inference meets deep learning: a hybrid type inference approach for python. In Proceedings of the 44th International Conference on Software Engineering (pp. 2019-2030).

[3] Ore, John-Paul, et al. "Assessing the type annotation burden." Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 2018.


summary_of_the_review: This paper presents a novel technique, TypeT5, to tackle the type inference challenge for Python. Also, treating the type inference task as code completion is new. Overall, I believe that this paper has good potential to become an influential work in ML-based type inference research. However, I have some (major) comments regarding the soundness of the evaluation of TypeT5, which hampers the paper from reaching its full potential and impact. I briefly mention the main comments here as they are explained in detail later in the review.

- Human-provided type annotations are used as ground truth, which is not always valid [3]. This can harm the soundness of the obtained results in the evaluation.
- The state-of-the-art approaches, i.e., Type4Py and Typilus, are not fairly represented and evaluated.
- Lack of comparison with a very similar published work, HiTyper [2], which is also a very recent SOTA type inference model for Python.

To increase the recommendation score to the acceptance level, all three raised concerns regarding the evaluation should be addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: With the aim of type inference for untyped code, the paper feeds, in addition to tokenized source code, contextual information gained from static analysis to the seq2seq-based code completion model called CodeT5. CodeT5 itself can infer types of elements in untyped code (including user-defined and parametric types) successfully, but incorporating the contextual information improves the accuracy performance marginally. Furthermore, the paper provided the idea of sequential decoding, which refines the contextual information gradually using the inference result of the model, and showed that the use of sequential decoding makes type inference more coherent.
strength_and_weaknesses: Strengths:

+ The proposed approach, especially the use of CodeT5, succeeds in inferring user-defined and parametric types.

+ The paper experimentally demonstrated that contextual information and the sequential decoding improve accuracy and make type inference more coherent.

+ The documentation is well written.

Weaknesses:

- The paper could provide a finer-grained analysis of the experimental result. Specifically, I am curious about the model's performance for parametric and under-defined types because this is the first work that addresses both kinds of types. Unfortunately, I cannot find it in the paper because all the accuracy numbers in Tables 2, 3, 4, and 5 are for the entire dataset, which includes primitive types such as int.

- The paper does not report the cost of constructing user graphs. Because the construction needs to examine the overall codebase and the libraries it uses in the worst case, it might take too much time. Perhaps in practice the graph construction is not problematic, but I'm unsure whether it is the case.

- (Minor) The notion of coherence needs to be better explained in the body of the paper. The footnote seems far from the formal definition of coherence provided in A.6.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow. The most significant contribution to achieving the reported performance is using CodeI5. However, the paper provides several inventions to improve the performance and makes type inference more coherent. The appendix also provides more information to explain the part unclear in the paper.
summary_of_the_review: The paper used CodeT5 to infer types and provided methods to improve the performance of the model with respect to accuracy and coherence. It would be nicer to provide a finer-grained analysis of the experimental result and the cost of constructing user graphs.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present a new approach to type inference by using T5 architecture to generate types in a seq2seq fashion. Their proposed model, TypeT5 is CodeT5 fine-tuned on a subset of ManyTypes4Py named BetterTypes4Py. TypeT5 uses static analysis to compose its contextual input with a preamble, main code, users, and usees. The authors found that such static analysis can improve TypeT5's type generation across a series of metrics and across the type error count. The authors also wanted TypeT5 to generate consistent type information and exploit its previous type predictions (as additional context at inference time), so they introduced sequential decoding where passes of inference are made from usee-to-user and vice versa. The model is trained normally (like T5) with teacher-forcing to directly optimizing on ground truth labels rather than its own predictions.

The results show that pre-training (CodeT5 and TypeT5) improves accuracy measures across all, simple, and complex types. The authors demonstrate that their techniques for introducing more context was helpful for the model at inference time. The static analysis required for the context appears to be lightweight. The authors introduce a type error count metric that was not previously introduced as a metric but is practically useful; consistency is equally important to accuracy and type accuracy does not indicate type consistency, thus, such a metric is helpful.

The authors found some interesting attributes to their model, namely, the importance of preamble information (which could improve other pre-trained approaches that do not use preamble information), and that user-to-usee decoding performed worse than independent predictions.

Finally, the authors took advantage of TypeT5s conditioning on previous predictions as an opportunity to correct incorrect predictions. In other words, during the sequential decoding process, if the decoding was incorrect, an human in the loop could correct the context and allow the model to continue inference with a higher precision rate. 
strength_and_weaknesses: Strengths
- Design: the design of the model is important in the field of type inference. The application of CodeT5 was overdue and the implementation of static analysis was fitting for the model architecture
- Comparison with SOTA: Comparison with state of the art and corresponding metrics in previous works was ok. 
- Use of type error count is good.
- Examples in Appendix were good.
- Ablation study was appropriately done
- Application of Human in the loop was well done

Weaknesses
- Comparisons on previous datasets in full and BetterTypes4Py/InferTypes4Py would strengthen results. The release of these datasets with the paper submission would help reviewers understand the distribution and demographic of types that are being evaluated. Table 1 is good, but the dataset would be best.

- Authors do not evaluate Typilus model on types that are not common. Authors claim that released Typilus model is not capable of predicting types outside of common (top100). The authors should verify this with Allamanis etal as it is a SOTA across rare types.

- The authors should validate claim that Type4Py inference is lower because of leakage. But in the appendix, the authors state that the Type4Py discrepancy between train and test in the figure on their dataset is due to (1) Type4Py evaluation on local variables (not just APIs) and (2) compiler inferable types (easier type subset). Both of these reasons would contribute to a higher evaluation on the original ManyTypes4Py dataset but does not explain the discrepancy between train and test on their dataset. With the availability of Type4Py and ManyTypes4Py, validating this claim should not be difficult; the authors might find that their intuition led to a tangential discovery maybe a degraded performance on novel types not available in the KNN search, for example, and not data leakage.
clarity,_quality,_novelty_and_reproducibility: Quality of work is good, the paper was well written and the evaluation was transparent. Given the model weights and the dataset, the paper would be reproducible.

The originally of the work was fine as it is moving the needle on SOTA type inference using the latest and greatest model architectures. The application of static analysis was a good touch.
summary_of_the_review: My recommendation of this paper is based on a strong understanding of this works position in neural type inference. This paper does contribute to the existing body of research of neural type inference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work adopts a language modeling approach to tackle program type prediction. Specifically, it fine-tunes CodeT5 on a type annotation task, and extends the standard approach by both including richer sets of context inferred via static analysis, and ordering the generation process to simulate information flow through a program usage graph. The resulting model provides reasonable improvements over a baseline approach that just uses CodeT5, which in turn performs far better than prior work.
strength_and_weaknesses: **Update:** The authors have provided additional results in response to my concerns below, which underscore that the TypeT5 consistently improves over CodeT5 on rare types, and provide both additional ablations under the requested conditions and information on the training and inference _cost_. I have thus increased my score.

This proposed approach balances the use of powerful neural language models with a clear understanding of how developers type-annotate their projects. The result is an intuitive and rigorous approach that yields improvements over all alternative methods. Working against it is the fact that the improvements over a plain CodeT5 model are quite small, which stands in especially stark contrast to how vastly that model in turn improved over existing baselines. This is no fault of the authors -- pretrained language models are hard to beat -- so I am inclined to not consider this a major problem, but it does imply that the paper should make a stronger argument for its current performance. Specifically, three parts of the reporting should be expanded:

1. The emphasis on (or alternatively, the definition of) adjusted accuracy feels unwarranted. While I could see filtering out `None`/`Any` labels as a reasonable step, replacing fully qualified types with simple ones and dropping `Optional` types would alter the type distribution quite significantly. This matters because the gain of TypeT5 over CodeT5 is quite a bit larger under the "adjusted" metric as it is under the "full" one, suggesting that it does better at inferring comparatively less precise annotations. Given this, please provide a second ablation for Tab. 4 focusing on full accuracy, so as to capture the factors that make TypeT5 outperform CodeT5 in the more challenging setting.

2. One of the paper's stated goals (starting from the abstract) is to yield better type predictions on rare and complex types. That it realizes the "complex" part of this felt convincing, given the results in Tab. 2 & 3. It is unclear, however, why Tab. 3 does not show the accuracy solely on _rare_ types, but rather combines them with the common type results that were already shown separately before. Given that common types are far more common than rare ones and thus contribute predominantly to these results, I assume that the accuracy drop between tables 2 & 3 signals a far lower accuracy on rare types for all models. Please provide an equivalent table with the rare type results only (and, preferably, replace Tab. 3 in the paper with said table). There is no harm at all in reporting low accuracies on this task; it is objectively hard for other models as well. But the current setup makes it very hard to gauge whether the paper achieved this sub-goal -- if anything, in some cases it looks like it failed to; e.g., on InferTypes4Py, CodeT5 drops about 9% (points) from Tab. 2 to Tab 3. (full accuracy) while TypeT5 drops 10%, suggesting that it performs *worse* on rare types than the former, or at least, that its performance improvement over CodeT5 is much slimmer on rare types. This is important information.

3. A modest performance gain should be presented along with the increase in cost, if any, that accompanies using the more performant technique relative to its baseline. Please discuss how much more expensive TypeT5 is, both for training (in terms of data collection, training time), and inference (time, memory), than CodeT5.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is quite well written and reasonably novel. I list a few typos below. The data processing steps and model implementation details are reasonably well documented, including in helpful appendices.

Typos:
- P1: missing space between "sythesis" and "(Li". Same on previous line
- P3: extra space after "usees".
- P4: "and edge" -> "an edge"
- P8: "it allow" -> "it allows"
- P8: "On BetterTypes4Py ..." -> "On the BetterTypes4Py"

summary_of_the_review: The work is well written and provides a natural approach that balances the performance potential of a pretrained language model with a series of improvements that provide the model with access to information that is pertinent to type annotation. The resulting model performs better than CodeT5, but only by a modest margin, and the evaluation leaves several open questions about the interpretation of this improvement. My review asks for expanded results that, if in line with the narrative in the paper, should make the improvements more convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a method for predicting types in code using a T5-based model. Additionally, using static analysis TypeT5 includes additional context that is helpful for the problem.

Strengths:
- A simple, yet powerful approach to a useful application.
- A interesting combination of static analysis and T5.

Weaknesses:
- No novelty with respect to deep learning models and methods.

Overall, this is a good paper and I suggest to accept it.
justification_for_why_not_higher_score: While this is a useful approach, the problem is not of wide interest to the community nor are there not many learnings that can be transferred to other subdomains of machine learning.
justification_for_why_not_lower_score: There is no good reason to reject this work. It's technically correct, improves the state-of-the-art, and provides some value to the learning-for-code community.
Decision: Accept: poster