TypeT5: Seq2seq Type Inference using Static Analysis
Authors: Jiayi Wei
Greg Durrett
Isil Dillig
Venue: ICLR 2023 poster (2023)
Keywords: Type inference
Code completion
Static analysis
Transformers
Pre-training

Abstract: There has been growing interest in automatically predicting missing type annotations in programs written in Python and JavaScript. While prior methods have achieved impressive accuracy when predicting the most common types, they often perform poorly on rare or complex types. In this paper, we present a new type inference method that treats type prediction as a code infilling task by leveraging CodeT5, a state-of-the-art seq2seq pre-trained language model for code. Our method uses static analysis to construct dynamic contexts for each code element whose type signature is to be predicted by the model.  We also propose an iterative decoding scheme that incorporates previous type predictions in the model's input context, allowing information exchange between related code elements. Our evaluation shows that the proposed approach, TypeT5, not only achieves a higher overall accuracy (particularly on rare and complex types) but also produces more coherent results with fewer type errors---while enabling easy user intervention.

Published as a conference paper at ICLR 2023
TYPET5: SEQ2SEQ TYPE INFERENCE USING STATIC ANALYSIS
Jiayi Wei, Greg Durrett, Isil Dillig Department of Computer Science
University of Texas Austin
{jiayi, gdurrett, isil]0cs. isiljecs.utexss.dd utexas.
ABSTRACT
There has been growing interest in automatically predicting missing type anno- tations in programs written in Python and JavaScript. While prior methods have achieved impressive accuracy when predicting the most common types, they of- ten perform poorly on rare or complex types. In this paper, we present a new type inference method that treats type prediction seq2seq pre- pre-trained language model for code. Our
code infilling task by leverag-
ing CodeTT5, a state-of-ther method uses static analysis construct dynamic contexts for each code element whose type signature is to be predicted by the model. We also propose an itera- tive decoding scheme that incorporates previous type predictions in the model's ss input context, allowing information exchange between related code elements. Our evaluation shows that the proposed approach, Type'T5, not only achieves a higher overall accuracy (particularly on rare and complex types) but also produces more coherent results with fewer type errors errors-wiile enabling easy user intervention.
INTRODUCTION
In languages like Python and JavaSeript, the lack of static type system makes it harder to maintain and analyze codebases. To address this issue, gradual typing (Siek & Taha 2007) was proposed allow type annotations to be incrementally added to untyped codebases, thereby marrying the benefits of static typing with the convenience of easy prototyping. As result, many mainstream programming languages, including Python and JavaScript, have already adopted this idea, and re- searchers have also developed learning- baaded techniques to predict missing type annotations (Ray-
chev et al. 2015 Hellendoorn et al.) 20188 Wei et al. [2020] Pradel et al. 20200] Allamanis et al,
2020 Pandi et al. 2020 Jesse et al. 2021 Mir et al. 2022 Jesse et al. 2022 Peng al. 2022).
Meanwhile, with the advent of large-scale pretraining and explosion transformer architec- tures, seq2seq models have proven be very effective for programming tasks like code comments generation (Panthaplackel et al.) 22020), completion (Wang et al.) 2022 Ahmma et al. 2021). ). and synthesis (Li et al.) (022). One particularly attractive feature of such models is that, due to the of subword tokenization (Gage) 1994: Schuster & Nakajima 2012 Sennrich et al. 2016). they can generate arbitrary code expressions- including novel identifier names and AST structures-at test time. However, unlike code completion tasks that can often work well with just the surrounding code as context, effective type inference generally requires non-local information, including code fragments that may belong to an entirely different file. For instance, consider function that passes a generically named parameter x directly into another function g. It can be hard to figure out the type of by just looking at J's body. When programmers find themselves in such a situation, they often inspect the callers and callees of f. sometimes even transitively, in order to figure out the intended type of Thus, in many cases, looking at the immediate context of given variable may be insufficient for accurately predicting its type.
Our approach, TypeT5, solves this challenge by using static analysis to identify which parts of the codebase are useful for each prediction. In particular, we construct a so-called usage graph, where nodes correspond to code elements (i.e., functions or variables whose types we want to predict) and edges denote a potential user-ure usee relation between them. Given such a graph, we then encode the users and usees of given code element in form that resembles normal code and feeds them as

---

Published as a conference paper at ICLR 2023
additional contexts to the transforeer model. To take full advantage of the seq2seq paradigm, also propose an iterative decoding scheme that pass in previous type predictions using the contexts, allowing information to be propagated between distant code elements across the entire codebase.
We have implemented TypeTS on top of the popular CodeT5 model and use it to synthesize type annotations for untyped Python code. Our evaluation compares Type'T'S with three state-of-the-arr type inference tools (Allamanis et al. 2020 Mir et al.) 2022 Peng et al. 2022) and a CodeTT base- line that does not leverage static analysis. The results show that TypeT5 outperforms all baselines by a large margin, while drastically improving the accuracy on rare and complex types. Our ablation studies confirm the benefits of the various modifications we made to the CodeT5 baseline, while an additional type checking experiment shows that the proposed iterative decoding scheme also im- proves the coherence of the produced type assignments, resulting fewer type constraint violations. Finally, we explore an alternative use case of our model, where the user interactively inspects the model's predictions and makes necessary corrections. The result demonstrates the usefulness of our approach developer tool to annotate entirely untyped projects-on on average, the user only needs to to correct one every five model predictions.
To summarize, this papers makes the following contributions:
We apply CodeTT to infer Python type annotations and show significant improvement over prior approaches. To our knowledge, this is the first ML-based technique capable of predicting both parametric and user-defined types.
We improve the vanilla CodeT5 model by applying static analysis techniques help the model reason about information beyond local contexts, further boosting its performance.
We propose an iterative decoding scheme that particularly helps with coherence, measured by the number of type errors reported by the type checker. We additionally propose the novel setting that combines the seq2seq decoding scheme with user intervention.
2 OVERVIEW
In this section, motivate the design of TypeT5 using the example shown in Figure This example features a method predict and two functions feal_oonddttses and chuck_sres, each of which is implemented in different file. Given an untyped version of this code, our goal is to automatically infer the type annotations (highlighted in green). This example is chal- lenging for existing type inference techniques due to the heavy use of user-defined types (such as ChunkedDataset, PythonIype, and Mode 1Wrapper) and complex parametric type like dict ot [int, , 1ist [BythonType]].
model.py
eval.py
class ModelMrapper:
1
def L wwaloon_dttseetl datesetl
other nethods and attributes onitted
model: Wwrper,,,,,,
3 data: Tokenized5
predict(
window size: Optional(int]
5
self, CrunkedDataset,
5
dictfint, List PythonTyyeel:::
window copy (rodee wtt.roeeeeeeautttiiiiin Defaultwiinddow
7
r_seos: Optional None,
7
bialexxindww not None:
8
dietlint, List [PythonType]]::
window. left_tokens windox_size
pred_types dict() for batch in data.data:
window. right_tokens window_size
18
10
batch("input_id"s
inputt tof(eeeic)))
sresidate,
12
preas, selffppredicton_baatee brttoonbtbtbbabttthhheaa n_seas)
return model, predittccsss n_seqs=None)
1, 5_id enunerate(batchh "chunk_te"))
data.py
14
if n_sees is None:
def chunk_srest
pred_types [c_id) preds li]
2
date: TokenizedBrcSet,
else:
3
window: WindowArgs
17
span 1 n_segs (1 1) n_sogs
4
ChunkedDataset:
pred_types (c_id] [span]
5
# ixplementation oxitted
19
return pred_types
ChunkedDstaseetl...
Figure 1: Simplified code snippets taken from our own codebase. The eval_on_dataset function first calls the chunk_srcs function convert the given textual data into equally sized chunks, and it then feed them into the ModelWrapper predict method.
2

---

Published as a conference paper at ICLR 2023
Output types
<extra_ ModeIWrapper
sextra_ Serrrr_dd_a 1> Token SSeSeett
ModelWrapper
<extre_id 2d_2> Opt Mitaallinnt
DefaultWindow
decoding
t
sextra, i
Token
1zed
5rc
-extra, ia
Pass 1: usee-to sase-to-une -s-usrr
Pass 2: user-t tori-to-uu usee
predict betch
CodeT5 Decoder
CodeT5 Encoder
predict
U
(window
size
Textra_ia 2
None
tokenitation
chunk_ srcs
Input code element
evailon_ddaasee" (o_ddaasse(
model: <аxtra 1d Rr, data: sextra, 1d_ wi window_s12ed size: <extra_id_e None,
eval_on_
wal_on_dataset dataset
Figure 2: How C5deTT encodes and decodes source code using BPE. Marker tokens (highlighted in blue) indicate gaps (input) and their corresponding fillers (output).
Figure 3:
The two-pass iterative de-
coding process.
Type inference code infilling In this work, advocate new approach that views type infer- ence as an instance of code infilling. Because type annotations can be viewed as missing code fragments, fine-tune a state-of-the-arr code infilling model, namely CodeT5, as our starting point. Since CodeT5 produces sequences of subword tokens using Byte Pair Encoding (Radford al. Gage 1994) (see Figure 2}. it can, in principle, predict arbitrary code snippets to fill masked gaps including type annotations with complex parametric types and user-defined classes.
Incorporating context through static analysis By using surrounding code as the prediction con- text, fine-tuned version of CodeT5 can relatively casily predict the correct type annotations of some the variables. For example, based on the names and default values of n. sege (model.py, line window.sizz (eval.py, line 4), CodeT5 eters. However, for other parameters such as model in line 2 of eval.py, the surrounding context does contain enough information make reasonable prediction. To see why, observe that ModelWrapper is a new class defined in a separate file, so, (1) it has never been seen during train- ing, and (2) its definition not available as part of the context. Similarly, is also very difficult to predict the return type of wow.ooonntttteeee since it directly returns the result of podee..predioe whose definition also not available to the model.
figure the correct types of these param-
To address this issue, our approach enhances the prediction context of CodeT5 using static analysis. The details of how we construct the context from static analysis will described in subsecction 3.3 but, in nutshell, our approach analyzes the user-usee relations among code elements and pulls in relevant definitions into the context. For example, when the model making a prediction for eval_on lol__oddttteeet the context includes the definitions of De befaa Fuutwindos and predict, which are defined in
and invoked at line and line 12 eval.py, respectively.
TypeT5 architecture As there many dependencies between different code elements, making independent type predictions for each code element is not ideal. For example, to infer the return type of eval_on_dat aset, we would need to know the return type of Mcdel arapeer wwwraaper.peee.ee. predict, which depends on the return type of the sel predict_on _otton_baccccc (model.py, line 12). However, since there is limited context that can be fed to the model, it not feasible to include all transitive users and usees. To deal with this problem, TypeT5 utilizes an iterative decoding scheme that allows conditioning on previous type predictions.
more detail, our decoding scheme first sorts the user- usee graph topologically and then performs two sequential prediction passes, first from usee to users and then going in the reverse direction, as illustrated in Figure 3 To see why both these directions useful, observe that the return type of loll.onndattset depends predict, which in turn depends on predict ct.on bat.tttch.h.h.h. Thus, propagating information from callees to callers clearly useful. Conversely, consider predicting the type of data, the first parameter of predict. Since the return value of chunk sres is passed as the first argument of predict, propagating information in the reverse direction can also be helpful.
3

---

Published as a conference paper at ICLR 2023
ModeIWrapper
WindowArgs
Legend
globa lunction
DefaultWindow
chunk_srcs
left tolkans
Claso mathod
right tokens
deee sttribate
predict batch
aval. dataset
class
ChunkedDataset
contain usago
predict
data
urage
Figure 4:
The usage graph corresponding to the code snippets in Figure WindowArgs and ChunkedDataset assumed be defined elsewhere in the same project.
METHODS
3.1 USING CODET5 FOR TYPE PREDICTION
We formulate type prediction as a sequence-to-sequenee (seq2seq) task. Let u = (u....., Up represent sequence of code tokens, where each token single untyped code element e (function or variable). We insert into u indexed marker tokens (<extra re iidi>> at each point where we wish to predict types and the model predict (41,..., lm). the token sequence that encodes types for the marked locations in u. Note that t only contains the types, other code tokens, in the format <ext ra_id_1> [type 1 tokens] <extra_id_2>> [type 2 tckens], etc. We use the same tokenizer CodeT5, which allows encoding any Python expression (typically short) sequence of subword tokens.
Our baseline CodeT5 model is a Transformer seq2seq model P(t ū) placing distribution over type sequences given the raw code sequence u and its surrounding code u. This process is shown in Figure 2 and is omitted for clarity.
Our improved model, TypeT5, replaces the surrounding code ù with a context constructed from static analysis, which we denote both models, we remove all comments and Python docstrings as pre-processing step, as was done in prior work. We now show how is constructed from static analysis.
Thus,
write the model P(t u, s). Note that, in
3.2 BUILDING THE USAGE GRAPH
To overcome the limitations of using only the surrounding code as context, our approach relies on static analysis to extract relevant global information about the codebase. particular, our analysis constructs usage graph, whose nodes correspond to code elements and edges represent direct usage. For example, æ and y are both functions, an edge from x to y means that æ calls y. Similarly, for variables, an edge from to y means that the value of depends on y.
The usage graph for our previous example is shown in Figure 4 We show two types usages: certain usage, shown solid arrwws, and potential usage, shown dotted anrows. A certain usage is one that can be statically determined without knowing the types of the involved variables. For example, the global function chunk_sres is directly used by eval_on_dal _on_dataset and also know that like a non-self attribute access or method call, the target depends on the type of the receiver, so we first collect all attributes and methods with matching name from the current project and then generate a potential usage for each of them. We give more details of usage graph construction in subsection A.I
wtt.cttoooattoooo is called by predict (self method call on line 12). For other cases
3.3 CONSTRUCTING MODEL INPUTS
TypeT5 leverages the usage graph to construct the inputs to our model. In particular, we define the model input for code element to be concatenation of four parts: the preamble Spze. usees Susees main code u, and users 8 Suser which are constructed as follows (see subsectioo A.9 for a concrete example).
4

---

Published as a conference paper at ICLR 2023
The main purpose of Spre is to help the model map each local name to the definition that it is referring to, so the preamble includes all import statements. Additionally, to help the model see which types available in the current scope, the preamble also includes the headers of all class definitions as well as all the type variable declarations from the current file.
Preamble:
Main code: For the code element e of interest, we use its definition to construct u. If e is a function, is just its source code, and if e is a variable, includes all top-level assignment statements in which e appears as the left hand side. Additionally, if e is a method or attribute of a class C, we also indent u and prepend it with the header of C, shown in Figure
Given the usage graph G, construct user by including the source tokens of all elements from users(G,es e). Since these elements can come from different source files, we also prepend each element with special comment denoting the Python module it comes from. Note that these elements can optionally contain the types predicted by our Type'T5 model, as described later in subsection 3.4
Users:
Susee contains not just the direct users of e, but also anything that is used in the user context Suser- i.e., Susee contains the elements from usees(G, e) U Ue' usees(S, e'). Since this generally contains many more elements than 8 Suser, we only use the (typed or untyped) signatures of the elements to construct Susce-
Usees:
We limit the total input size to be at most 4096 subword tokens and cut off any exceeding tokens from both left and right, centered around the main code. Context elements involving certain usages are arranged to be closer to the center so that they are always prioritized over potential usages.
3.4 ITERATIVE DECODING INFERENCE
We now describe how to conduct inference in our base model as well as our context-augmented model using an iterative decoding scheme.
Given our trained model P(t u. ú), we can infer a most likely set of types t for (with surrounding context ū) using beam search. Our implementation performs joint prediction of the output types for a single code block u. since later types in the block condition on the predictions of carlier types. However, note that both u and û are always completely untyped code: while we condition on previous types as we predict, these are not inserted into the prediction context for the next element
CodeTS decoding:
TypeT55 iterative decoding: Part of our motivation for including the context is to exploit its type information at inference time. Crucially, this requires to be typed. However, the contexts that are drawn from the original codebase are not typed, so Type'T5 iteratively adds type signatures to these contexts using its own predictions. Let M be the type assignment produced by the model, which maps each code clement to its predicted type te, and denote M(s) as the context obtained by annotating the elements in s according to M. Starting with an empty A (which leaves any context unchanged), Type'55 then iteratively updates M using an iterative decoding scheme that traverses the usage graph 9 twice, as shown in |Figure 3] The first prediction pass follows the usee-to-usee order while the second pass goes in the reverse direction allow for bidirectional information flow. At any given time step, we can denote the model's prediction for element as drawn from P(t. ue: M(ss)) and the predicted types E are then used to update M such that M(e) t
3.5 TRAINING
To save computation and improve parallelism during training, we use the available human annota- tions as a gold type assignment M instead of letting the model condition on its own prediction. Note that this type assignment is generally incomplete and may not contain a label for every miss- ing type. We train the model to maximize the log likelihood of predicting the ground truth, i.e., log P( Ue; (s.)))), for every element where t is available, using teacher-forcing. We train the Code'T5 baseline model similarly on the same dataset
In our experiments, including predicted types actually hurts the performance due to exposure bias. This requires performing a topological sort over G. When & is not DAG, we hreak the cycles arbitrarily. Note that without this training (fine-tuning) step, the original CodeT5 model performs very poorly it tends predict arbitrary Python expressions that are not types.
5

---

Published as a conference paper at ICLR 2023
Table 1: Basic statistics of our datasets.
BetterTypes4k
InferTypes4PP
train
valid
test
test
Projects
S73
40
50
Nonempty files
16.5K
1098
949
Lines of code
2.4M
174K
139K
21K
Top-level type slots
541K
38.2K
28.4K
4.6K
Top-level user-added types
275K
19.3K
15.8K
2.7k
Rare type ratio
25.7%
23.3%
35.0%
33.8%
Complex lype ratio
20.4%
16.6%
20.8%
33.4%
Average type size
1.42
1.33
1.43
1.72
EXPERIMENTS
We implement TypeT5 in Python, whose the source code and model weights be found GitHub We list the hyperparameters and hardware details in subsectioo A.6 Below, we first com- pare Type'T5 against three state-of-the-art type inference systems for Python, namely Typilus (Alla- manis et al.) 2020), Type4Py (Mir et al. 2022). and HiTyper (Peng et al. (022). We then present ablation studies evaluate different factors contributing to the model's performance.
4.1 EVALUATION SETUP
Datasets In evaluation, predict the type annotations for top-level code elements of each Python project. These include all class attributes, methods, top-level functions, and global variables. We treat any existing user-added type annotations as the ground truth, and we use per type accuracy as the main performance metric. Following a setting similar to that of Allamanis et al. (2020) and Wei al. (222)), we split our dataset per Python project. This way, types newly defined in the test projects will not have been seen during training. Such setting is more challenging than splitting the dataset per file, as done Type4Py and other work (Pradel al. 2020 Mir et al., 2022), but more closely reflects the model's ' real-world performance (Gruner et al.
2022
Our main dataset, BetterTyype4PPs is constructed by selecting a high-quality subset from the Many- Types4Py dataset (Mir et al.) 2021), which was used to train Type4Py. We describe the selection criteria in lsuussscttion A.2 Since our model is fine-tuned from the CodeT5 model (which may have already been pre-ne some of the test repositories in the aforementioned dataset), we addition- ally construct TyyesPPyyy a test set derived from the source code of Typilus, Type4Py, and our own tool, none of which were used CodeT55s pure-)training data. We further discuss the potential code duplication issue (subsection A.3) and label quality issue (subsection A.4) in the appendix.
We summarize key properties of both datasets in Table 1 We define the size of type the number of type constructors in its body and categorize a type as simple its size is 1, and complex otherwise. We also categorize type as rare common depending on whether contains rare type constructor that is not from the top-100 most frequent type constructors in our training set.
Accuracy metrics. Since Python allows the same type be written different syntactic forms we first perform type normalization to convert both the predicted and ground-truth types into canonical form. The details of this normalization step can found in subsecttion A.5) and we use the term full accuracy to refer to the accuracy against all human annotations after normalization. To better compare with prior work, we also define adjusted accuracy (our main metric), which (1) filters out all None and Any labels (as in prior work); (2) converts fully qualified names to simple names (c.g., Tensor instead of torch. since some prior approach does not output correctly qualified types; (3) rewrites any outermost Optional [T] and Final [T] into T since they
"Aaailable at httttt::// //gi thub. com wtt.:/o/ooooaarrrroooooooo
".gg, both int and foo. Bar has a size of I, whereas di et [str, foo. Har] has size of 3. ec.g., both Uni on ni, None] and Opt onal [int] refer to integer that can also be None, and both 1 sist and st [Any] refer python list with untyped elements

---

Published as a conference paper at ICLR 2023
Table 2: Accuracy comparison on common types.
BetterTypes4Py adjusted simple
InferTypes4Py adjusted simple
full
base all
full
base
all
all
complex
all
all
complex
all
Typilus
n/a
54.05
55.12
33.23
60.37
n/a
52.33
52.19
53.91
64.67
Type4Py
n/a
50.34
51.91
32.14
47.51
n/a
32.08
33.47
16.54
29.83
HrTyper
59.20
54.28
57.70
26.44
59.01
45.67
43.54
46.00
19.27
47.99
CodeTS
76.74
78.04
82.43
53.03
82.44
77.83
78.06
85,31
63.41
81.87
TypeT5
79.24
81.43
85.69
56.75
84.82
81.75
82.95
87.62
72.78
84.17
Table 3: Aceuracy comparison on rare types.
BetterTypes4PPy adjusted simple
InferTypes4Py adjusted simple
full
base all
full all
base
all
complex
all
complex
all
Type4Py
n/a
12.37
13.17
4.05
14.15
n/a
0.25
0.14
0.98
0.17
HrTyper
10.30
25.51
27.59
9.79
29.33
9.36
9.36
10.79
1.19
12.33
CodeTS
49.47
52.95
57.28
34.26
57.65
51.64
53.28
59.97
30.62
66.47
TypeT5
58,56
61.47
65.21
40.22
68.44
53.44
56.27
61.50
36.92
69.23
tend not to be used consistently across programmers Finally, we also define a base accuracy metric that is the same as adjusted accuracy except that it only checks the outermost type (c.g., Dict [str, List] will match any Dict but, for example, not Mapping.)
4.2 COMPARING TYPETS WITH OTHER APPROACHES
We compare TypeT5 with the basic CodeT5 model described in subssccttion 3,4 well the released versions of three other state-of-the-an approaches from prior work Typilus (Allamanis et al. 2020 models the program as graph and applies a graph neural network to compute the types of variables. The original Typilus model can predict from set of common types as well as (nonparametric) user- defined types, but their released model can only predict common types, so we only evaluate its performance on common types. Type4Py (Mir et al. (022) uses variable names and the surrounding code to compute an embedding and performs type prediction via a nearest-neighbe search in the type embedding space. We run the released Type4Py model via its web interface. HiTyper (Peng ctt al. 2022) combines the strengths of rulc- based type inference algorithm with ML type inference models by only invoking the ML model places where the inference algorithm gets stuck and deducing the types elsewhere using typing constraints. Its implementation uses Type4Py as the run HiTyper.
default ML backend, which we
We show each tool's accuracy in Table (on common types) and Table 3 (on rare types), and we put the combined results in appendix (Table 6). Since HiTyper only able make predition for about 67% of all labels, we report its performance on this subset. From Table 2 we can make the following observations. (1) Our CodeT5 baseline model already outperforms all prior approaches by large margin, demonstrating the advantage of using a seq2seq pre-trained language model. (2) TypeTs further improves the adjusted accuracy by 3.4% and 4.9% on the two datasets. (3) Type4Py's performance on InferTypes4Py dataset is significantly lower than on BetterTypes44Py. likely because Type4Py was originally trained on some of the test files in BetterTypes4Py Looking at Taall 6) we see that both CodeT5 and TypeTT dramatically outperform the other approaches on rare types. Moreover, TypeT5 achieves the largest improvements on types that are both rare and complex, improving upon CodeT5 by about 6% on both datasets, suggesting that that global information is especially important in these cases. For a qualitative analysis, we also show TypeT55ss outputs real-world example in subsection A.9
7 c.g., the type checker mypy has an option to enable implicit Opt ona types, so would not be possible for the model to know if it should output Opt onal 1 [T]] or T just from the untyped code.
SSince our approach henefits from CodeT5's large-scale pre-training across 8 different programming lan- guages, we use smaller training set than prior work and not retrain these prior approaches on our dataset.
'The accuracies reported in the original Type4Py paper are much higher than measured here. We analyze this discrepency in subsecctionn A.7

---

Published as a conference paper at ICLR 2023
Table 4: Performance of different model mod- ifications. All models are retrained with the corresponding inputs.
Table 5: Performance of different decoding strategies. The same TypeTS model weights are used for different decoding strategies.
Modification
Accuracy
Type Error
Strategy
Accuracy
Type Error
No Preamble
64,20
6067
Independent
71.68
6876
No Users
71,20
7053
Random
71.66
6215
No Usees
67.25
7332
UserToUsce
70.67
7415
Nonincremental
72.52
5720
TseeToUser
72.65
6402
Original (TypeT5)
73.02
5087
TwoPass (TypeTS5
73.02
5087
4.3 ABLATIONS ON TYPET5
We next present a series of ablations that evaluate how various factors contribute to TypeT5's per- formance. In addition to accuracy (on all types), we also report the type error count as a way to estimate how coherent the model's predictions are [0 Note that we only report errors that are directly related to type coherence (since the type checker also reports errors unrelated to type coherence, such unresolved import statements, We give more details about this metric in subsection A.8)
How do different components contribute to the model's performance? To evaluate the impact of each context element, we remove one component at a time and retrain the model accordingly. In particular, the No Preamble, No Users, and No Usees ablations correspond to removing the Spre. Suser, and usee context elements (introduced infsubsection 3.3) from the model's input, respectively. The Nonincremental model does not perform iterative decoding and trained to condition on an untyped context. We use the same input size limit (4096 subword tokens) for all models, so a model that does not utilize one kind of information have more space for other kinds. We show both the adjusted accuracy on all types and type enror count in Table 4 (and show other accuracy metrics in Table 7). We can see that (1) all components improve the overall accuracy, with the preamble having the largest impact, and (2) while the iterative decoding scheme only improves overall accuracy slightly, significantly improves the model's coherence, resulting in 12% fewer type errors.
How do different decoding strategies compare? addition to the two-pass iterative decoding strategy introduced in Isubsection 3.4] also test four other decoding strategies: (1) Indepen- dent, which independently predicts the type signature for each element without conditioning on the model's prediction (same the Nonincremental model except not retrained); (2) Random, which visits each element once following a random order; (3) UserToUser which visits the users before the usees; (4) Useer, which visits the usees before the users. The results are shown in Table 5) (and show other accuracy metrics in Table 8). We can see that our proposed TwoPass de- coding scheme yields the largest accuracy and type error improvement over Independent; whereas UserToUsee performs worse than Independent, suggesting that bad decoding ordering can have ad- verse effects on the model's performance.
4.4 USER-GUIDER GUIDED INTERACTIVE DECODING
Compared to prior work, our approach has a unique strength: because the model can condition on previous types, allows easy user intervention by conditioning any corrections made by the user. We thus explore an alternative use case of our model where the user interactively inspects each type signature predicted by the model and makes any necessary corrections before the model moves on to the next prediction. We emulate this interactive process using the ground-truth human annotations M and we modify the usee-to-user decoding process to let the model predict the types of each element e (as before), but then override the predicted types t. with the corresponding human annotations Me) if M On the BetterTypes4Py dataset, this interactive decoding process achieves a full accuracy of 78.04 % and an adjusted accuracy of 79.37 % m that
10 Accuracyy does not always correlate with coherency. e.g., when we have x = and and y are equally likely to be s or an int coherent type assignment needs ensure that and always have the same type, even if this requirement does not lead to higher accuracy in expectation.
8

---

Published as a conference paper at ICLR 2023
average, it only requires the user to correct one in every five model-predicted types to fully annotate an entirely untyped project from scratch. Note that only 59% of the test set contains a user type annotation,
some incorrect type annotations may not be corrected immediately and can get propagated to other places. Thus, in an interactive real-world setting, we can expect the actual accuracy to be even higher.
RELATED WORK
Deep Learning Type Inference Most prior approaches predict user-defined types via some form of type embedding matching. For example, LambdaNet (Wei et al., 2020) tackles this challenge by combining graph neural networks with pointer network. Typilus (Allamanis et al. (020) performs nearest-neighhor search in the type embedding space. However, neither approach can handle the unbounded type space induced by parametric types. TypeBert (Jesse et al. 202T) is the first type in- ference method based pre-trained transformer models and has demonstrated superior performance in predicting common types. Jesse et al. (2022) later improved TypeBert using deep similarity learn- ing to better support user-defined types. Different from our work, TypeBert does not model or construct context from static analaysis. Apart from just using the source code, other forms of information have also been utilized for type prediction. Both TypeWriter (Pradel et al. 2020) and HiTyper (Peng et al., 2022) combines type checking with inference-time search to avoid generating type errors. Optlyper (Pandi et al. 2020) explicitly models type constraints by turning them into continuous optimization objective. NL2Type (Malik et al. 2019) focuses on predicting types from natural language information such code comments and documentation.
seq2seq
Retrieval-based models in NLP Similar to context augmentation with static analysis, num- ber of methods have been developed to augment pre- pre- trained models with context in natural lan- guage processing (NLP). For open-domain question answering (Chen al. 2017), approaches like REALM Guu et al. (2020) and DPR (Karpukhin et al., (020) can retrieve knowledge relevant to a query, and retrieval-augmeened generation (Lewis al. 2020) and its extensions (e-g., Fusion-in- Decoder (Izacard & Grave 20200) have shown that it is possible to generate longer outputs using this information. RETRO (Borgeaud et al. 202T) and WebGPT (Nakano al. 202T) take this
web-scale extreme. However, we are also able to leverage static analysis based on the usage graph, which has no analogue for text.
Linking with T5 While our of CodeT5 for type prediction is novel to knowledge, T5 (Raffel et al. 2019) has been applied to a range of NLP tasks like summarization. Most similar ours is its use for entity linking (Petroni al. 202I): Systems in this vein generate names of entities token by token (De Cao et al. 2020) and are able to generalize to new entities like TypeTTS. However, the presence of ad hoc types for each new context and the types of context clues needed are very different in the code setting than in natural language.
Structured Prediction Our iterative decoding process can be viewed as applying a learned policy to perform structured prediction (Baklr et al.) 2007; Daumé et al. 2009). In this work, train- ing scheme can be viewed as behavior cloning since the model directly conditions on ground-truth human annotations, which lead to distributional mismatch between training and inference time. Applying more advanced training schemes such as Scheduled Sampling (Bengio et al., 2015), DAg- ger (Ross al. 2011). or reinforcement learning (Sutton Barto 2018 may help further boost the performance of iterative decoding.
6 CONCLUSION
In this work, we showed that TypeT5, a system integrating CodeT'S with lightweight static analysis and a new decoding scheme, is effective at predicting types for untyped codebases. We show that fine-tuning pre-trained code model already results in a strong baseline, outperforming systems from past work, and that incorporating the right context with static analysis is particularly helpful to enable accurate prediction on more challenging cases. We believe that even larger models can further improve the accuracy, but they will still need to leverage the methods in this work to achieve high accuracy on rare and complex types.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENT
This work is partly funded by NSF Award CCF-1918889. We would like to express our sincere gratitude to Miltiadis Allamanis and Amir Mir for generously helping us in setting up their tools and completing the comparison experiments. We would also like to thank members of the UTOPIA group, especially Jocelyn Chen, for their assistance managing the GPU server for the experiments.
REFERENCES
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre- pre-training for pro- gram understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp- 2655-2668, 2021.
Miltiadis Allamanis. The adverse effects code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposiun New Ideas, New Paradigms, and Reflections on Programming and Software, pp. 143-153, 2019.
Miltiadis Allamanis, Earl T Barr, Soline Ducousso, and Zheng Gao. Typilus: Neural type hints. In PLDI, 2020.
G. Baklr, Neural Information Processing Systems Foundation, T. Hofmann, A.J. Smola, B. Schölkopf, and B. Taskar. Predicting Structured Data. Advances in neural information pro- cessing systems. MIT Press, 2007. ISBN 9780262026178 htLthtts:://// //bokks.//oo google.
? id bLEFKUOFF8IIC
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran As- sociates, Inc., 2015, URL @995f98d5696 (922444444447 211.6///44111111177777110111111.
neuriss /ttp:a/ppp...ppppprr//////////// pdff
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol
Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Im- proving language models by retrieving from trillions of tokens. In arXiv, 2021. URL https: www.v.orrg/abaa.... 22 04426
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to Answer Open- Domain Questions. In Proceedings of the 55th Anmual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), 2017. URL https://cclanthollogg/
P17-1171
Hal Daumé, John Langford, and Daniel Marcu. Search-based structured prediction. Machine learn- ing. 75(3):977-32, 2009.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. In Proceedings of the International Conference on Learning Representations (ICL.R, 2020. URI.
https::/arxxtt org/ abs 2010 0 00904
Philip Gage. A new algorithm for data compression. Users Journal, 12(2):23-38, 1994.
Bernd Gruner, Tim Sonnekalb, Thomas S Heinze, and Clemens-Alexande Brust. Cross-domain evaluation of deep learning-based type inference system. arXiv preprint arXiv:2208.09188, 2022.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. REALM: Retrieval- Augmented Language Model Pre- Training. In Proceedings of the 37th International Conference Machine Learning, 2020.
10

---

Published as a conference paper at ICLR 2023
Vincent J Hellendoom, Christian Bird, Earl T Barr, and Miltiadis Allamanis. Deep learning type inference. In Proceedings of the 2018 26th acm joint meeting on european software engineering conference and symposium the foundations of software engineering, pp. 152-162, 2018.
Gautier Izacard and Edouard Grave. Leveraging Passage Retrieval with Generative Models Open Domain Question Answering. In arXiv, 2020. URL btttp:://axx...orrg/abb/////
01282
Kevin Jesse, Premkumar T Devanbu, and Toufique Ahmed. Learning type annotation: is big data enough? In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1483-1486, 2021.
Kevin Jesse, Premkumar Devanbu, and Anand Ashok Sawant. Learning to predict user-defined types. IEEE Transactions on Software Engineering, 2022.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. httptt/ss///ccllee
ogy org/ 2020. emn p-mnlp-maiin 1...mainn5555.. 550
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intense NLP Tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.078// 207012080.. 2022.
Rabee Sohail Malik, Jibesh Patra, and Michael Pradel. N12type: inferring javascript function types from natural language information. In 2019 JEEE/ACM 41st International Conference on Soft- ware Engineering (ICSE), pp. 304-315. IEEE, 2019.
A. M. Mir, E. Latoskinas, and G. Gousios. Manytypes4py: A benchmark python dataset for machine baarning-based type inference. In IEEF/ACM 18th International Conference on Mining Software Repositories (MSR). pp. 585-589. IEEE Computer Society, May 2021. doi: (0.1109/MS52258. 2021 100210
Amir M Mir, Evaldas Latoškinas, Sebastian Proksch, and Georgios Gousios. Type4py: practical deep similarity learning-based type inference for python. In Proceedings of the 44th International Conference on Software Engineering, pp. 2241-2252, 2022.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Bloun- dou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. WebGPT: Brosser-assisted question- answering with human feedback. In arXiv, 2021. URL
httpss:/aaxiy> wttp://////a//////op//p 09332
John-Paul Ore, Sebastian Elbaum, Carrick Detweiler, and Lambros Karkazis. Assessing the type annotation burden. In Proceedings of the 33rd ACM/IEEE International Conference on Automated
Software Engineering, pp. 190-201, 2018.
Irene Vlassi Pandi, Earl T Barr, Andrew D Gordon, and Charles Sutton. Opttyper: Probabilistic type inference by optimising logical and natural constraints. arXiv preprint ar Xrvv:22004 003400 2020.
Sheena Panthaplackel, Pengyu Nie, Milos Gligoric, Junyi Jessy Li, and Raymond Mooney. Learning to update natural language comments based on code changes. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1853-1868, 2020.
Yun Peng, Cuiyun Gao, Zongjie Li, Bowei Gao, David Lo, Qirun Zhang, and Michael Lyu. Static inference meets deep learning: A hybrid type inference approach for python. In Proceedings of the 44th International Conference Softwre Engineering, ICSE '22, pp. 2019-2030, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 1781450392 doi: 10. 1445/351000333500000 URL htt.::tttts/ettt http:///.....0r111.1114.444///// 0003. 35105 3510038

---

Published as a conference paper at ICLR 2023
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: Benchmark for Knowledge Intensive Language Tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, 2021. URL https:
//aclanthology.o org.y/.rg//22221 naacl main. 200
Michael Pradel, Georgios Gousios, Jason Liu, and Satish Chandra. Typewriter: Neural type predic- tion with search-based validation. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 209-220, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou. Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to- Text Transformer. In arXiv, 2019. URL https: http://aaavvvvva org/ absp///ss11111.../.. 10683
Veselin Raychev, Martin Vechev, and Andreas Krause. Predicting program properties from" big code". ACM SIGPLAN Notices, 50(1):111-124, 2015.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 JEEE interna- tional conference on acoustics, speech and signal processing (ICASSP), pp. 5149 -1522 IEEE, 2012.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1715 1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.18653/11/P16-11666 URL https:
//aclanthologg wwwpp//og///P./....1
Jeremy Siek and Walid Taha. Gradual typing for objects. In European Conference on Object- Oriented Programming, pp. 2-27. Springer, 2007.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yue Wang, Weishi Wang. Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-awar unified pre- trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6606,8708, 2021.
Jiyy Wei, Maruth Goyal, Greg Durrett, and Isil Dillig. Lambdanet: Probabilistic type inference using graph neural networks. In International Conference on Learning Representations, 2020.
URL htttps::/cpeereeieeette net/ forum? Hixx
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-ther the natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 45, Online, October 2020. As- sociation for Computational Linguistics. URL https:/ /www. acl aclweb. orgganthooogyyy.. 2020. emnlp demos.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 CONSTRUCTING THE USAGE GRAPH
To resolve the user-usee relations at the project level (used in subsecttionn 3.2). we built a custom static analysis pipeline on top of the libest library. We use 1ibest to parse Python files and also rely on its utilities to resolve symbol references within the same file (i.e., it tells us whether a local name refers to a function defined in the current file or comes from import statement, etc.) We then implement custom import resolution logic (following Python's is module rules) and combine it with 1 bibctt resolve all certain usages within the current project. For unresolved usages with syntactic form matching class attribute/method usage, we generate a potential usage to each class members with matching name. Since the involved static analysis operations are fairly lightweight and we parallelize the construction of different graphs on CPUs, the time spent on constructing the usage graphs only makes up a tiny fraction of the total training or inference time. For example, using 28 Python processes, it only takes about 8 minutes to process the entire BetterTypes4Py training set, including the time spent on other tasks such as parsing, code transformation, and tokenization.
While our implementation limits the analysis scope to the current project, it is trright-fooward to extend the analysis to also include usages involving library definitions, which may help the model see type signatures of uncommon library APIs. We did not do this in our experiments mainly due to the manual effort would require to install the correct library dependencies for all the projects in datasets.
A.2 CONSTRUCTING THE BETTERTYPES4TH DATASET
Our dataset is constructed from the ManyTyyes44Py dataset follows: we first filter out the GitHub repositories that are no longer accessible or that fail to download within 10 seconds. This leaves us with 4890 out of the 5996 original projects. Then, we discard projects that have not been updated for more than one year (to avoid outdated library APIs), reducing the number of repositories to 1218. To limit the influence of any particular project, we also filter out the 37 projects with more than 50K lines of code. Finally, to exclude those projects that have very few type annotations, we compare the number of type annotations n, of each project with its lines of code n. and filter out those with a n ratio less than 1:20. This gives us our final set of 663 projects. We then randomly select 50 test and 40 validation projects and use the rest for training.
A.3 CODE DUPLICATION
Code duplication (Allamanis, 2019) has been shown to have adverse effects on the performance of ML models and can blur the evaluation results. Hence, prior work like Type4Py applies file-level deduplication to remove duplicated source files. However, this is hard to do in our project-based setting since need all files to be present during inference. To address this, have manually verified that our Tyfer4yy dataset does not contain files that are copy-pasted from elsewhere. We also run the popular code duplication tool jscpd on our test set detect duplicated code blocks. The analysis shows that there is relatively little duplication in the dataset (only around 4% of duplicated lines), and the majority of these duplications came from the same project rather than across projects, so we believe code duplication is not a major issue under our by-project evaluation.
A.4 LABEL QUALITY
In our evaluation, developer-provided type annotations are used as the ground truth, which are not always accurate or coherent (Ore et al.) 2018). This partially motivated us to construct the Infer- Types4Py dataset, which consists of high-quality type annotations with a relatively low error rate. In particular, our own codebase (which are included in InferTypes4Py) makes heavy use of type annotations throughout the development process and is continuously type-checked by VSCode. As a very rough metric to approximate label quality, we report the coherence error (defined in inssubsec- tion A.8) of the human labels on both datasets: On BetterTypes4Py and InferTypes4Py, the average coherence error per human annotation is 0.019 and 0.045, respectively.
13

---

Published as a conference paper at ICLR 2023
Table 6: Accuracy comparison all types (common rarr).
BetterTypes44Py adjusted simple
InferTypes4Py adjusted simple
full all
base all
full
base
all
complex
all
all
complex
all
Type4Py
n/a
34.52
35.87
19.68
35.61
n/a
21.11
22.35
9.61
22.64
HiTyper
42.53
41.95
44.86
19.02
47.88
34.83
32.50
35.11
11.40
39.23
CodeTS
67.07
67.47
72.12
44.05
73.44
68.80
68,95
75.14
54.04
77.78
TypeT5
71.89
73.02
77.07
49.72
78.87
71.98
73.15
77.16
62.66
80.20
A.5
TYPE NORMALIZATION
To compute the accuracy metrics in subsection 4.1 normalize a Python type:
we recursively apply the following steps to
1. Rewrite any Opt Optional [T] Union [T, Nonel.
2. Sort the arguments Union types and flatten any nested Unions.
e.g., rewrite Union [B, Union [C,A]] into Union [A, B, Cl-
3. If all type arguments are Any, drop them all. e.g., rewrite List [Any] to List.
4. Capitalize the names of basic types. e.g., rewrite list to List.
A.6 HYPERPARAMETERS AND RUNNING TIMES
We initialize our model's weights from the CodeT5 model provided by Huggingface Transformers library (Wolf et al., 2020) and train our model for exactly one epoch using the library's default optimizer configuration (AdamW with a base learning rate of 2e-5 and weight decay of 0.01.) During inference, we use beam search with beam width of 16 and diversity penalty of 1.0. We adaptively set the maximal output sequence length to be 16n + 10, where the number of types to be predicted in the input.
We the size limit of preamble, usee Context, main code, and user Context to 1000, 2048, 512, 1536, respectively, both during training and test time. Note that preamble uses the space within usee context, so the total maximal input size is 4096.
Training the model took about 11 hours on single Quadro RTX 8000 GPU with 48GB memory. Performing the two-pass inference on the BetterTypes4PP test set takes about hours, whereas performing single-aass inference (e.g.. UseeToUser) takes about half the time. For comparison, training CodeT5 model on the same machine takes about 3.7 hours, and the corresponding evaluation takes about 0.5 hour.
A.7 WHY TYPE4PY HAS MUCH LOWER PERFORMANCE OUR DATASET
We have discussed our experiments with the authors of Type4Py and believe that the discrepancy we observe is likely due to the combination of two reasons: First, while our evaluation only counts the type annotations on all top-level APIs, et al. (2022) includes all local variables in their evaluation as well. Second, while we only evaluate on human annotations, they also include machine-inferred type annotations (via the type checker Pyre) as ground-truth labels. As result, the distributions of labels reported by the two papers are significantly different: in our setting, function annotations (parameters return types) constitute the majority (87%) of the labels, whereas in Mir et al. (2022), their portion is significantly smaller, merely 18%. This suggests that their label set is likely in- flated by simple labels inferrable from the type checker, which explain the performance drop when evaluated on our datasets.
A.8 MEASURING TYPE COHERENCE USING TYPE ERRORS
To estimate the type coherence (subsection 4.3). we call the type checker MyPs on codehases annotated with the types predicted by the model. Since not all emors reported by MyPy are type
1 hhtp://mypyylang.orr
14

---

Published as a conference paper at ICLR 2023
Table 7: Detailed comparison of different model modifications BetterTypes4Py. All models
retrained with the corresponding input format. Accuracy is measured on all types.
Accuracy adjusted simple
Modification
full all
base all
Type Error
all
complex
No Preamble
63.03
64.20
71.22
33.51
69.98
6067
No Usees
67.70
67.15
70.27
48.09
73.01
7053
No Users
70.22
71.20
77.87
41.34
77.26
7332
Nonincremental
71.86
72.52
77.23
47.26
78.47
5720
Original (SypeTS)
71.89
73.02
77.07
49.72
78.87
5087
Table 8: Detailed comparison of different decoding strategies BetterTypes4PP.. The same Type model weights are used for different decoding strategies. Accuracy is measured on all types, and the UserGuided decoding introduced in subsection 4.4
Accuracy adjusted simple
Modification
full all
base all
Type Error
all
complex
Independent
70.87
71.68
75.99
46.92
77.58
6876
Random
70.68
71.66
75.68
47.89
77.65
6215
UserToUsee
69.98
70.67
74.48
47.60
76.46
7415
Useerr
71.58
72.65
76.80
48.88
78.48
6402
TwoPass (Type' T5)
71.89
73.02
77.07
49.72
78.87
5087
UserGuided
78.09
79.36
83.27
58.61
84.02
n/a
errors or are related type coherence, we only count the errors with the following error codes, whose meaning according to MyPy's documentation are:
attr-defined checks that an attribute defined in the target class module when using the dot operator.
arg-type checks that argument types in a call match the declared argument types in the signature of the called function.
return-value checks that the returned value is compatible with the type signature of the func- tion.
assignment checks that the assigned expression is compatible with the assignment target.
name-defined checks that a name is defined the current scope.
Note that this metric does have its limitations. One undesired property we found is that it can favor predicting a non-existing type over an incorrect type. For instance, when the model predicts an incorrect type a function, the type checker will check all the usages of that function against this declared type and will thus likely report multiple errors. However, if the model predicts a non- existing type, the type checker will only report single name-def ned error at the declaration site and will skip checking its usages. This effect has caused the No Preamble variant in Table 4oo have lower error count than other two other variants since tends predict lot more non-existing types. But we have verified that it was not the cause of the type eтror count improvement by our two-pass iterative decoding model.
A.9 REAL EXAMPLES PRODUCED TYPETS
We run TypeT5 on the actual code corresponding to the example shown in Figure and show the obtained preamble (Figure 5), usee context (Figure 6), main code (Figure 7). and user context (Fig- ure 8h for the Model wrapper predict element. For each type predicted by the model, we indicate whether correct with a green or red marker, and not, also show the ground truth type in red. Note that these outputs were generated by the model in the second pass of the iterative decoding process (described subsecttoo 3.4), so all the elements have already been annotated at least once (but some may have not been annotate the second time).
15

---

Published as a conference paper at ICLR 2023
modellinput..y
Preamble
1
import random
2
from import deepcopy
import numpy np
4
from collections import Counter
from mypy_extensions import mypyc_attr
6
from torch import Tensor
from 7
oo collator import DataGollattoFoorSeeZeer
from typing import NamedTuple, overload
9
from batasets.arrow_dattee import Dataset
10
from torch. data import DataLoader, RandomSampler
from. import
12
ChunkedDataset,
13
CtxÅrgs,
TokenizedSrcšet,
15
sitput_ids_as_typee
16
preds_to_accuracies,
17
18
from. type_env import AccuracyMetric, PythonType
from.utils import
20
@dataclass
21
class DecodingArgs:
23
edataclass
24
class DatasttPaaieesiltteeeer(((1]]]]
26
@dataclass
class Worpperp
28
Figure 5: The preamble gathers the important statements and class headers from the current file. This helps the model see which types are available and where each symbol comes from.
16

---

Published as a conference paper at ICLR 2023
71
spot.modet
Usee Context
72
edataclass
73
class DecodingArgs:
74
sampting_nax_token token:: int
ctx_args: CtxArgs
76
77
utils
def assert_eq(x: TI, *xs: T1, extra_nessage: Callable((), strl lam
79
80
moot.
@dataclass
82
class DetasstPredfesutt((eeeerii]]]]]
chunks: ChunkedDataset
predictions: list[T1]
tist(tust(Pythonyyee]
85
extra_info: list[dict] fieiddderfuutt_fccttrysssss
List[T1]
87
spot. model
88
@dataclass
class ModelWrapper:
90
model: PythonType X NodelSPOT
tokenizer: PythonType TokenizerSPOT
92
args: CtxArgs DecodingArgs
93
def predict_on_batch(
self,
95
batch: dict,
96
nun_return_sequence int None, tuple (yyyhllTitttyt Tensor]:...
98
tuptelUsst(lss((pphhhnypp]]]] Tensor]
99
mpot.
101
def dynamic_data @ottel
102
dataset: ChunkedDataset, Dataset
103
nax_tokenst int,
104
collate_ DataCo latoororSeeg2eq, shuffle: bool False,
105
106
>> DataLoader:...
107
108
Used above
Figure 6: The usee context shows the signature of the elements that are used by the main code or by elements from the user context. By seeing their predicted type signatures, the model can understand the type-level behavior of these definitions without having dive into their implementation.
17

---

Published as a conference paper at ICLR 2023
189
# spot.
Main Code
@dataclass
Worpperapper:
predict(
113
self,
114
dataset: ChunkedDataset, Dataset tqdn_args: dict O,,
115
116
nuu_return_seqeecces: int None,
117
List(PythonType]] List[List(Pytthonyyea) model self.model
119
collator lataccllatrroorsseeeeessss... tokeeni,,, model)
120
loader dynamic_
121
dataset, ttrkemmsssel...ags..smmmml collate_ faacote_rrcollattr
122
123
Shuffle=True,
126
device model.device
127
pred_types- dict(int, list]
128
with tędm!
129
total=len(dataset), desc="predict", smoothing=0.01, **tqdn_args
130
tędm_bar:
batch loader:
n_chunks batch "erchhiintttidds"..oo
133
batch ""inn"t_ids""" batchh""nput__dd""" preds, for i, c_id in enumerate(batchh ("chunk_id")
134
prreeict_on_bbrcchhbatthh nun_return nun_retturnssquences)
135
136
c_id int(c_id)
137
if nun_return_sequences None:
pred_ pred_types(ccidd preds [1]
139
else:
140
podd_types_c_ide = preds
141
i nun_returnsequences : (i + 1) num_return_seque
142
143
tqdn_bat bar. update(n_chunks)
[pred_ [pred_types (int(c_id)) for c_id dataset ("chunk_id"]]
Figure 7: The main code is the element that is being annotated by the model at the current decoding step. The model has made two errors in this example, both of which can be directly attributed previous two errors made in the usee context (line 102 and line 97). This shows that the model is making coherent predictions according to the context, and such errors can be avoided the user has corrected the previous errors (as described in subsection 4.4)
18

---

Published as a conference paper at ICLR 2023
147
Users bewow spot.model
User Context, first half
@dataclass
150
class Mode WWrapper:
151
def eva eval_on_dataset(
152
self, src_data: TokenizedSrcSet, labels:
153
155
tqdn_args: dict O - DatasetPredResult::
156
157
ctx_args self.args.ctx_arg
158
if nax_labels is not None:
159
ctx_args copy(ctx_args)
ctx_args.mxx_lbbls max_labels
161
162
chunks src_data. wot_cchnnssctx_xeeee t
163
preds self.predict(
164
chunks. data, srr.rttrrn_squnncssstnnnnn grgn
return DatasettrreResuutttChhnkk,, preds)
167
169
# spot. decode
User Context, second half
178
def mmmple_andiddtess
wrapper: Lirapper,
src_data:
TokenizedSrcSet,
173
n_samples: int,
tuple[ChunkedDatssst, [istliss([Let((xyyonntee]]]]]]]
174
>> tuple [ChunkedDatoset, list (ysthoyyhoyyyee)
175
ctx_args = wrapper.args.ctx_ee
176
do_sample wrapper.args.dd_sempt.
177
not do_sample:
wrapper.args.nun beans is None, "nun_beams needs
assert n_samples wrapper. hag_s.nnu_eaama
188
181
chunks arc_data.to www.ttooccchcccccceeeeo (t__chhnns(ccxxrrrooooo (ctx_args)
182
n_ chuhukks = Len data
183
184
da se_sapple:
sanples
wrapper. punperr.predicttchunnss..d.e tędn_args=())
187
for in (n_range(n_smmleesss desc="Samp ling")
188
189
else:-
197
198
def get_preds(chunk_tt sample_id):
return
samples (samplees [chunk_id] do_sample samples [chunk_idd [samp]
201
202
203
pred_candi.cn candidates (get_preds(cid, sid) for sid in range(n_samples) for cid in range(n_chunl chunk
204
205
206
return chunks, pred_candidates
Figure 8: The user context shows two callers of the predict method from the main code. We see that the model successfully predicts all user-defined types, despite the fact that these are all new classes defined in the current project.
19


Reviews:
[{"review_id": "jinlwlZzEu", "reviewer": "Reviewer_8rav", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a new approach to type inference by using T5 architecture to generate types in a seq2seq fashion. Their proposed model, TypeT5 is CodeT5 fine-tuned on a subset of ManyTypes4Py named BetterTypes4Py. TypeT5 uses static analysis to compose its contextual input with a preamble, main code, users, and usees. The authors found that such static analysis can improve TypeT5's type generation across a series of metrics and across the type error count. The authors also wanted TypeT5 to generate consistent type information and exploit its previous type predictions (as additional context at inference time), so they introduced sequential decoding where passes of inference are made from usee-to-user and vice versa. The model is trained normally (like T5) with teacher-forcing to directly optimizing on ground truth labels rather than its own predictions.\n\nThe results show that pre-training (CodeT5 and TypeT5) improves accuracy measures across all, simple, and complex types. The authors demonstrate that their techniques for introducing more context was helpful for the model at inference time. The static analysis required for the context appears to be lightweight. The authors introduce a type error count metric that was not previously introduced as a metric but is practically useful; consistency is equally important to accuracy and type accuracy does not indicate type consistency, thus, such a metric is helpful.\n\nThe authors found some interesting attributes to their model, namely, the importance of preamble information (which could improve other pre-trained approaches that do not use preamble information), and that user-to-usee decoding performed worse than independent predictions.\n\nFinally, the authors took advantage of TypeT5s conditioning on previous predictions as an opportunity to correct incorrect predictions. In other words, during the sequential decoding process, if the decoding was incorrect, an human in the loop could correct the context and allow the model to continue inference with a higher precision rate. ", "summary_of_the_review": "My recommendation of this paper is based on a strong understanding of this works position in neural type inference. This paper does contribute to the existing body of research of neural type inference.", "strength_and_weaknesses": "Strengths\n- Design: the design of the model is important in the field of type inference. The application of CodeT5 was overdue and the implementation of static analysis was fitting for the model architecture\n- Comparison with SOTA: Comparison with state of the art and corresponding metrics in previous works was ok. \n- Use of type error count is good.\n- Examples in Appendix were good.\n- Ablation study was appropriately done\n- Application of Human in the loop was well done\n\nWeaknesses\n- Comparisons on previous datasets in full and BetterTypes4Py/InferTypes4Py would strengthen results. The release of these datasets with the paper submission would help reviewers understand the distribution and demographic of types that are being evaluated. Table 1 is good, but the dataset would be best.\n\n- Authors do not evaluate Typilus model on types that are not common. Authors claim that released Typilus model is not capable of predicting types outside of common (top100). The authors should verify this with Allamanis etal as it is a SOTA across rare types.\n\n- The authors should validate claim that Type4Py inference is lower because of leakage. But in the appendix, the authors state that the Type4Py discrepancy between train and test in the figure on their dataset is due to (1) Type4Py evaluation on local variables (not just APIs) and (2) compiler inferable types (easier type subset). Both of these reasons would contribute to a higher evaluation on the original ManyTypes4Py dataset but does not explain the discrepancy between train and test on their dataset. With the availability of Type4Py and ManyTypes4Py, validating this claim should not be difficult; the authors might find that their intuition led to a tangential discovery maybe a degraded performance on novel types not available in the KNN search, for example, and not data leakage.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fE2iqm-OWx", "reviewer": "Reviewer_JQ8L", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work adopts a language modeling approach to tackle program type prediction. Specifically, it fine-tunes CodeT5 on a type annotation task, and extends the standard approach by both including richer sets of context inferred via static analysis, and ordering the generation process to simulate information flow through a program usage graph. The resulting model provides reasonable improvements over a baseline approach that just uses CodeT5, which in turn performs far better than prior work.", "summary_of_the_review": "The work is well written and provides a natural approach that balances the performance potential of a pretrained language model with a series of improvements that provide the model with access to information that is pertinent to type annotation. The resulting model performs better than CodeT5, but only by a modest margin, and the evaluation leaves several open questions about the interpretation of this improvement. My review asks for expanded results that, if in line with the narrative in the paper, should make the improvements more convincing.", "strength_and_weaknesses": "**Update:** The authors have provided additional results in response to my concerns below, which underscore that the TypeT5 consistently improves over CodeT5 on rare types, and provide both additional ablations under the requested conditions and information on the training and inference _cost_. I have thus increased my score.\n\nThis proposed approach balances the use of powerful neural language models with a clear understanding of how developers type-annotate their projects. The result is an intuitive and rigorous approach that yields improvements over all alternative methods. Working against it is the fact that the improvements over a plain CodeT5 model are quite small, which stands in especially stark contrast to how vastly that model in turn improved over existing baselines. This is no fault of the authors -- pretrained language models are hard to beat -- so I am inclined to not consider this a major problem, but it does imply that the paper should make a stronger argument for its current performance. Specifically, three parts of the reporting should be expanded:\n\n1. The emphasis on (or alternatively, the definition of) adjusted accuracy feels unwarranted. While I could see filtering out `None`/`Any` labels as a reasonable step, replacing fully qualified types with simple ones and dropping `Optional` types would alter the type distribution quite significantly. This matters because the gain of TypeT5 over CodeT5 is quite a bit larger under the \"adjusted\" metric as it is under the \"full\" one, suggesting that it does better at inferring comparatively less precise annotations. Given this, please provide a second ablation for Tab. 4 focusing on full accuracy, so as to capture the factors that make TypeT5 outperform CodeT5 in the more challenging setting.\n\n2. One of the paper's stated goals (starting from the abstract) is to yield better type predictions on rare and complex types. That it realizes the \"complex\" part of this felt convincing, given the results in Tab. 2 & 3. It is unclear, however, why Tab. 3 does not show the accuracy solely on _rare_ types, but rather combines them with the common type results that were already shown separately before. Given that common types are far more common than rare ones and thus contribute predominantly to these results, I assume that the accuracy drop between tables 2 & 3 signals a far lower accuracy on rare types for all models. Please provide an equivalent table with the rare type results only (and, preferably, replace Tab. 3 in the paper with said table). There is no harm at all in reporting low accuracies on this task; it is objectively hard for other models as well. But the current setup makes it very hard to gauge whether the paper achieved this sub-goal -- if anything, in some cases it looks like it failed to; e.g., on InferTypes4Py, CodeT5 drops about 9% (points) from Tab. 2 to Tab 3. (full accuracy) while TypeT5 drops 10%, suggesting that it performs *worse* on rare types than the former, or at least, that its performance improvement over CodeT5 is much slimmer on rare types. This is important information.\n\n3. A modest performance gain should be presented along with the increase in cost, if any, that accompanies using the more performant technique relative to its baseline. Please discuss how much more expensive TypeT5 is, both for training (in terms of data collection, training time), and inference (time, memory), than CodeT5.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "C-qVeC7dHc2", "reviewer": "Reviewer_2PXr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "With the aim of type inference for untyped code, the paper feeds, in addition to tokenized source code, contextual information gained from static analysis to the seq2seq-based code completion model called CodeT5. CodeT5 itself can infer types of elements in untyped code (including user-defined and parametric types) successfully, but incorporating the contextual information improves the accuracy performance marginally. Furthermore, the paper provided the idea of sequential decoding, which refines the contextual information gradually using the inference result of the model, and showed that the use of sequential decoding makes type inference more coherent.", "summary_of_the_review": "The paper used CodeT5 to infer types and provided methods to improve the performance of the model with respect to accuracy and coherence. It would be nicer to provide a finer-grained analysis of the experimental result and the cost of constructing user graphs.", "strength_and_weaknesses": "Strengths:\n\n+ The proposed approach, especially the use of CodeT5, succeeds in inferring user-defined and parametric types.\n\n+ The paper experimentally demonstrated that contextual information and the sequential decoding improve accuracy and make type inference more coherent.\n\n+ The documentation is well written.\n\nWeaknesses:\n\n- The paper could provide a finer-grained analysis of the experimental result. Specifically, I am curious about the model's performance for parametric and under-defined types because this is the first work that addresses both kinds of types. Unfortunately, I cannot find it in the paper because all the accuracy numbers in Tables 2, 3, 4, and 5 are for the entire dataset, which includes primitive types such as int.\n\n- The paper does not report the cost of constructing user graphs. Because the construction needs to examine the overall codebase and the libraries it uses in the worst case, it might take too much time. Perhaps in practice the graph construction is not problematic, but I'm unsure whether it is the case.\n\n- (Minor) The notion of coherence needs to be better explained in the body of the paper. The footnote seems far from the formal definition of coherence provided in A.6.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B3t7aEkJUlL", "reviewer": "Reviewer_SGAP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Type inference is a challenging task for the Python language given its dynamic nature. Scholars have recently proposed machine learning (ML)-based techniques to infer types for Python. Previous techniques infer types based on seen examples during training, i.e, they cannot synthesize types, which hinders their ability to infer project-specific types. This paper presents TypeT5, a CodeT5-based model, whose main idea is to create a usage graph to capture usee-user relationships with static analysis, which provides global information to the CodeT5 model about code elements that need to be typed. Also, the proposed model has a two-pass sequential decoding that is conditioned based on previous type predictions. This allows bidirectional information flow from usees to users given a usage graph. Overall, the experimental results show that TypeT5 outperforms the state-of-the-art type inference models, namely, Type4Py and Typilus.", "summary_of_the_review": "This paper presents a novel technique, TypeT5, to tackle the type inference challenge for Python. Also, treating the type inference task as code completion is new. Overall, I believe that this paper has good potential to become an influential work in ML-based type inference research. However, I have some (major) comments regarding the soundness of the evaluation of TypeT5, which hampers the paper from reaching its full potential and impact. I briefly mention the main comments here as they are explained in detail later in the review.\n\n- Human-provided type annotations are used as ground truth, which is not always valid [3]. This can harm the soundness of the obtained results in the evaluation.\n- The state-of-the-art approaches, i.e., Type4Py and Typilus, are not fairly represented and evaluated.\n- Lack of comparison with a very similar published work, HiTyper [2], which is also a very recent SOTA type inference model for Python.\n\nTo increase the recommendation score to the acceptance level, all three raised concerns regarding the evaluation should be addressed.", "strength_and_weaknesses": "### Pros\n- A novel type inference technique based on CodeT5, which leverages global information in code and performs two-pass sequential decoding for inference.\n- The paper is well-written and has a good presentation. The proposed approach is explained with sufficient detail to reproduce the work.\n\n### Cons\n- The evaluation of the proposed approach is not fully sound and rigorous. The evaluation has some (major) shortcomings that need to be addressed. I have explained the shortcomings in “feedback for the authors”.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "1SJ0OvE_-6", "reviewer": "Reviewer_7ekP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a system to predict types in untyped or partially typed code. The authors propose to treat type prediction as a code completion problem, which can be solved using transformer models trained for code. However, just applying a transformer model \"out of the box\" would fail because there can be caller-callee dependencies in the code, sometimes across files, and these dependencies are essential to correctly predict types.\n\nTo solve this problem, the authors use static analysis to form a usage graph of these dependencies, and encode these as strings to provide additional context for the model while making predictions. They traverse the usage graph according to a topological ordering, and make predictions for each function sequentially. When the model makes a prediction for types in a function, these are carried forward to be used while predicting types for the next function in the sequence. Thus, the model is \"conditioned\" upon its previous type predictions.\n\nThey train their model using the \"ground truth\" type predictions, and evaluate it on a dataset of Python code without any type annotations. They show that their model achieves a large improvement in prediction accuracy (from ~55% to ~80%) as compared to other state of the art models.", "summary_of_the_review": "This paper presents an approach to automatically annotate types in Python code. The approach is conceptually simple and I have some concerns about the construction of the dataset, but a) the paper is well written and well structured, b) the evaluation is very thorough, and c) the performance gains are impressive. Given these factors, I would recommend acceptance.\n\nI am slightly less than confident in my evaluation because this is a very active research problem (predicting missing type annotations), and I am not familiar with all the related work. It is possible that I have overlooked a very similar approach or a relevant baseline that should have been compared against.", "strength_and_weaknesses": "**Strengths**\n\n1. Very precisely written. Excellent paper structuring that makes it easy to understand the approach. The motivating example is insightful and clearly illustrates both the problem and the benefits of the proposed solution.\n\n1. I appreciate the thorough dataset setup as described in 4.1. I think it's very important to ensure that train and test data are properly separated. Interesting to see that Type4Py's performance falls by so much.\n\n1. Very large performance increases. 55% to 80% is quite impressive.\n\n**Weaknesses**\n\n* Why do you evaluate only on public APIs, especially when you mention that that is a major reason for the drop in performance of Type4Py (Appendix A.5)?\n\n* I agree with the point about not including \"simple\" labels inferable from Pyre. But just to be thorough, it would be nice to include the results for the exact same setting as Type4Py. It would be okay even if Type4Py performed better than TypeT5 in this setting (no free lunch - every model has an inductive bias, but not all biases are bad!)\n\nEDIT - adding one more crucial point that slipped my attention in the initial review.\n\n* Do you re-train the baseline models on your custom (filtered) dataset? As in, when you present the results of Type4Py etc, do you retrain/fine-tune them on your collected dataset or do you just use their provided weights? It is vital to retrain. I am now wondering whether some of the extremely large performance improvement is due to these models not being trained on your dataset.\n\n**Questions for the authors**\n\n* Why not evaluate the coherence error (Table 4 and 5) on the baseline models Typilus and Type4Py?\n\n* UserToUsee performs worse than independent. Why is it a \"bad ordering\"? As shown in your motivating example, aren't there cases where you need User info before Usee, and vice-versa?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster