TypyBench: Evaluating LLM Type Inference for Untyped Python Repositories
Authors: Honghua Dong
Jiacheng Yang
Xun Deng
Yuhe Jiang
Gennady Pekhimenko
Fan Long
Xujie Si
Venue: ICML 2025 poster (2025)
Keywords: Benchmark
Type Inference
Untyped Python Repo
Large Language Models
Long-Context
Repo-Level
Software Engineering
Primary area: general_machine_learning->evaluation
TL;DR: We introduce TypyBench to evaluate LLMs' Python type inference, with new metrics for type similarity and consistency. While LLMs achieve good performance on individual types, most of them struggle with complex types and cross-file consistency.

Abstract: Type inference for dynamic languages like Python is a persistent challenge in software engineering. While large language models (LLMs) have shown promise in code understanding, their type inference capabilities remain underexplored. We introduce `TypyBench`, a benchmark designed to evaluate LLMs' type inference across entire Python repositories. `TypyBench` features two novel metrics: `TypeSim`, which captures nuanced semantic relationships between predicted and ground truth types, and `TypeCheck`, which assesses type consistency across codebases. Our evaluation of various LLMs on a curated dataset of 50 high-quality Python repositories reveals that, although LLMs achieve decent `TypeSim` scores, they struggle with complex nested types and exhibit significant type consistency errors. These findings suggest that future research should shift focus from improving type similarity to addressing repository-level consistency. `TypyBench` provides a foundation for this new direction, offering insights into model performance across different type complexities and usage contexts. Our code and data are available at \href{https://github.com/typybench/typybench}.

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Honghua Dong"
Jiacheng Yang" 12 Xun Deng" 1 Yuhe Jiang 12 Gennady Pekhimenko 123 Fan Long'
Xujie Si 12
Abstract
written by developers. This capability has become increas- ingly important as codebases grow in size and complexity.
Type inference for dynamic languages like Python is a persistent challenge software engineer- ing. While large language models (LLMs) have shown promise code understanding, their type inference capabilities remain underexplored. We introduce TYPYBENCH, benchmark designed to evaluate LLMs' type inference across entire Python repositories. TYPYBENCH features two novel metrics: TYPESIM, which captures nu- anced semantic relationships between predicted and ground truth types, and TYPECHECK, which assesses type consistency across codebases. Our evaluation of various LLMs curated dataset of 50 high-quality Python repositories reveals that, although LLMs achieve decent TYPESIM scores, they struggle with complex nested types and exhibit significant type consistency errors. These findings suggest that future research should shift focus from improving type similarity to addressing repository-level consistency. TYPY- BENCH provides foundation for this new direc- tion, offering insights into model performance across different type complexities and usage contexts. Our code and data are available at
The significance of type information in modern software development cannot be overstated. Type annotations serve multiple crucial purposes: (1) they enhance code clarity by making developers' intentions explicit, (2) prevent type- related errors through early detection, (3) enable rich IDE features like autocompletion, and (4) facilitate maintenance and refactoring operations. The introduction of type hints through PEP 484' marked a pivotal moment for Python, acknowledging the growing importance of static typing in large-scale software development.
While the benefits of type annotations are clear, manually adding them to existing codebases is time-consuming and error-prone. This challenge has sparked interest in devel- oping automatic type inference tools like Mypy (Lehtosalo et al.), Pyright (Microsoft) and Monkey Type (Instagram), as well as learning- based algorithms (Wei et al., 2020; Al- lamanis et al., 2020). Moreover, recent advances in large language models (LLMs) have shown promising results in code understanding tasks (Brown et al., 2020; Achiam et al., 2023; Chen et al., 2021), and in type inference tasks (Wei et al., 2023; Peng et al., 2023) with better performance than previous methods (Shivarpatna Venkatesh et al., 2024). Such tools can significantly reduce developer effort while improving code quality and maintainability.
Despite these advances, current evaluation benchmarks and approaches (Mir al., 2021; Allamanis et al., 2020; Shiv- arpatna Venkatesh et al., 2024) for type inference meth- ods face significant limitations. Traditional evaluation met- rics rely heavily on exact matching (or up to parametric type (Allamanis et al., 2020)), which fails to capture impor- tant semantic and functional relationships between types- for instance, the functional similarity between List and Sequence, where developers may them interchange- ably. Furthermore, existing benchmarks often evaluate type inference in isolation, focusing on individual functions or files rather than considering type consistency across entire codebases. This disconnect between local correctness and global coherence makes it challenging to reliably assess the real-world effectiveness of type inference methods.
1. Introduction
Type inference, the ability to automatically deduce the types of variables and expressions in a program, has been a long- standing challenge in programming language research (Ray- chev al., 2015; Hellendoomn al., 2018). In dynamically- typed languages like Python, where explicit type annotations optional, type inference involves analyzing code de- termine appropriate type annotations that could have been
Equal contribution, alphabetically ordered. 'University of Toronto è Veector Institute *CIFAR Al Chair. Correspondence to: Xujie Si <six @cs edu
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
https://pop..pyhhooor/////

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
greeting.py
TypeSim
TypeCheck
class Greeter:
def
Init (self, name:str)
None:
mypy
class Greeter:
consistency
self.name def greet(self) ~ str:
Typed Repo
Predicted Types
def Init (self, name: str)
None: ...
return HHHllo,, (selt.namej) def get_ nee_nefe(sei)) str. return self.name
def greet def get name(self) List[str]
Remove
greeting wwtp_wtttttttttee withoutt_tyes..py types.py
typing hints
class Greeter:
User
def (self, name):
self.name = name def greet(self): return ('Hello, (self.name)]! def get name(self): return self.name
Given python file, stub file is a pyy file that contains only the typing information. Your task to infer missing typing information generate stub file(,py)). ppy)..
Cleaned Repo
Figure 1. Overview of TYPYBENCH. We collect well-typed Python repositories and remove their typing information as the inputs. The outputs predicted by type inference methods are then evaluated using TYPESIM and TYPECHECK, where TYPESIM measures the functionality similarity between predicted and auman-annotated types, and TYPECHECK evaluates type consistency across entire codebases through static type checking.
To address these challenges, we introduce two novel eval- uation metrics, TYPESIM and TYPECHECK, as illustrated in Figure 1. TYPESIM measures the functional similarity between predicted and human-annotated types by incor- porating both structural relationships and type hierarchies, providing more nuanced evaluation than exact matching. Complementarily, TYPECHECK measures type consistency across entire codebases via static type checking?, verifying that inferred types integrate coherently.
high-quality Python repositories, designed to evaluate both local and global aspects of type inference.
An extensive empirical study of state-of-the LLMSs on type inference, revealing critical gaps between type prediction accuracy and consistency.
These contributions establish a foundation for future re- search in automated type inference, providing both the tools and insights needed develop more effective type inference systems.
We then present TYPYBENCH, a benchmark of 50 high- quality Python repositories selected based on their type coverage, complexity, and domain diversity. Unlike existing benchmarks (Pradel et al., 2020; Mir al., 2021), TYPY- BENCH emphasizes repository- levell evaluation, allowing the assessment of both local type accuracy and global type consistency.
2. Related Work
2.1. Type Inference Methods
Conventional Methods. Traditional type inference relies on static analysis and runtime tracing, as implemented in tools like Mypy (Lehtosalo al.). Pyright (Microsoft), and Monkey Type (Instagram). These approaches offer high precision but are limited in coverage and require explicit type annotations or runtime information.
Our extensive evaluation of state-of-the-art LLMSs on TYPY- BENCH reveals key insights. While modern LLMs achieve decent TYPESIM scores (up to 0.80), most of them strug- gle with TYPECHECK, highlighting gap between local type accuracy and global consistency compared to human annotations. The main contributions of this paper are:
Learning-based Methods. Early learning approaches like JSNice (Raychev et al., 2015) pioneered using probabilistic models to learn from existing codebases. This direction evolved to leverage various program representations, from natural language information (Malik al., 2019) to graph structures (Hellendoomn al., 2018; Wei al., 2020; Alla- manis et al., 2020; Cassano et al., 2023), though struggling to handle complex type structures and rare types.
Two novel metrics for evaluating type inference: TYPE- SIM for measuring semantic similarity berween types, and TYpECHECK for assessing repository-wide type consistency.
TYPYBENCI:: A comprehensive benchmark of 50
"We the number of Mypy check estimate the consistency of types.
LLM-based Methods. Recent work has shown that large language models can match or exceed traditional approaches
2

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
in type inference tasks (Jesse et al., 2021; Wei al., 2023; Peng et al., 2023). These methods benefit from pre-training on large code corpora and can leverage natural language understanding for improved type prediction.
def greet (name: stx) -> ste: return r'Hlello, sttello,,taane [nann]] )
These optional annotations do not affect runtime behavior, serving primarily as documentation and enabling static anal- ysis tools to catch type-related errors before execution.
2.2. Type Inference Benchmarks
Previous type inference benchmarks (Pradel al., 2020; Mir et al., 2021; Allamanis et al., 2020) primarily relied on exact match accuracy for evaluation, with Typilus (Al- lamanis et al., 2020) introducing a relaxed "Match up to Parametric Type" metric that compares only the outermost type constructors. However, these metrics still fall short of capturing full semantic similarity between type annotations. Our benchmark advances this by (I) introducing semantic similarity metrics that better capture the hierarchical and structural relationships between types, and (2) evaluating practical usability through type checking. By requiring pre- dictions in the form of PEP 484 stub files (pyi), we enable direct validation using production -aaadd type checkers, pro- viding more realistic assessment of type inference quality.
3.2. Type Inference
Type inference is the process of automatically deducing appropriate type annotations for variables and expressions in a program. In our context, given a Python repository without type annotations, the goal is to infer types that could have been written by developers:
Original untyped -ccoe def greet (name): return f f'Hollo,_ 3
After zype inference def gree. (name: str)
2
-+> str: return #"Hello,u (*amme)
(nare)
Here, the return type is str based on the returned value. The parameter name is likely to be str based on the se- mantic information, since the name of the variable is name and the function is greet.
2.3. Other Coding Benchmarks
The evolution of code-related benchmarks reflects a progres- sion from isolated to context-dependent evaluations:
3.3. Type Stub Files
Python uses pyy stub files to separate type information from implementation. These files contain only function signatures and type definitions:
Function-level Benchmarks. Traditional benchmarks fo- cused on self-contained programming tasks (Chen et al., 2021; Jain et al., 2024; Zhuo et al., 2024), evaluating spe- cific capabilities like code generation and problem-solving.
greetinggs ingo-pyl def greet (name: str)
stE:
Repository-level Benchmarks. Recent work has shifted toward repository-scale assessment, with each benchmark evaluating distinct aspects of code understanding: Re- poBench (Liu et al., 2024b) focuses on code completion, SWE-bench (Jimenez et al., 2024) tests bug fixing capabil- ities, and Tran (Wang et al., 2024) evaluates cross-language translation. Our work contributes to this ecosystem by examining models' ability to perform consis- tent type inference across entire repositories, adding another erucial dimension repository-leve model evaluation.
Stub files enable type checking without modifying source files and are commonly used in library distributions to pro- vide type information.
3.4. Static Type Checking
Static type checking verifies type consistency before pro- gram execution. Tools like mypy² analyze code to detect potential type errors, which include but are not limited to: (1) verifying type compatibility in assignments and function calls, (2) checking subtype relationships (e.g., List [int]] is subtype of Sequence [int]] ), (3) ensuring consistent usage of types across modules, In our example, mypy would
3. Background
This section introduces key concepts in Python's type sys- tem and type inference that are fundamental to our work.
detect errors like:
mag = greet ("TypyRench") Pasa, "TypySench" is str mag greet ([2, 2, Errors Expected atx, got list [int]
3.1. Gradual Typing in Python
Python supports gradual typing. allowing developers to in- crementally add type annotations while maintaining com- patibility with untyped code. Introduced in PEP 484, type hints enable specifying types for function parameters, return values, and variables:
Mypy performs this analysis by constructing a type depen- dency graph and propagating type constraints through the program, identifying violations of type rules defined in PEP 484 and related specifications.
hhmps:/mmypyrraatthcoccc.i
3

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Arguments
Permutable Argumanta
Sequence adds index, count, and length opera- tions Iterable, supporting indexed access.
List (uple [ar, In
List Duplle jis, stc, Roodd
Union (int,
V.S. Union [lat, float, xtr)
List
List
Est
List adds mutable operations like append, extend, pop to Sequence, enabling list modifi- cation.
Rost
Empty
Figure 2. Examples of type similarity computation. Left: List-wise comparison for generic types, where arguments are compared in order. Right:
This leads to meaningful similarities: s (Iteraalee, Saaee,eeeenee) = 0.92 as they share core iteration functionality, while (Sequence, List) = 0.7 reflects their additional differences mutability, Similarly, s(int, float) = 0.6 captures their shared arithmetic operations, while s(int, fundamental differences. See Appendix B for the TYPESIM between builtin types.
Set-wise comparison for Union types, where optimal matching computed between members.
4. Metric Design
0.06 reflects their
To effectively evaluate type inference systems, we introduce two complementary metrics: TYPESIM, which measures type prediction quality, and TYPECHECK, which assesses type coherence across codebases.
4.1.2. STRUCTURAL SIMILARITY
ListCompare. For generic types (e.g.. List [int], Dict [str, intl), we compute similarity recursively through their type tree structure, as shown in Figure This allows us handle nested types of arbitrary depth while considering both the container types and their type argu- ments.
4.1. Type Similarity
Traditional evaluation methods for type inference rely on exact matching, which fails capture the nuanced relation- ships between types. For example, Sequence [int] and Iterable [int] share most functionality but would re- ceive score of 0 under exact matching, even though both support iteration operations. Similarly, int and float would be considered completely different despite sharing most arithmetic operations. To solve this issue, we propose TypeSIM (Algorithm 1), continuous similarity metric that considers both functional similarity and structural relation- ships.
As shown on the left of Figure 2, for non-union types T and T', their similarity is:
1 STTTT) 2 ( (s Stsat args argss())))
where s(rool, rool') ) is the base type similarity, and Swot
(Algorithm 2) compares type arguments in order:
1. 1. BASE TYPE SIMILARITY
For non-generic types, we compute similarity based on their supported operations and methods. Given two types t and t', their similarity is:
(L,) Eismin(z).zny) S(Li, L1)
max([4], (41)
For example, comparing List[ints
with
stt, E (tttrs(t) (tttrstt))
Sequence [float]:
(ttrrs()UUlltrsss)
Base similarity: s(L.i Sequence) = 0.7 (shared sequence operations)
where attrs(t) represents the set of methods and operations supported by type 1, excluding those common to all types that inherit from object (like -.str-- or -iinit_.). We then the Jaccard index to measure functional similar- ity, where two types are similar if they share most of the same methods and operations. This approach captures the intuition that types supporting similar operations should be considered similar.
Argument similarity: float) 0.6 (shared
numeric operations)
Overall: S = (0.7+0.00 = -0.65
We choose average the similarity of the root and the argu- ments instead of multiplying them to give more weight to root types. This design aligns with common development practices where developers often annotate only the root type (c.g., List) without specifying arguments, especially dur- ing initial typing efforts. For example, when computing
Consider Python's 's collection types hierarchy as an example:
Iteerble provides _iter.. for iteration, enabling for loops.

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Algorithm 1 TYPESIM
Algorithm 3 SetCompare
Input: types T, T
Input: type sets A, B // Set-wise comparison for Union types
if At least one of T T" is Union then
Return: SetCompare( set(T), set(
for i 1 to | do
end if
for j 1 to [ do
score sTooo T", .root)
Gy
GefTypeSimilarityy B2)
if Both T. T have arguments then
end for end for Find optimal matching M using cost matrix C' =[cis]
score 1/ (score
DistCCoppaeeT.argy T" "aa,,""arrr... .args))
else One of T, T" has arguments then
score score
S >=E(IJEM Cij Return: max (I)
end if
g
Return: score
Algorithm 2 ListCompare
predicted types form a coherent system across the codebase. Since type annotations serve not only as documentation but also as a mechanism for early error detection and code maintenance, is crucial that predicted types work together consistently across the entire codebase.
Input: type lists L'
List-wise comparison for generic type arguments
S = 0
for: i min([[[,,|T'))
S S GelTypeSimilarityy L1)
We use the number of mypy errors proxy for type consistency because these errors directly reflect what devel- opers would need to fix before the type annotations become practically useful. For example, if a function is predicted to return Li List [int]] but is used in context expecting Li st [str], this inconsistency would prevent effective static type checking and IDE support two key benefits of type annotations. Specifically, we focus meaningful type errors that affect code correctness, such as incompatible re- turn types and invalid argument types. These errors indicate real issues that would hinder code maintenance and refac- toring. complete list of counted error types is provided in Appendix C.2.
end for
Return: max (
S(List, List [ [int]), while the argument similarity is 0 due to the missing argument, the base similarity is for matching root types. Averaging yields 0.5, acknowledging the partial correctness of the annotation, whereas multiplica- tion would give completely penalizing this common and often acceptable practice in gradual typing.
SetCompare. As shown on the right of Figure 2, we treat union types (e.g., Union [int, str]) as sets of possible types and compute an optimal matching between their mem- bers. This approach accounts for unordered union members and allows partial matches.
5. Dataset Curation
We curate a benchmark dataset containing 50 popular Python repositories from GitHub and PyPI to evaluate type inference capabilities. The dataset construction involves two key steps: repository selection and cleaning.
Given types T and T', where at least one is union:
S(T, (') = Seee(assset(T) (e)
where (T) converts type to its member set:
5.1. Repository Selection
[T] if not Union.
We select candidate repositories from GitHub's trending repositories and frequently downloaded PyPI Packages. In the initial filtering stage, we enforce the constraints of maximum of 1.5M tokens, minimum of 30 Python files, at least 50% typed functions, and valid mypy configurations to ensure the suitability of type inference evaluation. We then define quality score as below and rank the candidate repositories by their score:
as_set(T)= (Targs)) if is Union
Seet (Algorithm 3) finds the optimal matching between members:
Sset Eusees S(A,, max([A], [BI)
4.2. Type Consistency
==aSooverag + BS poppularity + S complexity>
While TYPESIM measures how close the predictions are human annotations, TYPECHECK evaluates whether the
where Scoverage measures the percentage of typed functions, Spopalarity calculated from the number of GitHub stars
5

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Algorithm 4 Type Removal Process
Web/API 12.0%
Dala Science 14.0%
Input: Repository It with typed Python files
Security 4.0%
Output: Repository R' with types removed for each Python file f E R do
Parse AST of f to locate type annotations for each function/variabbe declaration d do if d has type annotation t then Remove t from d in (e.g.. def foo(x: int) - def }
18.0%
Developer Tools
26 0%
end if
ML/AI
if the doestring c contains type hint for the function arguments returns (or variable annotations) then Remove the type hint
Figure 3. Distribution of repository categories in our benchmark dataset, showing coverage across major Python application do- mains.
end if end for Save modified file to R' end for
5.3. Benchmark Statistics
Return: R²'
To facilitate the potential evaluation of learning Searning-based based meth- ods, we split the 50 repositories into Train/Varn Validation/TTes splits, with 20/10/20 repositories correspondingly, The 20 test repositories are selected and further split into two sets with 10 repositories each based on the date created esti- mate the level of data contamination.
Table 1. The total number of tokens, functions, and variables to be inferred for different splits.
Repos
Tokens
Functions
Cases
Table summarizes the number of tokens, functions, and variables to be inferred for different splits. We also depict the diversity of TYPYBENCH in Figure 3 by classifying the repositories into domains spanning Developer Tools, MI/AI, Web/API, and Security. For more comprehensive details about each repository, please refer Appendix A.
Train
20
8403760
31161
59966
Validation
4037666
13988
Test
20
4983025
25101
46106
and PyPI downloads, and Scomplexity considers the depth and variety of type annotations. The top 50 repositories with the highest candidate scores are selected into the dataset.
6. Experiments
In this section, we conduct comprehensive experiments to answer the following key questions:
5.2. Repository Cleaning
To ensure the best quality of type inference evaluation, we remove non-P non-Python files, testing files, and irrelevant files, to only keep the Python files the source folder containing the main functionality. To ensure the quality of type annotations, run mypy check the original repository, and manually resolve errors that stop the check from running.
Model Readiness:
How well do current LLMs per- form on type inference? Are SOTA models ready for production use on untyped repositories? What are their key limitations? Does a longer context length help mitigate the limitations?
After initial cleaning, as illustrated in Figure 1, we remove type annotations using scripts while preserving code func- tionality to create input and ground truth pairs for evaluation. The type removal algorithm handles function signatures, and variable declarations as shown Algorithm 4.
Metric Effectiveness: How do our proposed TYPE- SIM and TYPECHECK metries compare to traditional exact matching? What additional insights do they pro- vide?
To maximize consistency and similarity with the original repository, every single part of each Python program is kept unchanged except for the type annotation. We rebuild each processed repository to ensure that the modified code remains syntactically valid and the runtime behaviors are preserved.
Performance Factors: How do factors like type com- plexity and repository age affect model performance? What do these patterns reveal about LLMs' type infer- ence capabilities?
6

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
1,400
SPM4
dlaude3.6
despseeks
5o44mmii
ap54o
0.800
deepoekky
0.780
grok-2
gpt4amini
TypeSim
deapsesko2
Qwie2.5
Uniedd 1-bb
X claudeS.5
Ulemad-ib
0.760
deepscekv2
0.250
150.0
200.0
250.0
300.0
350.0
400.0
450.0
TypeCheck
300.0
Figure 4. Comparison of TYPESIM and TYPECHECK metrics across models. SOTA models BLAUDE-3.5-SONNEE DEEPSEEK-V? GPT-40, GPT-4O-MiN MINI,, and GROK-2) achieve similar high TYPESIM scores while varying in TYPECHECK. showing TYPECHECK's additional discriminative power. Small local-hosted models (LLAMA-3-88, LLAMA-3. 1-8B, QWEN-2.5-7B) perform poorly.
6.1. Experimental Setup
Table 2. Average TYPESIM and TYPECHECK scores of all reposi-
We evaluate type inference capabilities across diverse set of SOTA LLMs, including APl-accessed models and local-hosted small models. The API-based LLMs include GPT-40, GPT-4O-MINI (Achiam et al., 2023), CLAUDE-3.5- SONNET (Anthropie, 2024), DEEPSEEK v2.5, DEEPSEEK- v3 (Liu al., 2024a), and GROK-2 (XAI, 2024). For bocal-hosted models, we evaluate popular models includ- ing L.LAMA3--8B I.LAMA-311 1-8B (Dubey et al., 2024), and QWEN-2.5-7B (Yang et al., 2024).
tories for various models. CLAUDE-3.5 -.UD--.5-SOONEL shows the best TYPECHECK score while top models share similar TYPESIM scores. The ground truth has a TYPECHECK score of 141.8.
MODEL
TYPECHECK
TYPESIM
WO MISSING
RATE L
LLAMA-3-888
187.5
0.363
0.731
0.508
LLAMA-3.1-88
465.7
OWEN-2.5-76
411.0
0.604
0.787
0.238
GPT-40
366.0
0,804
0.893
0.099
GPI-4O-MINI
3.AEDD-3.--SONETT
127.1
0.788
0.893
0.1
DEEPSBEK-V2.5
328.7
0.754
0.907
0.169
While our benchmark provides train/validation/tess splits for future learning-batt methods, we evaluate pre-trained LLMs on all repositories since fine-tuning is not performed the training We separately analyze potential pre- training data contamination through temporal analysis (Sec- tion 6.3).
214.6
0.795
GROX-2
190.1
0.787
0.903
0.129
overall TYPECHECK score than the ground truth (141.8 errors on average), though it still performs worse than the ground truth on 29 repositories. This consistency could be particularly valuable when humans need to fix the type- checking errors manually.
Since most repositories contain more number of tokens than the context length most LLMs, LLMs are tested in a file- by-file method, where we require LLMs infer the pyy stub file from the type-removed file with the same prompt (see Figure 9 and Appendix C for more details).
6.3. Factors Analysis
Impact of Type Complexity. We first compare TYPESIM with exact match metrics for types with different depths. Table 3 reveals an increasingly widening gap at higher depths. While both metrics show declining trends, exact match scores drop more precipitously - nearly vanishing for types of depth and above. In contrast, TYPESIM still captures semantically valid predictions that would be completely rejected by exact matching. This demonstrates TYpeSim's value in providing more nuanced evaluation, particularly for complex types where multiple valid type annotations may exist. As shown in Figure 5, model per- formance consistently slightly degrades as type complexity increases with increased variance. Even SOTA models strug- gle with deeper nested types (depth > 2), suggesting that
6.2. Main Evaluation Analysis
As shown Table 2. our analysis reveals both promising advances and concerning limitations in L.LM- type inference. While SOTA models achieve decent TYPESIM scores around 0.80, their noglligiig missing rates sug- gest systematic limitations in stably generating correct stub files. Moreover, as illustrated in Figure 4, the TYPECHECK metric reveals an interesting pattern model consistency. While most models struggle with type checking. CLAUDE- keep the type predictions
3.5- is the best consistent despite not having the highest TYPESIM. It is worth noting that CLAUDE-3,0 -AA-D--..5-SOEE 3.5-SONNET achieves a better

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Table Compare TYPESIM and exact match by type depth, the difference increases with depth.
TYPESIM
EXACT MATCH f
MODEL
DEPTHI
DEPTH2
DEPTH3
DEPTH4
DEPTH5
DEPTH
DEPTH2
DEPTH3
DEPTH4
DEPTH5
LLAMA-3-8B
0.415
0.271
0.235
0.209
0.136
0.406
0.023
LLAMA-3.1-8B
0.635
0.555
0,499
0,478
0.454
0.610
0.427
0.279
0.201
0.118
QWEN-2.5-78
0.658
0.525
0.475
0.443
0.438
0.640
0.420
0.277
0.191
0.113
GPT-40
0.822
0.775
0.753
0.733
0.776
0.792
0.633
0.497
0,417
0.350
GPT-4O-MINI
0.813
0.747
0.703
0.721
0.705
0.782
0.616
0.420
0.370
0.209
L.AUDE-3.5-SONNEE
0.801
0.768
0,746
0.749
0.746
0.769
0.652
0.517
0.427
0,303
DEEPSEEK-V2.5
0.771
0.722
0.697
0.627
0.676
0.745
0.609
0,462
0,320
0,218
DEEPSEEK-V3
0.809
0.769
0.747
0.702
0.708
0.774
0.644
0.295
GROK-2
0.802
0.757
0.737
0.710
0.743
0.771
0.642
0.496
0.387
0.274
AVERAGE
0.725
0.654
0.621
0.597
0.598
0.699
0.540
0.394
0.307
0.211
DIFF
-0.026
-0.114
-0.228
-0,289
-0,387
0.8
Typasiin Score on Rtar 1ypss (-5)
0.6
TypeSim
0.4
0.6
0.2
0.0
2
3 Depth
4
05
06
0.7
0.9
Overall Typesim Score
Figure 5. TYPESIM consistent high scores for simple types (depths 1) but show de- graded performance and incrcased variance for more complex types (depths 2+).
type complexity. Models achieve
Figure 6. Overall TYPESIM
TYPESIM
types. Scatter plot of average TYPESIM score on rare types and all types for each repository. A clear drop the observed on 40 out of 50 repositories (under the red line).
complex type inference remains a significant challenge in some cases.
ward way is to use the whole repository the context given to LLMs. However, this approach faces two major chal- lenges, long context length and long output length, due to the size of the whole repository. Nevertheless, we tested this approach with GPT-40 on repositories with total num- ber tokens smaller than 64k. shown Table 4, the TYPECHECK errors do decrease significantly with the whole repository context, with the cost of worse TYPESIM scores, potentially due to harder to exact the correct information from the long context and harder to generate all the predic- tions in the correct format. This suggests that more context is helpful for enhancing the type consistency, but the long input and output challenges need be addressed.
Impact of Type Frequency. As pointed out in previous work (Allamanis et al., 2020), predicting rare types that are less frequent in the repository is a challenge. As shown in Table we still observe gap between the TYPESIM scores on all types and the scores on rare types, but the gap that large overall. However, when it comes specific repositories, shown in Figure 6, the drop is significant in many repositories, indicating predicting rare types is still challenging for LLMs. An interesting observation is that TypESIM on rare types could also be higher than the overall TYPESIM in a minority of repositories.
Impact of Data Contamination. As observed in previous work (Roberts et al., 2023), data contamination a potential issue when evaluating the performance of L.MMs. To verify
Impact of Revo-Level Context. To reduce the TYPE- CHECK errors in the predicted types, the most straightfor-
8

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
TYPYBENCH will facilitate progress in LLM-based type inference by providing a standardized evaluation framework and highlighting key areas for improvement.
Table Comparision between full repo context and single file con- text with GPT-40. The TYPECHECK errors decrease significantly with the whole repository context, but the TYPESIM scores also decrease significantly some repositories.
Impact Statement
PULL Rero
SINGLE PILE
REPo TyreSt: f
TyssCHick
TereSto 1
TresCHbcc
This paper presents work whose goal is to advance the field of Machine Learning and Software Engineering. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
681
0.855
606
0.915
Acknowledgments
0.803
TypoSim
We thank Jialiang Sun for his valuable contributions to data collection and curation, Shiwen Wu for her insightful discus- sion, and Chris J. Maddison for his support on this project. This research project has benefitted from the Microsoft Ac- celerate Foundation Models Research (AFMR) grant pro- gram.
0.700
AV0=0.70 0.705
AVG
0.403
01/120112 01/1/011 01/1/2018
01/1/2018 01/1/2000 01/1/002 01/12024
Date Created
References
Figure 7. TYPESIM scores v.s. repository creation dates. Better performance observed on older repositories (pre-2024 avg: 0.705) compared to newer ones (2024+ avg: 0.649), indicating potential data contamination effects.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 087744 2023.
Allamanis, M., Barr, T., Ducousso, S., and Gao, Z. Typ- ilus: Neural type hints. In Proceedings of the 41 acm sigplan conference on programming language design and implementation, pp. 91-105, 2020.
this, we compare the TYPESIM scores of repositories cre- ated in different years. As shown in Figure 7, the TYPESIM scores of repositories created in 2024 and after are lower than that of repositories created before 2024, suggesting po- tential data contamination effects. So we suggest that future work should use the test set that contains relatively recent repositories when comparing the performance of different models, and we include the results of the test set in Table 8 in Appendix D.
Anthoopic Claude 3.5 sonnet, 2024.
Avail-
able at: https://www httts:://ww.anthro.i.c. anthropic com/news) al 1ude-3-5-sonne
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
7. Conclusion
We present TYPYBENCH, a comprehensive benchmark for evaluating LLMss Python type inference capabilities. Our evaluation reveals that while SOTA models achieve promising TYPESIM scores, they still face significant chal- lenges: poor handling of complex nested types, and sub- stantial type consistency errors. The proposed TYPESIM and TYPECUECK metrics provide complementary insights, with TYPESIM capturing semantically valid predictions and TYPECHECK revealing critical consistency issues. The ex- perimnntal finding suggests that the focus of type inference should turn to the repo-level consistency since the similar- ity is already high. We further find that increased context length improves type consistency but creates challenges in handling long inputs and outputs, suggesting the need for more efficient context handling mechanisms. We hope
Cassano, F,, Yee, M.-H., Shinn, N., Guha, A., and Holtzen, S. Type prediction with program decomposition and fill- in-the-type training, 2023.
Chen, M.. Tworek, J., Jun, H., Yuan, Q. Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Dong, H., Su, Q. Gao, Y., Li, Z., Ruan, Y., Pekhimenko, G., Maddison, C. J., and Si, X. Appl: A prompt pro- gramming language for harmonious integration of pro- grams and large language model prompts. arXiv preprint arXiv:2406. 13022466.33000 /3/61, 2024.

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Mir, A. M., Latoškinas, E.,, and Gousios, G. Manytypes4py: A benchmark python dataset for machine learning- based type inference. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), PP. 585-589 IEEE, 2021.
Hellendoorn, V. J., Bird, C.,, Barr, E. T.,, and Allamanis, M. Deep learning type inference. In Proceedings of the 2018 26th acm joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp. 152 162-162,22 2018.
Peng, Y., Wang, C., Wang, W., Gao, C., and Lyu, M. R. Generative type inference for python. In 2023 38th IEEE/ACM International Conference on Automated Soft- ware Engineering (ASE), pp. 988-999. IEEE, 2023.
Pradel, M., Gousios, G., Liu, J., and Chandra, S. Typewriter: Neural type prediction with search- based validation. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 209-220, 2020.
Instagram. Monkeytype: A python library that generates static type annotations by collecting run- time types. https: http:://gtttubcccoo hatto//ccoonnattaaaaeaaa Instagram/ MonkeyType.
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F,, Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint 20Xi::4403.07974. 2024.
Raychev, V., Vechev, M., and Krause, A. Predicting program properties from" big code". ACM SIGPLAN Notices, 50 (1):111-124, 2015.
Roberts, M., Thakur, H., Herlihy, C.,, White, C.,, and Dooley, S. Data contamination through the lens of time. arXiv preprint urXiv:2310. 10628, 2023.
Jesse, K., Devanbu, P. T., and Ahmed, T. Learning type annotation: Is big data enough? In Proceedings of the 29th ACM Joint Meeting on European Softare Engineer- ing Conference and Symposium on the Foundations of Software Engineering, pp. 1483-1486, 2021.
Shivarpatna Venkatesh, A. P., Sabu, S., Wang. J., M. Mir, A., Li, L.. and Bodden, E. Typeevalpy: micro- benchmarking framework for python type inference tools. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Pro- ceedings, pp. 49-53, 2024.
Jimenez, C. E., Yang, J.. Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
Wang, Y., Wang, Y., Wang, S., Guo, D., Chen, J., Grundy, J., Liu, X., Ma, Y., Mao, M., Zhang, H., et al. Repotrans- bench: A real-world benchmark for repository-level code translation. arXiv preprint arXiv: 177444 2024.
Lehtosalo, J., van Rossum, G., Levkivskyi, I., and Sullivan, M. J. mypy. http: http:://ypy-//tet /mppy-aapyyyyaappoo org/. Available at http:///// /mypy//ppp/////// lang.arg. org/.
Wei, J., Goyal, M., Durrett, G., and Dillig, I. Lambdanet: Probabilistic type inference using graph neural networks. In International Conference on Learning Representations, 2020.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseck- v3 technical report. arXiv preprint 19344.7412.19437, 2024a.
Wei, J.. Durrett, G., and Dillig, I. Typet5: Seq2seq type inference using static analysis. In The Eleventh Interna- tional Conference on Learning Representations, 2023.
Liu, T., Xu, C., and McAuley, Repobench: Benchmarking repository-lever code auto-completion systems. In The Twelfth International Conference Learning Represen- tations, 2024b.
XAI. Grok-2 beta release, 2024. Available at: https: hwol///gggoo///////////// //x //x.aix/bilieee/ wtt.sii/bblg//// grok- 2.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu,
Malik, R. S., Patra, J., and Pradel, M. N12type: Inferring javascript function types from natural language informa- tion. In 2019 IEFF/ACM 41st International Conference on Software Engineering (ICSE), pp. 304-315. IEEE, 2019.
B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2.5 technical report. arXiv preprint 4:X1v22412.15115, 2024.
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, N. B., Zhan, H., He, J., Paul, I., et al. Big- codebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint AXXv:2440615877, 2024.
Microsoft. Pyright: Static type checker for python. ht https: / //gi thub. cwm/mi mi crosoft/pyrightt
10

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
A. Repository Information
The statistics of repositories are shown in Table We use the tokenizer wwwtoktn.cl wwwp:kkeenn.11kk..aaeea... to count the number of tokens.
A.1. TypeCheck Results
Table 6 shows the results of running mypy check in the original repository for all repositories (except the ones are slow running mypy). We still include some repositories that have a few mypy errors making room for improvement.
B. Basic Type Similarities
We illustrate the TYPESIM between builtin types in Figure 8, and list the TYPESIM scores for some similar types in Table 7,
Similarity Score for Build-in Python Types
int-
6.49
0333
bool
0.11
6:02
5:93
000
5:30
bytes
5:2
2.00
1.03
545
002
200
0:2
3:20
EsCAnv|
dictAayy Anyl
0000
200
S01
5.77
100
AUT
C22
3.20
tuplelAry, ...
>e[[nny]
612
E:7
021
Nonc
am
530
201
5:33
5:50
bytes
BatiAny)
diaffny Any)
mntAnm]
Nome
Am
00
Cuplot Ama.
Figure 8. TYPESIM scores between builtin types.
C. Experimental Settings
We use Python version 3. 12 and Mypy version 1 11. for all our experiments. For API with multiple versions, 2-21-2-1212, and DEEPSEEK-
use 209-44-2024-08-066 APT-40-MINN--0044-77-1
v3 (12-26 VERSION).
We use APPL (Dong et al., 2024) to implement the evaluation for LLMs.
C.1. Prompt
Figure illustrates the prompt used for single-file type inference in our experiment. The prompt includes Python file along with its corresponding pyi file as an example. Additionally, it specifies the expected answer format and emphasizes that the generated output must be free of syntax errors.

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Table The statisties of all repositories in TYPYBENCH, including the number of tokens, functions, variables be inferred, the ratio of functions being annotated, the category, and the date created. The test sets are further split into two testl and test2 based on the date created test contamination issue.
FUNC. ANNOT.
AVG DEPTH
REPOSITORY
# TOKENS # FUNC
# CASES
CATEGORY
DATE CREATED
TRAIN
PRE-COMMIT-HOOKE
16268
105
810
1.00
1.57
DEVELOPER TOOLS
201 2014/03/13
FLAKES
34684
209
747
1.00
1.4
DEVELOPER TOOLS
2014/09/13
VNPY
47762
415
909
0.99
1.11
DATA SCIENCE
2015/03/02
GHUNT
63135
284
2185
0.62
1.91
SECURITY
2020/10/02
SPOTIFY-DOWNLOADEER
89384
232
2143
0.90
1.32
OTHERS
2016/07/06
DEEPFACE
90831
194
3263
0.57
1.4
OTHERS
2020/02/08
URLLIB3
98963
551
1844
1.00
1.49
DEVELOPER TOOLS
201 1/09/18
FASTAPI
137361
307
1983
0.93
1.7
WEB/API
201 8/12/08
POETRY
153087
1022
3556
1.00
1.32
DEVELOPER TOOLS
8/02/28
HAYSTACK
209305
2035
1.57
ML/AI
2019/11 9/11/11
RICH
244573
948
518
0.95
1.21
DEVELOPER TOOLS
2019/11/18
PYDANTIC
359486
1780
565
0.98
1.26
DEVELOPER TOOLS
2017/05/03
PHiDATA
552011
1799
2687
0.93
1,23
DEVELOPER TOOLS
2022/05/04
PILLOW
607377
3460
3923
0.99
1,35
DATA SCIENCE
2012/07/24
SPHINX
612272
4647
10285
1.00
1,27
DEVELOPER TOOLS
2015/01/02
FACESWAP
732572
3352
441
0.75
1.46
OTHERS
2017/12/19
STREAMLIT
806972
3236
3991
0.83
1.34
WEB/API
2019/08/24
GRADIO
874060
1845
1400
0.62
1.88
WEB/API
2018/12/19
1176602
5369
4789
0.62
1.36
DEVELOPER TOOLS
201 1/03/06
BLACK
1497055
11892
0.99
1.28
DEVELOPER TOOLS
2018/03/14
VALIDATION
TYPER
38236
167
1588
0.96
1.16
DEVELOPER TOOLS
2019/12/24
PRE-COMMIT
50380
345
2391
1.00
1.42
DEVELOPER TOOLS
2014/03/13
FLASK
73148
364
834
1.00
1.51
WEB/API
2010/04/06
PDM
172537
1291
532
0.98
1.37
DEVELOPER TOOLS
2019/12/27
MANIM
179218
1733
415
0.82
1.29
DATA SCIENCE
2020/05/19
NICEGUI
206638
1217
6018
0.98
1.09
WEB/API
2021/05/07
OPEHON
274146
1085
263
1.00
1.35
ML/AI
2020/10/25
403846
2356
2064
0.75
1.41
WEB/API
2022/02/18
OPENRB
1290501
3771
3137
0.70
1.8
DATA SCIENCE
2020/12/22
CAPA
1349016
1659
2935
1.53
SECURITY
2020/06/16
TESTI
PRIVATE-GUE GPT
45562
197
257
0.98
1,36
ML/AI
2023/05/02
GPTME
58715
319
512
0.79
1.51
MI/AI
2023/03/24
PAPER-QA
73284
353
764
0.95
1.64
MIJAI
2023/02/05
PANDAS-AI
127754
996
1145
0.66
1.21
ML/AI
2023/04/22
SUPERVISION
150793
505
1101
0.90
1.49
DATA SCIENCE
2022/11 1/28
GPT4FRBE
168395
679
808
0.76
1.11
ML/AI
2023/03/29
AUTOGPT
306046
1797
2235
0.78
1.34
ML/AI
2023/03/16
384359
1698
3182
0.83
1.25
MI/AI
2023/04/29
DB-GPT
817402
5329
9732
1,26
ML/AI
2023/04/13
VLLM
1037766
12064
1.22
ML/AI
2023/02/09
TEST2
SCREENSHOTT-OO-COSS
44482
60
102
0.73
1.58
OTHERS
2023/11/14
EXO
69991
406
721
0.61
1.28
OTHERS
2024/06/24
TEN-AGENT
71448
412
1076
0.76
1.07
OTHERS
2024/06/19
GPT-PILOT
94918
516
1101
0.82
1.19
MIJAI
2023/08/16
APPWORLD
156441
1125
2185
0.90
1.42
OTHERS
2024/06/23
1S6679
966
1956
0.88
1.14
OTHERS
2023/10/19
LEROBOT
183740
612
1068
0.57
1.69
OTHERS
2024/01/26
LLAMA-FACTORY FACTORY
194043
520
1778
0.88
1.52
ML/AI
2023/05/28
COMPOSIO
345846
1059
1963
1.44
DATA SCIENCE
2024/02/23
UNSTRACT
495361
2281
2416
0.89
1.13
DATA SCIENCE
2024/02/21
12

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Table 6. Repositories and their TypeCheck Errors. The repositories with slow nypy speed reported with N/A.
Train
Dev
Test1
Test2
Repo
Errors
Repo
Errors
Repo
Errors
Repo
Errors
pre-commit-hooks
pre-commit
gptme
screenshot-to-code
flake8
flask
private-gpt
14
composio
urllib3
nicegui
paper-qa
24
agents
haystack
typer
AutoGPT
48
unstract
62
rich
OpenBB
4
mic-lim
63
TEN-Agent
87
black
openai-python
8
supervision
136
exo
89
fastapi
taipy
52
pandas-ai
452
LLaMA-Factory
182
Pillow
pdm
76
vllm
466
gpt-pilot
332
poetry
capa
151
gpt4free
795
lerobot
N/A
spoiity-downloader
manim
N/A
DB-GPT
1222
appworld
N/A
faceswap
11
streamlit
13
deepface
24
phidata
34
vnpy
137
pip
165
sphinx
218
pydantic
339
GHunt
589
gradio
846
Average 119.45
Average
33.0
Average
322.0
Average
94.75
Table 7. A list of similar type pairs. It can be seen that the TYPESIM score reasonably exhibits the similarity hetween the two types.
Original
Predicted
TypeSim
list [Any] None
[str] None
0.75
diet [Any, Any] | None
dict[stx: Any] None
0.875
diet [str, tuple [Any, ...]] None
[ict, tuple[int, ...]] None
0.9375
Any dict [str, diet [Any, Anyl] dict [str, 1ist [int]]
Any dict [str, Any] dict [str, Union [tople [int, ...), Any]] ]
0,875
0,8385
float ndarray [Any, Any] pathlib.Path None
np. nd.nray [Any, Any]
0.5
str nathlib.Path None
0.6667
Figure 10 illustrates the prompt used for full-repo context type inference in experiment. We first input the structure of the repository and followed by the content of the source code. Additionally, we specifies the expected answer and path format.
C.2. TYPECHECK Score Calculation
We classify the following Mypy error types as indicators of consistency score: attr-defines assignment, arg-type, union-attr, and index.
D. Supplementary Experimental Results
D.I. Results on Test Set
We further summarize the results for the test set in Table 8.
13

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
User
Given python file,
is .pyi that contains only the typing information.
Here is an example of an input python file that is missing typing information: (py example)
Here is the correct stub file generated for the above python file:
{pyi_exanee example)
Your task is to infer the missing typing information and generate stub file(.pyi) for newly given python file.
You must generate the stub file markdown format: "apraat"""pphttaaa python <stub content>
The generated content must be a valid python file with no syntax error.
Note that relevent import statements should be included in the stub file.
Here is the python file for your task:
ffile_com content)
Assistant
Figure 9. Prompt Template for Single File Context
User
You are given repository with repository directory structure and the source files. # Inputs
directory structure ===-
(director structure)
==== source ====
The contents of the source code are as follows:
(source code)
# Task
stub file is a pyi file that contains only typing information. Your task is to generate stub file(. pyy) for all the pytho files in the repository.
Stub files must be free syntax error.
Your output should follow the format example below: stubfilepath: <relltive path> python <stub content> d
Assistant
Figure 10. Prompt Template for Whole Repo Context
D.2. Results for Each Model
We present the detailed results in Table 9 17 for each model on all repositories.
14

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Table 8. Average TYPESIM scores and TYPECHECK scores of the repositories in the test set for various models.
MODEL.
TYPECHECK 4
TyPeSIM
TYPeSIM wo MISING t
MISSING RATE 4
LLAMA-3-8B
150.6
0.396
0.747
0.470
1AAMA-311-8B
400.2
0.634
0.815
0.225
QWEN-2.5-7B
405.7
0.649
0.817
0.207
GPT-40
367.9
0.787
0.883
0.108
GPT-40-MINI
251.7
0.779
0.882
0.117
CLAUDE-3.5-SONNET
173.3
0.790
0.876
0.098
DEEPSEEK-V2.5
301.4
0.748
0.893
0.161
DEEPSEEK-V3
192
0.793
0.889
0.107
GROK-2
199.5
0.769
0.885
0.131
15

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Table TYPECHECK and TYPESIM for GPT-40 each repository.
TYPESIM BY DEPTH
REPO
TYPE CHECK
TYPE SIM
TYPESIM WO MISSING
MISSING RATIO
DEPTH1
DEPTH2
DEPTH3
DEPTH4
DEPTH5
AGENTS
202
0.774
0.883
0.123
0.766
0.814
0.694
0.000
APPWORLD
N/A
0.835
0.897
0.068
0.876
0.747
0.729
0.742
1.000
AUTOGPT
235
0.755
0.801
0.057
0.746
0.786
0.728
0.794
0.809
BLAAKK
131
0.807
0.898
0.101
0.822
0.771
0.692
CAPA
546
0.883
0.902
0.021
0.922
0.786
0.781
0.817
COMPOSIO
107
0.646
0.665
0.029
0.602
0.722
0.819
0.908
DB-GPT
2482
0.803
0.874
0.081
0,808
0.792
0.822
0.790
0.781
DEEPFACE
0.898
0.963
0.068
0.886
0,940
0.875
0.990
1.000
EXO
0.746
0.928
0.196
0.729
0.810
0.666
0.875
0.750
FACESWAP
67
0.829
0.973
0.149
0.863
0.745
0.595
0.600
0.737
FASTAPI
290
0.810
0.876
0.075
0.893
0.746
0.782
0.917
0.930
FLAKE8
0.793
0.936
0.153
0.720
0.765
FLASK
0.876
0.931
0.059
0.903
0.863
0.576
0.424
GHUNT
0.755
0.860
0.122
0.734
0.732
0.975
0.997
GPT-PILOT
267
0.813
0.885
0.082
0.828
0.812
0.692
0.851
GPT4TREE
413
0.835
0.898
0.071
0.691
0.695
0.965
GPTME
84
0.877
0.926
0.053
0.869
0.963
0.984
GRADIO
1161
0.655
0.836
0.216
0.715
0.629
0.546
0.457
0.709
HAYSTACK
335
0.692
0.910
0.239
0.696
0.755
0.871
LEROBOT
N/A
0.903
0.131
0.778
0.806
0.761
0.892
LLAMA-FACTORY
0.820
0.903
0.092
0,740
0.789
1,000
MANIM
0.835
0.943
0.114
0.691
0.562
0.000
MLC-LLM
0.919
0.196
0.874
0.858
0.742
NICEGUI
426
0.863
0.040
0.786
0.726
0.871
0.403
OPENAI-PYTHON
434
0.696
0.783
0.110
0.633
0.704
OPENBB
45
0.840
0.857
0.020
0.791
0.775
0.963
PANDAS-AI
209
0.729
0.787
0.073
0.737
0.731
0.663
1,000
PAPER-QA
0.867
0.904
0.041
0.846
0.726
0.825
0.943
PDM
0.880
0.932
0.056
0.920
0.773
0.754
PHIDATA
1333
0.895
0.014
0.908
0.844
0.818
PILLOW
0,715
0.877
0.185
0.754
0.611
0.514
0.530
0.323
PIP
976
0.816
0.936
0.128
0.851
0.720
0.686
0.709
0.857
POETRY
0.864
0.957
0.097
0.742
0.663
0.431
0.958
PRE-COMMIT
33
0.900
0.910
0.011
0.910
0.872
0.951
PRE- TEE-COMMIT-HORE
0.942
0.000
0.934
0.945
0.973
PRIVATE-GPT
0.855
0.924
0.075
0.886
0.801
PYDANTIC
1143
0.887
0.081
0.875
0.700
0.647
0.701
0.000
45
0.785
0.957
0.179
0.794
0.756
0.688
0.969
SCREENSHOT-TO-CODE
0.915
0.915
0.000
1.000
0.847
0.611
1.000
SPHINX
0.836
0.231
0.631
0.708
0.718
0.697
0.588
DETIIY-DDWNLLADER
0.876
0.911
0.039
0.887
0.837
0.947
0.913
29
0.804
0.882
0.089
0.807
0.708
0.941
SUPERVISION
0.872
0.971
0.102
0.865
0.876
0.900
0.851
0.917
804
0.742
0.883
0.159
0.779
0.671
0.706
0.787
0.875
-EN-AGENT
0.698
0.833
0.163
0.683
0.776
0.720
0.802
94
0.701
0.937
0.252
0.761
0.625
0.617
0.000
251
0.812
0.867
0.064
0.827
0.788
0.779
0.475
0.819
0.901
0.091
0.877
0.723
0.749
0.962
1,000
0.759
0.885
0.143
0.764
0.758
0.704
0.681
0.972
46
0.907
0.909
0.002
0.904
0.933
0.976
0.583
16

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Table 10. TYPECHECK and TYPESIM for GPT-4O-MINI each repository.
TYPESIM BY DEPTH
REPO
TYPE CHECK
TYPE SIM
TYPESIM WO MISSING
MISSING RATIO
DEPTH1
DEPTH2
DEPTH3
DEPTH4
DEPTH5
AGENTS
53
0.756
0.887
0.148
0.751
0.793
0.599
0.000
APPWORLD
N/A
0.571
0.921
0.380
0.626
0.429
0.512
0.496
1.000
AUTOGPT
77
0.786
0.848
0.073
0.802
0.763
0.687
0.679
0.807
BLAAKK
0.731
0.881
0.171
0.759
0.635
0.615
0.913
CAPA
332
0.762
0.806
0.055
0.793
0.732
0.644
0.817
COMPOSIO
98
0.732
0.779
0.061
0.738
0.724
0.663
0.983
DB-GPT
2033
0.815
0.886
0.080
0.819
0.809
0.815
0,803
0.818
DEEPFACE
9
0.894
0.949
0.058
0.898
0.876
0.990
0.688
EXO
74
0.730
0.901
0.190
0.713
0.783
0.699
0.859
0.719
FACESWAP
91
0.835
0.978
0.146
0.867
0.755
0.632
0.583
0.571
FASTAPI
722
0.777
0.861
0.098
0.872
0.725
0.688
0.824
0.922
FLAKE8
28
0.776
0.923
0.159
0.656
FLASK
0.784
0.922
0.150
0.704
0.565
0.236
GHUNT
0.750
0.854
0.122
0.735
0.717
0.966
0.944
GPT-PILOT
66
0.866
0.907
0.045
0.895
0.849
0.712
0.942
GPT4TREE
341
0.796
0.886
0.102
0.643
0.655
0.975
GPTME
28
0.888
0.917
0.031
0.860
0.888
0.984
GRADIO
1088
0.629
0.834
0.246
0.612
0.510
0.397
0.580
HAYSTACK
448
0.707
0.915
0.227
0.699
0.799
LEROBOT
N/A
0.752
0.896
0.160
0.767
0.777
0.623
0.892
LLAMA-FACTORY
0.802
0.865
0.074
0.751
0.709
0.994
MANIM
0.799
0.926
0.137
0.670
0.450
0.502
0.000
MLC-LLM
0.730
0.919
0.207
0.681
0.878
0.817
0.756
NICEGUI
318
0.831
0.898
0.075
0.741
0.683
0.482
OPENAI-PYTHON
633
0.726
0.807
0.100
0.760
0.691
OPENBB
277
0.898
0.914
0.018
0.914
0.774
0.946
PANDAS-AI
0.804
0.091
0.732
0.760
0.623
1,000
PAPER-QA
0.881
0.080
0.769
0.741
0.863
0.948
PDM
0.859
0.909
0.055
0.917
0.711
0.640
PHIDATA
1552
0.899
0.916
0.018
0.950
0.829
0.822
0.729
0.768
PILLOW
218
0.720
0.918
0.216
0.772
0.561
0.527
0.492
0.461
PIP
718
0.768
0.928
0.173
0.801
0.673
0.624
0.717
0.835
POETRY
67
0.855
0.951
0.101
0.712
0.561
0.513
0.825
PRE-COMMIT
0.879
0.900
0.024
0.913
0.796
0.881
PRE- TEE-COMMIT-HORE
0.905
0.905
0.000
0.920
0.930
0.808
PRIVATE-GPT
0.819
0.905
0.095
0.829
0.824
0.690
PYDANTIC
765
0.699
0.879
0.204
0.749
0.594
0.620
0.472
0.000
0.742
0.927
0.200
0.760
0.681
0.631
0.719
SCREENSHOT-TO-CODE
0.905
0.905
0.000
0.969
0.855
0.616
1.000
SPHINX
1157
0.624
0.808
0.227
0.622
0.645
0.627
0.475
0.525
DETIIY-DDWNLLADER
0.837
0.902
0.073
0.820
0.860
0.919
0.938
21
0.877
0.932
0.060
0.882
0.763
0.973
SUPERVISION
0.796
0.966
0.175
0.848
0.727
0.726
0.854
0.301
704
0.760
0.878
0.134
0.809
0.667
0.712
0.784
0.813
-EN-AGENT
28
0.716
0.838
0.145
0.713
0.743
0.699
0.583
0.902
0.932
0.032
0.955
0.830
0.840
0.958
91 0.807
0.864
0.066
0.817
0.784
0.811
0.913
0.820
0.909
0.097
0.876
0.737
0.727
0.655
1,000
1136
0.677
0.874
0.226
0.671
0.710
0.605
0.521
0.394
75
0.934
0.934
0.000
0.931
0.933
0.973
1.000
17

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Table 11. TYPECHECK and TYPESIM for CLAUDEF-3.-SONNTE each repository.
TYPESIM BY DEPTH
REPO
TYPE CHECK
TYPE SIM
TYPESIM WO MISSING
MISSING RATIO
DEPTHl
DEPTH2
DEPTH3
DEPTH4
DEPTH5
AGENTS
21
0.646
0.852
0.242
0.643
0.673
0.514
0.000
APPWORLD
N/A
0.660
0.927
0.287
0.725
0,502
0.584
0.512
0.000
AUTOGPT
60
0.769
0.856
0.102
0.749
0.821
0.793
0.752
0.604
BLAAKK
0.767
0.925
0.171
0.785
0.741
0.531
CAPA
61
0.887
0.915
0.031
0.898
0.836
0.879
0.821
COMPOSIO
0.578
0.684
0.155
0.569
0,581
0.634
0.960
DB-GPT
1767
0.797
0.870
0.084
0.809
0.777
0.809
0.802
0.806
DEEPFACE
0.930
0.955
0.027
0.934
0.941
0.886
0.990
1,000
EXO
36
0.825
0.897
0.080
0,810
0.869
0.799
0.984
0.930
FACESWAP
0.824
0.965
0.146
0.848
0.770
0.651
0.650
0.571
FASTAPI
137
0.780
0.887
0.120
0.851
0.732
0.770
0.754
1.000
FLAKE8
0.830
0.903
0.080
0.774
0.709
FLASK
0.889
0.948
0.062
0.909
0.858
0.758
0.757
GHUNT
0.853
0.853
0.000
0.920
0.703
0.975
0.993
GPT-PILOT
93
0.848
0.886
0.043
0.856
0.848
0.765
0.973
GPT4TREE
267
0.835
0.887
0.058
0.690
0.641
0.960
GPTME
0.906
0.915
0.010
0.907
0.891
0.969
0.984
GRADIO
0.722
0.846
0.147
0.774
0.701
0.629
0.700
HAYSTACK
462
0.726
0.889
0.183
0.708
0.705
0.876
0.788
LEROBOT
N/A
0.822
0.916
0.103
0.892
LLAMA-FACTORY
73
0.746
0.874
0.147
0.771
0.697
0.697
1,000
MANIM
N/A
0.752
0.934
0.195
0.779
0.687
0.560
0.423
0.750
MLC-LLM
0.901
0.224
0.688
0.759
0.576
0.625
NICEGUI
0.806
0.870
0.074
0.782
0.567
0.369
OPENAI-PYTHON
0.519
0.804
0.355
0,518
0.603
OPENBB
0.866
0.875
0.010
0.871
0.770
0.787
0.967
PANDAS-AI
0.773
0.018
0.708
0.648
0.969
PAPER-QA
0.830
0.897
0.075
0.804
0.758
0.943
PDM
89
0.870
0.122
0.692
0.701
PHIDATA
339
0.928
0.939
0.012
0.961
0.827
PILLOW
0.754
0.916
0.178
0.776
0.682
0.680
0.597
0.915
PIP
264
0,809
0.921
0.122
0.838
0.725
0.661
0.820
0.857
POETRY
31
0.960
0.102
0.779
0.722
0.734
0.825
PRE-COMMIT
0.915
0.035
0.869
0.883
PRE- SEL-COMMIT-HOON
0.914
0.914
0.000
0.884
0.964
0.981
PRIVATE-GPT
0.879
0.946
0.857
0.934
0.866
PYDANTIC
433
0.684
0.868
0.212
0.702
0.675
0.000
0.760
0.952
0.202
0.760
0.767
0.669
0.719
SCREENSHOT-TO-CODE
0.937
0.937
0.000
0.970
0.924
0.611
1.000
SPHINX
0.626
0.872
0.282
0.611
0.704
0.725
0.652
0.582
DETIIY-DDWNLLADER
0.845
0.896
0.057
0.841
0.836
0.957
0.938
0.564
0.910
0.380
0.530
0.796
0.993
SUPERVISION
26
0.884
0.964
0.084
0.895
0.861
0.896
0.926
0.660
0.652
0.887
0.265
0.694
0.574
0.589
0.767
0.563
-EN-AGENT
0.799
0.841
0.049
0.801
0.810
0.728
0.531
0.674
0.862
0.218
0.713
0.616
0.669
0.000
0.803
0.833
0.036
0.814
0.788
0.763
0.525
0.817
0.927
0.119
0.857
0.734
0.890
0.954
1,000
615
0.743
0.891
0.166
0.733
0.774
0.733
0.637
0.797
28
0.928
0.928
0.000
0.925
0.930
0.970
1.000
18

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Table 12. TYPECHECK and TYPESIM scores for DEEPSEEK- on each repository.
TYPESIM BY DEPTH
REPO
TYPE CHECK
TYPE SIM
TYPESIM WO MISSING
MISSING RATIO
DEPTH1
DEPTH2
DEPTH3
DEPTH4
DEPTH5
AGENTS
33
0.728
0.885
0.178
0.719
0.778
0.534
0.000
APPWORLD
N/A
0.721
0.910
0.208
0.798
0.503
0.755
0.572
0.500
AUTOGPT
61
0.788
0.825
0.046
0.781
0.800
0.827
0.764
0.656
BLAAKK
0.899
0.929
0.032
0.914
0.891
0.640
0.975
CAPA
133
0.817
0.869
0.059
0.875
0.740
0.614
0.773
COMPOSIO
95
0.717
0.766
0.064
0.718
0.721
0.664
0.903
DB-GPT
1705
0.816
0.895
0.088
0.828
0.797
0.831
0.761
0.839
DEEPFACE
0.829
0.951
0.128
0.807
0.907
0.818
0.656
0.500
EXO
41
0.739
0.898
0.178
0.802
0.699
0.859
0.734
FACESWAP
62
0.857
0.982
0.127
0.881
0.797
0.694
0.650
0.737
FASTAPI
591
0.433
0.772
0.439
0.424
0.495
0.339
0.073
0.680
FLAKE8
0.824
0.931
0.115
0.751
0.763
FLASK
0.856
0.927
0.077
0.829
0.639
GHUNT
0.743
0.846
0.122
0.742
0.678
0.981
GPT-PILOT
76
0.865
0.908
0.048
0.879
0.855
0.787
0.963
GPT4TREE
376
0.810
0.881
0.080
0.664
0.650
0.913
GPTME
23
0.895
0.926
0.033
0.930
0.984
GRADIO
1518
0.681
0.842
0.190
0.723
0.673
0.567
0.503
0.691
HAYSTACK
0.738
0.935
0,210
0.726
0.827
LEROBOT
N/A
0.796
0.916
0.131
0.821
0.795
0.892
LLAMA-FACTORY
29
0.844
0.907
0.069
0.813
0.815
1,000
MANIM
0.767
0.944
0.187
0.636
0.491
0.563
0.000
MLC-LLM
0.749
0.930
0.195
0.700
0.896
0.811
0.693
NICEGUI
0.907
0.078
0.851
0.801
0.710
0.643
OPENAI-PYTHON
381
0.665
0.822
0.190
0.716
0.668
OPENBB
0.883
0.929
0.050
0.912
0.789
0.913
0.973
PANDAS-AI
0.792
0.093
0.721
0.722
0.652
0.969
PAPER-QA
0.867
0.915
0.052
0.913
0.817
0.799
0.958
PDM
0.920
0.067
0.906
0.731
0.702
PHIDATA
1372
0.931
0.948
0.018
0.976
0.866
0.860
0.853
PILLOW
53
0.828
0.924
0.104
0.865
0.709
0.721
0.672
0.756
PIP
482
0.936
0.142
0.736
0.686
0.254
0.839
POETRY
24
0.966
0.094
0.770
0.670
0.825
PRE-COMMIT
35
0.897
0.908
0.012
0.911
0.855
0.954
PRE- TEE-COMMIT-HORE
0,912
0.923
0.012
0.898
0.910
0.977
PRIVATE-GPT
0.825
0.910
0.093
0.826
0.830
0.801
PYDANTIC
253
0.861
0.187
0.667
0.739
0.757
0.000
63
0.754
0.939
0.197
0.754
0.762
0.662
0.719
SCREENSHOT-TO-CODE
0.927
0.927
0.000
1.000
0.872
0.611
1.000
SPHINX
374
0.621
0.780
0.203
0.611
0.668
0.724
0.761
0.527
DETIIY-DDWNLLADER
35
0.865
0.921
0.060
0.854
0.882
0.916
0.950
0.882
0.934
0.056
0.886
0.769
0.993
SUPERVISION
137
0.826
0.967
0.146
0.866
0.762
0.822
0.807
0.301
492
0.775
0.879
0.119
0.799
0.722
0.784
0.815
0.625
-EN-AGENT
0.631
0.832
0.242
0.628
0.651
0.670
0.406
106
0.675
0.815
0.171
0.737
0.579
0.675
0.000
207
0.803
0.861
0.067
0.810
0.782
0.843
0.875
0.846
0.922
0.082
0.878
0.805
0.767
0.641
1.000
391
0.719
0.908
0.209
0.698
0.767
0.735
0.702
0.947
43
0.897
0.911
0.015
0.896
0.918
0.970
0.375
19

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Table 13. TYPECHECK and TYPESIM scores for DEEPSEEK-V2.5 each repository.
TYPESIM BY DEPTH
REPO
TYPE CHECK
TYPE SIM
TYPESIM WO MISSING
MISSING RATIO
DEPTH1
DEPTH2
DEPTH3
DEPTH4
DEPTH5
AGENTS
120
0.715
0.894
0.200
0.720
0.720
0.550
0.000
APPWORLD
N/A
0.542
0.918
0.409
0.608
0.385
0.438
0.440
0.000
AUTOGPT
161
0.764
0.796
0.040
0.763
0.774
0.735
0.791
0.754
BLAAKK
53
0.749
0.937
0.201
0.788
0.612
0.614
0.879
CAPA
382
0.823
0.886
0.071
0.841
0.750
0.798
COMPOSIO
32
0.671
0.757
0.114
0.683
0.659
0.569
0.813
DB-GPT
2153
0.776
0.899
0.137
0.778
0.774
0.768
0.730
0.837
DEEPFACE
15
0.932
0.965
0.034
0.925
0.972
0.906
0.741
1.000
EXO
90
0.811
0.935
0.133
0.816
0.814
0.649
0.833
0.710
FACESWAP
111 0.802
0.969
0.172
0.838
0,716
0.561
0.467
0.737
FASTAPI
462
0,533
0.961
0.445
0.498
0.650
0.336
0.073
0.680
FLAKE8
20 0.832
0.950
0.125
0.748
0.749
1.000
FLASK
0.799 28
0.940
0.150
0.843
0.735
0.545
0.257
GHUNT
47
0.766
0.880
0.129
0.742
0.751
0.974
GPT-PILOT
129
0.871
0.911
0.045
0.894
0.856
0.754
0.947
GPT4TREE
0.759
0.898
0.155
0.787
0.654
0.649
0.903
GPTME
0.898
0.924
0.027
0.900
0.883
0.984
GRADIO
0.574
0.855
0.328
0.603
0.574
0.448
0.481
0.772
HAYSTACK
477
0.708
0.949
0.254
0.700
0.708
0.736
0.875
LEROBOT
N/A
0.682
0.925
0.262
0.729
0.757
0.892
LLAMA-FACTORY
0.803
0.927
0.133
0.748
0.811
0.987
MANIM
N/A
0.635
0.870
0.270
0.481
0.398
0.493
0.000
MLC-LLM
0.951
0.396
0.537
0.696
0.574
0.601
NICEGUI
0.801
0.901
0.111
0.822
0.767
0.623
OPENAI-PYTHON
0.819
0,302
0,573
0.606
OPENBB
193
0.790
0.927
0.148
0.778
0.707
0.758
PANDAS-AI
0.817
0.087
0.737
0.789
0.658
1,000
PAPER-QA
0.885
0.275
0.653
0.631
0.634
0.506
0.943
PDM
0.865
0.116
0.638
0.592
PHIDATA
0.883
0.939
0.059
0.810
0.443
PILLOW
201
0.857
0.265
0.666
0.530
0.471
0.387
0.484
PIP
694
0.925
0.148
0.724
0.664
0.253
0.741
POETRY
0.876
0.970
0.097
0.773
0.682
0.463
0.958
PRE-COMMIT
0.943
0.953
0.011
0.887
0.960
PRE- SEL-COMMIT-HOOE
7
0.952
0.960
0.008
0.983
0.907
0.871
PRIVATE-GPT
0.923
0.186
0.764
0.736
0.686
PYDANTIC
751
0.667
0.912
0.268
0.562
0.631
0.431
0.000
291
0.618
0.934
0.339
0.631
0.563
0.659
0.719
SCREENSHOT-TO-CODE
0.887
0.887
0.000
0.980
0.813
0.611
0.833
SPHINX
666
0.498
0.840
0.407
0.476
0.625
0.610
0,513
0.590
DETIIY-DDWNLLADER
0.910
0.053
0.856
0.858
0.973
0.900
0.910
0.980
0.071
0.916
0.782
1.000
SUPERVISION
0.803
0.969
0.171
0.847
0.747
0.756
0.762
0.315
773
0.790
0.897
0.120
0.810
0.750
0.775
0.783
0.875
-EN-AGENT
0,590
0.869
0.322
0.584
0.621
0.631
0.469
0.676
0.903
0.252
0.736
0.581
0.678
0.000
84
0.812
0.872
0.068
0.823
0.793
0.788
0.738
0.848
0.933
0.091
0.896
0.776
0.790
0.641
1,000
1372
0.661
0.874
0.244
0.660
0.669
0.646
0.573
0.576
63
0.898
0.914
0.018
0.895
0.940
0.976
0.375
20

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Table 14. TYPECHECK and TYPESIM scores for GROK-2 2 each repository.
TYPESIM BY DEPTH
REPO
TYPE CHECK
TYPE SIM
TYPESIM WO MISSING
MISSING RATIO
DEPTHl
DEPTH2
DEPTH3
DEPTH4
DEPTH5
AGENTS
37
0.700
0.900
0.222
0.695
0.742
0.493
0.000
APPWORLD
N/A
0.535
0.931
0.426
0.596
0.395
0.413
0.491
0.000
AUTOGPT
81
0.755
0.803
0.060
0.756
0.752
0.762
0.773
0.750
BLAAKK
0.825
0.925
0.107
0.847
0.807
0.489
0.958
CAPA
89
0.905
0.924
0.021
0.936
0.794
0.851
0.852
COMPOSIO
35
0,482
0.593
0.188
0.454
0,563
0.445
0.897
DB-GPT
2066
0.820
0.898
0.086
0.834
0.805
0.789
0.764
0.841
DEEPFACE
0.868
0.961
0.096
0.915
0.808
0.990
1.000
EXO
49
0.778
0.928
0.161
0.763
0.838
0.696
0.875
0.750
FACESWAP
0.885
0.973
0.091
0.904
0.833
0.784
0.817
0.737
FASTAPI
707
0.659
0.940
0.299
0.751
0.646
0.468
0.577
0.979
FLAKE8
0.917
0.938
0.023
0.930
0.884
0.903
1.000
FLASK
22
0.900
0.935
0.037
0.933
0.847
0.692
0.861
GHUNT
0.751
0.894
0.160
0.732
0.735
0.921
GPT-PILOT
20
0.839
0.891
0.059
0.843
0.841
0.777
0.948
GPT4TREE
0.839
0.883
0.050
0.690
0.705
0.917
GPTME
22 0.908
0.937
0.031
0.907
0.912
0.894
0.984
GRADIO
328
0.695
0.861
0.192
0.739
0.677
0.612
0.569
0.630
HAYSTACK
0.933
0.199
0.773
0.837
LEROBOT
N/A
0,851
0.922
0.077
0.764
0.892
LLAMA-FACTORY
48
0.837
0.889
0.059
0.785
0.797
0.987
MANIM
N/A
0.650
0.872
0.255
0.667
0.619
0.461
0.641
0.000
MLC-LLM
32
0.946
0.224
0.703
0.819
0.837
0.905
NICEGUI
0.871
0.114
0.765
0.815
0.576
OPENAI-PYTHON
208
0.814
0.149
0.691
0.776
OPENBB
0.864
0.928
0.070
0.875
0.764
0.738
0.967
PANDAS-AI
0.817
0.093
0.745
0.759
0.642
0.833
1,000
PAPER-QA
0.857
0.903
0.051
0.808
0.791
0.838
PDM
0.920
0.055
0.741
0.701
0.688
PHIDATA
1234
0.899
0.943
0.047
0.945
0.844
0.798
0.443
PILLOW
0.818
0.923
0.114
0.863
0.688
0.629
0,533
0.864
PIP
240
0.814
0.936
0.130
0.765
0.704
0.848
POETRY
0.867
0.961
0.098
0.752
0.652
0.519
0.825
PRE-COMMIT
0.886
0.946
0.063
0.895
0.858
0.966
PRE- DEE-COMMIT-HOON
0.905
0.000
0.888
0.924
0.952
PRIVATE-GPT
0.645
0.939
0.313
0.609
0.712
0.724
PYDANTIC
564
0.920
0.113
0.811
0.827
0.837
0.551
0.500
0.628
0.856
0.266
0.628
0.630
0.601
0.719
SCREENSHOT-TO-CODE
0.936
0.936
0.000
0.990
0.904
0.616
0.833
SPHINX
243
0.638
0.803
0.205
0.633
0.679
0.639
0.531
0.594
DETIIY-DDWNLLADER
21
0.868
0.920
0.057
0.864
0.867
0.969
0.788
42
0.954
0.979
0.025
0.967
0.769
0.993
SUPERVISION
24 0.852
0.967
0.119
0.904
0.795
0.779
0.708
0.301
528
0.737
0.875
0.158
0.763
0.687
0.712
0.797
0.875
-EN-AGENT
0.468
0.912
0.487
0.475
0.425
0.561
0.323
175
0.690
0,923
0.252
0.744
0.608
0.689
0.000
33
0.809
0.877
0.078
0.818
0.791
0.807
0.738
0.854
0.910
0.062
0.902
0.774
0.862
0.608
1,000
696
0.732
0.869
0.158
0.717
0.754
0.809
0.731
0.972
56
0.860
0.913
0.057
0.871
0.600
0.976
1.000
21

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Table 15. TYPECHECK and TYPESIM scores for I.L.MMA3-88 each repository.
TYPESIM BY DEPTH
REPO
TYPE CHECK
TYPE SIM
TYPESIM WO MISSING
MISSING RATIO
DEPTH1
DEPTH2
DEPTH3
DEPTH4
DEPTH5
AGENTS
119
0.415
0.809
0.487
0.426
0.408
0.154
0.000
APPWORLD
N/A
0.177
0.901
0.804
0.194
0.131
0.176
0.168
0.000
AUTOGPT
82
0.451
0.749
0.398
0.500
0.370
0.180
0.227
0.000
BLAAKK
30
0.186
0.805
0.769
0.201
0.133
0.142
0.388
CAPA
161
0.235
0.606
0.612
0.246
0.147
0.258
0.219
COMPOSIO
115
0.382
0.573
0.333
0.458
0.218
0.198
0.068
DB-GPT
1032
0.317
0.604
0.476
0.381
0.228
0.266
0.289
0.216
DEEPFACE
0.231
0.630
0.634
0.298
0.141
0.107
0.000
EXO
0.377
0.782
0.519
0.390
0.343
0.341
0.208
0.458
FACESWAP
42 0.326
0.867
0.624
0.374
0,180
0.092
0.250
0.323
FASTAPI
1792
0.054
0.754
0.928
0.089
0.032
0.062
0.000
0.000
FLAKE8
28
0.573
0.815
0.297
0.649
0.441
0.181
0.250
FLASK
0.346
0.764
0.547
0.392
0.242
0.259
0.257
GHUNT
21 0.481
0.849
0.434
0.543
0.338
0.690
0.005
GPT-PILOT
68
0.574
0.768
0.253
0.620
0.523
0.438
0.828
GPT4TREE
0.543
0.809
0.329
0.418
0.343
GPTME
0.567
0.827
0.315
0.491
0.196
0.000
GRADIO
772
0.227
0.576
0.605
0.326
0.169
0.084
0.150
HAYSTACK
0.308
0.558
0.448
0.114
0.135
LEROBOT
N/A
0.216
0.754
0.714
0.309
0.096
0.000
LLAMA-FACTORY
0.366
0.730
0.498
0.401
0.325
0.225
0.828
MANIM
0.299
0.697
0.570
0.335
0.198
0.103
0.445
0.000
MLC-LLM
0.738
0.670
0.260
0.200
0.180
0.000
NICEGUI
0.514
0.811
0.367
0.527
0.506
0.269
0.100
OPENAI-PYTHON
296
0.317
0.668
0,525
0,270
0.213
OPENBB
102
0.298
0.589
0.495
0.473
0,237
0.129
0.194
0.350
PANDAS-AI
0.585
0.196
0.558
0.396
0.205
0.250
0.000
PAPER-QA
0.715
0.601
0.352
0.201
0.255
0.000
PDM
0.781
0.472
0.246
0.187
PHIDATA
0.629
0.296
0,236
0.117
0.000
PILLOW
0.382
0.885
0.569
0.427
0.254
0.181
0.142
0.053
PIP
253
0.861
0.489
0,356
0.245
0.058
0.134
POETRY
0.600
0.873
0.312
0.662
0.433
0.224
0.492
PRE-COMMIT
65
0.584
0.851
0.314
0.634
0.447
0.761
PRE- TEE-COMMIT-HORE
0.649
0.887
0.269
0.657
0.737
0.500
PRIVATE-GPT
0.410
0.814
0.497
0.386
0.440
0.513
PYDANTIC
223
0.652
0.826
0.051
0.091
0.000
49
0.170
0.664
0.744
0.191
0.091
0.173
0.000
SCREENSHOT-TO-CODE
0.532
0.952
0.441
0.633
0.456
0.306
0.000
SPHINX
229
0.097
0.620
0.844
0.097
0.096
0.083
0.103
0.207
DETIIY-DDWNLLADER
0.395
0.631
0.374
0.255
0.294
0.654
237
0.079
0.232
0.658
0.077
0.111
0.059
SUPERVISION
48
0.203
0.644
0.685
0.260
0.114
0.189
0.142
0.000
144
0.357
0.589
0.394
0.501
0.129
0.088
0.029
0.188
-EN-AGENT
0.393
0.827
0.525
0.412
0.307
0.287
72
0.300
0.917
0.673
0.314
0.265
0.369
0.000
0.573
0.759
0.245
0.638
0.435
0.518
0.763
27 0.462
0.837
0.448
0.537
0.362
0.281
0.079
0.000
413
0.185
0.531
0.651
0.214
0.132
0.112
0.072
0.169
37
0.590
0.772
0.235
0.652
0.117
0.079
0.000
22

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Table 16. TYPECHECK and TYPESIM for I.LAMA-3. 1-88 each repository.
TYPESIM BY DEPTH
REPO
TYPE CHECK
TYPE SIM
TYPESIM WO MISSING
MISSING RATIO
DEPTH1
DEPTH2
DEPTH3
DEPTH4
DEPTH5
AGENTS
153
0.636
0.809
0.214
0.634
0.669
0.410
0.000
APPWORLD
N/A
0.347
0.789
0.560
0.381
0.258
0.326
0.392
0.000
AUTOGPT
273
0.676
0.783
0.137
0.698
0.629
0.604
0.679
0.500
BLAAKK
189
0.518
0.782
0.338
0.554
0.398
0.345
0.763
CAPA
497
0.574
0.816
0.297
0.588
0.575
0.498
0.906
COMPOSIO
283
0.580
0.693
0.163
0.626
0.495
0.377
0.847
DB-GPT
2591
0.642
0.769
0.165
0.685
0,586
0.595
0.571
0.605
DEEPFACE
38
0.716
0.923
0.224
0.753
0.680
0.612
0.616
1.000
EXO
0.734
0.938
0.218
0.708
0.812
0.732
0.943
0.714
FACESWAP
199
0.710
0.908
0.219
0.745
0.629
0.427
0.533
0.722
FASTAPI
2260
0.275
0.573
0.519
0.231
0,359
0.175
0.008
0.000
FLAKE8
0.757
0.882
0.142
0.653
0.750
FLASK
85
0.639
0.772
0.173
0.687
0.556
0.363
GHUNT
221
0.755
0.863
0.126
0.749
0.717
0.964
0.672
GPT-PILOT
398
0.799
0.854
0.065
0.814
0.789
0.706
0.951
GPT4TREE
0.747
0.885
0.156
0.782
0.621
0.957
GPTME
0.586
0.850
0.311
0,531
0.504
0.984
GRADIO
2414
0.456
0.593
0.231
0.552
0.404
0.299
0.249
0.296
HAYSTACK
0.243
0.721
0.663
0.212
0.183
LEROBOT
N/A
0.830
0.352
0.337
0.446
0.250
LLAMA-FACTORY
0.794
0.209
0,597
0.574
0.389
0.961
MANIM
0.681
0.833
0.183
0.738
0.524
0.360
0.000
MLC-LLM
208
0.569
0.856
0.336
0.543
0.661
0.553
0.423
NICEGUI
0.824
0.297
0.573
0.612
0.557
0.507
OPENAI-PYTHON
906
0.539
0.702
0.233
0,504
0.405
OPENBB
237
0.599
0.839
0.286
0.607
0.607
0.479
0.339
0.200
PANDAS-AI
431
0.779
0.141
0.673
0.723
0.507
0.601
0.875
PAPER-QA
0.742
0.198
0.486
0.592
0.948
PDM
0.601
0.800
0.249
0.427
0.306
PHIDATA
1412
0.751
0.868
0.135
0.693
0.656
0.136
PILLOW
0.784
0.369
0.550
0.329
0.275
0.294
0.296
PIP
1161
0.831
0.285
0.631
0.505
0.475
0.201
0.272
POETRY
241
0.922
0.182
0.633
0.499
0.371
0.958
PRE-COMMIT
46
0.813
0.878
0.074
0.895
0.640
0.535
PRE- DEE-COMMIT-HOON
0.827
0.885
0.066
0.884
0.800
0.605
PRIVATE-GPT
0.750
0.809
0.073
0.852
0,559
0.542
PYDANTIC
786
0.565
0.486
0.358
0.142
0.198
0.111
0.000
309
0.458
0.705
0.350
0.464
0.433
0.521
0.250
SCREENSHOT-TO-CODE
7
0.912
0.930
0.020
0.960
0.883
0.616
0.833
SPHINX
0.022
0.537
0,959
0.021
0,030
0.016
0.029
0.000
DETIIY-DDWNLLADER
0.722
0.891
0.189
0.714
0.745
0.712
327
0.593
0.885
0.330
0.569
0.722
0.980
SUPERVISION
306
0.508
0.875
0.420
0.465
0.562
0.559
0.653
0.000
834
0.571
0.769
0.256
0.623
0.495
0.405
0.663
0.438
-EN-AGENT
0.655
0.778
0.158
0.652
0.690
0.551
458
0.508
0.889
0.429
0.555
0.446
0.460
0.000
263
0.709
0.821
0.137
0.743
0.647
0.633
0.650
129
0.702
0.812
0.136
0.761
0.621
0.535
0.515
1.000
858
0.326
0.634
0.485
0.337
0.313
0.273
0.186
0.175
150
0.827
0.924
0.105
0.828
0.788
0.939
0.375
23

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
Table 17. TYPECHECK and TYPESIM for QWEN-2.5-7P on each repository.
TYPESIM BY DEPTH
REPO
TYPE CHECK
TYPE SIM
TYPESIM WO MISSING
MISSING RATIO
DEPTH1
DEPTH2
DEPTH3
DEPTH4
DEPTH5
AGENTS
158
0.695
0.865
0.196
0.706
0.683
0.507
0.000
APPWORLD
N/A
0.462
0.921
0.498
0.502
0,380
0.350
0.368
0.000
AUTOGPT
252
0.680
0.784
0.132
0.703
0.633
0.596
0.626
0.750
BLAAKK
101
0.491
0.651
0.247
0.543
0.292
0.411
0.325
CAPA
511
0.576
0.757
0.240
0.589
0.402
0.656
0.793
COMPOSIO
108
0.546
0.655
0.167
0.639
0,358
0.211
0.636
DB-GPT
2245
0.596
0.693
0.141
0.702
0.460
0.440
0,508
0.481
DEEPFACE
0.492
0.840
0.415
0.582
0.214
0.323
0.000
EXO
176
0.682
0.881
0.226
0.669
0.713
0.747
0.708
0.714
FACESWAP
89
0.643
0.844
0.238
0.748
0,329
0.151
0.229
0.380
FASTAPI
863
0.191
0.484
0.605
0.315
0.118
0.155
0.065
0.690
FLAKE8
35
0.662
0.887
0.253
0.715
0,546
0.571
0.750
FLASK
0.715
0.836
0.145
0.797
0.560
0.400
0.167
GHUNT
67 0.696
0.851
0.183
0.701
0.613
0.959
0.998
GPT-PILOT
489
0.776
0.872
0.110
0.778
0.805
0.623
0.739
GPT4TREE
0.795
0.900
0.116
0.702
0.701
GPTME
0.652
0.847
0.231
0,583
0.529
1.000
GRADIO
2008
0.459
0.613
0.251
0.580
0,394
0.248
0.260
0.177
HAYSTACK
0.326
0.528
0.383
0.493
0.136
0.078
0.296
LEROBOT
N/A
0.752
0.417
0,267
0.283
0.000
LLAMA-FACTORY
251
0.608
0.789
0.230
0,555
0.537
0.000
MANIM
0.732
0.877
0.166
0.522
0.319
0.778
0.000
MLC-LLM
0.527
0.777
0.322
0.585
0,388
0.232
0.358
0.137
NICEGUI
640
0.790
0.882
0.105
0.733
0.644
0.325
OPENAI-PYTHON
0.654
0.168
0.462
0.319
0.187
OPENBB
294
0.409
0.453
0.097
0.776
0,260
0.352
0.359
0.346
PANDAS-AI
462
0.793
0.090
0.720
0.791
0.518
0.719
1,000
PAPER-QA
0.776
0.249
0.597
0.490
0.833
PDM
0.891
0.188
0,580
0.566
0.765
PHIDATA
0.804
0.894
0.100
0.856
0.741
0.671
0.361
PILLOW
0,574
0.841
0.318
0.634
0.402
0.300
0.390
0.417
PIP
0.717
0.902
0.206
0.761
0.615
0.550
0.166
0.804
POETRY
132
0.854
0.932
0.084
0.903
0.708
0.619
0.676
0.958
PRE-COMMIT
0.916
0.068
0.828
0.752
PRE- SEL-COMMIT-HOOE
0.958
0.039
0.927
0.890
0.934
PRIVATE-GPT
0.873
0.444
0.459
0.534
0.546
PYDANTIC
0.789
0.514
0.445
0.269
0.201
0.000
255
0.579
0.834
0.305
0.587
0.545
0.667
0.750
SCREENSHOT-TO-CODE
0.756
0.918
0.177
0.837
0.683
0.616
0.833
SPHINX
421
0.373
0.669
0.443
0.377
0.373
0.255
0.274
0.393
DETIIY-DDWNLLADER
31
0.637
0.327
0.497
0.286
0.386
0.167
195
0.287
0.426
0.326
0.246
0.528
0.882
SUPERVISION
288
0.554
0.827
0.331
0.645
0.392
0.562
0.612
0.315
853
0.487
0.704
0.308
0.600
0.328
0.225
0.046
0.000
-EN-AGENT
0.686
0.808
0.152
0.683
0.726
0.532
0.472
328
0.472
0.891
0.470
0.493
0.454
0.417
0.000
216
0.796
0.859
0.073
0.812
0.767
0.779
0.650
0.662
0.794
0.167
0.755
0.524
0.522
0.231
1,000
1631
0.513
0.673
0.238
0.569
0.400
0.389
0.299
0.750
82
0.816
0.841
0.030
0.857
0.808
0.140
0.000
24

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
D.3. Type Inference Examples
We code snippet from the repo DB-GPT demonstrate the type inference accuracy of each LLM. We have conducted necessary code formatting for better visualization.
SOURCE CODE
class RerankEmbeddi (ABC) @abstractmethod
def predict (seif, query: ste, cand Taat (stz)) Tiat (1loa/flot]]
anyze def spredicttself, quezy: str, candidates: List (stx)) -> List [float]:
8
class Embeddinge
11
@abetractmethod
embed
texts: (str() List [uist [float]]:
13
15
Babetraclmethod
def embed_ query (selE, text: stz) -> 1ist [float]) :
18
asyze sembed_doumentt (self, texts: [ste]] List (List [float]]:
20
22
def qenbyd_query (self, text: str) >> List :
23
GPT-40 (SCORE: 0.874)
class Rerankkmbeddings (ABC): gabstractmethod
def predict (self, query: Any, candidates: List [Any]) > List (fleat];
asyre def apredic: (self, query: Any, candidates: M4st (Any]) +> List (float:):
8
10
class Enbeddings (AB2):
11
@abstractmethod
def documenn (salf, texts: list [str]) List [List [float]]
13
15
Rabstractmathod
ambod_query (self, (solf,
sext: ste) [float]:
17
18
asyre daf aambed documents (self, texts: Mist [ste]] Maat (Tdat [float]]:
20
22
asyze def aembed (self, tex:: str) >> Jist [float]:
25

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
GPT-4O-MINI (SCORE: 0.886)
class TerankEmbedding (ABC)
Babetractmethod
def predict (self, query: Any, candidates: List (Any]) List (float):
asyze def apredict (self, query: Any, condidates: List (Any]) +> List [float]:
class Embeddings (ABC)
@sbstractmethod
11
def mmbed_document (self, texte: List (etr]) -> List [List [float]]:
13
Babetractmethod
15
def embed_quezy (self, sext: stz) List [float];
16
18
a5yze def
(self, texts: List [ste]] -> [Lsst [float]]:
20
21
asyne def query (self, text: stx) b1st [float]:
22
CLAUDE-3.5-SONNE (SCORE: 0.870)
class Rer Embediinng (ABC) Rabetractmethod
def predict (self, query: Any, candidates: List [Anyl) List (float):
asyze def apredic: (self, query: Any, candidates: List (Any]) +> List (float]:
class Embeddings (ABC)
@abetractmethod
11
def umbed_docuretd (self, texts: List (etr]) -> List [List [float]]:
13
Babetractmethod
15
def embed_quezy (self, stz) List [float]:
16
18
asyze def sembed_documente (self, texts: List [ste]] -> List (List [float]]:
20
21
asyne def aembed (self, text: str) bist [float]:
22
DEEPSEEK-V3 (SCORB: 0.898)
class Rereannnbeeduing (ABC)
Rabetractmethod
def pradict (self, query: str, candidates: List (stz)) List (float))
asyre def apredic: (self, query: str, candidates: List (stx)) +> Awaitable (1ist [float]]:
class Embaddings (ABE)
11
Rabstractmathod
embed ambod_docute isolf, texts: list (str)) List (List [float]]
13
15
Rabstractmethod
16
def qmery_ sext: str) >> List (float]:
18
asyre
(self, texts: [ste]] Awaitable at (tist [#loat]]]
20
21
22
asyze def sembed_query(self, >> Avsitable[List [float]]: (1loot))))
26

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
DEEPSEEK-VE 5 (SCORE: 0.899)
class RerankEmbeddiing (ABC) Babetractmethod
def predict (self, query: candidates: List (str]) List (float):
anyze def apredict (self, query: str, candidates: List (stx]) +> List [float]:
class ambaddings (AB2)
11
Rabstractmethod
embed iself, texts: dist(str) List [uist [float]]
13
15
Rabetractmethod
16
def query text: str) -> List [float];
18
asyze bembed_documente (self, texts: [ste]] (List :
20
21
22
asyne def sembei_querr(eeef. text: -> List [float]:
GROK-2 (SCORE: 0.898)
class RerankLmbedddings (ABC):
@abstractmethod
def predict (self, quezy: str, candidates: [ist(str]) >> [iscat))t)):
apredict (self, query: sts, candidatess 51st [ste]) -> asyncio. Future (List [float]]
class Embaddings (ABE)
10
Rabstractmethod
embed_ documenss iself, texts: List (str)) List [List [float]]
12
14
Rabetractmethod
def query (self, sext: str) >> List [float]:
17
acmbed doc.uontoustslllooooo texts: (stx)) -> sayncio. (List (List [float]]] ]]]
19
21
Puture def aembed query (self, text: stx) asyncio. Future [List [float]]
LLAMA-3-8B (SCORE: 0.604)
class RerankEmbeddingg (ABC):
Eabstractmethod
def predict (self, quezy: object, candidates: List [object]) >> object:
asyne def apredics tself, query: object, candidates: Llat [object]] Awaitable (object]]]
class Embaddings (ABE)
10
Rabstractmathod
ambod_ documense isolf, texts: (object)) (object:)
12
14
Rabstractmethod
def emery sext: object] >> object:
17
asyre
(self, texts: [object)) -> Awaitable st [object]]
19
21
asyze def sembei_queryyseef,, tex:: object) waitable(obbeett)
27

---

TYPYBENCH: Evaluating LLM Type Inference for Untyped Python Repositories
LLAMA-3.1-8 (SCORE: 0.769)
class ngg (ABC)
Babetractmethod
def predict (self, query: Any, candidates: List (Any]) Any
asyze def apredict (salf, query: Any, condidates: List (Any]) => Awbitable [Any]
class Embeddinge (ABC)
@sbstractmethod
11
def embed (self, texte: List [Any]) ) -> List [Anyl:
13
gabstractmethod
15
def embed_quezy (self, sext: Any) >> Aay:
18
asyze def sembed_documente (self, texts: List [Any]] -> Awaitable [List [Any]
20
21
asyno def aembed (self, text: Any) Awaltable [Any]:
22
QWEN-2.5-7B (SCORE: 0.693)
class RorankEmbeddi (ARC):
#abstractmethod
def pradict (self, query: str, condidates: List (str)) List (fleat):
asyzc def apredicz (self, quezy: str, candidates: (atel) List [fleat]:
class Embeddings (ABC)
@abetractmethod
12
def embee_ddcuren::: (self, texte: List [ete]) >> List [List [float]]:
14
#abetractmethod
16
def embed_quezy (selE, sext: stz) List [float];
17
19
asyzc def sembed_documente (self, texts: List [ste]] >> List [fllo]]]
21
asyno def query (self, text: str) b1st [float]:
23
28


Reviews:
[{"review_id": "3oZ6Q6lyPS", "reviewer": "Reviewer_SnQQ", "summary": "Summary: The paper introduces TYPYBENCH, a benchmark designed to evaluate the type inference capabilities of large language models (LLMs) across entire Python repositories. The benchmark features two novel metrics: TYPESIM, which measures the semantic similarity between predicted and ground truth types, and TYPECHECK, which evaluates type consistency across codebases. The authors evaluate various LLMs on a dataset of 50 high-quality Python repositories and find that while LLMs achieve reasonable TYPESIM scores, they struggle with complex nested types and exhibit significant type consistency errors. The findings suggest that future research should prioritize improving repository-level type consistency over type similarity. TYPYBENCH provides a foundation for this new research direction, offering insights into model performance across different type complexities and usage contexts.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and suggestions. We would like to clarify the metrics design and address your remaining concerns with the following responses. We will improve the manuscript accordingly to address them.\n\n> The motivation behind this work is worth discussing. … Why do we need evaluation metrics that are specifically designed for LLMs?\n\n[Q1] First, we want to clarify that our metrics are not only designed for LLMs but all predictions made by any type inference method (including the program-based ones) can be evaluated using these metrics. These metrics are calculated over a set of type predictions.\n\n**For TypeSim**, as stated in the Introduction (L38), exact matching fails to capture important semantic relationships between types.\n\n[Q3] For *match up to parametric type*, it is a special case of our TypeSim metric as formulated in L187:\n\n$S(T, T')=\\alpha~s(root, root')+(1-\\alpha)S_{list}(args(T),args(T'))$\n\nwhere in TypeSim $\\alpha=0.5$ and $s$ is Jaccard on attributes, and in *match up to parametric type* $\\alpha=1$ and $s$ is exact matching (equivalent to taking floor operation on the result of Jaccard Index). Therefore, TypeSim is a more general similarity metric than *exact match* and *match up to parametric type*.\n\nFor example, for a target type `list[str]` and the prediction is `list`, exact matching gives a score of 0 would be too harsh, and *match up to parametric type* gives a score of 1 is too benign. TypeSim gives a score of $\\alpha=0.5$ reflects the similarity between two types.\n\n[Q2] The main goal of the TypeSim metric is not limited to detecting the types that could be used interchangeably (which are illustrative examples) but also providing a better measurement for the semantic similarity between complex types by considering their functional attributes (Section 4.1.1 L209-219) and the structure (Section 4.1.2) within the types.\n\nAs shown in the experiments (Section 6.3.1), TypeSim provides a more nuanced semantic-based evaluation compared with exact matching, especially for more nested types (e.g., depth > 3).\n\n|Original|Predicted |TypeSim|\n|-|-|-|\n|`list[Any] \\| None`|`list[str] \\| None`|0.75|\n|`dict[Any, Any] \\| None`|`dict[str, Any] \\| None`|0.875|\n|`dict[str, tuple[Any, ...]] \\| None`|`dict[str, tuple[int, ...]] \\| None`| 0.9375|\n|`dict[str, list[int]]`|`dict[str, Union[tuple[int, ...], Any]]`|0.8385|\n|`float \\| np.ndarray[Any, Any]`|`np.ndarray[Any, Any]`|0.5|\n|`pathlib.Path \\| None`|`str \\| pathlib.Path \\| None`| 0.6667|\n\nAbove is a list of similar type pairs. It can be seen that the TypeSim score reasonably exhibits the similarity between the two types.\n\n**For TypeCheck**, though Mypy is not a perfect type checker and could produce false positives or incorrect error reports, it is currently a well-adopted best practice for software engineering. Repositories passing Mypy check (or having fewer Mypy errors) tend to have fewer bugs in practice, so we introduce this as a good starting point for evaluating type inference results. We will discuss this limitation and leave it as future work to find a better checking mechanism.\n\nWe hope these clarifications resolve your questions. Please let us know if you have further questions.\n\n> The evaluation lacks comparisons with more diverse models such as GPT-4, CodeLlama, StarCoder, CodeT5, and its variants.\n\nWe did not evaluate these LLMs since they are already outdated models compared with the SOTA LLMs we tested (e.g., GPT-4o, Claude-3.5-sonnet). Moreover, SOTA general-purpose models (e.g., GPT-4o) have already surpassed code-focused models in coding tasks. Nonetheless, we tested CodeLlama on the test sets as suggested, which gives similar results as Llama-3-8B:\n\n|Model|TypeCheck|TypeSim|TypeSim wo missing|Missing rate|\n|-|-|-|-|-|\n|Llama-3-8b|44.0|0.396|0.747|0.470|\n|CodeLlama|25.0*|0.3558|0.735|0.516|\n\n*: only count on private-gpt\n\n**Other questions**,\n\n> [Q4] Is it reasonable to evaluate the type inference capabilities of LLMs using such simple, naive calculations? Are there more suitable evaluation methods that better align with the way LLMs actually reason about code?\n\nWe’d like to clarify that our proposed metrics — TypeSim and TypeCheck — are not designed to mimic or replicate how LLMs internally reason about code or types. Instead, they are metrics measuring the semantic quality and repository-wide consistency of predicted types (regardless of how those predictions are produced, by LLMs or symbolic tools). TypeSim provides an interpretable view of type prediction quality, especially when predictions are partially correct. Its formulation using Jaccard similarity and structural decomposition is a pragmatic and interpretable approach that aligns with how types are composed and used in practice.\n\n> I could not find any links to the open-source code or dataset in this paper.\n\nWe will open-source the code and dataset when published.", "claims_and_evidence": "This paper presents clear and compelling evidence. For instance, Section 2.1 cites relevant references to conventional methods, learning-based methods, and LLM-based methods.", "methods_and_evaluation_criteria": "As far as I know, there are currently no approaches that rely entirely on large language models for type inference, primarily due to concerns around soundness. Given this, why do we need evaluation metrics that are specifically designed for LLMs?\nFor TypeSim, which aims to evaluate the semantic similarity between predicted and ground-truth types — how frequently does such semantic similarity actually occur at the function level in real-world code? In other words, what proportion of function-level type annotations can be considered semantically interchangeable?\nFor TypeCheck, which evaluates consistency of types within a codebase — what is the proportion of types in real-world repositories that exhibit this kind of internal consistency? Without quantitative insights into these questions, it's hard to assess how broadly useful or necessary these two criteria are.", "theoretical_claims": "The paper does not provide specific theoretical proofs, as it is not a theoretical paper.", "experimental_designs_or_analyses": "The experimental design appears reasonable, as it includes both the Main Evaluation Analysis and the Factors Analysis.", "relation_to_broader_scientific_literature": "This paper proposes two metrics for LLM-based type inference: TypeSim and TypeCheck.", "essential_references_not_discussed": "The related works are sufficiently discussed in this paper.", "other_strengths_and_weaknesses": "Strengths:\n+ The dataset curation in this paper is a valuable contribution. Overall, it is necessary and well-justified, especially considering that the widely-used dataset currently available is ManyTypes4Py.\n\n+ Clear structure. The structure of this paper is clear.\n\nWeaknesses:\n- The motivation behind this work is worth discussing. As far as I know, there already exist a wide range of metrics for evaluating type inference, such as exact match, match to parameter, and others as mentioned in your paper. So why do we need a dedicated metric specifically for LLMs? This point is not clearly addressed in the introduction. Moreover, you mention in the introduction that types like List and Sequence can be used interchangeably. However, this comes across as a special-case observation rather than a generalizable insight. Have you conducted any statistical analysis to quantify how often such interchangeable usage occurs in real-world codebases? Moreover, can other type pairs like str and bool also be used interchangeably? If so, how frequent are such cases? \n\nIn addition, match to parameter seems to perform better compared to the proposed TypeSim, as it can match the outer type structure more effectively.Can I then understand TypeSim as a suboptimal alternative to match to parameter, rather than a fundamentally better metric?\n\n-The methodology behind the proposed criteria seems overly simplistic. For TypeSim, the base type similarity is computed using the Jaccard index, followed by a basic additive scheme to estimate structural similarity.But is this really how large language models perform type inference — through such naive calculations? This method appears to have little to do with the actual mechanisms or capabilities of LLMs, and does not take into account any of the unique characteristics or behaviors of language models.\nThis paper uses the number of mypy errors as a measure of Type Consistency, but this approach is rather naive. mypy itself can produce false positives or incorrect error reports, meaning that the evaluation may not accurately reflect true inconsistencies. \n\n\n- As a benchmark metric, the current selection of foundation models is insufficient. Notably, the evaluation lacks comparisons with more diverse models such as GPT-4, CodeLlama, StarCoder, CodeT5, and its variants. Including these models would provide a more comprehensive and convincing assessment of the proposed metrics.\n\n-I could not find any links to the open-source code or dataset in this paper. Providing access to the implementation and data is essential for reproducibility and for allowing others to validate or build upon this work.", "other_comments_or_suggestions": "None", "questions_for_authors": "1.why do we need a dedicated metric specifically for LLMs?\n2.Have you conducted any statistical analysis to quantify how often such interchangeable usage occurs in real-world codebases?\n3.Can I then understand TypeSim as a suboptimal alternative to match to parameter, rather than a fundamentally better metric?\n4.Is it reasonable to evaluate the type inference capabilities of LLMs using such simple, naive calculations? Are there more suitable evaluation methods that better align with the way LLMs actually reason about code?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jjqc3gIvWt", "reviewer": "Reviewer_zv9m", "summary": "The paper introduces TypyBench, a benchmark aimed at evaluating large language models (LLMs) on their ability to perform type inference for Python code. Recognizing limitations in existing benchmarks and exact-matching metrics, the authors propose two novel evaluation measures: TypeSim, which assesses semantic similarity between predicted and ground truth types, and TypeCheck, which evaluates type consistency across repositories using static type checking. Through experiments on a curated dataset of 50 Python repositories, the study reveals that while state-of-the-art LLMs can achieve strong semantic similarity scores, they struggle with maintaining consistent type annotations at a repository scale, especially for complex or nested types.\n\n## update after rebuttal\nI appreciate the authors' responses, which conceptually addressed my concerns, though open-sourcing the whole benchmark at this moment will be more appreciated than 1-2 examples in the rebuttal to convince me regarding the benchmark design and quality. \n\nI am raising my score to support this paper's acceptance for now, though I am still not fully convinced without seeing the real examples in the benchmark.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and suggestions. We would like to address your remaining concerns with the following responses. We will improve the manuscript accordingly to address them.\n\n### Need to Justify Why We Need LLMs to Do Type Inference\n\nAs shown in [previous work](https://github.com/secure-software-engineering/TypeEvalPy) [1], LLMs already showed better type inference results compared with symbolic tools such as Pyright and [Jedi](http://jedi.readthedocs.io/). Moreover, code-completion tools like Github Copilot and Cursor are making progressively better type hints completions in Python. These motivated us to mainly focus on using LLMs for type inference.\n\n> the paper should more clearly articulate what specific limitations of symbolic approaches LLMs are intended to address\n\nThanks for the suggestion. We use Jedi to illustrate the limitation of symbolic approaches.\n\nThe original code snipet in the `black` repo is:\n```python\n@dataclass\nclass BracketTracker:\n    \"\"\"Keeps track of brackets on a line.\"\"\"\n\n    depth: int = 0\n    bracket_match: dict[tuple[Depth, NodeType], Leaf] = field(default_factory=dict)\n    delimiters: dict[LeafID, Priority] = field(default_factory=dict)\n    previous: Optional[Leaf] = None\n    _for_loop_depths: list[int] = field(default_factory=list)\n    _lambda_argument_depths: list[int] = field(default_factory=list)\n    invisible: list[Leaf] = field(default_factory=list)\n```\n\nThe predicted types by *jedi* are:\n```python\n@dataclass\nclass BracketTracker:\n    depth: int\n    bracket_match: dict\n    delimiters: dict\n    previous: None\n    _for_loop_depths: list\n    _lambda_argument_depths: list\n    invisible: list\n```\nThe TypeSim score on `depth` is 1 and 0.5 for others. The overall score for this file is 0.6645.\n\nthe predicted types by claude-3.5.sonnet.\n```python\nclass BracketTracker:\n    depth: int\n    bracket_match: Dict[tuple[int, int], Leaf]\n    delimiters: Dict[int, int]\n    previous: Optional[Leaf]\n    _for_loop_depths: List[int]\n    _lambda_argument_depths: List[int]\n    invisible: List[Leaf]\n```\nThe TypeSim scores for these variables are all 1.0 and the overall score of the file is 0.8906.\n\nAs shown in the example above, the type predicted by Jedi lost fine-grained information, while Claude recovered most of that with a much higher similarity compared with the original codes.\n\n### Usefulness of Applying TypyBench as a Mainstream Evaluation Benchmark\n\n> what general model capabilities type inference serves as a proxy for\n\nThanks for the suggestion. TypyBench mainly introduces the following challenges (see Table 1 for concrete numbers):\n\n1. Code understanding in long input context length (repo-level, could be >1M tokens)\n2. Long completion length (each function argument and return value need to be typed)\n3. Consistency among lots of outputs (i.e., cases, the predicted types)\n\nAs shown in Figure 4 and Table 2, the relative performance of tested LLMs (claude-3.5, gpt-4o, grok2, and deepseek-v3 were top-tier at that time) generally matches the relative performance on other coding tasks like the code completion in [LiveCodeBench](https://livecodebench.github.io/leaderboard.html) (claude-3.5: 32, gpt-4o: 30, gpt-4o-mini: 27.7, date range 8/1/2024 - 2/1/2025), indicating it is a meaningful diagnostic benchmark that pressure tests the code understanding ability of long-context LLMs.\n\n### More Illustration Regarding the New Metrics\n\n> it’s not entirely clear how well it aligns with human judgment of correctness or practical utility in real-world development workflows.\n\n| Original | Predicted | TypeSim  |\n|-|-|-|\n| `list[Any] \\| None` | `list[str] \\| None`| 0.75|\n| `dict[Any, Any] \\| None` | `dict[str, Any] \\| None`| 0.875 |\n| `dict[str, tuple[Any, ...]] \\| None` | `dict[str, tuple[int, ...]] \\| None`| 0.9375|\n| `dict[str, list[int]]`| `dict[str, Union[tuple[int, ...], Any]]`| 0.8385|\n| `float \\| np.ndarray[Any, Any]`| `np.ndarray[Any, Any]`| 0.5|\n| `pathlib.Path \\| None`| `str \\| pathlib.Path \\| None`| 0.6667 |\n\nAbove is a list of type pairs that are similar but not exactly the same. It can be seen that the TypeSim score reasonably exhibits the similarity between the two types.\n\n> it would be helpful to better justify why this is a comprehensive proxy for type quality.\n\nMypy is a widely adopted best practice in software engineering to improve code quality and detect potential bugs with static checks. For many well-maintained Python open-source repositories, the number of Mypy errors should be almost 0. Therefore, the number of Mypy errors reflects the developer's efforts to fix the inconsistency after using type inference methods.\n\n[1] Shivarpatna Venkatesh, Ashwin Prasad, et al. \"Typeevalpy: A micro-benchmarking framework for python type inference tools.\" Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings. 2024.", "claims_and_evidence": "Claims in this paper are mostly supported by empirical evidence.", "methods_and_evaluation_criteria": "__Need to Justify Why We Need LLMs to Do Type Inference__\n\nWhile this paper presents a well-structured benchmark and introduces valuable evaluation metrics (TypeSim and TypeCheck) for assessing LLMs' performance on type inference, it lacks a compelling justification for the necessity of using LLMs in this context. Given the existence of mature and widely used symbolic tools such as Mypy, Pyright, and MonkeyType, the paper should more clearly articulate what specific limitations of symbolic approaches LLMs are intended to address. Without a thorough discussion of the practical advantages of LLMs—such as their ability to generalize to uncommon patterns, handle incomplete code, or infer types in low-resource settings—it remains unclear why LLM-based inference is worth pursuing over traditional methods.\n\nTo strengthen the contribution, I encourage the authors to (1) provide a clear motivation for using LLMs for type inference, including a comparison of their theoretical or practical advantages over symbolic systems, and (2) include an empirical baseline where symbolic tools are evaluated on the same benchmark. Demonstrating that LLMs can outperform symbolic methods—either in terms of TypeSim, TypeCheck, or overall utility in real-world development scenarios—would substantially reinforce the paper’s significance and help clarify the role LLMs should play in the type inference landscape.\n\n__Usefulness of Applying TypyBench as a Mainstream Evaluation Benchmark__\n\nAnother key concern lies in the broader motivation for evaluating LLMs specifically on the task of type inference. As a benchmark, TypyBench would benefit from a clearer articulation of what general model capabilities type inference serves as a proxy for. Unlike tasks such as code generation or execution reasoning—where LLMs are uniquely positioned to outperform symbolic methods and improve downstream applications like self-debugging or automatic code generation —type inference is a narrow and well-defined task that symbolic tools already handle effectively in many practical settings. Without a strong argument for how improving LLMs' performance on type inference translates into broader gains in language model capabilities, it remains unclear why this task warrants dedicated benchmarking and why LLM developers should prioritize it as part of their evaluation pipeline.\n\nTo enhance the benchmark’s relevance, the authors should consider framing type inference within a more holistic view of code understanding or software engineering tasks that LLMs uniquely enable. Alternatively, they could provide evidence that performance on type inference correlates with performance on broader tasks like code editing, completion, or static analysis augmentation—thereby justifying the benchmark as a meaningful diagnostic tool. Without this contextualization, the current scope risks appearing too narrow and decoupled from the more impactful capabilities that developers and practitioners typically seek in modern LLMs.\n\n__More Illustration Regarding the New Metrics__\n\nWhile the proposed TypeSim and TypeCheck metrics are novel and address some limitations of exact match evaluation, their design and practical significance could benefit from further clarification. TypeSim, in particular, introduces a similarity-based measure that may capture more semantic nuance, but it’s not entirely clear how well it aligns with human judgment of correctness or practical utility in real-world development workflows. Similarly, TypeCheck measures consistency via mypy errors, but it would be helpful to better justify why this is a comprehensive proxy for type quality. I encourage the authors to provide more rationale or empirical validation for these metrics—perhaps through user studies, ablation analysis, or correlation with downstream developer effort—to strengthen confidence in their effectiveness and generalizability.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See \"Methods And Evaluation Criteria\".", "relation_to_broader_scientific_literature": "This is related to program analysis literature in PL/SE domain.", "essential_references_not_discussed": "References are well discussed.", "other_strengths_and_weaknesses": "See \"Methods And Evaluation Criteria\".", "other_comments_or_suggestions": "See \"Methods And Evaluation Criteria\".", "questions_for_authors": "See \"Methods And Evaluation Criteria\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HvEpMwI30m", "reviewer": "Reviewer_MFpm", "summary": "In this work, the authors evaluate the ability of LLMs to perform type inference in Python codebases. They introduce two type inference evaluation metrics: (1) TypeSim, which extends prior work focused on exact matching to consider semantic similarity between LLM-inferred vs. human-annotated types, and (2) TypeCheck, which assess repository-level type consistency (e.g., detection of incompatible return types and invalid argument types). They also introduce a type inference benchmark dataset, TypyBench, which consists of a curated set of 50 Python repos. Their empirical results indicate that LLMs are better at local type inference (as indicated by TypeSim scores) than global consistency (as indicated by the TypeCheck scores).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and suggestions. We would like to address your remaining concerns with the following responses. We will improve the manuscript accordingly to address them.\n\n> Have you considered alternative methods for assessing type/argument similarity? (eg, embedding-based approaches?)\n\nOur attribute-based method (L198) can also be viewed as an embedding-based approach. \nFor example, if we regard each attribute as a column (1 means has and 0 means not), then each type gets an embedding representation, then TypeSim is naturally defined as the Jaccard Similarity over such a representation.\n\nFor example:\n\nType | __iter__ | append | __add__ | … |\n|-|-|-|-|-|\nint | 0 | 0 | 1| …|\nlist | 1 | 1 | 1| …|\nSequence | 1 | 0 | 0| …|\n\nSuch embedding representation can naturally work for user-defined types and the similarity is based on the functionality of types rather than the semantic meaning of the name of the types.\n\n> It would be helpful to provide error analysis/insights into how/why LLMs struggle with code consistency. This can help to inform mitigation efforts that do not require passing the whole repo in as context.\n\nThanks for the suggestion. We did a preliminary case study and found that a common cause of type inconsistency comes from missing context (defined in other files or imported from third-party libraries). The patterns include but not limited to:\n\n1. Missing base class \n```python\nclass A(B):\n    def func(vars):\n        super().func(vars)\n```\n\n2. Missing function information\n```python\ndef func(vars):\n    return another_func(vars)\n```\n\n3. Missing decorator information\n```python\n@some_decorator\ndef func(vars):\n    …\n```\n\nThe most straightforward way is to provide the whole repository as the context, but limited by LLM's context window and might hurt TypeSim as shown in Table 4.\n\nAn alternative way is to copy-paste the necessary context following the `import` statements but requires engineering efforts. We tried to provide such context manually for the examples we found, and the inconsistency issues were resolved after the context was provided.\n\nAnother potential method could be iteratively refining the stub files while using all stub files (and the implementation of the current file) as context. This would require much less context window since the stub files are relatively small.\n\n> It might be helpful to provide technical definitions for consistency and coherence when these terms are used early in the paper.\n\n> If possible, it would be helpful to place algorithm blocks after the text that describes them, rather than before.\n\nThanks for both suggestions. We will improve the manuscript accordingly.", "claims_and_evidence": "The authors' empirical results largely support their claims and validate the benefits of the metrics they propose.", "methods_and_evaluation_criteria": "The metrics that the authors introduce (e.g., TypeSim and TypeCheck) and the ways in which they operationalize and evaluate them seem appropriate and useful.", "theoretical_claims": "N/A, the authors’ claims are largely empirical.", "experimental_designs_or_analyses": "I read their empirical evaluation section, but did not take further steps to verify/replicate the soundness of their results.", "relation_to_broader_scientific_literature": "This paper is related to prior work on type inference methods, including static/dynamic analysis, ML-based approaches that leverage structured and/or unstructured representations of code, and existing type inference and programming-related benchmark datasets (e.g., for code generation, code completion, debugging, etc.).", "essential_references_not_discussed": "I am not aware of any essential but omitted references.", "other_strengths_and_weaknesses": "Strengths\n-  The paper is well-written and well-motivated.\n- Consideration of the semantic similarity of type annotations meaningfully improves upon evaluation based on exact matching. \n- The authors conduct empirical analysis on a representative set of LLMs and a well-curated benchmark dataset. \n\nWeaknesses\n- It would be helpful to provide error analysis/insights into how/why LLMs struggle with code consistency. This can help to inform mitigation efforts that do not require passing the whole repo in as context.", "other_comments_or_suggestions": "- It might be helpful to provide technical definitions for consistency and coherence when these terms are used early in the paper.\n- If possible, it would be helpful to place algorithm blocks after the text that describes them, rather than before.", "questions_for_authors": "1. Have you considered alternative methods for assessing type/argument similarity? (eg, embedding-based approaches?)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TLw1LVqH30", "reviewer": "Reviewer_KRyD", "summary": "The paper introduces TYPYBENCH, a benchmark for evaluating the capability of Large Language Models (LLMs) in type inference for Python repositories without explicit type annotations. It defines two novel metrics: TYPESIM, which captures semantic relationships between predicted and ground truth types using syntax-level features, and TYPECHECK, which assesses type consistency across entire codebases. The benchmark includes 50 well-typed Python repositories from diverse domains. Experimental results show that while LLMs achieve reasonable type similarity, they struggle with complex nested types.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and suggestions. We would like to address your remaining concerns with the following responses. We will improve the manuscript accordingly to address them.\n\n> LLMs are evaluated file by file. Can the authors discuss the potential limitations of using a single file for type inference and possible future directions in improving LLM's type inference via giving more constructive contexts?\n\nThanks for the suggestion. We did a preliminary case study and found that a common cause of type inconsistency comes from missing context (defined in other files or imported from third-party libraries). The patterns include but not limited to:\n\n1. Missing base class \n```python\nclass A(B):\n    def func(vars):\n        super().func(vars)\n```\n\n2. Missing function information\n```python\ndef func(vars):\n    return another_func(vars)\n```\n\n3. Missing decorator information\n```python\n@some_decorator\ndef func(vars):\n    …\n```\n\nThe most straightforward way is to provide the whole repository as the context, but limited by LLM's context window and might hurt TypeSim as shown in Table 4.\n\nAn alternative way is to copy-paste the necessary context following the `import` statements but requires engineering efforts. We tried to provide such context manually for the examples we found, and the inconsistency issues were resolved after the context was provided.\n\nAnother potential method could be iteratively refining the stub files while using all stub files (and the implementation of the current file) as context. This would require much less context window since the stub files are relatively small.\n\n> On the lack of comparison with static type inference tools. From my knowledge, there are other tools than Mypy such as Pyre developed by Meta are doing Python's type analysis. While I am not suggesting additional experiments, could the authors discuss the limitations of relying solely on Mypy as the baseline? How could alternative static analysis tools influence the evaluation results?\n\nThanks for bringing up this point. While Mypy is imperfect and could produce false positives or incorrect errors, it is still a popular choice for software engineering best practices to improve code quality. Evaluation results may be different when switching to another static type inference tool like Pyre, but since the majority of static checks should be similar, the change in the evaluation results should be small. TypeCheck metric could switch to another static type inference tool if there is a better and well-adopted one.\n\n\n> TYPESIM only relies on the similarity of syntax features. While TYPESIM effectively relaxes exact type matching, it primarily relies on syntax-level features, such as attribute comparisons. Have you considered incorporating other features related to code semantics? For example, the similarity between the control flow graphs of two functions.\n\nThanks for your question. If our understanding is correct, you are suggesting not only comparing whether two types have the same functions, but also examining the code semantics, like how similar are their implementations for the same function?\n\nWe think it is an interesting direction for future work (especially for complex user-defined types), but current TypeSim already reasonably captures the similarity between types with convenient and robust calculations.\n\n| Original | Predicted | TypeSim  |\n|-|-|-|\n| `list[Any] \\| None` | `list[str] \\| None`| 0.75|\n| `dict[Any, Any] \\| None` | `dict[str, Any] \\| None`| 0.875 |\n| `dict[str, tuple[Any, ...]] \\| None` | `dict[str, tuple[int, ...]] \\| None`| 0.9375|\n| `dict[str, list[int]]`| `dict[str, Union[tuple[int, ...], Any]]`| 0.8385|\n| `float \\| np.ndarray[Any, Any]`| `np.ndarray[Any, Any]`| 0.5|\n| `pathlib.Path \\| None`| `str \\| pathlib.Path \\| None`| 0.6667 |\n\nAbove is a list of type pairs and their TypeSim score. It can be seen that the TypeSim score reflects the compatibility of two types, which roughly matches human's impression on the type similarity.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Not applicable, the paper is about benchmarks and there are no theoretical claims or proofs.", "experimental_designs_or_analyses": "I checked how TYPESIM works in detail and appreciate the authors’ efforts to move beyond exact type matching by incorporating semantic similarity. \nHowever, while TYPESIM aims to capture the semantic relationships between types, it still primarily relies on syntax-level comparisons and focuses on shared fields and attributes between types. \nThis raises concerns about whether it fully captures deeper semantic equivalences beyond structural similarity.\n\nAdditionally, the study acknowledges that LLMs are evaluated file by file due to token size limitations. \nIt raises a concern that the capability of LLMs in type inference could be undermined due to the limited size of context windows.\nWhile I acknowledge that the experimental results are persuasive, it would be great if the authors could discuss using techniques like using techniques as program slicing to feed related context of a type to LLMs.", "relation_to_broader_scientific_literature": "The paper mainly introduces a benchmark for Python's type analysis on the repository level, which could highlight future research in studying the how LLMs can help type inference of industry length.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "### Strengths \n\n- The type analysis is performed across both built-in, user-defined, and generic types.\n\n- The benchmark is targeted at the type inference on the repository level, across 50 well-typed Python repositories from diverse domains.\n\n- The evaluation is comprehensive, including multiple state-of-the-art LLMs.\n\n\n### Weaknesses\n\n- TYPESIM still relies on syntax-level comparisons.\n- Lack of comparison between baseline static type inference tools but involving mypy only.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "- LLMs are evaluated file by file. Can the authors discuss the potential limitations of using a single file for type inference and possible future directions in improving LLM's type inference via giving more constructive contexts?\n\n- On the lack of comparison with static type inference tools. From my knowledge, there are other tools than Mypy such as Pyre developed by Meta are doing Python's type analysis.  While I am not suggesting additional experiments, could the authors discuss the limitations of relying solely on Mypy as the baseline? How could alternative static analysis tools influence the evaluation results?\n\n- TYPESIM only relies on the similarity of syntax features. While TYPESIM effectively relaxes exact type matching, it primarily relies on syntax-level features, such as attribute comparisons. Have you considered incorporating other features related to code semantics? For example, the similarity between the control flow graphs of two functions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Meta-Review for “TypyBench: Evaluating LLM Type Inference for Untyped Python Repositories”**

---

### Summary of the Paper

The submission introduces **TypyBench**, a new benchmark for evaluating Python type inference—particularly with large language models (LLMs)—on entire code repositories rather than just on isolated files or functions. The authors propose two metrics:
1. **TypeSim**, which measures how semantically similar a predicted type is to the “ground truth” by looking beyond exact string matches (e.g., capturing that `List[str]` and `Sequence[str]` share functional attributes).
2. **TypeCheck**, which evaluates repository-level consistency by counting how many type errors (e.g., incompatible return types, invalid arguments) a static type checker (mypy) reports when using the inferred annotations.

An dataset of 50 curated Python repositories is collected and processed so that all original type annotations are removed (thus forming the benchmark’s input). The repositories are chosen to be sufficiently diverse (e.g., developer tools, ML libraries, security, web APIs) and to have meaningful, correct, and relatively complete annotations in their original form. The authors demonstrate that state-of-the-art LLMs can achieve decent scores on local type similarity (TypeSim) but do not ensure consistency well at the repository scale (TypeCheck), particularly for nested or complex types.

---

### Strengths

1. **Type Inference Metrics**  
   - **TypeSim** captures partial correctness. This is an improvement over pure exact-match or “match up to parametric type” metrics that might over-penalize cases where the model predicts, for example, `List` instead of `List[int]`.  
   - **TypeCheck** indicates how well an inferred set of types holds together globally, which is important in real-world usage.

2. **Type Inference Dataset**  
   - A collection of 50 repositories across diverse domains.  
   - The dataset is large enough (over 100k total lines in many repos) to test long-context and consistency challenges.

3. **Evaluation on Multiple Models**  
   - The experiments compare both top-tier API-based models (GPT-4 variants, Claude, Deepseek, Grok) and smaller local models (Llama variants, Qwen).  
   - Results show that some models excel at local similarity (TypeSim) but suffer from consistency errors (TypeCheck).

---

### Weaknesses

1. **Comparisons to Static Analysis Tools**  
   - Comparison with other static-analysis tools would improve the evaluation.

2. **Drawbacks of Proposed Metrics**  
   - **TypeSim** uses syntax level comparisons comparing sets of supported methods and structural features. While this is an improvement over exact matching, some reviewers expressed skepticism as to whether it is enough to fully capture “semantic” type correctness.  
   - **TypeCheck** is limited by mypy’s correctness. False positives or negatives do exist, although it is considered a fair standard for everyday Python usage.

---

### **Recommendation**:

Based on reviewers evaluations, I recommend weak accept.
Decision: Accept (poster)