Dataflow-Guided Neuro-Symbolic Language Models for Type Inference
Authors: Gen Li
Yao Wan
Hongyu Zhang
Zhou Zhao
Wenbin Jiang
Xuanhua Shi
Hai Jin
Zheng Wang
Venue: ICML 2025 poster (2025)
Keywords: type inference
neuro-symbolic
language models
dataflow analysis
Primary area: applications->everything_else
TL;DR: This paper presents Nester, a neuro-symbolic approach to enhance LMs for type inference by integrating symbolic learning without increasing model size.

Abstract: Language Models (LMs) are increasingly used for type inference, aiding in error detection and software development. 
Some real-world deployments of LMs require the model to run on local machines to safeguard the intellectual property of the source code. This setting often limits the size of the LMs that can be used. We present Nester, the first neuro-symbolic approach that enhances LMs for type inference by integrating symbolic learning without increasing model size. Nester breaks type inference into sub-tasks based on the data and control flow of the input code, encoding them as a modular high-level program. This program executes multi-step actions, such as evaluating expressions and analyzing conditional branches of the target code, combining static typing with LMs to infer potential types. 
Evaluated on the ManyTypes4Py dataset in Python, Nester outperforms two state-of-the-art type inference methods (HiTyper and TypeGen), achieving 70.7\% Top-1 Exact Match, which is 18.3\% and 3.6\% higher than HiTyper and TypeGen, respectively. For complex type annotations like typing.Optional and typing.Union, Nester achieves 51.0\% and 16.7\%, surpassing TypeGen by 28.3\% and 5.8\%.

Dataflow-Guided Neuro-Symbolic Language Models for Type Inference
Gen Li
Yao Wan
Hongyu Zhang? Zhou Zhao
Wenbin Jiang' Xuanhua Shi Hai Jin
Zheng Wang"
Abstract
cult to debug (Oh & Oh, 2022). Therefore, automated type inference, which infers the type of variable automatically, is attractive. Recently, there has been growing interest in using Language Models (LMs) for type inference to reduce the manual effort in developing type inference systems.
Language Models (LMs) are increasingly used for type inference, aiding in error detection and soft- ware development. Some real-world deployments of L.s require the model run on local machines to safeguard the intellectual property of the source code. This setting often limits the size of the LMs that can be used. We present NESTER, the first neuro-symbolic approach that enhances LMs for type inference by integrating symbolic learning without increasing model size. NESTER breaks type inference into sub-tasks based on the data and control flow of the input code, encoding them modular high-level program. This program executes multi-step actions, such evaluating ex- pressions and analyzing conditional branches of the target code, combining static typing with LMs to infer potential types. Evaluated on the Many- Types4Py dataset Python, NESTER outperforms two satte-of-the-eeee type inference methods (Hi- Typer and TypeGenn), achieving 70.7% Top-1 Ex- act Match, which is 18.3% and 3.6% higher than HiTyper and TypeGen, respectively. For com- plex type annotations like "yyinng.ptionall and "typing. Union", NESTER achieves 51.0% and 16.7% surpassing TypeGen by 28.3% and 5.8%.
Large Language Models (LLMs), such as GPT-4 (Koubaa, 2023) and OpenAI-ol (Temsah al., 2024), have shown impressive results in type inference. However, these mod- els typically rely on cloud-based infrastructure with sub- stantial computational resources, raising privacy concerns when user data is sent to untrusted providers (Ray, 2023). This is especially problematic for organizations that treat their source code as valuable intellectual property. To mit- igate risks, many prefer deploying smaller models locally developer machines or private clusters due hardware resource constraints.
Running moderately sized LMs, like Code Llama with 7B parameters (Lu et al., 2023), locally can address privacy and confidentiality concerns. It also delivers faster response times compared to cloud solutions employing fully homo- morphic encryption (Chen et al., 2022) or multi-party com- munication techniques (Yao, 1986; Rathee et al., 2024). However, downsizing LMs from hundreds of billions to tens of billions of parameters significantly reduces their ability to model and reason about complex program structures. As a result, there is a need to improve the capability of LMs in reasoning code for type inference without relying on cloud-based solutions.
1. Introduction
A Motivation Example Figure illustrates motivating example for inferring the return type of the Python func- tion learning.rate. As shown in Figure I(a), mypy', ' a rule-based tool, fails to infer the type correctly due to its lack of knowledge of custom functions or third-party libraries (e.g., normalize ir line 7). Similarly, pre- trained CodeLlama 7B (Lu et al., 2023) predicts the type as flol-float-noo
Dynamically typed languages like Python and JavaScript are popular for rapid prototyping and have the flexibility of not requiring explicit type declarations (Mir al., 2022: Srinath, 2017). However, this flexibility can compromise reliability, often leading to runtime type errors that are diffi-
'National Engineering Research Center for Big Data Tech- nology and Systems, Services Computing Technology and Sys- tem Lab, Cluster and Grid Computing Lab, School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China *Chongging University "Zhejiang University 'University Leeds. Correspondence to: Yao Wan <wanyao@ hust.edu.cn>>
while the correct type is float-none as seen in Figure 1(b). This error stems from the LM relying on the textual structure of the code rather than its underly- ing semantics, such as control and data flow. Additionally, LMs can be misled by irrelevant nearby information (e.g.. return Fals se in line 3). Specifically, since Ir never exceeds 1.0 and line I unreachable, the return type cannot
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
hhtps://mypy-lang.org

---

Dataflow-Guided Neuro-Symbolic Language Models for Type Inference
ajstaticc Analyzer (Symbolic)
def learning Baatinn__rtt__ppppe (opt, base_lr, f):
The return type leaming.
1r = nin(base__n, 0.5)
Static Analyzer
none
if > 1.00
return False
(b)LMs (Neural)
Control Flow Graph
if Ir <= 0.5:
type learning. rate boo foaa nons.
tgt_ base_ir
Prompt
Code Llama
tgt Igt_ir 1r= 10
(tt 28)
c]Nester (Neuro Symbolic)
opt ]
High-Lovel Program
return 1r
Firua
10
return None
Fbi
Data Graph
Program
Recognizer
Interprater
Figure 1. Example showing the 7B CodeLlama LM and static analyzer (mypy) fail to infer the return type of Learning.rate.
be boolean. This issue can be addressed by incorporating
and explainability by incorporating symbolic rules.
control flow analysis via static techniques.
We introduce data flow analysis of the target program to guide LMs in inferring types more precisely.
Our Solution: A Neuro-Symbolic Approach Our work aims to enhance LMs' ability to reason about program con- trol and data flow, essential for accurate type inference, without relying on larger models. We propose novel neuro- symbolic framework, NESTER, which decomposes type inference into simpler symbolic steps encoded as a Python- like high-level program. This high-level program is then processed by program interpreter by leveraging static type inference and LMs. As shown in Figure (cc), we first break down type inference into a number of sub-tasks according to control flow, where each sub-task corresponds to potential type of the execution path. The analysis of the sub-task is realized through number of API calls like f anall 1 ys (inputs, condit tions, terpreter processes each line of the high-level program to combine static type inference and LMs to infer the type at each step. Our neuro-symbolic approach also enhances interpretability by embedding reasoning rules into LMs, improving decision makinn transparency.
Extensive experiments on Many'Types4Py demonstrate the effectiveness of NESTER, and VSCode plugin developed to showcase its enhanced interpretability.
Online Materials The source code and data as- sociated with this work are available at: https: //github. com coc.occcooccc/cccoooeeeoossso /oc.l-oooeee/ntuuall///eeeeeeeee naturalcc/tree! main/ httn//xxmmllee/eeeeeee
2. Preliminaries
2.1. Type Inference
Then, program in-
This work focuses on the type inference task, which pre- diets identifier's type based on its context in code snip- pet. Given code snippet c, let {x. sent the V distinct identifiers needing type assignment. Let p(T: clx)) denote the type distribution for identifier x.., de- termined by its context ctz. Our goal is to develop a model that, given an unannotated or partially annotated code snip- pet c, outputs a type distribution for each missing annotation. Recent studies have used L.MSs for type inference (Siddiqui & Kellogg, 2024; Peng al., 2023). These approaches frame the task as a generative one through natural language prompts, which describe the code and identify relevant iden- tifiers for type prediction. TypeGen (Peng et al., 2023) is the saate-of-the-an LM-based type inference, using a specific prompt based on In-Context Learning (ICL). The prompt strueture is [Code, ICL EXAMPLES, TYPEHINTS), where ICL EXAMPLES provide demonstrations, and TYPEHINTS are derived from static analysis. TypeGen also uses CoT reasoning to improve accuracy, with the prompt structured as follows: "First, the variable error message is as- signed from a string. Therefore, the type of the variable error message "str"."
1... V} repre-
Key Results and Contributions We evaluate NESTER by applying the widely-used ManyTypes4P dataset (Mir et al., 2021). Experimental results show that NESTER outper- forms the advanced Type4Py baseline by 3.8% for argument types and 3.9% for return types in Top-1 Exact Match ac- curacy, using an LM with just 7B parameters. Additionally, NESTER surpasses the state-of-the-art TypeGen baseline by 3.6% across all categories. This demonstrates NESTER's ef- ficiency, achieving strong performance using small LM by today's standards. Case studies on real-world code snippets further highlight NESTER 's ability to accurately infer types with natural-language explanations.
The key contributions of this paper are as follows:
We propose NESTER, the first neuro-symbolic framework for type inference using LMs, which enhances reasoning
2

---

Guuddee Neuro-Symbolic Language Models for Type Inference
demonstration examples, whose type answers are identical to those in the test set, with examples from pool of can- didate inference examples that have differing type answers from the test set. These new demonstration examples, like those selected by TypeCien, exhibit high code similarity to the training set.
1.0
Typegen Similar Examples Examples
0.8
0.72
0.64
0.6
8.57
0.4
Figure shows the performance of TypeGen in type in- ference over the aforementioned two settings. From this figure, can observe that incorporating examples with identical result types into TypeGen surely enhanced ac- curacy, reaching 72.5% compared to 57.2% without any examples, marking an increase of 15.3%. However, after replacing with similar examples, the accuracy of examples provided to LMs decreased 64%. This marks a reduction of 8.5% from the original TypeGen scenario, effectively halving the initial improvement. The experimental results indicate that LMs do not truly understand the problem; they merely generate text and append the type result at the end, resulting in misleadingly high accuracy. However, when replaced with similar examples, LMs become confused by cat-of-distribution test sets, which hampers their ability to perform generalization reasoning and ultimately leads to a halving of accuracy. Consequently, while TypeGen's 's infer- ence examples misleadingly improve accuracy, they lack reasonableness in the LM type inference process.
0.2
0.0
Typegen
Similar Examples
Examples
Figure 2. The reasoning ability study on TypeGen.
2.2. Neuro-Symbolic Reasoning
Neuro-symbolic reasoning enhances interpretability by breaking tasks into subtasks (Gupta & Kembhavi, 2023; Cunnington et al., 2022; Nye et al., 2021; Zhang et al., 2021). It divides the solution function f into two phases:
= h.g.
Here, g is a perception network that processes raw data, imposing human-like constraints define conditions for the problem, while h is reasoning model that analyzes the data to infer the solution.
Consider Visual Question Answering (VQA) as an exam- ple, where the goal is to answer a question based on an image (Gupta & Kembhavi, 2023). The image processed into a structured format by the visual perception network g. serving as static information for the question. For example, given the query "How many cubes behind the cylinder are large?", , the context-awer awwre model h parses the question into a high-level program. Key terms like cube, cylinder, and large are extracted as parameters. The program then uses APIs such as fitter-shape and count to generate a solution. Finally, the program is executed using the database from the visual perception network.
3. NESTER: Our Approach
Figure provides an overview of our proposed system, NESTER, which comprises two main components: (a) high- level program generation and (b) program interpreter. The high-level program generation component involves crafting prompt (e.g.. "Convert the code into a high-level pro- gram "' along with several demonstration examples to guide the LMs generating high-level programs composed of predefined APIs. the program interpreter component, we sequentially implement each API derived from the high- level programs. Each API utilizes tandem of the type recognizer and LMs to perform type inference by traversing the dataflow starting from the API's parameter.
2.3. Do LMs Demonstrate Reasoning Ability in Type Inference?
Before introducing NESTER, first conduct prelimi- nary study to explore whether LMs demonstrate reasoning ability in type inference. While LMs have shown promis- ing results in type inference, we attribute their successful type predictions primarily to the examples provided in ICL demonstrations, rather than to the reasoning over symbolic type rules. To verify our suspicion, we select TypeGen, a state-of-the-arr LM-based type inference approach, our target for analysis, over the widely utilized Many Types4Py dataset (Mir et al., 2021). In implementation, we select Code Llama 7B as the LM backbone, and design two settings: (1) No Examples: We remove all demonstration examples, keep- ing only the query code and the question asking for the type of code. (2) Similar Examples Replacement: We replace the
3.1. High-Level Program Generation
NESTER first decomposes the code snippet into high- level program, which represents reasoning substeps in the structure of the code. Each line of the high-level program is an API call to a function supported by NESTER, which is then processed by our program interpreter. The main function of the high-level program is guiding the program interpreter to focus on specific variables and execution paths, while the API performs detailed analysis on a variable, con- ditions, or a string. It is noteworthy that we preserve the original program's 's control flow information based on the high-level program. On the one hand, the high-level pro-
3

---

Mataflow-Guider Neuro-Symbolic Language Models for Type Inference
(al High lovel Program Generation
(b) Progremt Intergrater
Targel Program
High-Lavel Progrem
Dataflow Graph
LMs
Lenalysis(r,, 1,8, >):
3r
iin 0.5)
OO Llama3
def Zeaming (opt,,,, bese_ir, f):
return_naalystt (F
Condition Evaluation API
Tr 1.0:
ircs1
1r winghese_irr _r, 0.5)
f_analysts(ir, 8.5, reture _enllysss((s)
If le 1.8:
T3 reture_enslllaeeeeeee ( combine(12, 12)
vin(case_ir, 0.5)
1rc=1
return False
11 6:5:
if Ir <- 0.5:
1K(3)
1gt_2r Sa *
1pt_ir
Executor
1et 1at
1tt_ Ir hase_rr Jir
CodeLiama
#loat
www.ooooloeeeeeenm Inttxt. 20)
Inference API
Type
Dormolte__in(ttt__. 20)
Type Recognizer
goroooo 1n'] <30
retuin (r)
10
None
mone
Optional|foatt
Figure 3. Neuro-symbolic type inference pipeline. blue highlights the target program, purple highlights the high-level program generated by LMs, and green highlights the output by our program interpreter. @ to @ are the line numbers of the high-level program. O. @ are executed by the condition evaluation API. O, O are executed by the type inference API.
gram can utilize the type information implicitly contained in the control flow; the other hand, it contributes to the final type inference results combination. A running example of the high-level program is given in Figure 3. Moreover, we also provide concrete definition of the high-level program, as detailed in Appendix B.I.
tially corresponds to a type inference sub-task.
Supported APIs NESTER offers two types of APIs be used in the high-level programs: (i) Condition evaluation APIs to evaluate if condition in the input program can be satisfied to determine the reachable execution paths; (ii) Type inference entry-point APIs to retain identifiers (e.g.. ir in Figure 1) that guard the evaluation outcomes of a condition. NESTER is extensible new APIs can be added by creating and registering a class. The current implementation of NESTER provides four APIs, detailed in Appendix B.3.
A Running Example Figure 3(a) shows an example high- level program generated by NESTER for the target program in Figure 1. Here, we generate the high-level program by prompting an LM (e.g., Code Llama 7B), although this can also be done through traditional parser. We use LMs be- cause a general-purpose LM can adapt multiple program- ming languages, reducing the manual efforts in building a language-specific parser. We use a prompt consisting of code and high-level program pairs to enable LMs to under- stand the translation task. For details on the specific prompt, please refer Appendix B.2.
3.2. Program Interpreter
The high-level program interpreted and executed by a NESTER program interpreter, which takes two inputs: (1) the target program for type inferencing and (2) a high-level program generated from the target program. The program interpreter first performs data flow analysis on the target program to construct a data flow graph. Then, we traverse the dataflow graph, beginning at the entry point of each API call of the high-level program.
NESTER generates the high-level program by translating each conditional branch of the input code into an analy- sis unit in the high-level program, realized through API calls. We do so by providing examples in the LM prompt to ask it to generate API calls for given code input. For the example given in Figure 3(a), we start by feeding if 1r > 1.0 at line 2 to an LM to generate an API call, if if
Dataflow Graph Figure 4 illustrates an example of constructing the dataflow graph for the target pro- gram. Specifically, for the identifier 1r, we first lo- cate the initial definition in the code snippet, namely ir min (base_lr, 0. on line 1. Consequently, we add the line as a node to Der( denoted as Def( = [Line 1}. After defining Def(Ir), We proceed to identify the lines of code that directly use each member belonging to Def(Ir). These are the User(Ir) nodes, explicitly denoted as [Lines 2, 4, 7, 8 and 9}. After identifying all the User(Ir) nodes, we apply the same method to find the Usee(Ir) nodes, which capture the lines of code that utilize the users from both User(Ir) and Drr( denoted as {Lines 0, 5}. We use the Graised Dataflow Graph (CGDG) as the implementation of this data flow graph. For details of the
The source if expression is bro- ken down into identifiers, operators, and literal values: 1r 1.0. and Similarly, if ir <= 0.5 is also parsed by the LM to 1r. 0.5, , and <= For the code in lines 3, 9, and 10, the LM parses this code as return False to return 1r can transform to T2 = return.analysis (1r) and return None to neturn_analyse (None) re- spectively. The API calls in the high-level program guide NESTER for type inference, where each analysis unit essen-
T2- = return analysis (False)

---

Dataflow-Guided Neuro-Symbolic Language Models for Type Inference
learning.rate = min (base_lr, 0.5) and con- sults LM to determine the range of values for ir In this case, the LMM returns the range 1 <= 0.5, which is correct return result. Then we utilize the results to evalu- ate the condition ir > 1.0, which not satisfied, so we jump over 2 to execute 3 in the high-level program. The code if 1r 0. also parsed by the LM into
0
def learning learning_rateeppt,, base factor):
1
(base 1r,-0:5)
2
Ir
1.0:
User
Def return False
3
Usee
4
1r <= 0.5:
5
tgt_ Ir base 1n factor
6
tgt = tgt_le 1r 10
if if.analysis (lr, 0.5, <=) Using the result 1r
7
1r wonnollleeeeeeeeeme 1n(ttt Ir, 0.01)
<= 0,5, we evaluate that 1r < 0.5 holds true. We utilize dataflow graph to identify code lines in the target program, which are related to 1r These lines are initially type recognizer. Upon failure, the lines are for- warded to LMs, which then return the type "float". 5 The type recognizer directly returns the type "none" to our API. Finally, the API Combine integrates the two types, "float" and "none", based on the return values provided by return.analysis Consequently, derives the final correct type "Optionalftlatt".
8
opt ppt.pprrmms0][[1111 1r
9
return 1r
10
return None
Identifier
Def
User
Usee
opt
line 0
1ine 1,5
base
line 9
line 1,5
factor
line e
line
Line D Tine 2,4,7, 7,8 8,8,9
line 0,
tg 1r
line
line 6,7
line0
When handling multiple if ana malysss APIs, the combine () module detects and discards unreturned type T before merging the remaining types.
Note:
Mp
query
T2 return analysis
Float
Figure The construction of dataflow graph code snippet.
4. Experiments
CGDG construction, please refer to Appendix C.2.
4.1. Evaluation Setup
Type Recognizer NESTER uses the type recognizer to identify the types of a specific node (which include multiple code lines) in dataflow graph, including Def(r,,), us.)( and )., a line of code contains type information, the type recognizer uses regular expressions to recognize and return the corresponding type. These types include basic types such as "int", "float", "bool", "str", and "bytes", as well as more complex generic types such as "tuple", "list", and "dict". Appendix C.I presents the denotational semantics of the regular expressions, which is used by the type recognizer NESTER.
Dataset We follow the methodology from previous stud- ies (Allamanis al., 2020; Peng al., 2023; Lukasczyk et al., 2023; Zhang al., 2023) and evaluate approach using the widely-used 2021), which contains around 5,500 Python projects with over 870,000 type annotations. The dataset includes over 880,000 functions (return types). 1.5 million arguments (ar- gument types), and 2.1 million variable declarations (local types). To ensure fairness in comparison, we normalize outputs by treating similar terms, such as "int" and "integer" or "bool" and "boolean" as equivalent.
dataset (Mir et al.,
LMs System NESTER employs two LMs for program in- terpretation: the Condition Eval LM and the Type Infer LM. The Condition Eval LM assesses the possible value ranges for specified identifiers, while the Type Infer LM identifies the type of given code snippets, as a supplement for the type recognizer. We utilize prompts to elicit responses from the LLM as follows: [Def( U User(x.)U Usee(z,))(odee) QUESTION CONDITION] and (Def(x..) U User(x,)U (see(zz)((ooe)) TYPEHINTS, QUESTION.TYPES re- spectively.
LM Backbones We select two LMs as backbones.
D Code Llama 7B (Lu et al., 2023) is LLM specifically designed for code-related tasks. This variant, with 7 billion parameters, is pre-trained on a diverse set of code and natu- ral language data, enabling to effectively understand and generate code across multiple programming languages. The pre-training process includes extensive datasets from open- source repositories, which helps Code Llama 7B develop robust understanding of syntax, structure, and various programming paradigms.
Figure 3(b) shows an example of program inter- preter. In terms of condition evaluation, API execu- tion. Olt looks for identifiers in the API call of the high-level program and uses data flow analysis on the target program to locate relevant code segments
> Llama 3 8B (Das et al., 2025b) is versatile foundation model designed for multilinguality. coding, reasoning, and tool usage. This variant, with 8 billion parameters, leverages dense Transformer architecture, enabling to handle com-
5

---

Datdflow-Guided Neuro-Symbolic Language Models for Type Inference
plex tasks across different languages and domains. Llama 3 8B is pre-trained on diverse data sources, including text, code, and structured information, which enhances its capa- bility to perform well in various applications, from language understanding to code generation.
Implementation Details The model is configured with a 4,096-token context window and maximum batch size of 8, which is adequate for most dynamic type inference tasks, To improve accuracy while keeping computational demands manageable, we set the temperature to 0.2. This configura- tion minimizes output randomness, enabling consistent and reliable type inference across diverse coding scenarios. All experimental tasks are carried out on one Linux platform, running an Ubuntu 22.04 system, powered by an Intel Xeon 12-core CPU running at 2.40GHz, coupled with an RTX 4090 GPU, and equipped with TB of RAM.
Baselines We compare NESTER with eight baselines.
>> Hityper (Peng et al., 2022) combines static analysis and well-designed static rules for type inference.
>> TypeBERT (Jesse et al., 2021) treats type inference as a -illin-theeblank task, pre-training a style model on a large-scale code corpus for this purpose.
4.2. Overall Performance
To evaluate NESTER 's performance, compare it against several baselines: a symbolic method (HiTyper), four deep learning models (TypeBERT, TypeWriter, UniXcoder, Type4Py), and eight LM-based methods (InCoder [1.3B, 6.7BJ. CodeTT [base, large], Code Llama, Llama3, Type- Gen). Table 1 presents the results based on Exact Match and Match to Parametric metrics. NESTER is competitive with the cop-performing deep-learning method, Type4Py, across nearly all Match to Parametric categories (Top-1 to Top-5). Specifically, NESTER scores 81.5% for Top-1, surpassing Type4Py's ' 80.2%. In Exact Match, NESTER exceeds Type4Py by 3.8% and 3.9% for arguments and returns, respectively. In Match to Parametric, NESTER outperforms Type4Py by 7.7% and 5.1%. Interestingly, NESTER lags behind Type4Py in Exact Match for both lo- cal and overall evaluations, due to Type4Py's s strength in handling short code segments, despite its limited reason- ing ability. However, NESTER 's performance is based on 7B 1B-parameter LMs, and we expect further improvements with larger models. We also compare NESTER with several LM-based approaches, including TypeGen, InCoder (1.3B, 6.7 6.7B), UniXcoder, CodeT5 (base, large), Code Llama (7B), and Llama3 (8B). NESTER generally outperforms these methods, achieving 70.7% accuracy for Top-1 Exact Match, surpassing TypeGen by 3.6%. It also exceeds TypeGen across all metrics, with a 5.3% improvement in Match to Parametric. This highlights the benefit of combining LM generative capabilities with the reasoning strengths of static rules for type inference.
>> TypeWriter (Pradel et al., 2020) applies a classification method for type inference. It segments code into identi- fiers, tokens, comments, and types, then uses a sequence-to- sequence RNN model (Hochreiter & Schmidhuber, 1997) for classification.
UniXcoder (Guo et al., 2022) is a pre-trained model for both code understanding and generation, applied to type inference tasks.
> Type4Py (Mir et al., 2021) employs clustering for type inference by mapping types into continuous space, allowing rare types to be predicted by analyzing cluster distances.
InCoder (Fried al., 2022) is code generation LM that treats type inference as cloze-style task, completing type annotations within code segments.
> CodeTS (Wang et al., 2021) is an LM for type inference, using the T5 architecture and fine-tuned on code datasets to generate type annotations with improved accuracy.
TypeGen (Peng et al., 2023) uses a generative approach for type inference via the cfain-of-thought method. It struc- tures the prompt into code snippets, type hints, questions, and answers to guide LMs in generating type inference responses.
Evaluation Metrics Following previous studies (Allama- nis et al., 2020; Mir et al., 2021; Peng et al., 2023), we adopt two evaluation metrics: Exact Match and Match to Paramet- ric. These metrics assess the proportion of outcomes that: 1) Exact Match: fully correspond to human annotations, for example, "Dict(int: int)" matches exactly with "Dict(int: int)" and "Dict(str: str)" with "Dict(str: str)"; and 2) Match to Parametric: meet the criteria of exact match, excluding type parameters, thus treating types like "Dict(im) (nt: int]" and "sict(str: str)" as equivalent to a generic "dict" under this metric. The experiment utilizes Top-1, Top-3, and 5 accuracy metrics to assess the performance com- prehensively.
4.3. Effectiveness of Neural Understanding
To explore how NESTER learns control flow via neural un- derstanding, compare it with TypeGen, focusing on the typing module, which humans use for optional typ- ing. Figure 5 shows the precision of NESTER and TypeGen across different typing categories: "Optional", "Union", "Pattern". "Match". "Defaultdict", "Type", "Deque", "IO". In common categories like "typing. Optional" and "typing. Union", NESTER achieves precision scores of 51.0% and 16.7%, significantly outperforming TypeGen (22.7%
6

---

Datdflow-Guided Neuro-Symbolic Language Models for Type Inference
Table The performance of NESTER along with the baselines under four types of variables of Top-1,3, 1,3,5 Exact Match (%) and
Match to Parametric (%).
Metrie|
Cat.
Approach
Top-1 (%)
Top-3 (%)
Top-5 (%)
Arg.
Ret.
Var.
All
Arg.
Ret.
Var.
All
Arg.
Ret.
Var.
All
ST
Hityper
8.0
43.5
65.7
52.4
8.0
43.5
65.7
52.4
8.0
43.5
65.7
52.4
TypeBERT
28.0
38.5
51.1
45.4
34.8
52.6
55.8
51.4
57.1
58.6
54.1
TypeWriter
53.3
52.8
61.1
60.7
65.8
65.3
DL
UniXcoder
55.0
49.2
35,9
40.9
66.9
64.6
42.1
49,0
70.6
69.8
45,2
52,4
Type4Py
66.5
56.1
82.0
76.6
72.0
59.2
83.8
79.3
73.8
60.7
84.3
80.1
InCoder-1.33
20.9
20.5
15.1
16.7
21.3
20.8
15.5
17.1
21.3
21.0
15.6
17.2
Exact
InCoder-6.7B 6.7B
24.1
42.0
18.7
21.9
24.6
42.7
19.1
22.3
24.7
43.1
19.2
22.4
Match
CodeT55bbse
51.1
57.6
21.7
30.7
59.3
64.4
28.0
37.4
62.0
66.9
30.7
40.1
(%)
--rrge
56.2
60.2
44.7
48.4
61.6
64.5
50.4
53.9
63.9
66.3
53.4
56.6
Naive-CL
33.5
22.1
32.2
31.7
48.9
35.1
46.2
46.0
52,8
36.4
49.8
49.5
LMs
Naive-L3
31.3
41.6
40.2
38.3
45.9
49.4
55.0
52.3
46.0
49.4
56.3
53.4
TypeGen-CL
61.8
58.0
69.7
67.1
71.5
61.0
76.6
74.5
75.9
65.3
77.4
76.3
TypeGen-LL3
56.9
52.2
60.5
59.0
69.8
65.6
73.6
72.4
71.5
68.2
75.3
74.2
NESTER-CL
70.3
60.0
72.2
70.7
75.6
67.7
77.5
76.3
76.5
69.5
78.5
77.3
NESTER-L3
58.7
55.7
71.9
67.8
66.2
65.9
78.4
74,9
66.7
67.7
79.5
75.9
ST
Hityper
8.4
52.7
70.2
56.5
8.4
52.7
70.2
56.5
8.4
52.7
70.2
56.5
TypeBERT
29.8
41.4
54.0
48.1
36.0
55,9
58.0
53.5
37.7
60.8
61.2
56.5
TypeWriter
54.4
54.1
63.4
63.5
68.8
69.3
DL
UniXcoder
61.8
44.3
49.3
72.3
76.0
51.2
57.6
75.0
80.1
53.8
60.4
Type4Py
68.0
59.0
86.2
80.2
74.1
64.1
88.3
83.3
75.9
66.3
88.8
84.3
InCoder-L.3B
22.8
18.7
19.9
23.3
19.1
20.3
23.4
23.3
19.2
20.4
Para-
InCoder-6.78
28.8
51.6
25.0
28.1
29.3
52.1
25.3
28.5
29.4
52.5
25.3
28.6
metric
CodeT5--bal
54.8
66.7
27.7
36.6
62.9
74.2
34.4
43.6
65.6
76.4
37.1
46.3
(%)
CodeT5-large
61.4
69.4
55.7
58.0
66.8
74.3
61.2
63.5
68.9
76.2
63.7
65.9
Naive-CI.
33.9
26.0
33.3
32.9
51.5
45.5
50.1
50.1
56.2
48.1
54.7
54.6
LMs
Naive-I.3
44.2
43.9
41.3
46.4
53.2
59.9
56.3
47.2
54.5
61.0
57.4
TypeGen-CI.
68.4
62.0
80.1
79.0
78.0
66.4
85.5
82,4
81.9
70.7
86.2
84.1
TypeGen-L3
64.9
59.5
68.9
67.2
78.5
74.5
83.1
81.6
80.3
77.5
85.1
83.6
NESTER-CU
75.7
64.1
85.3
81.5
80.0
73.8
90.5
87.2
81.7
75.8
91.5
88.2
NESTER-L3
65.5
63.8
84.6
79.0
74.9
76.4
91.0
86.6
76.3
78.3
91.9
87.8
1.0
1:00
4.4. Performance of High-Leveel Program Generation
TypeGen
To evaluate the accuracy of high-level programs generated by LMs, and to assess the potential of larger LMs for this task, we employ five widely-use models: Code Llama 7B, Llama 8B, GPT-3.5, ChatGPT-4, and GPT-4 on 100 Many- Types4Py samples. Three master's students specializing in Computer Science, each with five years of Python pro- gramming experience, are invited to evaluate the generated programs. Drawing on prior research in visual question answering (Hu et al., 2024), we developed evaluation crite- ria for each generated program. Human annotators scored the model responses based on: (1) Correctness: Is the pro- gram execution result accurate? (2) Explainability: Does the LM clarify the type inference process? (3) Factuality: Are all steps factually correct compared to the original code snippet? (4) Consistency: Are the results consistent across multiple executions?
Procision
0,21
0.4
0:20
122
2:28
b.11
0.64
Optional
Defdict
Dequue
Figure 5. Comparison between NESTER and TypeGen.
and 10.9%). This demonstrates NESTER's ability to synthe- size control flow. In rarer categories like "typing. Pattern" and "yp"", NESTER also excels with scores of 99,4% and 90.0%, compared to TypeGen's 92.0% and 36.4%. These results highlight NESTER's program syn- thesis capabilities and its ability to extract type information from control flow.
Table 2 shows the human evaluation results of five LMs across four metrics. GPT-4 and Llama 8B achieve the

---

Dudddle Neuro-Symbolic Language Models for Type Inference
Python
Case Study
Table Evaluation of high-level program generation accuracy
HitpeesonsehoFoonn
across different LMs.
def donniaad(/ngg, 14, rane):
nedts badia.grt, Sratarce()
Model
Acc. (%)
Exp. (%)
Fact. ( (%)
Cons. (%)
1f vol. se mones
Ropert
#ttptroeraaeeeeeee Commnnt (14[-243) sot Exptinec? inte milattasemseieee dacoddd uoc 873) nea "Neterrrrffalayy "oragnn"
Code Llama 7B
80
85
81
73
Naster HttpResponse
LLama 8B
96
88
96
100
GPT- mini
95
87
97
95
GPT-4o
87
95
91
92
GPT-4
96
87
97
96
Figure 7. Case study of a real-world type inference example using NESTER.
1.0
Def
Average Scores
0.8
1873
6.78
6.77
Usee
0.6
6.59
0.88
o039
We detail the results of this three-step analysis across nine categories, including "int", "float", "bool", "str", "bytes" "list", "tuple", "dict", and "user diseerdeeined", in Figure 6. From this figure, we can see an improvement in accuracy across all categories. Notably, in the step of inference Def, NESTER infers over 50% of the types, indicating that initial type inference is relatively easy due to the simplicity of the types. Furthermore, there is a significant performance im- provement from Def User compared the transition from User Usee. This suggests that the step-by-step inference is particularly effective in the early stages, but more static knowledge is incorporated, the types become increasingly difficult to infer. We also observe that certain categories, such as "tuple" and "user", do not show substantial im- provements from Def' to User. This can be attributed to the specific categories that are most likely defined or utilized during the Def stage.
0244 (.40
0.4
0.2
0.0
int
float
b0ol
str
bytes
list
tuple
dict
user
Figure 6. Iterative reasoning average accuracy of NESTER method across various data types.
highest correctness, indicating that high-level programs gen- crated by NESTER can be effectively produced by models as small as billion parameters. In contrast, Code Llama 7B achieves only 80% correctness, likely due to Llama 3's su- perior handling of tasks that combine natural language and code, while Code Llama focuses more on code generation. GPT-4-4o mini scores 95% in explainability but only 87% in overall correctness, suggesting strong generative ability but occasional inaccuracies. This is supported by factuality rat- ings, where GPT-4 and GPT-4o score highest at 97%, while GPT-4o drops to 91%. Notably, Llama 3 8B achieves 100% consistency, likely due to its lower temperature setting.
4.7. Case Study
To compare our methods with TypeGen, we present one type inference example from real-world scenarios in GitHub repositories, as shown in Figure 7. In this case, which fo- cuses on the Python function download, the return type needs to be inferred. TypeGen incorrectly reasons the return type as SoppResspooseNoooonndd In contrast, NESTER first generates a high-level program, incorporating control flow in if anaassis and type information for T1 and T2 in assign- ment Janalysi. It merges this information based on control flow, and during program interpretation, LMS evaluate if the control flow can proceed. Due to insufficient information, LMs default to allowing the flow to proceed. The assign- ment of HtipResponseNotoonnd to T1 and HupResponse to T2 is done using LMs, and static analysis reveals that T1 is a subclass T2, concluding with the final type HttpReseonse.
4.5. Effectiveness of Symbolic Parsing
To evaluate NESTER'S symbolic parsing, we focus on the types inferred based on regular expressions. The results, shown in Table 3, highlight NESTER's 's coverage across local, argument, and return variables in various type categories: "int", "float", "bool", "str", "bytes", "list", "tuple". "dict", and "set".. For local variables, types like "int" (54.7%) and "str" (57.4%) show good coverage, while others like "float" (25.3%), "bool" (29.7%), and "list" (33.0%) are also effec- tively inferred. More complex types, such "set" (4.3%), less covered but still show meaningful inference, con- firming the validity of symbolic reasoning. For return types, coverage is generally lower (around 3.4%), with function arguments showing minimal coverage (0.1%). This suggests that symbolic reasoning is less effective for return types and arguments due to the higher frequency of local definitions.
5. Related Work
Type Inference Type inference reduces the programming burden in dynamic languages by minimizing explicit type annotations. While static-rule-based solutions have been widely used, they often have limited coverage due to the complexity of dynamic languages (Chen & Erwig, 2016;
4.6. Effectiveness of lterative Inference
To assess the efficacy of NESTER's iterative inference, we conduct experiments on NESTER for each reasoning step.
8

---

Duddelow-Guided Neuro-Symbolic Language Models for Type Inference
Table Coverage analysis of symbolic rules across different type categories.
Type Category
int
float
bool
str
bytes
list
tuple
dict
set
total
Var.
54.7
25.3
29.7
57.4
5.5
33.0
2.1
0.6
4.3
32.2
Arg.
0.1
0
0
0.3
0
0.09
0.1
0
0.1
Ret.
11.1
3.9
2.2
3.5
2.3
2.2
3.1
0
0
3,4
Furr et al., 2009; Pavlinovic et al., 2021; Ke et al., 2024). In contrast, machine learning's 's adaptability has led to the adoption of leep-learning techniques for type inference. These methods are typically divided into two categories: the supervised approach (Allamanis et al., 2020; Pradel et al., 2020; Mir et al., 2021), which treats type inference as classification task, and the cloze-style approach (Wei et al., 2023), which frames it as a fill-n--the-blant task. The rise of language models like GPT-3, using a chain-of-thouge approach (Peng et al., 2023), has further enhanced reasoning and interpretability in type inference tasks.
6. Discussion
Our work adopts neuro-symbolic approach to type in- ference by decomposing the task into simpler steps. We further enhance inference by integrating both dataflow and symbolic information into LMs, allowing for more accurate and interpretable type predictions. Despite these clear ad- vantages, the practical integration of symbolic components introduces certain limitations.
A key limitation of our approach is its restricted applicability to larger-scale LLMs. Although NESTER, integrated with the symbolic rules on the Code Llama 7B, successfully inferred types that Code Llama 7B alone could not handle, larger models like Code Llama 70B are able infer some of these types without the constraints of symbolic rules, as models with larger parameters generally possess better comprehension and generalization capabilities. In future work, we plan to extend NESTER to LMs of larger scale. We also intend to adapt our approach to more locally deployable models, such as Microsoft's Phi series, to explore a balance between inference accuracy and deployment efficiency.
LMs for Reasoning LMs for reasoning enhance human understanding of problem-solving and excel in complex tasks like deducing formulas, solving logical puzzles, and analyzing data patterns (Wang et al., 2023a; Mizrahi et al., 2023; Biswas Talukdar, 2024). To unlock their full zero-shot potential, techniques such as prompt tuning, in- context learning, chain-of-though reasoning, and Retrieval- Augmented Generation (RAG) have been developed. While effective in simpler tasks, LMs struggle with complex rea- soning, often producing "hallucinations' or inaccurate out- puts (Tonmoy et al., 2024; Ji et al., 2023; Yao et al., 2023). In RAG, for example, LMs may be influenced by retrieved examples, leading to flawed reasoning. Compared human cognition, LMS still require significant improvement in their reasoning abilities. This limitation becomes particularly evident when the model size is reduced from hundreds of billions to tens of billions of parameters.
Another limitation is the difficulty in extending our ap- proach to other programming languages. Although we uti- lized NESTER to tackle the issue of dynamic type inference within Python, integrating it into Java presented significant challenges due to Java's strict syntax; for instance, Java requires explicit type declarations at variable initialization and prohibits null values without explicit nullable decla- rations (Lanzinger et al., 2021). Consequently, undertake comprehensive redesign when adapting the type inference system designed for Python to Java, ensuring that high-level programs are restructured to meet Java's 's syntactic and type compatibility requirements.
will
Neural-Symbolic Reasoning Neural-symbol learning combines deep learning's 's representational power with the logical rigor of symbolic reasoning to address complex cognitive tasks. In 2018, Manhaeve al. (2018) intro- duced DeepProbLog, integrating neural networks with prob- abilistic logic programming to enhance reasoning. VIS- PROG (Gupta& & Kembhavi, 2023) leverages LMS for visual reasoning tasks without task-specific training, highlighting their potential. For more complex visual tasks, Embed2Sym utilizes symbolic optimization via embedding space cluster- ing enhance reasoning efficiency (Aspis al., 2022). Fur- thermore, Manhaeve et al. (2018) extended neural-symbou learning to natural language processing using advanced tech- niques such as inductive logic programming. Despite these advancements, challenges persist in integrating deep learn- ing with symbolic systems, particularly in terms of scalabil- ity and robustness.
7. Conclusion
We have presented NESTER, new neuro-symbolic method for type inference. NESTER leverages LMs to acquire high- level programs through in in-context learning. Subsequently, it employs a program interpreter to execute the program. By harnessing the dataflow information within the program, NESTER facilitates human-like reasoning. Moreover, each interface is designed be extensible and offers good inter- pretability. Experimental results demonstrate the effective- ness of NESTER and highlight the usefulness of its major modules.

---

Dutiflow-Guided Neuro-Symbolic Language Models for Type Inference
Acknowledgements
modal large language models. Journal of Artificial Intel-
ligence Research, 4(1):176-195, 2024.
This work is supported by the National Key Research and Development Program of China under Grant No. 2022YFB4501400, the National Natural Science Founda- tion of China under Grant No. 62372199, and the Major Program (JD) of Hubei Province (Grant No. 2023BAA024).
Chen, S. and Erwig, M. Principal type inference for GADTs. ACM Siglan Notices, 51(1):41 428, 2016.
Chen, T., Bao, H., Huang, S., Dong. I.., Jiao, B., Jiang, D., Zhou, H., Li, J., and Wei, F. The-x: Privacy-preserving transformer inference with homomorphic encryption. In Proceedings of the 22nd Association for Computational Linguistics. pp. 3510-3520, 2022.
Impact Statement
The proposed framework, NESTER, adopts a neuro- symbolic approach enhance the type inference capabili- ties of LMs. By integrating haaflow-guided reasoning, our method tackles key challenges in type inference, resulting in more accurate and reliable predictions. The high-level pro- gram we introduce not only improves type inference but also expands to other areas of code intelligence, including code summarization, code generation, and potentially automated debugging. This approach is part of the community's 's on- going efforts in combining traditional software engineering techniques with the powerful capabilities of modern LMs. Additionally, our framework paves the way for developing more robust, interpretable, and scalable LMs for software development, ultimately improving both the efficiency and safety of code-related tasks, The broader impact of this work extends to a wide range of applications, from improving de- veloper productivity to advancing research in LM-assisted programming and automated software engineering.
Cunnington, D., Law, M., Lobo, J., and Russo, A. Neuro- symbolic learning of answer set programs from raw data. arXiv preprint arXiv:2205. /2735, 2022.
Das, B. C., Amini, M. H., and Wu, Y. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 57(6); 2025a.
Das, B. C.,, Amini, M. H., and Wu, Y. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 57(6):1-39, 2025b.
Fried, D., Aghajanyan, A., Lin, J., Wang. S., Wallace, E., Shi, F., Zhong, R., Yih, W., Zettlemoyer. L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. arXiv preprint S.Xiv:2204.05990 2022.
Furr, M., An, J.-h., Foster, S., and Hicks, M. Static type inference for ruby. In Proceedings of the 2009 ACM Symposium on Applied Computing, pp. 1859-1866, 2009.
Although NESTER, when integrated with symbolic rules Code Llama 7B, successfully inferred types that Code Llama 7B alone could not, larger models such as Code Llama 70B are capable of inferring some of these types independently, further demonstrating the potential of combining symbolic reasoning with large-scale LMs.
Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., and Yin, J. Unixcoder: Unified cross-modal pre-training for code representation. arXiv preprint arXiv: arXiv:2203.03850, 2022.
Gupta, T. and Kembhavi, A. Visual programming: Compo- sitional visual reasoning without training. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953-14962. 2023.
References
Allamanis, M., Barr, E. T., Ducousso, S., and Gao, Z. Typ- ilus: Neural type hints. In Proceedings of the 41st ACM Sigplan Conference on Programming Language Design and Implementation, pp. 91-105, 2020.
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 9()) 1735--7557780, 1997.
Hu, Y., Stretcu, O., Lu, C., Viswanathan, K., Hata, K., Luo, E., Krishna, R., and Fuxman, A. Visual program distillation: Distilling tools and programmatic reason- ing into vision-language models. In Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9590-9601, 2024. doi: 10. 111109/VVPR2273.222.00000
Arakelyan, S., Hakhverdyan, A., Allamanis, M., Garcia, L.., Hauser, C., and Ren, X. Ns3: neuro-symbolic semantic code search. In Proceedings of the 32nd Advances in Neural Information Processing Systems, volume 35, pp. 10476-10491, 2018
Aspis, Y., Broda, K., Lobo, J., and Russo, A. Embed2sym- scalable neuro-symbon symoolic reasoning via clustered embed- dings. In Proceedings of the 19th International Con- ference on Principles of Knowledge Representation and Reasoning, volume 19, pp. 421-431, 2022.
Jesse, K., Devanbu, P. T., and Ahmed, T. Learning type annotation: Is big data enough? In Proceedings of the 29th ACM Joint Meeting on European Software Engineer- ing Conference and Symposium on the Foundations of Software Engineering, pp. 1483-1486, 2021.
Biswas, A. and Talukdar, W. Robustness of structured data extraction from in-plane rotated documents using multi-
10

---

Dataflow-Guided Neuro-Symbolic Language Models for Type Inference
Ji, Z.., Yu, T., Xu, Y., Lee, N., Ishii, E., and Fung, P. To- wards mitigating IIm hallucination via self reflection. In Proceedings of the 23nd Association for Computational Linguistics, pp. 1
Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., and Stanovsky, G. State of what art? A call for multi- prompt Ilm evaluation. arXiv preprint arX1v:2401.00595, 2023.
2023.
Ke, W., Shang, Z., Luo, Z., Wang, P., Guo, Y., Liu, Q., and Chen, Y. Unveiling factuality and injecting knowledge for I1ms via reinforcement learning and data proportion. Sci- ence China Information Sciences, 67(10)220910, 2024.
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. A comprchensive overview of large language models. arXiv preprint arXiv:2307.0644 2023.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners, Ad- vances in neural information processing systems, 35: 22199 22213, 2022.
Nye, M., Tessler, M., Tenenbaum, J., and Lake, B. M. Im- proving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. In Proceedings of the 35th Advances in Neural Information Processing Systems, volume 34, pp. 25192-25204, 2021.
Koubaa, A. GPT-4 vs. GPT-3.5: A concise showdown. Preprints. org, 2023030422, 2023.
Oh, W. and Oh, H. Pyter: effective program repair for python type enrors. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 922-934, 2022.
Lanzinger, F. Weigl, A., Ulbrich, M., and Dietl, W. Scalabil- ity and precision by combining expressive type systems and deductive verification. In Proceedings of the 2021 ACM on Programming Languages, volume 5, pp. 1-29, 2021.
OpenAL
OpenAI models
GPT-4-vision.
https://opents com/ /ooom/reearch//// wys.evv-sstemmmmees 2023.
Liu, P., Yuan, W., Fu, J., Jiang. Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-355 10,1113550200 2023.
Pavlinovic, Z., Su, Y., and Wies. T. Data flow refinement type inference. In Proceedings of the 48th ACM on Pro- gramming Languages, volume 5, pp. 1-31, 2021.
Lu, J., Yu, L., X., Yang, L., and Zuo, C. Llama-reviewer: Advancing code review automation with large language models through parameter-efficien fine-tuning. In Pro- ceedings of the 2023 IEEE International Symposium Software Reliability Engineering (ISSRE), pp. 647-658. IEEE, 2023.
Peng, Y., Gao, C., Li, Z., Gao, B., Lo, D., Zhang, Q., and Lyu, M. Static inference meets deep learning: hybrid type inference approach for python. In Proceedings of the 44th International Conference Software Engineering, pp. 2019-201110 2030, 2022.
Lukasczyk, S., KroiB, F,, and Fraser, G. An empirical study of automated unit test generation for python. Empirical Software Engineering, 28(2):36, 2023.
Peng, Y., Wang, C., Wang. W., Gao, C., and Lyu, M. R. Generative type inference for python. In 38th IEEF/ACM International Conference on Automated Software Engi- neering. ASE 2023, Luxembourg, September 11-15, 2023, pp. 988-999 IEEE, 2023. doi: 10.1109/ASE56229. 2023.00031, URL ASE56229. 2023. 00031.
Manhaeve, R., Dumancic, S., Kimmig, A., Demeester, T., and De Raedt, L. Deepproblog: Neural probabilistic logic programming. In Proceedings of the 32nd Advances in Neural Information Processing Systems, volume 31, 2018.
Pradel, M., Gousios, G., Liu, J., and Chandra, S. Typewriter: Neural type prediction with search-based validation. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 209-220, 2020.
Mir, A. M., Latoškinas. E., and Gousios, G. Manytypes4py: A benchmark python dataset for machine learning-based type inference. In Proceedings of the 2021 IEEE/ACM 18th International Conference on Mining Software Repos- itories (MSR), pp. 585-589. IEEE, 2021.
Rathee, D., Li, D., Stoica, I., Zhang, H., and Popa, R. MPC-minimized secure LLM inference. arXiv preprint PPXiv:2408.03560 2024.
Mir, A. M., Latoškinas, E., Proksch, S., and Gousios, G. Type4py: Practical deep similarity learning- baaning-based type inference for python. In Proceedings of the 44th Inter- national Conference on Software Engineering, pp. 2241- 2252, 2022.
Ray, S. Samsung bans ChatGPT among employees after sensitive code leak. https: http://wwwwww//w.///ttttttttttts com/ sites/ cites//si.addtyyaaa//033333
2023.

---

Dataflow-Guided Neuro-Symbolic Language Models for Type Inference
Siddiqui, K. A. I. and Kellogg, M. Inferring pluggable types with machine learning. arXiv preprint arXiv:2406.15676, 2024.
elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Srinath, K. Python-the fastest growing programming lan- guage. International Research Journal of Engineering and Technology, 4 4(12):35--357, 2017.
Wei, J., Durrett, G., and Dillig, Typet5: Seq2seq type inference using static analysis. arXiv preprint arXi::2303.0955 2023.
Temsah, M., Jamal, A., Alhasan, K., Temsah, A. A., and Malki, K. H. OpenAl ol ol-preview vs. ChatGPT in health- care: new frontier in medical AI reasoning, Cureus, 16 (10), 2024.
Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T. Next- GPT: Any-to-any multimodal LLM. arXiv preprint arXiv:2309.05579, 2023.
Yao, A. C. How to generate and exchange secrets. In Pro- ceedings of the 27th Annual Symposium on Foundations of Computer Science (SFCS 1986), pp. 162-167. IEEE, 1986.
Tonmoy, S., Zaman, S., Jain, V., Rani, A., Rawte, V., Chadha, A., and Das, A. A comprehensive survey of hal- lucination mitigation techniques in large language models. arXiv preprint arXiv:2401.0113/3 2024.
Yao, J., Ning, K., Liu, Z., Ning, M., and Yuan, L. LLM lies: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469, 2023.
Wang, K., Ren, H., Zhou, A., Lu, Z., Luo, S,, Shi, W., Zhang, R., Song, L., Zhan, M., and Li, H. Mathcoder: Seam- less code integration in Ilms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.0333// 2023a.
Zhang, J., Chen, B., Zhang, L., Ke, X., and Ding, H. Neural, symbolic and neural-symbon reasoning on knowledge graphs. Al Open, 2:14-35, 2021.
Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. D. Hypothesis search: Inductive reasoning with language models. arXiv preprint wrXiv:2309.05666. 2023b.
Zhang, X., Du, C., Pang. T., Liu, Q., Gao, W., and Lin, M. Chain of preference optimization: Improving chain- of-thought reasoning in LLMs. In Proceedings of the 38th Advances in Neural Information Processing Systems, volume 37, pp. 333-356, 2024.
Wang, Y., Wang, W., Joty, S., and Hoi, S. C. Codet5; Identifier- aware unified pre- pre-trained encoder-decoder mod- cls for code understanding and generation. arXiv preprint arXiv:2/09.005: 2021.
Zhang, Z., Chen, C.,, Liu, B., Liao, C., Gong, Z., Yu, H., Li, J., and Wang, R. Unifying the perspectives of NI.P and software engineering: A survey language models for code. arXiv preprint 23Xiv:2311.09988 2023.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F, Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting
12

---

Dutiflow-Guided Neuro-Symbolic Language Models for Type Inference
A. Preliminaries
This section presents several preliminaries essential to our work, including LMs, type inference, and neuro-symbolic reasoning.
A.1. Language Models
In the past few years, LLMs such as the GPT series (Koubaa, 2023) and Llama series (Das et al., 2025b) have made significant progress in text and code generation (Naveed al., 2023: Das et al., 2025a). LMs aim to predict the probability distribution of the next token based on given sequence of preceding tokens. Let s denote sentence or code sample tokenized into sequence (W1, W2, ..., w The LM estimates the joint probability p(s) as the product of conditional probabilities:
ss) II p(ws W1, Wa..... 1.-1)
Training is performed on a large corpus D = (181,8,--- | of size N, using the negative log-likelihood loss function:
(l) = Σ sED
log W1, u2, ...,, 1;;1;;)
where I represents the model parameters. Once trained, the language model can generate subsequent tokens during inference by iteratively selecting tokens with the highest predicted probabilities from the vocabulary.
The emergence of LLMs is transforming the learning paradigm from the traditional "pre-train and fine-tune" to "pre-train, prompt, and predict" (Liu et al., 2023). Instead of extensively fine-tuning LMs for each downstream task, tasks are now reframed using textual prompts to align with the models' original training objectives. Various prompting techniques have been developed fully leverage LMs' capabilities.
To maximize the potential of LMs, researchers have developed various prompting techniques that reframe tasks to align with the models' original training objectives. Zero-shot prompting (Kojima et al., 2022) leverages well-designed prompts to guide L.ss in handling new tasks without additional training data, such as generating class hierarchies in game design or solving math problems. In-context learning enhances task comprehension by providing a few input-output examples, enabling the model to infer solutions based on similar cases. For more complex reasoning, Chain-of- Thought (Co'T) prompting (Wei et al., 2022; Zhang et al., 2024) breaks down problems into step-by-step reasoning processes, improving accuracy in tasks like arithmetic. Finally, Least-to-Most (LTM) prompting further refines CoT by decomposing difficult problems into progressively simpler subproblems, ensuring robust and scalable reasoning. These techniques collectively optimize LM performance across diverse applications.
A.2. Type Inference
The type inference task studied in this work is to predict the type of an identifier based on its relevant context within a code snippet. In this paper, we focus Python, dyyamically-typed language, which is characterized by untyped programs but can utilize existing deterministic type system. Given code snippet c, let (x. E 11... V} represent the set of V distinct identifiers within that require type assignment. Let p(T: ctx:) denote the probability distribution(may not probability distribution) over types for an identifier ze, determined by the context cta of the code. Our objective is to develop a type inference model that can input an entirely or partially unannotated code snippet e and output a probability distribution of types for each missing annotation. Typically, the prediction space (cc consists of two components: the of all user-defined types (classes/interfaces) declared within e and fixed set of commonly-used library types. Let T represent the typing environment of the code snippet c, and T + indicate that the program is well-typed according the deterministic type system, with types for identifiers specified by T.
LM-based Type Inference Recently, as the LMs have achieved substantial progress in content generation, including text generation (OpenAI, 2023), code generation (Lu et al., 2023), and video synthesis (Wu et al., 2023), several studies have leveraged the capabilities of LLMs for type inference (Siddiqui & Kellogg, 2024; Peng et al., 2023). The key of LM-based type inference approaches lies in framing the task as a generative one by designing a natural language prompt. As shown in Figure& 8, the prompt describes the given code and identifies the relevant identifiers for type prediction. Such prompt
13

---

Datdflow-Guided Neuro-Symbolic Language Models for Type Inference
Question
filter shape(scene, cylinder) relate(behind)
High-level Program
A
Answer
How many cubes that behind the cylinder large?
Decocer
filter_shape (sceee cube) filten_sizer sitten_size(ccene, large) (scen())ene)
Decoder
Program Interpreter
Image
Size
Shape
Location
Small
Cube
(1,3,-5)
Encodar
Decoder
Large
Cylinder
(-1,2,44)
Structural Scene Representation
Encoder + Decoder
..*
Pigure Neuro-symbolic visual question answering.
can effectively guide LMs to perform type inference. TypeGen (Peng et al., 2023) represents the current state-of-the-ar LM-based type inference, employing a specialized prompt designed around ICL. The TypeGen prompt is structured as [Code, ICL EXAMPLES, TYPEHINTS], where ICL EXAMPLES refers to demonstration examples used for ICL, and TypeHINTs denotes prediction hints extracted through prior static analysis techniques. Additionally, TypeGGnn incorporates CoT reasoning enhance the accuracy of type inference. The CoT prompt in TypeGen is structured as follows: "First, the is assigned from string. Therefore, the type of the variable error messaag is 'str'.
variable
A.3. Neuro-Symbolic Reasoning
Neuro-symbole reasoning is innovative approach that en- hances the interpretability of tasks by breaking them down into manageable subtasks (Gupta & Kembhavi, 2023; Cunnington et al., 2022; Nye et al., 2021; Zhang et al., 2021). )). Specifically, divides the solution function, denoted as f, into two distinct phases:
Prompt
Python code:
error message 'Ensure this value has at most 64 characters'
Type hints:
Aecounts Login... Laairrfrrr_ogiffoomeetsss.
K Inference Exomples
Question:
f hag
(1)
What the local type of variable message??
Answer:
the variable error_ __sssagee assigned Therefore, the type of the variable error, message is 'str'. Python code:
CoT
Here, 9 represents perception network that processes raw data through human-like constraints to establish a set of conditions for solving the problem. h, on the other hand, functions as a reasoning model that analyzes the raw data to delineate logical inference pathway, leading to the solution. Neuro- symbolic reasoning aims to optimize the learning of these two functions identify the most appropriate f-
'Unauthorized"
Type hints:
Test item
Question:
What the local type of variable message?
(To generated]
The neuro-symbolic reasoning methodology has demonstrated significant promise across a range of applications, such as visual reasoning (c.g., visual question answering) (Gupta Kembhavi, 2023), mathematical problem-olllsssss (Wang et al., 2023b), and code search (Arakelyan et al., 2018).
Figure An examplar prompt of LMs for type inference (Peng al., 2023).
An Example of Visual Reasoning Figure 9 shows an example of neuro-symbolic visual question answering, the goal of which is to derive an answer to a specified question by analysing the provided image. Consider the query "How many cubes that are behind the cylinder are large?". For the perception function g, the image is processed into a structure analogous to dataframe, which serves as static information for addressing the question. As for the reasoning function context-aware model to parse the question text into high-level program. This model identifies the key terms of the question specifically, cube, cylinder, and large-whiee are utilized as parameters for the high-level program. The program then generates a sequence of steps to address the question using the provided APIs, such as filter shape and count.
employs
Finally, this high-level program executed by a program interpreter, utilizing the database created earlier by the perception function g before producing the final result answer 2. noteworthy that if errors occur in the high-level program, the program interpreter will randomly sample an answer from all possible outputs of the final module, In this case, the sampling
14

---

Datdflow-Guided Neuro-Symbolic Language Models for Type Inference
will be limited a numerical range that is contextually appropriate for the query, such as the number of cubes depicted in the image.
B. Details of High-Level Program Generation
B.1. High-Level Program Definition
High-Level Program Definition We define the structure of a high-level program formally, starting by modeling the original program as a flow graph G = (V, E),, where represents nodes and E' represents control flows. intra-procedurally. high-level program is represented as a flow graph G" (V', EE), where each node V in G corresponds
smaller flow graph G" (V",,, F²) within G'. Each node in V is defined by function F G G" that selects subgraph from G and represents it node in G". This ensures that UV' G and that for any two nodes V/: 1 V', U2 Nej = 0. We define control flows within V' as Sub- Control Flow (Feub, Frub, ,..., Frub), and edges E' as Main-Control Flow (Fmain), reflecting their hierarchical relationship.
B.2. Prompts for High-Level Program Generation
Local Type Prompt The local type prompt focuses transforming Python code that involves local assignments pressions. The goal is to use the transformation API Ass Anment. Anaalyiis () for assignments and If If-Analysis ()
for conditional statements to generate high-level program description for each line of code.
Example:
Code:
urlpatterns [path about_projett views. index, name= name='video" )
High-level Program: urlpatterns Assignment_analyee ([path (" aaaabbbo_tppoo phoout_project,, views. index, name=' nime='video" ])
video'
Code: buf self.getvalue()
High-level Program: but = Assignment anallssttnaaaliisssseeeee (self (self.getvalue
Code:
val is None:
app_name 'about'
High-level Program: If if_analysis (val, None, is)
Assignment aasighment_analysis about'))
Codo: If val is None:
123
else:
a 'abe'
High-level Program:
If_ it_analysis (val, None, is)
Assignment ansignentttanalsiisssss
analysis (123)
if_analysis (val, None, is not)
a Assignment_analysii. ('abc')
Code:
cetrics_to_rre return ()
if not www.feerror__aaaayeeeeee and name starsswtth
if is_empty_metric (metric):
if isinstance (metric, Categoricalaccuracy) High-level Program:
basiinment_anallyisst
If if_analysis (not arr.rerorraaalysii and name. startssthtt
If if_analysis (is_empyy_metrie (metric))
It analysis (isinstance (metric, Caterar)
15

---

Dutdflow-Guided Neuro-Symbolic Language Models for Type Inference
Arg Type Prompt The arg type prompt is designed for analyzing functions and arguments. It the Funct ion Analysis API for functions and the Argument analysis, () API for specific arguments values passed into functions, This prompt is intended to break down function definitions and their corresponding arguments. It is useful for situations where the focus is on understanding function signatures and the arguments they receive.
This kind of analysis particularly valuable real-world development, especially when dealing with complex codebases or performing code reviews. By thoroughly examining argument types, developers can significantly reduce runtime errors and improve code robustness. Additionally, structured analysis like this helps clarify function's 's intended behavior, preventing potential logical flaws. When combined with proper documentation, the benefits multiply-cleee doestrings and strict type hints are, after all, the foundation of high-quality code.
Example:
Code:
def boot (self, container):
provider container. (settings. Pwaeeiingg.Prps..... PropS.DD__RROIEEEE. PROVIDER)
High-level Program:
function_Analysil (boot (self, container))
Argument_Anal (provider container.ged container.ger(settinng (settinggs.. Props.
Code:
def init_ value)
self.value = value
High-level Program:
Funttion__Anllysiii_intt (self, value)) Tteumment_nnallsiisself....
value)
Code: def
sareee__odee___e mooeeereetoeerreeeee (restorer, convenience_methodl placeholdez__mdee/
crained_mocel. ckpt_id, capsys)
b_.check_oog (restorer, ckpt_id, capsys)
High-level Program:
runction_Analysis coret__oovenieece_mmeeeeeee
(restorer,
convenience_metheee alaceholder_modde trained_modell ckpt_id, capsys)
Argument_Analyste chhccccheeeh_eeeee (restorer, ckpt_id, capsys))
Code:
def got_mods_mani [ag. (manifest_url):
if n == 1:
return json. Loads (et_deegegrreesss__ehhocc (manifest_url) .tett)
High-level Program:
Punstioo_Annalystin (get_mods_man) fest (mani fest_ur1))
If If_Analysis (n--1) Ceruuent_AAaayyae (return json. Loads wwt.eeeeeuueeeeeeeeeeeeeeeee (manifest_ url) .text))
Code:
def save_mod1 (manifest_url, mods_details, mods_path): if mod_ids, modlines [modline] = [mods_details [mtd_illmmlmmdii]]] directorry_amee) for mod_id in mod_ids]
(el_ids.differe. difference (set (mods_details))
High-level Program:
Function Function_Analyste Analysis (save _erereeddiieessse (manifest_ure fest_url, mods details, mods_path).
If S_AAnalysis (mad_ide www.iidd....deeeee...ee.ee..eeee difference (set (meds moeds_details))
Argument Argument_Analysiis (modl ines [modline] = [mods_detaill] (mod id] d'directory_name! for mo in ids]
Return Type Prompt The return type prompt used for analyzing functions with return statements. The prompt uses the Return_Anal Analysis() () API to describe the function's return value, and Combine ne () is used for combining multiple return values if the function contains more than one conditional return. This type of transformation focuses on the function's return logic and how the return values are constructed based on conditions.
Understanding return behavior is crucial for debugging and ensuring predictable function outputs- after all, inconsistent return types can lead to nasty surprises down the line. By systematically analyzing return paths, developers can catch edge
16

---

Datdflow-Guided Neuro-Symbolic Language Models for Type Inference
cases early, enforce type safety, and maintain cleaner control flow. Whether function returns single value, multiple conditional results (r1, F2: m. Fn), raises exceptions, the Return Analysis API helps document and verify expected behavior.
In practice, this means fewer "Why is this function returning None?" moments and more confidence in code reliability. The relationship between input parameters and output values becomes more transparent when return types are properly analyzed.
Example:
Code: return TestApp High-level Program: Return_Typel Return_Anal Assis (TestApp (app
Return_Type Return_lypel
Code:
return json. Loads (get requests object (manifest_ url) .tett)
High-level Program:
Return_Typel Return_Analysis (json.Icads loads rget_reeuestt (rani .text)) Return_Type Return_Typel
Code: if == 1:
return mrossddttalls High-level Program:
If Is_Analysis (n == 1)
Beeurn_Typel Return_nnaaaaaeee (mods details) Return_Type Typel
Code: if code !- 0: return False if counter return True
High-level Program:
If_Analysis (code != 0)
Return_Typel Return_Analysis (False)
If If_Analysi (counter 3)
Return_Type2 Oeturn_Analyyee (True)
Return_Type Combine (Return_Typel, Return_Type2)
Code:
if payload.get (' return json_success ((' full_name": fu
('type') = 'auth':
'email'; user_profile..mai]. rid'; user_profile.id))
topic is None:
if topic None:
if content is None:
return jscn_succes succ_ss
High-level Program: g't typp''') 'auth')
If_Analysis
Return_Typel deeturn_Analysie (jstssisssssnnioo (json_success success name user_profile, full name, 'email': user_profile. If If_Analysis (topic is None)
rid': sser_profile.idd))
If if_Analysis (topic is None)
If Analysis
is None)
Return Typee Return Analysis (json_ success s ())
Return__yppe Combine (Return Typel, Return Type2)
A line of the high-level program, or a program step, consists of the name of an API, the API's input argument names and the API's output variable name. To aid LM generation, we use descriptive API names (e.g., if_anal aanalysis sis, return_ana neturn.analysis) argument names (c.g..lr, 1.0, >), and variable names (e.g., T1, T2) to allow LM to under- stand the input and output of each API. The LMs need identify the locations of these statements (e.g., conditional and
17

---

Dataflow-Guided Neuro-Symbolic Language Models for Type Inference
return), fill in the corresponding identifiers from each line arguments for the respective APIs and create new variable name output.
Through this method, we gradually guide from the simplest condition to more complex structures, increasing complexity step-by-step, which is in line with the "least-to-most prompting" instructional strategy. This approach helps the model (or learner) to gradually build up complex logical expressions, enhancing the accuracy and reliability of understanding and execution.
B.3. Details of Supported Modules in NESTER
NESTER currently supports four modules, including the If Statement Module, Assignment Module, Argument Module, and
These modules are designed to implement fundamental constructs and operations of high-level programming languages, facilitating the processing and execution of code within the interpreter. Each module has specific responsibilities and execution workflows, detailed as follows:
If Statement Module: This module handles conditional statements. It first determines the truth value of the condition expression through the condition iin. onjjungmentttt method. If the condition is true, the execute e method will carry out the code within the conditional block; otherwise, it will skip it.
> Assignment.Moone Module: Responsible for handling variable assignment operations. This module parses variables and expressions in assignment statements using the symboli c. parse method, then calculates the value of the expression and updates the corresponding variable in the state dictionary through the execute method.
Argument Modulee: This module manages the passing of arguments in function calls. During the execute e process, it parses the incoming arguments and calculates each parameter's 's value based on the current state dictionary, subsequently passing these values to the respective function or procedure.
& Return Module: Handles the return operations from functions or methods. By parsing return statements using the symbol 1 i c_pa rse method, this module calculates the return value in the execute method, updates the state dictionary, and returns the result the caller.
C. Details of Program Interpreter
C.1. Details of Symbolic Parsing
In the high-level program interpreter, we denote the parameters (always statement) received by the interpreter as S. To determine the type of S, we first use symbolic parsing to classify the input T.,, requiring type annotation. Figure 11 shows the denotational semantics of regular expressions used for symbolic parsing in this paper. These include simple types like int, float, bool, str, and bytes, as well as generic types such as tuple, list, and dict. Finally, if has not been generated by an LM, we parse it to identify 2, for iterative type inference.
Specifically, for statement S, we first use regular expressions to classify the type of input z,.. For instance, the statement return "Odd" is processed by our regular expression and recognized type str. If our designed regular expression fails to recognize the type, we proceed to parse it, such as in the statement return a + b, which we identify as two identifiers a and b.
C.2. Details of CGDG
Coarse-Grained Dataflow Graph (CGDG) To build a bridge from a non-structured code representation to structured intermediate code graph representation, a Coarse-Grained Dataflow Graph (CGDG) is introduced in NESTER framework. Compared to traditional Concrete Syntax Tree (CST) and Abstract Syntax Tree(AST), the CGDG has at most three hops for each identifier, resulting in a more concise graph structure. The CGDG is constructed from the CST of the target program. For an identifier Ne, we represent these three hops using three types of nodes: Def(r.). User(r,). and Usee(z,). while the edges are directed to capture the relationships between them.
For an identifier 2., Def(z,) represents the definition of 2u within the code snippet This includes all top-level assignment statements where T. appears on the left-hand side. Let Ue represent the set of all code constructs, i.e,, the individual nodes in the CST that correspond to elements such as variable declarations and operations. User(x,) includes all con- structs u in which the identifier I, is used for computations or condition checks. We define User(x,)) = {u E U
18

---

Duddel Neuro-Symbolic Language Models for Type Inference
If Statteeett Module():
class Assignment Nodule():
def Init (self):
def init _nit__seef))=
load trained model; nove GPU
load a trained nodel; move to GPU
def condition condition_judgent list input expression/variabl
step: str):
def Synbolic_pareeee parse(self, step: str):
parse step return
input values/varia /aariable nanes/operaton and output the type on variable nane
nanes/operator and output a bool value of the condition
def execute(self, step: str, state: Dict):
def execute(self, step: str, state: Dict):
step query table(query) inguts, input_var__anne
inputs, input_vare Sas__var__nneee _aness output_var_mat nane self. self.parse(step)
sutput_var_name - self self- self.parse(stop)
get values input variables from state for in input_ sarpt__a___aaee
# values of Input variables fron state
for var_nane in input var_nanes:
inputs.
I perform conputation using the loaded LLNs type sone sone_cooooueeteeeeeeeeeooooccoss (inputati
perform computation using the loaded LLMs
query, type cone_cooputaaiion((ppttt)))
# update state
# update state
state[output_ sar_name] output return output
state[output, var _aneaane output
return output
class Argument Argunent_Moduleee Module():
class Return Module():
def init (self):
def init (self):
load trained model;
GPU
load trained nodel;
GPU
def execute(self, step: str, step (uery_table(query) inputs, input_var_nanes output_var_mat self, parse-aassssss))
Dict):
synbolic parse(self, step: str):
I parse step and return 1ist of input velues/variabe nanes/operaton and output type variable nane
get values input
execute(self, step: str, step query ttttabbleqquryy inputs, input_var_neat output_vva__nne self.parse(step)
Dict):
for var_name in input_vare vaa__aanes: arnuts.appendsttateeee
# perform computation using the loaded LLMs
get values input variables from state in input_ sastt_aa___aeeee for poputssappen(stttte(va...mm
query, type aam__ccopuaatiininnuts
update
state[output___ sar_ya___naa ____nane] = output return putput
# per-oom computation using the loaded LLNs query, type seme__conutationninpeee
update state
state[output__ var_name] output return autput
Figure 10. Modules currently supported in NESTER.
[int]] = {u w = s - d, E {+, -, d E [digits]]
[float]] (w w = s .2, 8 E (+, )), di d2 E [digits]]
[string] {w w 81 ... 8n 8; E Ue), n 2 0)
[bool] {w
"Truc" V
"False")
[bytes] = w
MsI...sh sn), E (a U c), n 2 0}
[list]] = {w w [[s1, 32]] 82 E [string]] }
[dict]] = {w w = {[ s2]} 91,32 E [strin]]]
[[set]] = [w w {s1, 82, .,.
8n), 8; E [[strin]] 2 0)
[tuple]] [w ((s1, 82)), 81, 82 [string]]
digitss] {w
didg ... dn. d, [digit]. 2 1)
[digit]] {w w E 1, 4, 6,6, 9,9)
[<] {w w is valid escape sequence]
[[]] = [w w is any valid character)
Figure 11. Denotational semantics of regular expressions applied in NESTER.
I, is actively utilized in u). where active utilization indicates that the identifier x, is involved computational opera- tions or evaluations. Usee(x,) contains not just the direct users of øg, but also anything that is used in the user context Usee(x..) = U UseUser(z.) {el involved in u's computation), denotes variable actively engaged in u's operations.
19

---

end Dutielow-Guided Neuro-Symbolic Language Models for Type Inference
Algorithm 1 Coarse-grained Dataflow Graph Generation from CST
Input: CST of function, fumc_est
Output: Coarse-grained dataflow graph of the given function, eg
for n E fume_cst do if n.type = Token n.tokenT Identifier then if (a) then cy.aodNooe(De((()) reliir))
end
end
eg do
est_node + ge(CSTNode(z) CasCSSUUse(ccst.nooo then user uode + ge(CSTUser(cst node) if Def( _odee) E then (Deee() Del(user..node).0 cst-user-node <- gelCSTNNode if hasCSTTSeersss..serrrr
then
usee_mode + getCSTUserrest user wew.eusserrss mode) if usee_node cg then
| cg.addEdge(De((mm) Def 1)
else
end (gaddNNoe(Usee(()) use..nooe..ine "g.addEdge((eefll Usee(n),
end end else
eg (p.ad(NodoUUser(())). user node number) (g.add) User(n),
end
end
end
1.0
0.60
1.0
Code
Dataflow
Axagemmun,
prrset
(alculata(a): a.keys()
Assign
m
Consieea
Trow 10 15003
Keyword
0-11
str(x) "is result"
"exaxple result"
02
0.2
return y
re
TDG
0.0
Asogrmmnt
Roywords
Compleis Cids
0.
Look
Argument
(a) Code segment.
(b) Slice method effects on an
(c) Slice time each method.
Figure 12. Impact of code slice type inference performance.
CGDG Construction Given the input code's CST, NESTER constructs a CGDG for each function following the logic in Alg. first identifies all identifier definitions Def(x,)) and builds CGDG nodes by traversing the CST. The RedBaron library is used to visit each CST node. When an identifier node is found (line 2), NESTER creates corresponding CGDG node and preserves the line number (line 4). The process repeats recursively until all identifier nodes visited, reaching the CST's leaf nodes (line 6).
After node instantiation, NESTER traverses the CGDG. As each node is encountered, it identifies the corresponding CST node and checks for identifiers in use (line 9). If a node uses another, NESTER checks for an existing matching node the CGDG. If found, an edge is added (line 12). Otherwise, a new User(n) node is created, and an edge is added from Def(n) to User(n) (line 24). Similarly, Usee(n) nodes are created necessary, and a Usee edge added from Def(n) to Usee(n) (line 21).
D. Extended Analysis: Rethinking Type Dependency Graph (TDG) for Neural Type Inference
To the best of our knowledge, TypeGen (Peng et al., 2022; 2023) is the first to explore LMs for type inference using a Type Dependency Graph (TDG) to model variable type dependencies and static analysis for code slicing to improve efficiency. We argue that the TDG may be too detailed, impacting inference efficiency. To test this, we design an experiment to assess the TDG's role and the granularity of code slices. As shown in Figure 12a, code can be sliced different granularities,
20

---

method is needed to better Dataflow- Neuro-Symbolic Language Models for Type Inference
Sef COntOnetINN, ST,
def sninleedine,, 14, rans)
Prtseespone
select
redas getild)
Recommended type
SF VAL de cone:
returr insptesponnanit (utiaralioon)
of va3 ie Hore:
retum aetpoeepaaaicttee contest docadol
recE sereeeeee torigin"
retponepernaavee. decodat
select
Taterrr "alisr)) "arigin"
retaan
retur
toltres 1o, reed)
Reccisercas ced type
65
pertii-grrttt Tratance LA Corneett
click
ougeetia
@ptioreaaaar)
Sype inference procezs
VLF-2/33
(H0C-26)) fourd, (epiradl))
wwwpr//aaaaaatttttetaaaoooo (pptred))))
trJnnllaasaoaa hone, +)s
orage"
click
Le raturr waapysssal)))))
Detailed type inference process
Figure 1.3. Example from case study I solved by the NESTER plugin. You can directly select the type recommended by NESTER to complete the annotation. Alternatively, you click the ">' icon view the detailed type inference process.
such by assignments, keywords, or TDG. These slices then input into LM (Code Llama 7B) predict variable, argument, and return types.
Figure 12b shows that slicing by Assignment, Keywords, and TDG all improve accuracy compared to using the full code. The TDG method achieves the highest accuracy, but for local types, the Assignment method is nearly as accurate ( (69%). For argument types, the Keywords method reaches 70%, close to the TDG's performance. Figure 12c compares slicing times for 50 Python annotations, The TDG method takes an average of 0,95 seconds, while the Assignment and Keywords times faster. These results show that while the TDG performs
methods take just 0.02 and 0.05 seconds, respectively- well, it sacrifices time, even underperforming simple keyword searches. We suggest that LMs don't need such detailed static analysis; coarser slices can still provide good results with greater efficiency. Moreover, integrate LMs with static analysis to avoid reducing the potential of combining both.
E. Implementation
To show the interpretability of our method, we developed a plugin on Visual Studio Code (VSCode). Any developer can download the plugin their local system. This plugin crafted assist developers in obtaining type inference results and gaining deeper understanding of the type inference process. We use simple regular expressions to generate high-level programs, which can not handle complex code structures, but fortunately can handle most simple code, thereby facilitating seamless interaction.
Figure 13 depicts a screenshot of the VSCode plugin. The plugin's interface is designed to intuitive and interactive, featuring an arrow-based type annotation mechanism. When developers append the >' symbol after a function definition for type annotation, the plugin promptly activates a feedback interface. This interface displays five potential types associated with the annotated identifier. For instance, download is predominantly identified an HttpResponse type. Additionally, 'click' option is available beneath the recommended type, allowing developers view the inference process if desired. Upon clicking, detailed high-level program response is displayed, which elucidates the concrete reasoning steps of the type inference process.
21


Reviews:
[{"review_id": "IlKXlMtWxm", "reviewer": "Reviewer_ugzA", "summary": "This paper proposes Nester, a neurosymbolic tool for performing type inference with LLMs. It uses LLMs to generate high-level versions of the code in question, and determines the return type of a function by analyzing its data and control flow. It outperforms existing SOTA type inference tools for simple as well as complex types.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback. \n* * *\n**Q1:** What is the fundamental reason why Nester couldn't just be run with a larger LLM?\n\n**A1:** Nester can work with LLMs of various sizes. In this work, we use a modest-size LLM as we target scenarios where users prefer running models locally—on a developer PC or cluster—to avoid sending data to the cloud due to security and privacy concerns. In the final version, we will demonstrate how Nester can be integrated with a larger LLM (70B). \n* * *\n**Q2:** Does Nester only work with individual functions and local context, or can it draw from contexts outside the scope of the function?\n\n**A2:** Our current implementation performs type inference on individual functions and local context. This can be easily extended by - for example, inlining the callee functions. We will provide a discussion. \n* * *\n**Q3:** Other statistics about the functions input to Nester, such as the number of lines of code in each input.\n\n**A3:** Agree. The distribution of lines of code in our dataset is as follows.\n\n| Line Range     | Count  |\n|---------------|-------:|\n| ≤ 5 lines     | 68,860 |\n| 6–10 lines    | 13,108 |\n| 11–15 lines   | 2,340  |\n| 16–20 lines   | 557    |\n| > 20 lines    | 340    |\n\nWe will provide this information. \n* * *\n**Q4:** Have you thought about evaluating Nester with small language models like Microsoft's Phi family?\n\n**A4:** Thanks for your comments. Based on your suggestion, we simply evaluate Nester on 100 instances from ManyTypes4Py using top-1 predictions with Phi-3.5-mini-instruct, due to the limited time slot. The results are as follows:\n\n| Method             | Arg.  | Ret. | Var. | All  |\n|--------------------|------|------|------|------|\n| Exact Match       |      |      |      |      |\n| Naive            | 27.3 | 72.7 | 39.4 | 40.4 |\n| Nester           | 22.7 | 72.7 | 51.5 | 47.5 |\n| Match to Para.   |      |      |      |      |\n| Naive            | 31.8 | 72.7 | 43.9 | 44.4 |\n| Nester           | 22.7 | 72.7 | 60.6 | 53.5 |\n\nFrom this table, we can see that Nester outperforms the naive method in overall accuracy. However, it does not surpass the naive method in argument (Arg.) and return(Ret.) type predictions. This is likely due to the small number of return and argument types in the 100-instance dataset—if a single inference fails, it can result in a significant deviation. We leave further investigation of Nester on Microsoft's Phi family for future work.\n* * *\n**Q5:**  In Figure 3, for line 6 in the high level program, why is only T2 and T3 being combined? \n\n**A5:** In the paper, we simplified the presentation by combining T2 and T3 in Figure 3. In actual execution, however, the interpreter processes T1, T2, and T3 separately and checks whether the returned variable is empty. For this example, the runtime check confirms that T1 is unreachable. We will clarify.\n* * *", "claims_and_evidence": "I believe the claims are clear and the paper shows the evidence to support them.", "methods_and_evaluation_criteria": "Yes, the paper presents a good evaluation with relevant baselines for the problem.", "theoretical_claims": "Didn't verify.", "experimental_designs_or_analyses": "The experiments seem sound. As for the results, I would have liked a better explanation for why Type4Py exceeds performance in certain cases. The paper says that Type4Py is stronger in handling short code segments, but is it just based on lines of code in the function? There are also cases where CodeT5-Large exceeds performamce, so an explanation for that would be nice too.\n\nApart from that, the evaluation section tables use a lot of abbreviations which were not specified earlier (e.g. Arg., Ret., Var. in table 1), so please add definitions for those (unless I missed them somewhere in the paper).\n\nAlso, what are the main challenges preventing you from extending support to other datasets? Based on the paper, it should be relatively easy to just apply to any function defined in Python.", "relation_to_broader_scientific_literature": "This paper offers a marked improvement over previous SOTA type-inference methods, and uses techniques relevant to both the type-inference fields as well as the neurosymbolic fields.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "In general, the problem being tackled here is both important and difficult to solve, and the proposed technique is novel and elegant. The paper is also well explained and clear on its contributions.\n\nI do believe, despite what is written in threats to validity, that Nester should be evaluated over larger LLMs. What is the fundamental reason why Nester couldn't just be run with a larger LLM? It would be interesting to see if Nester + LLama-70B performs better than just LLama 70B, or if the gains made by the neurosymbolic reasoning diminish with larger models. I know the main motivation was to use smaller LLMs, but section 4.2 suggests an openness to evaluating Nester with larger models.\n\nThe paper also does not have a discussion on limitations of Nester wrt the input functions itself. For instance, does Nester only work with individual functions and local context, or can it draw from contexts outside the scope of the function? E.g., if there was a function call to some function custom defined in a repository, will Nester assume the return type of the called function or in turn perform type-inference for that function? Other statistics about functions input to Nester would also give a good idea about its abilities, such as how many lines of code are there in each input?", "other_comments_or_suggestions": "1. Have you thought about evaluating Nester with small language models like Microsoft's Phi family?\n2. In Figure 3, for line 6 in the high level program, why is only T2 and T3 being combined? The condition evaluation hasn't been run yet and it doesn't know T1 is unreachable, so should it not assume for now that T1, T2 and T3 should all be combined?", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ENd5E1W8Qr", "reviewer": "Reviewer_q6Mu", "summary": "The paper introduces a neuro-symbolic approach named NESTER, which integrates language models (LMs) with program analysis for type inference in dynamically typed languages like Python. With the help of LMs, NESTER translates target code into a high-level program composed of predefined API-driven analysis units and thus decomposes type inference into modular sub-tasks guided by \ndataflow analysis. This program is executed via a neuro-symbolic interpreter that combines static analysis (e.g., coarse-grained dataflow graphs) and LM-based reasoning to infer types. It uses a type recognizer to initially identify types from explicit type declarations in the code. When the type recognizer fails, two language models are involved: a condition-evaluation language model to assess conditional statements in the high-level program and a type-inference language model to infer types based on context and the dataflow graph. Evaluated on the ManyTypes4Py dataset using metrics like Exact Match and Match to Parametric, NESTER outperforms the selected baselines by 3-4% in Top-1 accuracy.\n\n## update after rebuttal\n\nThe authors did not provide enough evidence to show that the proposed method can perform well for user defined types or rare types, which is known as a common chanllenge in the type inference task. Moreover, the authors also did not provide how they construct the datasets from ManyTypes4Py. Based on this, I have concerns about whether the datasets are cherry-picked from ManyTypes4Py.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** How does the proposed method perform for complex parametric types and user-defined types?\n\n**A1:** In response to your concern, we will provide Nester's performance on **complex parametric types** (depth = 0, e.g., list; depth = 1, e.g., list[str]; depth ≥ 2, e.g., Dict[str, List[int]]) and **user-defined types**, as follows.\n \n| Method           | Depth=0 | Depth=1 | Depth>=2 | User-defined |\n|------------------|---------|---------|----------|--------------|\n| Exact Match      |         |         |          |              |\n| TypeGen-CL       | 40.8    | 39.3    | 7.0      | 74.7         |\n| Nester-CL        | 58.3    | 30.7    | 2.1      | 81.3         |\n| TypeGen-L3       | 35.7    | 31.3    | 2.1      | 69.3         |\n| Nester-L3        | 55.8    | 26.8    | 2.1      | 78.7         |\n| Match to Para.   |         |         |          |              |\n| TypeGen-CL       | 68.3    | 69.3    | 35.5     | 75.0         |\n| Nester-CL        | 79.9    | 75.2    | 58.7     | 81.2         |\n| TypeGen-L3       | 51.5    | 60.5    | 31.3     | 69.5         |\n| Nester-L3        | 66.3    | 73.8    | 61.7     | 78.6         |\n\nFrom this table, we can observe that Nester outperforms Typegen for user-defined types in terms of exact match and match to parametric metrics. Furthermore, Nester performs better with complex parametric types in terms of match to parametric metrics. Although Nester does not achieve higher exact match accuracy for depth ≥ 2 and depth=1, this can be attributed to Typegen's use of specially designed chain-of-thought operations for recursive type structures, which Nester has not yet incorporated. We believe that integrating such techniques into Nester could further enhance its performance, and we will explain how this can be done later.\n* * *\n**Q2:** The key differences between Nester and TypeGen (raised in the **Claims and Evidence** part).  \n\n**A2:** Nester differs from TypeGen in its approach to type inference. Nester integrates static data and control flow analysis with the capability of LLMs.  In contrast, TypeGen relies entirely on LLM-generated reasoning, treating code analysis as solely textual processing progress—a less reliable approach and more prone to hallucination. Our design also allows Nester to provide an intuitive, high-level program view, offering greater interpretability during type inference than TypeGen. \n* * *", "claims_and_evidence": "The authors claim their approach to be the first neuro-symbolic framework for type inference using LMs. The existing work TypeGen also follows combines static analysis and LLM. What are the key differences?", "methods_and_evaluation_criteria": "The proposed methods are evaluation criteria are reasonable.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental design is generally acceptable, with some issues which will be discussed in the weakness part.", "relation_to_broader_scientific_literature": "This paper is an increment compared to previous work, as the proposed method differs from existing work.", "essential_references_not_discussed": "The references are adequate.", "other_strengths_and_weaknesses": "Strengths:\n- NESTER effectively bridges neural and symbolic paradigms by leveraging LMs as neural sub-task parsers while retaining control-flow semantics through symbolic program decomposition.\n- The paper provides empirical analyses of practical challenges in LM-based type inference. For instance, it investigates the pitfalls of in-context learning (ICL), demonstrating that LM accuracy drops by 8.5% when exposed to misleading demonstrations.\n- The use of LMs to generate high-level programs avoids reliance on language-specific parsers, suggesting scalability to other dynamically typed languages (e.g., JavaScript).\n\nWeaknesses:\n- In the motivation example, it's more like that adding an unreachable statement to fool the CodeLlama model (from float—none to bool—float—none).\n- While NESTER reports strong results on common and Union-like types (e.g., 53.2% accuracy on `typing.Optional`), its evaluation lacks dedicated metrics for complex parametric types (e.g., `Dict[str, List[int]]`) and user-defined types. This contrasts with some previous work [1][2][3], which explicitly benchmarks rare/complex/user-defined types. This omission raises concerns about generalizability as real-world codebases heavily utilize such types, especially given ManyTypes4Py’s known dataset distribution bias on rare type, which was reported in [2].\n- The experimental improvement is relatively marginal.\n\nReferences\n[1] Allamanis, M. et al. \"Typilus: Neural Type Hints.\" PLDI 2020.\n[2] Wei, J. et al. \"TypeT5: Seq2seq Type Inference using Static Analysis.\" ICLR 2023.\n[3] Wang, X. et al. \"TIGER: A Generating-Then-Ranking Framework for Practical Python Type Inference.\" ICSE 2024.", "other_comments_or_suggestions": "- While the case study highlights NESTER’s ability to correct TypeGen’s errors (e.g., inferring HttpResponse instead of HttpResponseNotFound), it does not analyze failure modes.\n- The authors may consider to submit the paper to a software engineering venue, as the key contributions seems to be the high-level program generation and program interpreter parts.", "questions_for_authors": "How does the proposed method perform for complex parametric types and user-defined types?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qPCERU8UMY", "reviewer": "Reviewer_fVw9", "summary": "The paper presents Nester, a novel neuro-symbolic technique for type inference. Nester decomposes the type inference process into sub-tasks that are aligned with the data and control flows of the input code, encapsulating these into a modular high-level program. This program executes multi-step actions, such as evaluating expressions and analyzing conditional branches, thereby integrating static typing with language models (LMs) to deduce potential types. Implemented for Python, Nester is benchmarked against various models, including symbolic, deep learning-based, and LM-based systems, using the ManyTypes4Py dataset. Although Nester does not surpass Type4Py in exact match accuracy, it excels in both exact match and match to parametric metrics for inferring argument and return types. Additionally, Nester is available as a VSCode plugin, providing users with visibility into the high-level program generated by the LM and insight into the LM's reasoning process.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The key differences between Nester and TypeGen.\n\n**A1:** Please see  **Q2** of **Reviewer Q6Mu** for a discussion. \n* * *\n**Q2:** Could the author please provide a detailed formalization of high-level programs?\n\n**A2:** This is provided in Appendix D.1. We will better highlight. \n* * *\n**Q3:** Comparison between the introduced coarse-grained dataflow graph and the original one.\n\n**A3:** Thanks. This information is provided in Appendix E.2 in the context of CGDG (Coarse-Grained Dataflow Graph). Specifically, for an identifier $x_v$, we represent its three hops using three types of nodes: **Def**($x_v$), **User**($x_v$), and **Usee**($x_v$). In essence, our approach differentiates nodes based on their hop distance from $x_v$, categorizing them into three distinct types. Compared to traditional Concrete Syntax Tree (CST) and Abstract Syntax Tree (AST), CGDG constraints each identifier to at most three hops, resulting in a more concise and structured graph representation. We will explain this approach in more detail.\n* * *\n**Q4:** How does Nester compare to traditional symbolic execution approaches?\n\n**A4:** Nester is a static analysis framework for type inference and does not rely on symbolic execution—no symbolic execution traces are generated. Unlike traditional symbolic execution, which explores possible input spaces to produce traces, Nester operates purely on static source code information. We will clarify.\n* * *\n**Q5:** What is the coverage of the rules in Nester?\n\n**A5:** The coverage of rules is reported in **Table 3**. Our rules support primitive types (int, float, bool, str, bytes) and container types (list, tuple, dict, set),  covering the vast majority of type inference scenarios encountered in real-world code.\n* * *", "claims_and_evidence": "The paper presents an interesting claim regarding whether LMs perform reasoning in type inference. The experiments conducted in Section 2.3 provide convincing evidence to support this claim.", "methods_and_evaluation_criteria": "The proposed method, Nester, makes sense as it builds upon existing work in both rule-based and LM-based approaches. The evaluation criteria are convincing since the dataset, ManyTypes4Py, is commonly used in other studies, and the metrics—exact match and match to parameters—are standard for type inference task [1], [2].\n\n[1] Peng, Yun, et al. \"Generative type inference for python.\" 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2023.\n\n[2] Guo, Yimeng, et al. \"Generating Python Type Annotations from Type Inference: How Far Are We?.\" ACM Transactions on Software Engineering and Methodology 33.5 (2024): 1-38.", "theoretical_claims": "The paper does not provide specific theoretical proofs", "experimental_designs_or_analyses": "The experimental design is reasonable. The paper first presents the overall performance, then separately discusses the neural and symbolic rule-based components, and finally provides a case study. Additionally, it examines the effectiveness of high-level programs.", "relation_to_broader_scientific_literature": "This paper presents a novel neuro-symbolic method for addressing the type inference task, specifically focusing on tackling control flow.", "essential_references_not_discussed": "The related works are sufficiently discussed in this paper.", "other_strengths_and_weaknesses": "Pros:\n\n1. This paper introduces a novel methodology that seamlessly combines static analysis (symbolic) with neural perceptual representations (neural) via LMs to enhance the accuracy and effectiveness of type inference tasks. To the best of my knowledge, this work is the first to apply a neuro-symbolic approach in software engineering, specifically selecting type inference as a suitable application scenario. \n2. This paper reveals the limitations of traditional rule-based methods and recent LMs-based approaches by providing a motivating example. Motivated by this, the authors introduce a neuro-symbolic system that addresses the shortcomings that neither rule-based nor LMs-based methods can fully resolve. Additionally, a practical and easy-to-understand example is used throughout the paper to clearly illustrate its motivation and approach.\n3. This paper presents a high-level program interpreter that initially conducts data flow analysis on the target program to construct a data flow graph, which is reasonable and effective. \n4. To validate the effectiveness of the proposed Nester, the authors have conducted experiments and compared Nester against eight other systems, including three types of models: rule-based models, deep-learning models, and LM-based models.   \n\nCons:\n1. Nester seems to mainly use LLMs for type inference. How does it differ from other LLM-based methods, such as TypeGen?\n2. I would assume that LM-based high-level program generation might sometimes produce errors. How does the proposed Nester handle situations when the generated program is not entirely correct?\n3. Although the authors provide an example of a high-level program generation in the paper, a specific definition is lacking. Could the author provide a detailed formalization of high-level programs?\n4. One limitation of the approach is that Nester is only implemented for very simple code structures, neglecting complex code structures. Are there challenges in implementing it for more complex code structures?", "other_comments_or_suggestions": "None", "questions_for_authors": "1. I would like to understand your contributions in the area of type inference. How does your work differ from existing LLM-based approaches, such as TypeGen?\n2. Could the author please provide a detailed formalization of high-level programs?\n3. In the context of this dataflow graph, what specifically has been made coarse? How does this compare to the original dataflow graph? \n4. It seems that Nester employs a symbolic execution method for type inference. How does Nester compare to traditional symbolic execution approaches?\n5. Since rule-based methods are designed, there must be concerns about coverage. What is the coverage of your rules?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oE0PzBhV4E", "reviewer": "Reviewer_f94C", "summary": "The authors present a framework for enhancing language models’ ability to accurately infer types from code. This is achieved by decomposing the input into a high-level program composed of evaluations and type analyses using LMs. This high level program can then be deterministically executed to perform type inference.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback. \n* * *\n**Q1:** How much more expensive is Nester inference in terms of time and energy over the naive CL and L3 models?\n\n**A1:** We conduct additional experiments to evaluate Nester's computational cost in terms of inference time and energy consumption using an NVIDIA RTX 4090 GPU. The results are as follows:\n\n| Method                                      | Time (s) | Energy (J)  | Accuracy (%) |\n|---------------------------------------------|---------:|------------:|-------------:|\n| Naïve-CL                                    |    4910  |     219,317 |         31.3 |\n| Nester-CL                                   |   18,396 |   1,365,714 |         68.7 |\n| Nester-CL (w/o multi-step reasoning)        |    4391  |     228,023 |         65.7 |\n| Naïve-L3                                    |     436  |      20,818 |         42.4 |\n| Nester-L3                                   |    2906  |     233,378 |         63.6 |\n| Nester-L3 (w/o multi-step reasoning)        |     465  |      22,248 |         59.6 |\n\nNester incurs higher computational costs, primarily due to multiple invocations of large LMs during inference. However, when multi-step reasoning is removed, the additional overhead is negligible, making the inference time and energy consumption comparable to those of the naive models. In this case, Nester will only locate the relevant code lines (e.g., identifier definitions) from the source code according to the high-level program. It then leverages LLMs along with predefined rules in a single step to infer types without further iteration, still yielding practical results for type inference. This provides an optional solution with fast type inference time. \n* * *\n**Q2:** Is Nester extensible to compiled languages, or are interpreted languages the only candidate for type inference with Nester?\n\n**A2:** This is a good point. Nester can be adapted to lower-level compiled languages like bytecode and assembly instructions. This will require adjusting the high-level program prompts for LMs and developing language-specific rules for inference, taking into account the existing type systems of the specific languages. We will add a discussion. \n* * *\n**Q3:** Why not evaluate on ManyTypes4TypeScript as well? It seems that Nester should at least be extensible to JS/TS.\n\n**A3:** Our current implementation does not support certain JS/TS features due to the limitations of the underlying tools used by Nester. These include JavaScript's dynamic scoping (e.g., eval), prototype-based inheritance, event-driven asynchrony, and TypeScript's hybrid semantics, such as gradual typing and any escape hatches—none of which are present in Python. We will clarify this and discuss how to extend Nester to other languages beyond Python.\n* * *", "claims_and_evidence": "The authors primary claims are that their proposed framework improves type inference by applying smaller models (suitable for local deployment) in a structured manner. The claim is supported by their evaluation, which reports accuracy of various approaches on various tasks. The results of this evaluation indicate that their approach achieves greater accuracy in general, compared against other symbolic and neural methods.", "methods_and_evaluation_criteria": "Both the evaluation criteria and methodology make sense for the application; the authors use program generation and dataflow analysis to reconstruct the input as a series of modules to capture dependencies. This ‘program’ is passed to an interpreter for execution", "theoretical_claims": "There do not seem to be any rigorous theoretical claims or proofs in the main body of the paper.", "experimental_designs_or_analyses": "The experimental design and analysis is valid. Mild concerns are raised in a later section of this review, but by and large there are enough results presented to justify the claims made in the paper. I am referring specifically to Table 1 (the primary results table) and the subsequence analysis in section 4.", "relation_to_broader_scientific_literature": "This paper contributes to the trend of producing neuro-symbolic frameworks which apply a deep learning model(s), usually large language models, in a structured system that executes symbolic operations or algorithms on programs or data synthesized by the model. This paper supports an argument for this design pattern, which is that certain abilities that are strong in truly large models but weaker in smaller models can be achieved by the smaller models if this design pattern is applied correctly.", "essential_references_not_discussed": "To my knowledge, all relevant references are included.", "other_strengths_and_weaknesses": "Strengths: \n+ Broad range of LMs included in the evaluation.\n+ Detailed methodology makes the approach clear.\n+ Appendix addresses information omitted in the main body.\n+ Experiments and code are made available for reproducibility.\n \nWeaknesses:\n+ Only one purely symbolic method is included in the evaluation.\n+ Only one target language is used in evaluation.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "1. How much more expensive is NESTER inference in terms of time and energy over the naive CL and L3 models?\n\n2. Is NESTER extensible to compiled languages, or are interpreted languages the only candidate for type inference with NESTER?\n\n3. Why not evaluate on manytypes4typescript as well? It seems that NESTER should at least be extensible to JS/TS.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a neurosymbolic approach to type inference in dynamically typed languages. NESTER translates the code into a high-level program which can then be processed by a neurosymbolic interpreter. The evaluation suggests that results are improved over a large number of baselines.

Overall, the reviewers felt that this work proposes a novel technique that works well and the problem is interesting and relevant to the community. Thus, I recommend that this paper is accepted.
Decision: Accept (poster)