Monitor-Guided Decoding of Code LMs with Static Analysis of Repository Context
Authors: Lakshya Agrawal
Aditya Kanade
Navin Goyal
Shuvendu K Lahiri
Sriram Rajamani
Venue: NeurIPS 2023 poster (2023)
Keywords: Language models
code generation
correctness
program analysis

Abstract: Language models of code (LMs) work well when the surrounding code provides sufficient context. This is not true when it becomes necessary to use types, functionality or APIs defined elsewhere in the repository or a linked library, especially those not seen during training. LMs suffer from limited awareness of such global context and end up hallucinating.

Integrated development environments (IDEs) assist developers in understanding repository context using static analysis.  We extend this assistance, enjoyed by developers, to LMs. We propose monitor-guided decoding (MGD) where a monitor uses static analysis to guide the decoding. We construct a repository-level dataset PragmaticCode for method-completion in Java and evaluate MGD on it. On models of varying parameter scale, by monitoring for type-consistent object dereferences, MGD consistently improves compilation rates and agreement with ground truth. Further, LMs with fewer parameters, when augmented with MGD, can outperform larger LMs. With MGD, SantaCoder-1.1B achieves better compilation rate and next-identifier match than the much larger text-davinci-003 model.

We also conduct a generalizability study to evaluate the ability of MGD to generalize to multiple programming languages (Java, C# and Rust), coding scenarios (e.g., correct number of arguments to method calls), and to enforce richer semantic constraints (e.g., stateful API protocols). Our data and implementation are available at https://github.com/microsoft/monitors4codegen.

Monitor-Guided Decoding of Code LMs with Static Analysis of Repository Context
Lakshya A Agrawal
Aditya Kanade
Microsoft Research
Microsoft Research Bangalore, India sonedeaadtyyamiiiirrrssao
Bangalore, India
t- waaakkaaaaaaaaaa wwweraawwa@micooseteae cww.rooottccommm
Navin Goyal
Shuvendu K. Lahiri
Microsoft Research
Microsoft Research Redmond, United States
Bangalore, India
navingoomicrosofftccom com
shuvendu. lahiri www.coooott.commmmmm com
Sriram K. Rajamani
Microsoft Research Bangalore, India sriram@mi crosoft. com
Abstract
Language models of code (LMs) work well when the surrounding code provides sufficient context. This is not true when it becomes necessary to use types, func- tionality or APIs defined elsewhere in the repository or a linked library, especially those not seen during training. LMs suffer from limited awareness of such global context and end up hallucinating.
Integrated development environments (IDEs) assist developers in understanding repository context using static analysis. We extend this assistance, enjoyed by developers, to LMs. We propose monitor-guides decoding (MGD) where monitor uses static analysis guide the decoding. We construct repository-level dataset PRAGMATICCODE for method-complen models varying parameter scale, by monitoring for type-consistent object deref- erences, MGD consistently improves compilation rates and agreement with ground truth. Further, LMs with fewer parameters, when augmented with MGD, can outperform larger LMs. With MGD, SantaCoder-1. 1. achieves better compilation rate and next-identifier match than the much larger rext-davinei-003 model.
in Java and evaluate MGD on it. On
We also conduct a generalizability study to evaluate the ability of MGD to gener- alize multiple programming languages (Java, C# and Rust), coding scenarios (e.g., correct number of arguments to method calls), and to enforce richer semantic constraints (e.g., stateful API protocols). Our data and implementation are available at http:///w..ooooooooooooo//ooaaaa
1 Introduction
Language models of code (LMs), such as in Chen et al. (202): Nijkamp et al. (2023); ; Allal et al. (023) and many others, revolutionizing code generation. Many commercial offerings based on LMs, like GitHub Copilot, Amazon Code Whisperer and Replit, are now available. The LMs work well when the surrounding code in the vicinity of generation provides sufficient context. This
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

completed
toxt dw--d--0ndd- SantaCoder
parseServer
aränate Surveriade par aesarven String ard)
hesg (orr())
Iic start uel, Indexdl (Ste0" (nd
(goct (gort(Zitegggnpprrscee (errrger-pprssent(rrtLL)) build())
Stmte
Inc and and Lasttaresth (atel" url lengthi wilh wal.lastTaded (a 170
stritg etn sabeeeeeell lag shart, eedss
ExpeStmi
RetSimi
ttring El are
crateee "11
retur
Darel
SuntaCoder monitor guided
recure Serverficte, Basider cunder ctederrertion corficce))
Reszived Type (Nshhodinvocatioo ServerMode B Builder
[UNKHOW
(Integrr wtwp:ennnnnneeepeeeiiiteeoooo (er())) build();
"serverNode Bu www.derreeeedeeee"
(a) Example where text-davinci-003 and SantaCoder generate wrong (b) Annotated partial AST for the code to identifiers, but SantaCoder with MGD generates correct identifiers. the left.
Figure 1: Motivating example to illustrate monitor- guided decoding (MGD).
condition does not hold when it becomes necessary to use types, functionality or APIs defined in another module in the repository or external library, especially those not seen during training. In the absence of awareness of such global context, the LMs end up hallucinating, e.g., using types defined in other files incorrectly. Further, devoid of the repository context, the LMs may lack awareness of other semantic information like the number of arguments required by a called method, or the ordering constraints method calls (API protocols) be followed. Often, such type and semantic information can come from artifacts generated at build-time, like Project Lombok (lom) (009) and ProtoBuf pro 2008), and therefore may not even be present as code context in the repository.
As an example, consider the partial code (method to be completed) in Figure a To complete this code, an LM has to generate identifiers consistent with the type of the object returned by ServerNode Builder..... wwwilddernewweere newServerNode (). The method newServerNode and its return type, class ServerNode. Builder, are defined in another file. If an LM does not have information about the ServerNode Builder type, it ends up hallucinating.
We show completion generated by the OpenAl text-davinci-003 (Ouyang et al. 2022 and Santa- Coder (Allal et al. (023) models in the box decorated with in Figure [ a). The completion uses identifiers host and port, which do not exist in the type ServerNode Builder. The generated code therefore results in "symbol not found" compilation errors. The lack of awareness of other semantic information from the global context may result in other kinds of errors like compile-time errors (e.g., "actual and formal argument lists differ length" on using wrong number of arguments) runtime errors (e.g., illegalStateExceptiio in violation of API protocols).
Integrated development environments (IDEs) have been at the forefront of assisting developers. Our inspiration is the use of static analysis by IDEs to bring the global context at the fingertips of developers. Many analyses are integrated in IDEs (Fuhrer) 2013) to infer and enforce semantic constraints the code under development, e.g., resolving def-use, symbol references, and type hicrarchies. Recently, there has been a rise in the use of Language Server Protocol (LSP) (Isp), which is an open industry standard of communication between IDEs and programming language specific tools like static analyzers and compilers, called Language Servers. There are a large number of Language Servers available, targetting most programming languages (lan) 2023), and providing a variety of syntactic and semantic information. In this work, we focus on the type-directed code completion analysis available through LSP in language-agnssie manner, to provide guidance an LM.
We propose a notion of monitors as stateful interface between LMs and static analysis. A monitor observes the code generated by an LM and queries static analysis at pre-defined trigger points. The suggestions returned by the static analysis are converted to masks which are used for reshaping the logits (or equivalently, token-generation probabilities) produced by the LM in the subsequent decoding steps. We call our method monitor-guides decoding (MGD). Unlike an LM, static analysis operates on the entire repository and its dependencies. While the LM generates completions by conditioning on the local context, the static analysis ensures consistency with the rest of the code in the repository. Through MGD, we bring the two together without the need to retrain the LMM, and making minor and modular addition to the decoding stage of the LM.
Figure Ma) aa) also shows the code generated by the SantaCoder model with MGD in the box decorated with V. This code makes use of identifiers that are actually defined in the class

---

ServerNode Sectioode.uuiill -Builder. It compiles and matches the ground truth. In comparison, the same Santa- Coder model without MGD generates the erroncous code shown in the box decorated with x
Some recent approaches use static analysis (Shrivastava et al. 2022 Ding et al.
2022 Pei et al. 2023
or retrieval (Zhang et al.] (023) to extract relevant code fragments from the global context. These approaches expand the prompt (Shrivastava et al. 2022 Pei et al. 2023 Zhang et al.
2023 or require
architecture modifications (Ding et al., (022) and additional training (Ding et al. 2022 Pei al.
2023). In comparison, we provide token-level guidance to frozen LM by invoking static analysis on demand. Our method is complementary to these approaches they condition the generation by modifying the input to the LM, whereas we apply output-side constraints by reshaping the logits.
We make the following contributions in this paper:
Monitor-Guided Decoding (MGD) as stateful interface between LMs and static analysis. A monitor watches the LM generating code, queries static analysis in the background, and uses the information from the static analysis to effectively guide the decoding stage of LMs.
PRAGMATICCODE, publiely-released dataset of Java code repositories complete with their development environments and dependencies.
Instantiation of MGD for generating code having type-consistent identifier dereferences. Large scale evaluation on PRAGMATICCODE showing: (1) MGD consistently improves the ability of an LM (across varying parameter scales) to generate type-consistent identifiers, compilation rates and agreement with ground truth. (2) Further, LMs with fewer parameters, in conjunction with MGD, can outperform larger LMs. For instance, SantaCoder-1. 1B with MGD) achieves better compilation rate and next-identifier match than the much larger text-davinci-003 model, when both have a budget of generation each. With budget of 4 generations, it also surpasses agreement with ground truth of text-davinci-003. (3) We also evaluate how MGD complements different prompt augmentation and decoding strategies. Microbenchmark to demonstrate generalizability of MGD to different (1) programming lan- guages, (2) coding scenarios, and (3) use of other static analysis techniques for guiding with rich semantic properties. Notably, we demonstrate that small-LMs can be guided to adhere to rich static properties like satisfaction of stateful API protocols.
We open source our implementation and provide an extensible Python library called mul tilspy with static analysis bindings for multiple languages over LSP,, suitable for monitoring of LMs. can be used for other A14Code scenarios as well.
2 Monitor-Guided Decoding
Background. Static analysis of code (Nielson et al. 2015) is used widely in industry various applications such as in detecting bugs and optimizing code. While analysis is usually performed on complete code, IDEs have long applied static analysis on incomplete code under development (Reps)
ct al. 1983 Dagenais& & Hendren 2008), using algorithms for incremental parsing and semantic
analysis (e.g., type inference) of partial code (Hedin 1992. Wagner 1997, Maddox III] 1997). These
analyses have now become standard part of language servers.
key abstraction used in analysis of partial code is partial abstract syntax trees (ASTs) with special nodes to indicate incomplete parts of code. These AST's are further decorated by semantic information through attribute grammars (Reps et al. 1983) that decorate each AST node with attributes that capture the static semantics not captured in the context-free grammar (such as consistency of types among expressions in an assignment). This can range from computing the type-hicrarchy for object oriented languages, binding the variables in AST nodes to their declarations, resolving the types of expressions as well as computing the def-use relationships for resolved AST nodes (Fuhrer 2013).
Figure (b)) shows the partial AST for the incomplete code in Figure a All the statements upto the incomplete return statement are completely parsed and subtrees corresponding to them are constructed. The subtree for the return statement includes node [UNKNOWN] indicating the incomplete part. As shown in Figure [b), an incremental semantic analysis resolves the type of the partial expression ServerNode. Builder newServerNode () to ServerNode. Bui Ider. Later, we show how to use this type information to construct a monitor, which can then be used guide an LM to generate type-consistent identifier completions.
3

---

Basic Concepts and Notation. Consider an LM Lg operating on a vocabulary V. Let 21, ................................ be the partial code that has been generated by the LM and Zn+1 be a candidate next token. Though a vanilla (auto-regressive) prompt would consist only of æ1,..., Zn. today many approaches augment it with additional information. We use p to indicate this additional prompt, e.g., the suffix information used in all-in-the-midlle prompting (Donahue et al.) 2020 Fried et al. 2022; Bavarian et al., (202))
property yp specifies the constraints that a piece of code needs to satisty, A static analysis A checks whether the partial code satisfies If yes, it returns suggestions to extend it so that the extended code continues to satisfy yp The static analysis works on the repository context C' which not only includes code spread across multiple files in the repository, but also external dependencies and intermediate artifacts (e.g.. code bindings) generated during the build process. Such repository context is often very large, diverse and complex. Directly including it as an input to the LM will result in bloating and pass the burden of distilling useful information from it to the L.M.
A monitor Ma for a property yp is a tuple (Ag, fo, S, pre, update, maskgen). The monitor starts in the wait state 5g. If the partial code satisfies the pre-condition pre then the monitor is triggered and it invokes Ag on the partial code. The monitor maintains the suggestions returned by Ag in its state and uses them to guide sampling of the next token In+11 Let S be the set of states and s, E S respectively be the current and next states of the monitor. With each sampled token +1+ the monitor updates its state using a function update(s, Zn+1) to a new state s', which tracks the residual suggestions after the token Tx+1 is output. When the suggestions are exhausted, it reverts to the wait state sg. We explain the function maskgen below.
Decoding Process. Usually, the next token Fn++1 can be any token from the vocabulary sampled based on the logits e determined by the LM. Unlike the usual decoding, in monitor-guided decoding, we supervise the code generation using monitor M. for a property ş. We denote the composition of Le and Mi by Le MMp meaning that both the LM and the monitor are running concurrently and sampling the tokens jointly. The decoding is conditioned on the partial code II,..., In, the repository context C. the prompt p and the current state of the monitor.
Eq. (1) states that whenever the monitor is in the wait state so, we sample 2x+1 as per the logits l determined by the LM (Eq. (2)). Otherwise, the logits are combined with a mask using function @ such that if mn[r]] = 0 then {[x] is reset to a large negative value - K and is left unchanged otherwise. This mask is computed by the function maskgen in Eq. (3) guided by the current state s of the monitor. Eq. (4) defines how the state of the monitor evolves. When the pre-condition pre(s; F1,..., evaluates true, the next state s' of the monitor is determined by the suggestions returned by the static analysis Ag. Otherwise, it is determined by the update function.
(LolI la1,,.., Fni 8) = ssof [softmax()] [Zx+1] toft (6 + m)
if = so is the wait state otherwise
Lo( (.1....,
(2)
m maskgen V)
s' [Ap(o),.... Tni C) (update(s, In+1) otherwise
= A pre(s; In)
(4)
The specifics of the monitor state, and the pre, update and maskgen functions depend the static analysis Ag used by the monitor. Our formulation is general and even allows combining multiple static analyses by taking product of the spae-spaces of their respective monitors. In the following, we discuss a specific instantiation of this framework of monitor-guided decoding.
Monitoring for Use of Crpe-Consistent Identifiers. When an object obj of type is dereferenced, the next token (or more generally, the sequence of subtokens) should refer to an identifier of a field or method defined by the type T. It can otherwise result in "symbol not found" error. The type T could be defined in another package, imported file or in a library. Unless T comes from a popular library seen during training, the LM may not have knowledge about T'.
Our monitor M, is triggered when the partial code .1...., In ends with a partial object dereference expression "obj." "" where "." is the dereference operation. This is the pre-condition pre we use. We employ a static analysis A₂ which returns all the type-consistent identifiers that can be referenced through obj. For this, A, performs global analysis over the partial code, imported files, libraries used, and class hierarchies to infer the type of obj and the identifiers accessible through T. The set of type-consistent identifiers returned by Ag forms the state of the monitor (see Eq. (4)).
4

---

Given a state and the vocabulary V of the LM, maskgen identifies all the tokens in V that are consistent with the suggestions in s. The identifiers returned by the static analysis are tokens as per the programming language, whereas the vocabulary V may use its own space of subtokens (Schuster & Nakajima 2012 Kudo & Richardson 2018). The mask m (Eq. (3)) generated by string matching. For all tokens & E V that form prefixes of the identifiers in s, the mask value m[]] is set to 1, indicating that they can be sampled. Let E be the set of special symbols that indicate end of an identifier name, e.g., the symbol "(" in 'ggeNaame (" or ,," in 'x," Let w be a string in and E be the set of all possible characters. If a token & E V matches the regular expression w E > then its mask value m it] is also set to 1. For all other tokens t in V, the mask value m [t] is set to 0.
Let In+1 be the next token sampled according to the second equation in Eq. (1). If In+1 contains a symbol from E. indicating that a complete identifier name has been generated in accordance with the set returned by A.. the monitor reverts to the wait state to wait for the next trigger. Otherwise, the token 2n-1 must be prefix of member of s. The update function removes the members in s that are not prefixed by 2, and those prefixed by xn+1 are updated by pruning the prefix string Fx+1- The resulting set of character strings forms the next state s' (see the second equation in Eq. (4)). If Ag returns an empty set to start with, we abandon the current run. Note that a single identifier may need to be generated using multiple tokens. Figure 4 (see Appendix A shows the interaction between the LM and the monitor for the example in Figure [ and specifically illustrates how the complete identifier names suggested by the static analysis are gradually pruned by the monitor to corresponding suffixes in each successive state as the prefixes get generated as tokens.
Experimental Setup
Dataset Creation. In order to evaluate MGD, we need real-world repositories with their build environments and dependencies. Most published datasets are standalone, with the exception of CoderEval (Yu et al. 2023) and PyEnvs (Pei et al. 2023), both of which are not publicly available at the time of this writing. Hence we curated PRAGMATICCODE, dataset of real-world open-source Java projects complete with their development environments and dependencies. We ensure that these repositories were released publicly only after the determined training dataset cutoff date (31 March 2022) of the models which we use to evaluate MGD.
From PRAGMATICCODE, we identify a set of method-level completion task instances, creating DOTPROMPTS as a method-level code completion benchmark. Each testcase in DoTPROMPTS consists of prompt upto dereference location (using the "." operator in Java) within a target method, and the task is to complete the remainder of the method. We ensure sufficient complexity in the identified target methods in DOTPROMPTS by including methods that satisfy a set of complexity filters (e.g., the method should have at least 7 lines of code) described in detail in appendix B Overall, PRAGMATICCODA consists of 100 repositories, and DoTPROMPTS consists of 1420 methods and 10538 derefcrence prompts. Appendix B gives further details.
Models. We study the effect of performing MGD on code generation with the HuggingFace Trans- formers (Wolf al. 2020) implementation of Salesforce CodeGen family of models (CodeGen- (350M, 2B, 6B) MMulti, abbreviated as CG-(350M, 2B, 6B) hereafter) (Nijkamp et al. 2023) and BigCode SantaCoder-1.1I (SC or SantaCoder hereafter) (Allal et al.) 2023). We also evaluate OpenAl texi-davinci-003 (TD-3 hereafter) with and without MGD, available on Azure.
Prompting Strategies. We study the effect of different prompt augmentation techniques when combined with MGD: (1) Standard: Include the local file content up to the dereference point and truncate from left to fit the prompt budget. (2) classExprTypes For given target method belonging to class C, identify the type of all expressions occurring in C (after masking out the target method to prevent leakage) and include the concatenated file contents for the type definitions of all the identified files, truncating from the left as necessary. We assign a budget of 20% tokens of total prompt budget to classExprTypes. (3) RLPG: Use the prompt augmentation technique proposed in Shrivastava et al. (002). We use their released source code and model checkpoints to adapt RLPG to DOTPROMPTS.
Decoding Strategies. We experiment with two decoding strategies: (1) Autoregressive: for left- to-right decoding. (2) Fillinnthe-miidlee: Use fill-in-the middle (FIM) setting implemented in SantaCoder (Allal et al. 2023).
5

---

Table Summary of results with a budget generations per model: The numbers in parentheses
are relative improvements of the "-MGD" configuration over the respective base model.
Config / (Metric, score@6)
CR
NIM
ISM
PM
CG-350M
52.43
76.94
31.86
26.86
CG-350M-MGD
65.37 (24.69%)
83,80 (8.92%)
34.31 (7.70%)
28.69 (6.82%)
CG-2B
81.11
36.38
30,95
CG-2B-MGD
70,91 (24.38%)
87.32 (7.66 %
39,03 (7.29%)
33,06 (6.82%)
CG-6B
58.64
81.55
37.17
31.69
CG-6B-MGD
72.28 (23.25%)
87.35 (7.11% %
39,55 (6.41 %)
33.56 (5.89%)
SC
59,97
82.40
38.14
32.10
SC-MGD
73.03 (21.77%)
88,42 (7.31 %)
40.69 (6.68 %))
34,25 (6.72%)
SC-clasExprTyypes
64.57
84,91
39.67
33.55
SC=daasEEpr
75.01 (16.18%)
89.37
(4.78%)
34,98 (4.26%)
SC-RLPG
66.39
85.42
42.35
36.21
SC-RLPG-MGD
78.14 (17.70%)
89,89 (5.23 %)
44,47 (5.00%)
37.97 (4.87
SC-FIM
68.23
85.56
42.22
36.12
SC-FIM-MGD
80.19 (17.52%
(5.07% %)
(5.39%)
37.91 (4.95%
SC-FIM-class SC-FIM-chasSExprTypes
70.97
86.99
42.67
36.36
80,33 (13.18%)
90.42
(3.94
44.18
(3.54%
37.75 (3.82%)
TD-3
62.66
86.18
44.97
38.77
TD-3-MGD
74.26 (18.52%)
91.19 (5.81%)
47.33 (5.24%)
39.94 (3.03%)
Metrics. We use the following metrics to measure the quality of generated code: (1) Compilation Rate (CR): We replace the ground truth method with the generated method in the context of the complete repository and invoke a clean build. We assign a score of if the compilation succeeds, and 0 otherwise. (2) Match with the ground truth: We use three specific metrics measure how closely the generation matches ground truth, namely (a): Next Identifier Match (NIM): If the first Java token generated by the LM matches with the ground truth, assign score of 1, O otherwise; (b) Identifier Sequence Match (ISM): Longest prefix match between the ordered set of identifier names in the ground truth and generated completion, normalized by the number of identifiers in the ground truth; and (c) Prefix Match (PM): Longest prefix match between the ordered set of Java tokens (as obtained from a Java Lexer) between the ground truth and generated completion, normalized by the number of tokens in the ground truth. Except NIM, all other metrics namely CR, ISM and PM evaluate the complete method-level generation by the model.
In our experiments, for all the evaluated model configurations, we use nucleus sampling (Holtzman et al. 2020) with a top-p value of 0.95 to generate = 6 independent samples. For a budget of E vi samples, we compute the aggregate score score@a (see Appendix D) On the discrete- valued metrics (CR and NIM), it is identical pass@k, (Chen al. 2021), estimating the expected number of times the list candidates contains at least one successful compilation or match with ground-truth identifier. On the real-valued metrics (ISM and PM), estimates the expectation of the maximum value of the corresponding metric given k chances.
Python Library for MGD. We are releasing extensible Python library, multilspy, for interfacing between LMs and language servers using LSP. It can be used with multiple programming languages, static analyses and LMs. It also supports approximate mechanism for MGD of black-box LMs with limited gegii-masking support. Please refer to Appendix for more details.
Evaluation
Table D shows the summary of the results for all of our experiments and metrics. For the "-MGD" configurations, we also report the relative improvement over the base model in parentheses, where the base model is the same model configuration without MGD. Below, we present detailed evaluation.
4.1 Effect of MGD Models across Parameter Scale and Architectures
We present results for all the models on Standard prompts described in Section
Compilation Rate. As shown in Figure 2a all the base models with MGD, including the smallest model CodeGen-350M for k: 2 4, outperform the largest model text-davinci-005: by maximum relative margin of 16.55% achieved by SantaCoder. All the models with MGD outperform their respective base models on Compilation Rate, by a relative improvement ranging between 21.77% 24.69%. TD-3-MGD outperforms TD-3 by relative margin of 18.52%.

---

CG-35OM
G--30M-MMGD
CG-28
(G-28MMGD
05-68
06-68-M63
SC.MGD
TD-3
TD-3-MGD
(a) CR
(b) NIM
(c) ISM
(d) PM
Figure 2: score @k for models with MGD and Standard prompt compared against base models. The values of [1, 6] are marked on the X-axis.
Next Identifier Match. As seen in Figure 2b all the models with MGD outperform the respective base models, with relative improvement of 7. 9.2%. The smallest model CodeGen-350M with MGD outperforms the much larger CodeGen-6B with relative improvement
2. 2.76%. SantaCoder with MGD outperforms the larger CodeGen-6B by relative margin of 8 8.42%, and even the largest model rext-davinci-O0s by 2.60%. TD-3-MGD outperforms TD-3 with relative margin 5.81%.
Identifier Sequence Match. Figure 2c shows that all the models with MGD outperform their respective base models ISM, showing relative improvement ranging between 6.41 7.70%. SantaCoder and CodeGen-2B with MGD outperform the larger CodeGen-6BB with a relative margin of 9.47% and 5.00% respectively. TD-3-MGD outperforms TD-3 with relative margin of 5.24%
Prefix Match. Figure 2d shows percentage prefix match with ground truth. All the models with MGD outperform their respective base models with relative improvement of 88%%-682% Both SantaCoder and CodeGen-2B with MGD outperform the larger CodeGen 6B with a relative margin of 8.08% and 4.30%. TD-3-MGD outperforms TD-3 with relative margin of 3.03%.
SC-MGD TD-3. SC is 1.1B parameter model whereas TD-3 has 175B parameters. Being a much larger model and possibly due to other differences in training, TD-3 does better than SC across all metrics. Interestingly, with MGD, SC-MGD outperforms TD-3 on CR (Figure 2a) and NIM (Figure 2b). ISM and PM are method-level metrics and the relative advantage of the larger model prevails. Even then, with small increase in the sampling budget, from manages to surpass TD-3's performance with = on ISM (Figure 2ch and PM (Figure 2d).
k SC-MGD
Summary. MGD improves the compilability of code significantly. across architectures and parameter scale, leading even the smallest CodeGen-350M with MGD to outperform the largest LM, text- davinci-003. We also see improvements in all ground truth agreement metrics. Notably, smaller LMs with MGD outperform larger LMs (CodeGen-3550 with MGD outperforms text-davinci-003 in CR and NIM, CodeCien-2B with MGD outperforms CodeGen-6B ISM and PM) across all metrics.
4.2 Effect of MGD and Prompt Augmentation Strategies
We choose the best perforrrfnn base model, SantaCoder, from Section 4.1 to study the effect of prompting when combined with MGD. 3 results for SantaCoder- (Standard, classEx- prTypes, RLPG) and TD-3, compared against SantaCoder with respective prompts and MGD.
Compilation Rate. Figure shows the results for Compilation Rate. We observe improvements in compilation rate with both the prompting techniques, classExprTypes and RLPG, with RLPG marginally outperforming classExprTypes. We note that SantaCoder with Standard prompt and MGD is able relatively improve over both RLPG and MlasExxrrTppes augmentation by 10.01 % and 13.11% 1% respectively. Further, SantaCoder with RLPG and MGD is able to outperform SantaCoder- RLPG and wlweeorrereeserrrrrrreeleela with relative margin of 17.70% and 21.0210002 21.02% respectively, while increasing the margin of relative improvement over text-davinci-O03 to 24.70%.
Next Identifier Match. As seen in Figure 3b similar to compilation, both RLPG and classExprTypes prompt augmentation leads to improvement over the base model. However, SantaCoder with ei- ther prompt augmentations underperforms text-davinci-0053 SantaCoder with MGD outperforms

---

5C
SC=clasExprryppe
SC-RLIG
SC-MGD
Cysssssoooyypeeee MGD
SC-RPG-NGO
TD-3
1ing
vingt venge
vingo
vinge
varge
uengt segg
veng
Vrange
viigg
ving
(a) CR
(b) NIM
(c) ISM
(d) PM
Figure 3: score @k for models with MGD and prompt augmentation compared against base models.
TD-3, and consequently, both SantaCoder-RLPG and SantaCoder-classEpprr improvement 2.60%, 3.51% and 4.14% respectively. SantaCoder with prompting and MGD outperform their respective baselines (SantaCoder with prompting) by relative margin in the range of 5.23%-5.25%. We note that SantaCoder with RLPG and MGD increases the relative improvement with respect to the largest model, text-davinci-003 to 4.31%.
with relative
Identifier Sequence Match. On ISM, SantaCoder with prompt augmentation and MGD able to outperform its respective baseline by relative improvement of 4.78%-5.00%, while both the prompt augmentations result an improvement over the base model. SantaCoder with RIPG and MGD able to significantly reduce the gap with text-davinci-003. underperforming it by just 1.11%.
Prefix Match. As seen in Figure 3d SantaCoder with prompt augmentation and MGD able to outperform its respective baseline by 4.26%
Summary. While prompt augmentation techniques help in improving performance all metrics, see further improvement with MGD augmentation, and conclude that the contributions by prompt augmentation and MGD complementary. Notably, SantaCoder-RAl RLPG with MGD improves the relative margin for compilation rate with respect to text-davinci-000 to 24. 70% compared to the 16.55% improvement achieved by SC-MGI,, 5.95% improvement achieved by SC-RLPG.
4.3 Effect of MGD on Fill-in-the-middle (FIM) Decoding
Among the base models, SantaCoder supports the FIM modality. We evaluated SantaCoder with autoregressive and FIM decoding strategies and text-davinei-003, and compared them with respective configurations SantaCoder with MGD. Similar to our observations with prompt augmentation, while FIM modality leads improvements across all metrics, using both FIM and MGD. Due to space limitations, detailed results are in Appendix E Motivated by the complementary nature of FIM and MGD, we further evaluated CC-FII-classBxxpTYyyss-MM combining both prompt augmentation and FIM modality. Consistent with our findings, leads to further improvement over SC-FIM-ClasEEppTypee as seen in Figure (Appendix
continued improvement when
4.4 Effect of Identifier Complexity on Next Identifier Match
Identifier names in repositories can get specific and long (Karampatsis et al.) 2020). Due to this, while commonly used APIs may get tokenized into single tokens, identifiers specific to the context of individual repositories, especially in private settings, can span multiple subtokens in the LM vocabulary. We define the complexity of an identifier the mean number of subtokens required to decode it. We show that the ability of LMs to accurately predict the identifier name decreases sharply with an increase identifier complexity, while augmenting them with MGD improves their performance. MGD provides an improvement in the range of 21.79%-27.99 27.91 % compared to the base model without MGD, across parameter scales, for the highest identifier complexity, which is prevalant in more than 36% of the methods in DOTPROMPTS. The detailed results are available in Appendix F

---

5 Generalizability Study
We curate MGDMICROBENCH as micro benchmark (10 examples), spanning 3 programming languages (Java, C#. Rust), 4 coding scenarios, and requiring use of different static analyses, to evaluate generalizability of MGD. The detailed results are discussed Appendix H
Different Programming Languages. MGD utilizes static analyses that help infer and enforce semantic constraints the code under development. Such analyses are available through the
Language Server Protocol (LSP) (Isp) for most programming languages. c.g., clangd for C/C++ (cla 2020 Jedi for Python (Roeca 2019) and Rust Analyzer for Rust (rus) 2018). The monitor used in MGD can be instantiated as a thin client around LSP. Supporting new languages is easy and doesn't t necessitate changes to the monitor's static analysis interface. Hence, MGD is applicable to most programming languages. Results on MGDMICROBENCH demonstrates MGD for Java, C#f and Rust.
Different Coding Scenarios. A monitor under MGD triggers when specified precondition is met during code generation. Flexibility in defining preconditions allows MGD to be applied to a variety of code scenarios, as follows: 1) Instantiation of valid classes: A monitor triggers ' 'new ', invokes a static analysis that identifies instantiable classes from the local & global context to ensure only valid classes are instantiated. 2) switch over enum: A switch statement over enum uses named enums in case <val> to select branch in C# and Java. A monitor triggering on case is used to generate valid enums. 3) Correct number of arguments: A stack-based monitor is implemented to guide for generation of right number of arguments to methods, that also handles nested function calls. 4) Joint monitoring for multiple properties: Unlike previous examples that monitored for single static property, we instantiate 2 joint-moniton a) jointly guiding for type-correct dereferences & along with right number of arguments, b) jointly guiding for valid switch enum branches type-correct dereferences. MGDMICRO (appendix H.22) provides results over all of these scenarios.
Different Static Analyses. MGD is able to utilize results from various static analyses guide the generation of code with LMs. We demonstrate MGD over the following properties in MGDMI- CROBENCH: 1) Typestates (Strom & Yemini 1986): Many APIs are stateful, and require callers to follow specific ordering of calls as part of the API contract. For example, a type representing a file handle would have a contract disallowing calls to read after close has been called. Such contracts can be expressed as finite state machines (FSMs), called typestates. 2) Session Types (Jespersen et al. (2015) ensure that messages between concurrent programs are sent and received in the expected order, following protocol, and are specified as communicating FSMs. In MGDMICRO demonstrate a monitor for Rust that utilizes typestate and session-type design analyses.
we
6 Discussion
Limitations. Though static analysis is a mature field with strong thcoretical foundations and several robust implementations of tools, analyzing partial and incomplete programs is still difficult problem. In practice, editors such as Eclipse and Visual Studio support static analysis with heuristics. Though these heuristics are well-engineered and are widely used, they be both imprecise (they can give incorrect suggestions) and incomplete (they can leave out correct suggestions). Satisfying dunetional-correctnes specifications like pre/post-conditions and invariants is beyond the scope of this work. Consequently, though our results from guiding LMs using these analyses (through MGD) show improvements in quality metrics, additional steps such testing and human inspection are needed to guarantee correctness of generated code.
Societal Impact. Software pervasively affects all aspects of our lives. With LMs being widely deployed as copilots and intelligent assistants to help developers write code, it is crucially important to develop tools like MGD to improve the quality of code generated by LMs (even if humans review and accept the suggestions given by L.Ms). Without such tools, we risk introducing bugs in code due to incorrect suggestions made by LMs, which has the potential to impact all of our lives negatively.
Amount of Compute. Our experiments do not involve any training, and we only perform inferences. We used machines of the following configurations: (1) CPU: 24-core AMD Epyc with 220GB RAM, GPU: Nvidia A 100 80GB. (2) CPU: Intel Xeon(R) Platinum 8168 with 290GB RAM, GPU: Nvidia Tesla V V100 16GB. For the experiments to evaluate 1ext-davinci-003, we used the Azure API.

---

7 Related Work
Pre-trained Models of Code. Many powerful pre-trained models have been designed for code.
These include encoder-only models like CodeBERT (Feng et al.
2020), GraphCodeBERT [uoo et al.
(2020) and CuBERT (Kanade et al., 2020); encoder-decoder models like PLBART (Ahmad et al.
2021), CodeT5 (Wang et 2 I and AlphaCode (Li et al. )022); or decoder-only models like
Codex (Chen et al. (021),, GPT-J ( Wang & Komatsuzaki] [021), J. Austin et al.
(2021). GPT-Neo (Black
ett al.] 2021). GPT-NeoX (Black et al. 2002),, CodeParrot (Tunstall et al. 2022), PolyCoder (Xu
ct al. 2022). Incoder (Fried et al. ).22). CodeGen (Nijkamp et al. 2023), SantaCoder (Allal et al.
2023) and StarCoder (Li et al.,, 2023); or unified models like UniXCoder (Guo al. 2022). Our
monitor-guided decoding works only with logits and hence can be used with any model.
Global Context of Code. |Hellendorrn & Devanbu (2017) build an n-gram model with a cache to
track directory-level context. Xu et al. (2021 use locality based on directory-structure in retrieval-
augmented modeling (Khandelwal et al. 2019 Many approaches use static analysis or previous gen- erations (Zhag et al. (202)) to extract relevant code. They use the relevant context to either augment the prompt (Shrivastava et al. 2022 Pei et al.
2023 Zhang al. 2023) or embeddings (Pashakhan Loo et al. 2022; Ding et al. 2022) presented as imput to the LM. Due to the limit on the prompt or embedding size, these approaches filter information through random walks (Pashakhanloo et al. 2022). classification (Shrivastava et al. 2022) or using fixed pruning strategies (Ding et al. 2022)..
As we neither augment the prompt nor use extra embeddings, we do not need to prune the global context. We let the static analysis generate completion suggestions using the entire repository-level context. Zan et al. (2022) and Zhou et al. (2022) retrieve information from library documentation for
prompt augmentation. Our static analysis analyzes libraries along with the repository-level source code. Several of these techniques require architecture modifications (Pashakhanloo et al. 2022] Ding et al., 2022) or finetuning (Zan et al.) 2022] Ding et al., 2022; Pei et al., (003)). We use a simple interface between logits and static analysis with a frozen LM. Most of these approaches, excluding (Xu et al. 2021 Zhang et al. 2023), use one-time a priori retrieval. In contrast, we provide token-level guidance by invoking static analysis on demand. Our method is complementary to all the above approaches as they all try to condition the generation by modifying the inpur to the LM whereas we apply output-side constraints by reshaping the logits.
Syntactic and Semantic Constraints. There are two primary lines of work to enforce syntactic and semantic constraints on code generation, based on specialized modeling and through constrained decoding. GNN2NAG (Brockschmidt et al., 2019) and NSG (Mukherjee et al., 202T) are examples of the first and use attribute grammars. They respectively evaluated on expressions that do not use user-defined methods or on methods with class-level context. We consider repository context for method-leved completion. Unlike these approaches, work is applicable to aff-the-shelf LMs. PICARD (Scholak et al. (02T) and Synchromesh (Poesia et al.] 2002)) are constrained decoding approaches similar to ours. They use incremental parsing for syntactic validity and design domain- specific semantic checks to ensure semantic validity. Both are evaluated on SQL,, and Synchromesh additionally considers domain -emaii- languages for visualization and calendar applications. In comparison, we target generation of general-purpose programming languages with focus on semantic constraints like type-consistency, API protocols, etc. using static analysis over repository context.
Conclusions and Future Work
In this work, we show how use repository-wide information computed by static analysis (specif- ically, type-based analysis) using a stateful monitor as an interface, to improve quality of code generated by LMs. Our experimental results show the potential for significant quality improvements for code generation using this approach. Our approach is complementary to prompt augmentation techniques. It allows smaller models to achieve better or competitive performance compared to much larger models. This could open up the possibility of using smaller models directly within IDEs, alongside our monitor, as an alternative the use of remotely-hoster Large LMs (LLMs), reducing inference costs and improving privacy. Our method is general and applicable to various coding scenarios where LMs are used generatively, such as code refactoring, code repair, or code completion, even if the repositories are in a transient state. We plan to expand the scope of MGD to more languages and deeper semantic analyses such as pre/post-conditions for which advanced constrained decoding methods (including backtracking and beam-search) might be needed.
10

---

Acknowledgements
We thank the authors RLPG and of the LMs (CG, SC, TD-3) used in our work. We are grateful to Sheng Chen and Shi Chen from Microsoft, and the author of OLSP, Predrag Nikolic, for their generous help in answering our queries about interfacing with LSP. Our paper also benefited significantly from the constructive feedback by the reviewers.
References
Official page for Language Server Protocol. URL htt.htts////ccooo...ggtttto 1 Lo/language- erver- proer-protoco//
Protocol Buffers.
https::/protobuuf.c dev/ 2008. Accessed: May 14, 2023.
Eclipse JDT LS, httpe://projeccionbok... jdt. ls
Project Lombok.
2009, Accessed: May 14, 2023.
2016. URL https:
rust-analyzer, 2018. URL https://rus--aallzzer. www.olll..r..gttb.... io/
What is clangd?, 2020. URL httpp:::/cllg.........rrrr 1lvm.orgg/
Language Server Implementationss https://microsoff .inrtttggtttb....... 10/ http://gt......./aagggggggee//ee seweeaaaaeee---eee-epeaee emeetoors/eeeeeersaa.s taccl////mpeeeaatorrr//eeeeesss 2023. Accessed: May 14, 2023.
Ahmad, W. U., Chakraborty, S., Ray. B., and Chang, K.-W. Unified pre-training for program
understanding and generation, 2021.
Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu. A., Dey, M., and others. SantaCoder: don't reach for the stars! arXiv preprint arXiv:2301.03988 2023.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program Synthesis with Large Language Models, August 2021. URL sttp://arxiv. wrg. ab/////s.//../////////////// 07732 arXiv:2108 0et:21088.7773 [es].
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., and Chen, M. Efficient training of language models to fill the middle, 2022.
Black, S., Gao, L., Wang, P., Leahy, C., and Biderman, S. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite using these metadata, 58, 2021.
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., and others. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint (w..iv:224.0077:: 2022.
Brockschmidt, M., Allamanis, M., Gaunt, A. L.., and Polozov, O. Generative Code Modeling with Graphs. In International Conference on Learning Representations, 2019, URL https "opperrerier.net//oouudddd.dd..
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, d. O., Kaplan, Edwards, H., Burda, Y., Joseph, N., Brockman, G., and others. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374, 2021.
Crichton, W. Typed design patterns for the functional era. In Proceedings of the 1st ACM SIGPLAN International Workshop Functional Software Architecture, FUNARCH 2023, pp. 40-40, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702976. N0.1145/3609025.366
doi:
URL httpp::///oooo org/ 1045//169///5 /ttp//336000255594477
Crichton, W. Navarro, Semeniuta, D., Zeng. S., and Gupta, S. Session types, November 2019. URI. https: htttps:/sstanford-ioo cs242,
-tt topett htm]

---

Dagenais, B. and Hendren, L. Enabling static analysis for partial java programs. In Proceedings of the 23rd ACM SIGPLAN conference on Object-oriented programming systems languages and applications, pp. 313-328, 2008
Ding, Y., Wang, Z., Ahmad, W. U., Ramanathan, M. K., Nallapati, R., Bhatia, P., Roth, D., and Xiang, B. CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Context, December 2022. /ttt//////rrxi org/ 2twp/ 10007 arXiv:2212. 10007 [cs].
Donahue, C., Lee, M., and Liang, P. Enabling language models to fill in the blanks. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 2492-2501, Online, July 2020. Association for Computational Linguisties. doi: 10. (0.18653/112020..ak 2ttS5//112220aclmmmai.2222 mlll-mma..... URL attpps:/aat haap:////ccaattlooo/o hwwpillooyggggggg/oeee org.ooog0000
225
Duarte, J. and Ravara, A. Retrofitting typestates into rust. In Proceedings of the 25th Brazilian Symposium on Programming Languages, SBLP '21, pp. 83-91, New York, NY. USA, 2021, doi: 10. 00.114/3445501.34455000
Association for Computing Machinery, ISBN 978 7781450390600
URL httph://tool org/10. 1145/ 337774447700117755522 3475082
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., and others. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020.
Fried, D., Aghajanyan, A., Lin, J., Wang. S., Wallace, E., Shi, F., Zhong, R., Yih, W.-t., Zettlemoyer, L., and Lewis. M. Incoder: A generative model for code infilling and synthesis. arXiv preprint
arXiv:2204.05999 2022.
Fuhrer, R. M. Leveraging static analysis in an ide. Generative and Transformational Techniques in Software Engineering IV: International Summer School, GTTSE 2011, Braga, Portugal, July 3-9, 2011. Revised Papers, pp. 101-158, 2013.
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., and others. Graphcodehert: Pre-training code representations with data flow. arXiv preprini MXXi:2009.00366 2020.
Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., and Yin, J. Unixcoder: Unified cross-modal pre-training for code representation, 2022.
Hedin, G. Incremental semantic analysis. Department of Computer Sciences, 1992.
Hellendoorn, V. J. and Devanbu, P. Are deep neural networks the best choice for modeling source code? In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pp. 763-773, 2017.
Holtzman, A., Buys, J., Du, L.,, Forbes, M., and Choi, The Curious Case of Neural Text Degeneration. In International Conference Learning Representations, 2020. URLhttps //openreview.o pent//formm?e www.orriddrrroooooo idryygdyryvyi
Jespersen, T. B. L., Munksgaard, P., and Larsen, K. F. Session types for rust. In Proceedings of the 1 Ith ACM SIGPLAN Workshop Generic Programminng, WGP 2015, pp. 13-22, New York, NY, USA, 2015, Association for Computing Machinery. ISBN 9781450338100 doi: 10.1 10.1145/2808098 2808100. URL |tt. httttttss:///i.ttttteetttteeteee oreeii.rrg11..0 1ew.11115/28880 2808100
Kanade, A., Maniatis, P., Balakrishnan, G., and Shi, K. Learning and evaluating contextual embedding of source code. In International conference on machine learning, pp. 51 10-5121. PMLR, 2020.
Karampatsis, R.-M., Babii, H., Robbes, R., Sutton, C., and Janes, A. Big code!= big vocabulary: Open-vocabulary models for source code. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 1073-1082 1085, 2020.
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L.., and Lewis, M. Generalization through memorization: Nearest neighbor language models. arXiv preprint wtXiv:/911.00//12100 2019.
Kudo, T. and Richardson, J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.062200 2018.
12

---

Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone. M., Akiki, C., Li. J., Chim, J., al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 2023.
Li, Y., Choi, D.. Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, D., Hubert, T., Choy, P. d' A'tttmme C. d. M., Babuschkin, I., Chen, X.. Huang, P.-... Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Robson, E. S., Kohli, P., Freitas,, N. d., Kavukcuoglu, K., and Vinyals,
Competition-level code genera- tion with AlphaCode. Science, 378 37866224)1099-10977 2022. doi: 10.11 10.1126/science.ahg URL https: //www http://www..iiic.o. scwence or /ttp:///ee.r////////////a///// 1tt.:1......1111 w126/science....oo abq1158 _eprint:
Maddox III, W. H. Incremental static semantic analysis. University of California, Berkeley, 1997.
Mishra, A., Kanade, A., and Srikant, Y. N. Asynchrony-ave aware static analysis of android applications. In 2016 ACM/IEEE International Conference on Formal Methods and Models for System Design (MEMOCODE), pp. 163-172, 2016. doi: 10.1 10009//EMCCD.201...777.
Mukherjee, R., Wen, Y.. Chaudhari, D., Reps, T., Chaudhuri, S., and Jermaine, C. Neural Program Generation Modulo Static Analysis. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, W. (eds.), Advances in Neural Information Processing Systems, 2021. URL httph://opeerree view www.:eeeefforue wew.te/ooormmmiyyasssC""C idyyaissCyyaasass
Nielson, F., Nielson, H. R., and Hankin, C. Principles of program analysis. Springer, 2015.
Nijkamp, E., Pang, B., Hayashi, H., Tu, L.,, Wang, H., Zhou, Y., Savarese, S., and Xiong, C. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In The Eleventh International Conference on Learning Representations, 2023. /forum?id=iaYcJKpp22
.net
Ouyang, L.., Wu, J., Jiang, X., Almeida, D.,, Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022.
Pashakhanloo, P., Naik, A., Wang, Y., Dai, H., Maniatis, P., and Naik, M. Codetrek: Flexible modeling of code using extensible relational representation. 2022.
Pei, H., Zhao, J., Lausen, L., Zha, S., and Karypis, G. Better context makes better code language models: A case study function call argument completion. In AAAI, 2023. URL https: //www. amazon science httpc/////pbbaaaaaaaaaattteerrra http:///ttterrccooottttt---eeeee www.taaaeeeeeeeeeeeeeeeeeeeeeeee code- www.ooaneeeeeaeaoooaooaoaooooa age-models-a- case -tw.---tto--oo---utttoooo------ oog.aaaooaaaagaaaaaaeooooooooooo
Poesia, G., Polozov, A., Le, V., Tiwari, A., Soares, G., Meek, C., and Gulwani, S. Synchromesh: Reliable code generation from praitedi language models. In /CLR 2022, April 2022. URL
https::/www.. www.://oo...icrooot..ttt com/ en http://nseeeeeerrrrcooliiieeeeee oagp:///c/ppbalaaaaiiooocccoccea synchromesh reliabl -angudd----aggaag--mmddeesss wtwp:oorrraaaaadaaaaaagaggeeeaa.
e- -odee generation frrne--rommm- -room-rr----- prrom-pre-rraineede
Reps. T., Teitelbaum, T., and Demers, A. Incremental context-dependent analysis for language- based editors. ACM Transactions Programmingg Languages and Systems 5(3 1983.
Roeca, S. jedi-language-server, 2019. URL http://p////hhhooo.oo//aaaaaaaap
8 server.
Rust Embedded Devices Working Group al. The embedded rust book, 2018. URLhtts:/// rust must-lang.o htgp//eta/embbeddedbboooto wwweeeeddbooo///eeeeeeeeoedd statictttticccc httpaa/acccccttttteeeeeeeaoaataa
Scholak, T., Schucher, N., and Bahdanau, D. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. In Proceedings of the 202 Conference on Empirical Methods in Natural Language Processing, pp. 9895-9901, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 18653 /v 1/202 w.w.nl--ppppppe...m 779..779, 779, URL https: http:////aaaaaalo///aeo org/ 2021. emnlp-mail 779
13

---

Schuster, M. and Nakajima, K. Japanese and korean voice search. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 1142-5152. IEEE, 2012.
Sennrich, R., Haddow, B., and Birch, A. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computa- tional Linguistics. doi: 10.18653/v 1/P11/ 1162.
aettttss://aaaaanholooooo org/P16- 1162
Shrivastava, D., Larochelle, H., and Tarlow, D. Repository-Ievel prompt generation for large language models of code. arXiv preprint arXiv:2206. /2839, 2022.
Strom, R. E. and Yemini, S. Typestate: A programming language concept for enhancing software reliability, IEEE Transactions on Software Engineering, SE-12:157-171, 1986. URL https /aapi.semantccccheeee
org/ Corpus 15575346
Tunstall, L., Von Werra, L.,, and Wolf, T. Natural language processing with transformers. "O'Reilly Media, Inc.", 2022.
Wagner. T. A. Practical algorithms for incremental software development environments. University of California, Berkeley, 1997.
Wang, B. and Komatsuzaki, A. GPT-J-6B: A 6 billion parameter autoregressive language model, 2021.
Wang, Y., Wang, W., Joty, S. R., and Hoi, S. C. H. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. ArXiv, abs/2109.0085. 2021.
Wolf, T., Debut, L.,, Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q. and Rush, A. Transformers: State-of-the-Ar Natural Language Processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10. 10..8853/vvll http:///2000002eeemmeeeee https://aclantholl
URL
org/2020/ omnlp momml-demos.... demos 6
Xu, F. F., He, J., Neubig, G., and Hellendoorn, J. Capturing structural locality in non- non-parametric language models. arXiv preprint arXiv:2110.02870 2021.
Xu, F. F., Alon, U., Neubig. G., and Hellendoom, V. J. Systematic Evaluation of Large Language Models of Code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine
Programmiing, MAPS 2022, pp. 1-10, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978- 1 1-1501--52772335 doi: 10.1 (115//520312.3534482 URL httthtts///do...... g/ g/10. 1..011145/352012.35533 3534862 event-place: San Diego, CA. USA.
Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q. Ma, Y., Liang, G., Li, Y., Xie, T., and Wang, Q. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models, February 2023. htR/http::// httpp://wwwwr..p/p/aaa///a/a//// 2302 00288, arXiv:2302.0025. [cs].
Zan, D., Chen, B., Lin, Z., Guan, B., Yongji, W., and J.-G. When Language Model Meets Private Library. In Findings of the Association for Computational Linguistics: EMNLP 2022, 277-287 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL HRRLhttps: acl antholooy org/ 2022.2222..22 menddiing-emllp.... 21
Zhang, F., Chen, B., Zhang, Y., Liu, J., Zan, D., Mao, Y., Lou, J.-G., and Chen, W. RepoCoder: -epesitory-Level Code Completion Through Iterative Retrieval and Generation. arXiv preprint arXiv:2303.1 12500000....0. 2023.
Zhou, S., Alon, U., Xu, F. Jiang, 7.., and Neubig, G. Docprompting: Generating code by retrieving the docs. In The Eleventh International Conference on Learning Representations, 2022.
14

---

Appendix: Monitor-Guided Decoding of Code LMs with Static Analysis of Repository Context
Outline
Monitor for the Running Example
Data Set Curation Details
Experimental Setup Additional Details
Calculation of @core@k
Effect of MGD on Fill-in-themmidlle (FIM) Decoding Complete Results
Effect of Identifier Complexity on Next Identifier Match Complete Results
Impact of MGD on Inference Time
Generalizability aspects of MGD
CodeQL Query for Identifying Evaluation Target Methods
Examples of code generation with MGD
A
Monitor for the Running Example
Logi
Loptts
Mecified uppits
No
Co
spdate up')
spdatelar
pra
C EprSttring to) T Port Cintager port)
NO PerSing
Vacter
Figure Monitor guide generation of type-consistent identifiers for the code in Figure
Figure 4 shows how the monitor interacts with the LM decoder for the example Figure 1 Initially, the monitor, M is in the wait state 8g- Given the code completion input, I), 22, Trs Msp first evaluates pre(so; 21 .r.,,....,), Since F,
(the dot symbol indicating object dereferencing in Java), T1, .r,)) evaluates to true, and subsequently, in accordance with Eq. (4), the static analysis Ag is invoked, which determines the input prompt to be in accordance with the property so, and resolves the type for the completion point to be ServerNode. Builder, as shown in the annotated AST in Figure A.p then returns the set of identifiers consistent with the resolved type- [withIp, withPort, newServerNode, I. transitioning the monitor M. to state $1 M, then calculates m maskgen(31, V), which masks out, for example, the token host (as inferred by SantaCoder in Figure b)). Concurrnntyy, the input is tokenized and the LM Lg provides inferred logits, for the next token. The output logits from La and mask m are combined as @ m to obtain the modified logits, which is then softmaxed and a token is sampled wwith in this case. The monitor then invokes update(s), with with) to transition to 82. Note that with the state transition, newServerNode pruned from the set of identifiers, the sampled token with does not prefix it. Le provides logits for the

---

next token, and the composition of Le and M₂ is repeated to obtain modified logits, and finally we obtain the sampled token Ip. Since Ip is prefix of a member in $2, update(s22.) Ip) transitions to 83. with 83 = {+), a singleton consisting of the empty string. Note that the token "(" is consistent with the suggestions in state 83, since it matches the regular expression w F. Σ", where w = 6. Following the sampling of the token "(, the monitor transitions back to the wait state 80-
B Data Set Curation Details
For the evaluation of pragmatic code generation, we require real-world repositories, with their complete environments and dependencies. Most public datasets for code generation do not meet this requirement, due to their focus on standalone code generation, except the recent CoderEval (Yu et al. 2023) and PyEnvs (Pei et al. 2023), both of which are not publicly available at the time of this writing. Further, CoderEEval just evaluates 10 Java and 43 Python repositories. Since these datasets do not filter for repositories by their creation date, the test repositories could be a part of the training set for the evaluated LMs. Considering these, we describe the curation of PRAGMATICCODEL a dataset of real-world open-source Java projects complete with their development environments and dependencies. We ensure that the training repositories were released publicly only after the determined training dataset cutoff date (31 March 2022) for the CodeGen (Nijkamp et al. 2023). SantaCoder (Allal et al. 2023), and text-davinci-O0f (GPT-3.5) (Ouyang et al.) 2022) family of models. All the repositories PRAGMATICCODE are buildable with their respective build systems. The development environment also provides support for analysis over templated code generated with systems like ProtoBuf (pro) (008) and Lombok (lom) 2009). Further, we create DoTPROMPTS from PRAGMATICCODE for the evaluation of code generation in a pragmatic setting.
PRAGMATiCCODE. We queried the GitHub API on 25 March 2023 to obtain list of the top 1000- Java repositories on GitHub created after the determined cutoff date of 31 March 2022. We then attempt to download the latest snapshot of each of the top 1000-starred Java repositories and were able to download 731 repositories from GitHub. We create build environment for Java projects consisting of Oracle Java Development Kit 17.0.6, Apache Ant 1. 1.10.13. Apache Maven 3.8.7, Gradle 7.3.3, and GitHub CodeQL CLI 2. 12.5. In this build environment, we invoke the CodeQL. database create command for every software repository obtained from GitHub. The CodeQL database creation process identifies the build system used in the repository and invokes the command for clean build. The respective build systems use the project-level dependency information stored in configuration files like pom.xml and build. build.gradle to fetch the dependencies and store them locally. Next, we filter for repositories with permissive licenses, and filter out the repositories for which the CodeQL database creation failed, that indicates that the repository either uses unrecognized build system, some of its dependencies aren't It satisfied, or the repository is in a transient state. We are left with 302 GitHub repositories after filtering for successful builds. We store the CodeQL database created for each of these repositories. Finally, we invoke the initialization of Eclipse JDT.L. on each of the repositories, and filter for the repositories where JDT.LS could be successfully initialized. The final filtered list of 100 repositories, along with their CodeQL databases, comprise the PRAGMATICCODE dataset.
PROMPPSS Yu et al. 2023) show that standalone functions account for less than 30% of open source projects among the top 100 most popular open source projects on GitHub, with most functions referencing third-party APIs or variables/constants defined in cross-fil context. Henee, we create DoTPROMPTS for the evaluation of code generation in a pragmatic setting and aim to evaluate over rorldworld projects, not restricting to standalone function generation. Each task instance in DOTPROMPTS consists of prompt up to a dereference location (dereference '.' operator in Java) within a target method, and the task is to complete the remainder of the method. Since dereference locations might be the points of occurrence for cross-file entities in source code, a model's 's ability to cross- file context can be evaluated using DOTPROMPTS. Curation Details. We identify non-test class files that aren't t auto-generated and have at least cross-file dependency in the repository. From these files, we identify methods that aren't class and object initializers and have top-level statements spanning 2 7 lines of source code, to ensure sufficient complexity in target methods. The average number of lines of code in the ground truth completion in DoTP RRMPTS 12.7. We use the CodeQL Query listed in section to identify target methods from each of the repositories in PRAGMATiCCODe based on the above criteria. As shown by SShrivastaaa al. (202)) repositories are
"happs///ocs.githhhe wttp//////oooeeeecccetoeeeeeeeee htwpa//aw.....o.../o.//////aaaaa

---

quite uneven in terms of their size, so to avoid individual repositories from dominating our evaluation, we limit to including up 20 methods from each repository as identified by the CodeQL query. In order to simulate the real-world usage scenario, where developer may invoke code completion at different points within a method, we identify up 10 uniformly distributed dereference locations within each of the identified methods. Each such dereference location becomes data point for DOTPROMPTS.
C Experimental Setup Additional Details
Monitor Implementation. Language Server Protocol (LSPP (ss)) is an open industry standard of communication between IDEs and programming language specific tools like static analyzers and compilers, called language servers. Eclipse JDT.LS (eel. 2016) a language server for Java, Rust Analyzer provides support for Rust, OmniSharp supports C# and JEDI supports Python. All language servers provide access to results various static analyses over the same API. These can be accessed by implementing Language Server Client. We implement an extensible and cross- cross-platform language server client, multilspy with the aim to make it easy to setup and use these language servers in language- -agnostic manner. At the time of this writing, mult multilspy has been tested to work with language servers for Java, Python, Rust and C#, and we plan to extend the support to more programming languages and language servers.
Using multilspy, we specifically instantiate Eclipse JDT.LS as an engine that implements the static analysis Ap to check for type-consistency of identifiers. Notably, Eclipse JDT.LS supports reasoning with build time artifacts like those obtained from Project Lombok (lom) 2009) and ProtoBuf (pro) 2008), thus allowing to consider broad view of the code repository in the static analysis results provides. We implement our monitor M, as a thin layer around an LM, in accordance with Section 2 language server client that communicates with Eclipse JDT.LS. Since implementation based on LSP, and LSP is compatible with most major programming languages, our implementation can be easily ported to other languages besides Java, with the use of multilspy.
Hyperparameters. We use nucleus sampling (Holtzman al. 2020) with a top-p value of 0.95 to generate 6 samples in total-1 each with temperature 0.2 and 0.4, and 2 each with temperature 0.6 and 0.8. We fix a prompt budget (2048-512)=156 tokens, and generation budget of 512 tokens, for total context window size of 2048. If the text exceeds the prompt budget, we truncate it from the left for standard and classExprTyyps prompts, and from the right for FIM. For classExprTyypes augmentation with autoregressive decoding, we reserve budget of 20% of total prompting budget for classExprTypes, and the remaining for autoregressive prompt. For FIM decoding without prompt augmentation, we reserve 50% of total prompting budget for the suffix. For FIM decoding with classExprTyype prompt augmentation, we reserve 20% of total prompting budget for classExprTyppss, 40% for suffix, and the remaining for autoregressive prompt.
D Calculation of score@k
Let S (51, s2..... } be multiset representing metric scores obtained across n-independent trials. Without loss of generality, we order S in monotonic decreasing order S2 calculate score n as per the equation below:
(,, ..., $2) We
1+
n2 -
score
S2
max(1))
(5)
k
TE(*)
(s) = [VIV S.
(6)
where 1 k S n, and (9) is the set of all subsets of S having cardinality A.
3

---

SC-FIM
Sc-slassExpryype
SC-FIM-ciasEExpryypes
TD-3
SC-MGD
SC-FIM-MGD
Ec-classExprTypssMMM.
SCFIMcciassExprryyes-GGD
75
86
05
50
(a) CR
(b) NIM
34
42
40
38
<urug2
sooreg1
scoreg2
scorego
scoregp
scorego
scorego
(c) ISM
(d) PM
Figure 5: score k for models with MGD and FIM compared against base models
E
Effect of MGD on Fill-in-the-middle (FIM) Decoding Complete Results
Among the base models, SantaCoder supports the FIM modality. Figure 5 shows the results for SantaCoder with autoregressive and FIM decoding strategies and text-davinci-003, compared with respective configurations of SantaCoder with MGD.
Compilation Rate. Figure 5a shows that SantaCoder with MGD outperforms SantaCoder-FII by relative margin of 7.0 7.04%. We see significant improvement in compilation rate, when SantaCoder-FIM outperform text-davinci-003 with relative margin of 27.97%.
augmented with MGD, leading
SantaCoder-FFi with MGD relatively improves over SantaCoder-FIMM by 17.52%.
Next Identifier Match. Figure 5b shows that FIM modality boosts next identifier match but still underperforms text-davinci-00 000 and correspondingly. SantaCoder with MGD. Augmenting SantaCoder-Fi FIM with MGD leads relative improvement of 5.07%.
Identifier Sequence Match. On ISM, SantaCoder-Fii with MGD improves over SantaCoder- by 5.39%, which closes the gap with text-davinci-003, underperforming it by just 1.06%.
Prefix Match. Figure 5d shows that SantaCoder-FIM improves over SantaCoder, still under- performs text-davinci-003 by relative margin of 6.83%. SantaCoder-Fii CMM with MGD outperforms

---

SantaCoder-FIM by 4.95% showing continued improvements, while reducing the gap with text- davinci-003, underperforming it by just 2.21%. 1.
Summary. Similar to our observations with prompt augmentation, while FIM modality leads to improvements across all metrics,
continued improvement when using both FIM and MGD.
Motivated by the complementary nature MGD, we further combining both prompt augmentation and FIM modality, and consistent with our findings, it leads to a further improvement over AC-FIM-clasSEXprType as seen in Figure 5 We ElassExprTypes with FIM since RLPG also selects subset of post-lines lines in prompt augmentation in a large number of cases.
D.--IN--haSSExpPYyee---EEE
evaluated
Effect of Identifier Complexity on Next Identifier Match Complete Results
13.23
200 (14 0894
14.201
285 120 07%)
410 87%)
60
[3 41
(a) Distribution of methods by complex iden- tifier
[1.2]
[2.
[3
[6, 18)
(b) Standard prompt
90
85
70
86
[1
12.3)
[1. 4
[4. 1.0
(1.2]
(2.
(3.4)
[5. 18)
(c) Prompt augmentation
(d) FIM
Figure 6: (NIM, score across next-identifier complexity
Identifier Complexity. Identifier names in code repositories can often get specific and long (Karam- patsis et al. 2020). The vocabulary of LMs like CodeGen and SantaCoder is generally created by training a BPE tokenizer over complete or sample of the pretraining dataset (Sennrich et al.) 2016). Due to this, while commonly used APIs may get tokenized into single tokens, identifiers specific to the context of individual repositories, especially in private settings, can span over multiple subtokens in the LM vocabulary, making their accurate generation difficult for the LM (both due to

---

Table Statisties on inference time comparing CodeGen-Oo and Codegen-6B with MGD. Time in seconds.
CodeGen-6B
CodeGen-6B-MGD
Mean
22.57
41.34
Mean slowdown
83.16%
Standard deviation
12,44
22.39
Min
0,94
1.14
First quartile
21.23
26.76
Median
25.48
44,56
Second quartile
25.86
58.20
Max
79.06
111.73
the increased number of decoding steps leading to a larger search space and also due to the relative rarity of the identifier name). We define the complexity of an identifier as the mean number of subtokens required to decode it, using the tokenizers of all the models under study (CG, SC, TD-3). About 36.97% of methods in DOTPROMPTS dataset have at least 1 identifier of complexity [4, 18) as shown in Figure (6a] and therefore, for a model to generate those methods correctly, it is important for the model to be able to generate the complex identifier name.
Next Identifier Match. Figure shows the result for the NIM metric, across increasing complexity of the next ground-trutt identifier. We note that all models show a sharp degradation in performance with an increase in identifier complexity. The same trend holds true for prompt augmentation as well as FIM modality. Augmenting base models with MGD leads to significant improvement in the model's ability to predict the next identifier for the most complex identifier case [4, 18) with relative improvement in the range of 21 28%. The smallest model, CodeGen-350M with MGD achieves parity with the largest model, text-davinci-003 and outperforms the much larger CodeGen-6B with a relative margin of 11.95%. CodeGin-2BB with MGD outperforms CodeGen- 6B and text-davinci-003 by relative margin of 22.14% and 9.79% respectively. SantaCoder with MGD) improves over text-davinei-003 by a relative margin of 11.53% We further observe that both prompt augmentation and FIM-modality with MGD lead to a diminishing rate of degradation, as can be seen in the curves for SC-FIM-MGD in Figure 6d SantaCoder-FIM with MGD outperforms text-davinci-003 and SantaCoder- by a relative margin of 16.11% 1% and 17.04% respectively.
Summary. The ability of LMs to accurately predict the identifier name decreases sharply with an increase identifier complexity. All models with MGD outperform their respective base models with large relative improvement in the range of 21 2-%, with smaller I.Ms outperforming much larger LMS GoodeGien3300-MMGGD achieves parity with text-davinei-003, and outperforms CodeGen-6B, While prompt augmentation and FIM modality lead
CodeGen-2BB 2GG outperforms CodeGen-óil to improvement over baselines, they also suffer from a sharp decrease across identifier complexity, but augmenting them with MGD leads to similar large improvements observed with base models (relative improvement in the range of 15.92%-18.559%
G Impact of MGD on Inference Time
To study the impact of adding MGD inference time, we compare the code generation times by CodeGen-6B model and CodeGien66 with MGD. We select 500 prompts from DOTPROMPTS and generate up 512 tokens with CodeGen-6B as well CodeGen-6B with MGD. The inferencing performed with HuggingFace implementation of the model on machine configuration (1) as described in Section We use the same decoding scheme as described in Section 3 After generation, we filter out the cases where the number of tokens generated by CodeGen- 6B and Codegen- -6 with MGD is not the same, in order ensure parity both in the size of the input prompt and number of generated tokens. We are left with 161 instances, where the size of the input prompt, as well as number of generated tokens, is the same for both models. Figure shows statistics related to inference time for both the models in seconds. We observe mean slowdown of 83.1 in decoding time for CodeGen-6B with MGD. We note that there might be many opportunities to optimize our implementation though we have not explored them yet.

---

H Generalizability Study: Microbenchmark Results
Table 3 presents results over MGDMICROOENCH, detailing the scenarios evaluated for studying the generalizability of MGD.
H.1 Different Programming Languages
In MGNM,CROBENCI, NAI, VD1, are coding scenarios in Java, SE1, SE2 and J/ are coding scenarios in C#, and TSI, TS2 and STI are coding scenarios in Rust for which monitors under the MGD framework, targetting the specific languages are built. Section 4 provided detailed evaluation of monitor for Java, Hence, collectively, section and MGDMICROBENCH demonstrate that MGD is viable for Java, C# and Rust.
H.2 Different Coding Scenarios
Instantiation of valid classes. Scenario CII (abbreviation for "class instantiation") considers demo code for a university setting, where Person is declared as an abstract class, extended concretely by Student and Teacher classes. In the scenario, the prompt code is setup to declare an object p1 of type Person as follows: 'Person p1 new ;. The task for the LM is to generatively complete the code, assigning a concrete instantiation of Person. Following the keyword 'new', constructor should be invoked. SC configuration, without MGD, invokes the non-existen constructor Person (. (non-existent since it is an abstract class), whereas SC-MGD, with monitor that differentiates between abstract and concrete classes invokes Student (.. (...), which is valid as per the scenario.
switch over enum. Scenarios SE1, SE2 (abbreviation for "switch over enum") and JI (abbreviation for "joint monitoring") are representative scenarios obtained from real world emulation software written C#. In scenarios SEI and SE2, the prompt code has been setup to "switch(.... over enum values of type 'AccessSize' and 'Intrinsic' respectively. The model configurations used are SC and SC-MGD, While SC generated the 'case' branch '1', which is a violation of the enum type, SC-MGD correctly generated the case branch 'AccessSize .Byte' which is type-correct while also being consistent with the ground truth. In SE2, while SC was able to get the type-name of the enum value correct, i.e., it generated 'Itrinsscc', the enum value under the enum type it generated is non-existent, i.e., the symbol 'X86Comisdgt' does not exist, and hence invalid. SC-MGD generates valid case branch, 'Intrinsic. X86Comisdlt', consistent with the ground truth. Scenario SE2 is not completely solved due to the invalid dereference of LessThan' by SC-MGD, due to the absence of monitoring for dereferences, which is resolved in scenario J/ below, in the discussion of joint monitoring.
Valid number of arguments to methods. Scenarios NAI (abbreviation for "number of arguments") VDI (abbreviation for "valid dereferences") and J2 are derived by modifying the underlying API presented in Figure Ta (in the main paper). The task is to use the ServerNode Builder fluent API to create an instance of ServerNode. For these scenarios, we replaced individual methods withIp (ip) and withPort (port) (as seen in Figure Tab with single method wi withIpPort (ip, port) after which build() can be called to instantiate the ServerNode object. In scenario NAI, the prompt code has been setup with an open call to method wi withIpPort (, and the task for the LM is to generatively write code to pass the correct arguments to the method. SC configuration, without MGD generates single argument, and closes the method call, thus violating the API of the withIpPort method. It further generates a call to the non-existent method withPort. SC-MGD configuration guided by a monitor for valid number of arguments, generates two arguments, corresponding to ip and port respectively, ln-line with the contract of withIpPort. Further, it calls the method (u In scenario VDI, the setup is similar to NAI, except that the call to the method withIpPort is not made, and instead, the task for the LM is to generate the call to the right method wi withIpPort, and then generate the arguments for it. Unlike NAI, in this scenario, we use only the dereferences monitor. While the base configuration, SC without MGD, generates call to non-existent 'hostAddress', SC-MGD with monitor for dereferences calls the right method, 'wihhItPr'''. However, without the monitor for right number of arguments, it just generates one argument, thus violating the API. further calls withIpPort a second time, which is a type-correct dereference, however, not the expected response per the ground truth, since it is redundant call. Scenario VD1 is not completely
7

---

Table 3: Results over MGDMICROBENCH
ID DESCRIPTION GENERALIZATION ASPECT
WITHOUT MGD
WITH MGD
Code scenario Valid class instantiations
University Instantiation CII
Code Scenario, Java
Person pl Person ("John", ...)
Person pi
Student ("John", ...)
and assignment
to abstract
class
Code scenario Valid enum constants within switch statements
SEI
Emulation switch over
Code Scenario, C#
AccessSize Byte
AccessSize
cnum
SE2 Emulation switch over Intrinsic cnum
Code Scenario, C#
case Intrinsic. X86Comisdgt
case Intrinsic. x86Comisd1
X86Condition.
X86Condition.
Greater Than
LessThan
Code scenario Correct number of arguments in method calls NAI
Fluent API Code Scenario, Java Parse Server
[0])
arr [0] trimm)
withhort(....)
integer.parsele
arr [1] tr)
build()
Code scenario - type-correct dereferences
VDI
Fluent API
Code Scenario, Java
hostAddress (arr [0]) withIpPort (arr [o]]
Parse Server
port (Integer.
wwihhIpPottarr [1])
parseInt (arr (1]))
Joint monitoring for valid enum constants in switch and type-correct dereferences
JI Emulation
Code Scenario, C#
case Intrinsic.
case Intrinsic.
switch over Intrinsic cnum
X86Comi sdgt
x86ComiddIt
X86Condition.
X86Condition.
GreaterThan
Below
Joint monitoring type-correct dereferences and number arguments
J2 Fluent API Parse Server
Code Scenario, Java
hostAddress (arr [0])
withlpPort(
port (Integer. parseInt (arr (1]))
arr [0] trim()
Integer pargeent
arr [1] trimm))
build()
Static Analysis
Typestate API protocols
TSI
Android Me-
Typestate, Rust
stop ();
reset ();
diaPlayer
TS2
GPIO Pins
Typestate, Rust
set_ aet_input_ppul..
aet_input..igh.z0
Static Analysis - Session'Type 2-party communication protocol
Deposit
SessionTypes, Rust
recv ();
send (0) recv())
money to
ATM

---

solved due to the invalid number of arguments by SC-MGD, and will be resolved in scenario J2 below.
Joint monitoring for multiple properties. MGD framework can utilize results from multiple monitors simultaneously guide generation of code to follow multiple properties. Scenarios J/ and J2 explore 2 different instances of joint-monitors operating simultancously. J1 is the same coding scenario as SE2. In SE2, the monitor only monitored for generation of type-valid 'case' branches, whereas J1, the monitor for valid case branches and monitor for type-valid dereferences are used jointly to perform the decoding. The difference can be noted in generation of the correct dereference X86Condition Below' in JI (highlighted in green) for SC-MGD, which earlier was 'X86onniiiion LessThan' in SE2, resulting in a non-existent symbol error. In J1, SC-MGD with the joint monitor is able to generate both the correct case branch, as well as valid dereference. Scenario J2 is the same coding scenario as VDI. In VDI, only monitor for valid dereferences was used in SC-MGD, whereas in /2, the monitors for valid dereferences and monitor for correct number of arguments to methods, are both used jointly with SC-MGD. The difference can be noted in the correct number of arguments (2) generated by SC-MGD in J2. Further, unlike in VDI, SC-MGD in J2 subsequently calls build(), which is in-line with the ground truth, likely due to the improved code context that generating the right number of arguments provided.
H.3 Richer Static Analyses and Constraints
Each of the different coding scenarios discussed in MGDMICROBENCH require a different static analysis to be performed, and hence, MGD is applicable with a variety of static analysis techniques. In this section, we focus on richer properties that can be enforced with MGD. Incorrect use of symbol names a broad class of errors, and root cause of many compile-time and run-time errors. Invalid use of defined symbol/method names can also lead various compile-time and run-time errors, and hence, just having the symbols in context may not be helpful. For the following scenarios, we use the CC-FIM-cllasExpPTTy be the strongest configurations due to the additional context, in the detailed evaluation in section Consider the following scenarios:
and FeEMMclaasxxxrTypeee
configurations, which we found to
Typestate Protocols. The Android MediaPlayer has complex constraints on the ordering of the API calls F which might be hard even for developers to reason about (Mishra et al., 2016 Runtime violation of such contracts lead to exceptions like IllegalStateExxcetiooi MMshrra et al. (2016) proposes the use of typestate analysis (Strom & Yemini 1986) to detect violations of such protocols, through static analysis techniques, and prevent the bugs at runtime. Duarte & Ravara (202 show that typestate protocols can be enforeed by the rich analysis supported in the Rust type system itself. We utilize this, and instantiate a monitor to guide LMs to generate typestate-valid API calls.
TSI: The task in scenario TS/ (abbreviation for "typestate") is to generatively complete the partially- written code that iteratively play songs in a playlist using the Android MediaPlayer API. The completion is invoked at point where the MediaPlayer object is in the Stopped state. While the base configuration generates a call to stop() () ;, which is in violation to the API protocol, since the MediaPlayer is already in the Stopped state at the point of completion, the same model configuration with MGD generates reset (); (; which is the only legal transition at the state as per the implemented protocol.
TS2: Crichton 2023) and Rust on Embedded Devices Working Group et al. 2018) describe the representation of GPIO pin states and the transitions between them as typestate protocol. We use the Rust API provided in Rust on Embedded Devices Working Group et al. 2018) and task the LM to complete partially written code, that initializes a GPIO pin and transitions to the imput_high While the base configuration generates a call to set pet_innut___pe ppullpplllpppppppppppppppppplpp_ () ; which transitions to the prpuutppuled__aie e__high state, the same model augmented with MGD calls set_input, wtt.inntuuthhhh((((_ high_z(); which leads to the desired state.
state.
Session types can be used to capture ordering constraints on two party communication (Crichton al., 2019) as communicating finite state machines. [Jspeesen al. (2015) propose that Rust type system can be used to enforce session type contract at compile-time. We utilize the proposed approach, and build monitor, that guides LMs to generate method-calls in line with the contract.
Session Protocols.

---

STI: Interaction with an ATM machine is an oft used example for two-way communication protocol. We use the session type formalism for ATM machine, described in (Jespersen al., 2015). The task for the LM is to complete partially written code using the session type API, to deposit money and print the new balance. On the client side, after authentication money (from the ATM menu), the base model generated a call to recv () which would wait for the ATM to send a value, whereas the protocol expects the client to "send" the deposit value. The base model augmented with MGD is able to correctly generate the call to send (0) .rev () ;, which first sends amount to deposited, and then waits for the ATM to communicate the new balance, which will be reported to the user. We note that with MGD, the model was able to generate code that follows the protocol appropriately.
selecting the action deposit
I CodeQL Query for Identifying Evaluation Target Methods
/*-
@id jaaa///aaalle//iaa__atrttttttte target_methodl
Oname find_taree sanndttaget_methode
Odescription Identify target methods from Java repository along with classExprTypes information for DotPrompts dataset
*/
import java
predicate filterClase (Class c) not(c instanceof TestClass) and f.fromSource() and c.getFile() getRelativePath() www.hheeeeeccc... ("aecchess""geeaiii ("Xgeneratedd"" and not getRelattivePath() wwtcmes ("*test(")) not (c..getFile() getRellttivePath() www.maaeeeeeecc..mmmmmmmm ("XtargetX")) and not (o.getFile) getRelativePath() matches ("Xbuild"")) and
count (Method m1 m1 c.getAMethod() and f (mlterMethoddn1)
m1) >=
predicate filterMethod (Method m)
fromSource(() and not (m instanceof TestMethod) and not (a.hasamme (" ("<clinit>")) and not (m. hastammee ("aiiammmm"bbbiiii ("<obinit>")) and (ereBodyy)gee//WuSSttt(( geggeeeBdyy)) () getNumStmt(() () >= 2 and
(m. getBody getLocation)) getEnndLine() (etthaaaiioo)..ggttaadiineee))))
m. getgodyd getLocation() getstartLine())
}
predicate typelbccaaredIniiee(Tyee File file) { (t. (t.frrmSource() and t.getFile() file) (t. (t.gotErasure() fromSource() and t.getErasure (eetFFl file)
}
// Find classExprTypes files such that they contain type definition of any expressions defined in any callable in the class except for target_m predicate filtercllsskxrrryppeelleleeee classExprTypesFilee)
c, Method target_m, File classFile, File
classExprTypessFile I= classFile and
// classExprTypesFile contains type definition singly imported type
exists exise t, ImportType impt
impt. fmmt.ffromouuce() and c.getFile() impt. smpt.getFile() and t
- (etoroeeeymmorreeyyep(( typeDeclareddIffletts claseExprTypesFilee
) or
10

---

// classExprlypesFile contains type definition of return type param type of the target method
exists (Type t
(t Baaggt_...gtaPaaaatyyyo or t
(enorm..geteetuu(Tpe(o))) and ExpeDeclareddIFileltt, classExprTypesFilo)
// clasaExprTypesfile contains type definition of type of any expression within a callable (that is not the target method) in class c, return type of callable any of its parameters exists (Expr e, Type t, Callable m c.getACallable() m != target_ and
e get.nEhcllosingStttt m. m..eBBody (egrodyyy..eAAtttttt getAStmt and
) gegeType
t
m. getaPranTTye) or t m. getReeturnTyee
bypeDeclareddIPileett classExprTypeaFile)
) or
// classEaprTypesFile contains type definition of any field type exists (xyse t, Field f c.getAField() and g.getType and typeDeclaredInFilett,, classExprTypesFile)
}
predicate exgrrsscaffyyooonntiiiilllloooee (Expr e, Type t, BlockStnt e getånfnclosingStatt b.getAStmtt () and
getType()
}
File classFile, File classExprTypesfile, Class c, Method m, BlockStmt b, int startLine, int startCol, int endLine, int endCol
where
// Bind variables
m c.getAMethod() and b - am.getBodyy) and classFile c.getFile() and
// Apply filters filterClass(c) and filterMethod(m) and #iltorcuassxxprypeeille.. m, classFile, classExpr(ypessiiee) and
// Bind method boundary locations startLine b. getLocation() getsttartLine() and
startCol = b. getLocationn() getStartCollmm () and
endline b. getLocation()) getEndLine () and
endCol b. getLocation getEndColumn
select
classFile.ge (Lessllleggttbesouutetttt)) claasFFle.ggeRallatvvveeaaadd
SlaaxxxpTypsFille.efeellaaaaasa startLine,

---

startCol, endLine, endCol
Examples of Code Generation with MGD
We present examples of code generation with MGD and compare them to generations without MGD augmentation. The appearance of red-markers below identifier names indicate that the identifier name is not type-consistent with the target object/class for the dereference operator. We highlight the correct identifiers generated with MGD augmentation in light green.
TD-3
getliame())
Prompt
return envVo;
colollect(Coollectons... tolist());
ListResult<EnWW> list() {
return ListResult. Muccess(envVOS:
List<EnWO> Arrays.streas
(rrTypeEnun.values// (oaallsss)).nsp(evveeeeeemm
new EnvVO();
SC
@mWD.setCode(mmm (nll-oddeeenTyyeennuneeeett (enTTypeEnum,
ectDesc())
return envVi
(Colececrs LoLis
Ground Truth
return Listllsllt..g. (gsceulll..gccgggeeeo0
getDescription());
return envV;
SC-MGD
}) coo.llect(oolleetrrrstttl... return (istResu
gettescriptionn())
return envio; }) (Culle. Lars, tulsst))))))) return ListResult (
Figure 7: TD-3 and SC generate invalid identifier names: getName, getDesc for target object envTypeEnum and success for target class ListResult. Augmenting SC with MGD leads to the generation correct identifiers: getDescription, of for the respective objects, leading to complete agreement with the ground truth.
12

---

public AccounLNumber
Guard. Guard.guard guvrddeerv( aggains aggans UNu110rNhi wetorWhieessppcce Lewpppaeeeaaiiiiioooeeeeeeeeeeee www.sssssssssscc......... ACCOUNT MUMBER MUMBER_EMPTY) EMPTY);
SC
Prompt
ACCOUNT NOT FOUND);
@Override
public BankAccount Eindeequired(cccoutlllnn accountNunber)
{
Bankkccounttgee((]
var optionelBankkccount find (inacuucuu)))))))))
SC-MGD
if (smpptyy)) [
throw new VeliddetionExxepptiinolllllll wwt.://pppeeeeeeaaaaaaeeieeeeese
ACCOUNT JUMBER NOT EXIS
Ground Truth
return ogtiionelBmikccount.cee
ACCOUNT_ NWWBER_BBRRTTTT _OT__XXIS)
SC-classExpTTypes
return optionslBankkcccunn..eg
ACCOUNT_ BOAAATT_NMMBEREETR [EVIBBER_EETT))
ogtionnlBamkkeccuun...tt
TD-3
ACCOUNT NOT FOUND);
SC-clasSExprTypes-MGG
|ACCOUNT_ L___JNNNBRRRRR _OT LO___OOT__XSS___ _EXIST)
return ogtiiomalBem/kccount..e(((
ogtionolBanlkeccoun..eeo
Figure SC and TD-3 generate the identifier name: COCOUNT_NOT, L_T___OTT__OO FOUND, which invalid for the target class. The snippet of code presented above is from one of the files present in the clas- sExprTypes augmentation. Augmenting SC with classExprTypes leads the model to generate the identifier name CECOUNT_NNMBEES EMPTY, which consistent with the type: ValidationMessager and therefore compilable. However, still does not match the ground truth. Augmenting both SC and SC-llassxxrrTypes with MGD leads the model to generate the correct identifier name: AC- COUNT SOOVT_NUMBE__NAM NOT, _ achieving agreement with the ground truth.
Prompt
milli wetcon chsunidl))
www.eetitt tne wn vefeess/oooee www.h/pppaaeeeeccoo pros/oeen))
Capprppeen penkoouaarr shereddettiioe propperr
Ground Truth
SC-MGD
wnteeaeddeeeiii lagal www.ara...t.ooa..........oo
www.
Figure 9: TD-3 and SC generate code that match the ground truth, except for the identifier names for target objects: advancedSettings. advancedSetting wtn.eeeeetninnCoooeent..tt Augmenting SC with MGD leads to generation of correct identifier names, and therefore agreement with ground truth.
13

---

Prompt
SC
jublir losless cetrish haoelel, Ielatessoge mssogg)) {
fcoaest reseest nov billdor()
-ars(baceurt "/at//eccaae""
Enme nor berteeeercttoonn CCant seal park to bendel, 422
(esssage),
cell call Hts. (tt.:recall(eemsst)
3x23/0
SC-MGD
try (Beeponee recponee call e3]
httpte/tesssseeeesooeeee codtt, sessage.
Trees
catch (boeeption
Ground Truth
return trus;
SC-FIM
urespected while comuntcate server);
citec DDecettinn 6)
TD-3
SC-FIM-MGD
(Poneption
Srrkntttactraace) returo falles
seme-"fy
Figure 10: TD-3 and SC both generate code with invalid identifier names. SC augmented with MGD is able to use the correct identifier name and therefore generates compilable code. However, the generated code does not make of the defined exception class: Zol ZolaserverConnett.. FailedException for exception handling. Augmenting SC with Fill-in-the- thh-middle provides the model with the necessary context perform exception handling, however, the model still hallucinates the identifier name getQueueName for the target object message. Hence, to get correct generation, SC is augmented with both FIM and MGD, and this configuration is able match the ground truth.
Prompt
SC
gethrrrrrlggutnn inttrruurtt queyString, secreblie); reture eniduliingee (bseeir urtaath, ciaastane, parumeters, Hitpetted, type staen, stowtinst/sapo)
ittplei that bitafented, bodien sho belttee
She own sowe soloe wwwaroo ptle (illeee (sspettoedoe Dey) leuri kajunni byy cared to iell suayy")
cormectors aetlie toestten", instatitrr baticleettmiii linggg que pitring Seleperr sa
SC-MGD
gettigutheei sedtlg wileuts, sigutere, ittpflethed, Requottops SIDED, sboc.nttioape)
Ground Truth
Meslarattrelowersttrrae astretkey);
SC-classExprTypes
eigutirs,
Ittplethod, RequestTipee
returs renolciles Jeads, artheth, sigritant, peresserr, drp-itine.iine
TD-3
SC-classxppTTppes-KG
mnrretaiggattrr Strlng secnetloy);
reture endillllegues bas.iiiegett//aara... arbteth. siguture, parenaters, Ictaletted, yopest St8855, shov (elttae)
wwwlgg/bgggetttbbeeooo urlaats sigrature, ittotiched, RoguotTyy. 3t005, shor.indtissg))
Figure 11: Example of generation with classExprTypes prompt augmentation.
14

---

Prompt
@iveride
tubioLisd wettaa//eeerlloddsseereolee get.lccardaaaaooooooooooo reult: on reture reselt;
publis peecarrkfiiuissteeeeoi listoerriiincineeeeeeiiii istterr panen) B
Pagrolorkflowe kfoostesaane www.:/aannnngtrrreeeeeee pww.gerePPsettee))
Pegeclorkti srpecdorklloeesstteeee vorkllodnstanceepp wook:loodnstancceeppps
-list(page, setterrflladdiiiiiii parmm.aetttate())
Pageclarkd
SC-MGD
worket
te0oktoorrkilewannttttoooooo htwpe/ccwwwggggggccoooeeoo (ww.tccocddd)))).....o....o ()); result. sesRecords (ww.rriioooeeeeittotootrs ressit;;
Ground Truth
SC-RLPG
Settocerrrlwwnnssaaeeeeeee result. (et.eeecondrtteeeaaaaaaaiiiieeed return result;
toltos (werke www.wllwwsttttoo www.wwawnnnntntaaneooooom www.ee/wwwa.....geeeeooooooo (
result, www.ttt.tteecerrr.oooooooo www.:e/eeedeekkkeoooaaaaaisiieee return resslt;
TD-3
SC-RLPG-MGD
List (etfeenffiounnntnnnehearr htwpaaanchheeeeeeeeereeeeeeeeeee getRecords result, setiecoods (wtreeccoodsoooroooooooooooo foodffolooddliitiioooe...
petiecondss)));
set result;
Example of generation with RLPG prompl augmentation. Augmenting SC with RLPG leads to generation of toDtos as compared to toDTOList, where both are invalid identifier names for the target object. Augmeting both the configurations with MGD leads to generation of the correct identifier and match with ground truth.
Figure 12:
Prompt
TD-3
privete List<doodsitem> nockItens OX
ATB [B)))
Idienerator- Saamessttrgetttssaeeee]]
Littdoohchee Isend Condefren bedeettee.geertttteeeee www.ee/ttteeeeeoorrrttot.. Eaeeeett,,,,,,eetarrr
GoodsInfo goodsInfo - GoodsInfo.off
item (Hoe))) 573, relum Iiees
1o0dsId/1011, sku1d catsgoryddenull,
20 hame""
GoodsInfo goodsInfo2 GoodsInfo.oft
1001,, skurd120021, catsgorytdeinull,
SC
mmm2, 18 180, tomer "P33" pnodsttttaasslll)
List<Goodsiten itens Goodsiten. genersteitens ggodssinfo, goodini,, wtt.r/ttrr..x..ttxxttteee. (Cannttttt
gex.geetxtraa).... (ut(stotttttt..... $89.E_ ka10611)))) reture itees;
Ground Truth
UPOATEABLEPRICE, S.r.tc,,..tSStlerriieeo. (l
SC-MGD
addell (Goods ten, generatellees (ereeessssoooeee (goodstnfot, iddenerstor, www.getExtre().coo (ut((Cosstant UPOATENBLEPRICE, petp:eeeee..ettsleeeeeeeee)
add www.lliioodtteeee. www.eetetteeeeeeeeeiiiioo (diveeratoo,
itees;
gottatea() pertCCtttt (PSSITERLEPHIEE, 199 grt5derrieeii)) returs items
Figure 13: TD-3 and SC generate identifier names: ATTR, GROUP _EY respectively for the target class Constant, which are both invalid. MGD augmentation leads to generation of correct identifier: UPDATEABLEPRICE.
15


Reviews:
[{"review_id": "nWkoTJRTe4", "reviewer": "Reviewer_hw3j", "summary": "Inspired by the fact that IDE static analysis helps in code writing, this work proposes an MGD approach that uses a monitor to guide the decoding generation of the code language model. The authors started with the motivation that the code language model generation process does not perceive the global information of the repository thus leading to hallucination errors. The proposed MGD method uses static analysis to obtain the global context of the repository scope instead of the local context and uses this monitor as a stateful interface between LM. Experimental results show that the MGD method can significantly improve the quality of Java code generation.\n", "questions": "As mentioned in the weaknesses, can MGD be extended to other programming languages and other coding scenarios?", "limitations": "The authors discuss the limitations of their work and give ideas for improvement.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Compared to introducing global information through prompt engineering, language model architecture modification, incremental training, etc., MGD is a simpler and more effective way of limiting the output by utilizing construct masks. However, such an idea is not innovative.\n2. The experimental results given are good and can support some of their claims, especially the effectiveness of the MGD method in type-consistent identifiers.", "weaknesses": "1. Constructing masks to constrain the output space using grammar information, compilation information, and code definition information is classic in the code generation, Text-to-SQL Task. The authors also mention in related work that Text-to-SQL work such as PICARD has a similar constraint strategy, but MGD is targeted at general-purpose programming languages and focuses on static analysis through repository-level context. \nHowever, MGD cannot be theoretically justified to support general-purpose programming languages, or even to effectively limit output in only some coding scenarios. In this work, the MGD utilizes the \".\" as a trigger to update the state, is it possible to find a single unique trigger for all coding scenarios, are there scenarios to consider consecutive multiple words as triggers, or spanning words as triggers? Also, can MGD support nested programming languages? Again, this would require setting up special triggers, and complex state management.\n\n2. The experimental design, including the construction of the dataset and evaluation metrics, appears to be specific to \nmonitoring type-consistent identifiers. I understand that this is an instantiation of the MGD method, but my concern is that the MGD method only supports special instantiations from the methodological idea and experimental validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Authors mention ... MGD is targeted at a general-purpose programming languages and focuses on static analysis through repository-level context. However, MGD cannot be theoretically justified to support general purpose programming languages, or even to effectively limit output in only some coding scenarios\n\nWe have demonstrated that MGD does support general purpose programming languages (as shown by large improvements in both compilation and method level match for Java, in a dataset consisting of real world projects), however, we target specific properties that can be checked statically and enforced during decoding, like those supported by type systems. We have provided examples of additional coding scenarios and static analyses (typestate and session-types) in the common response. We have also built MGD for the Rust programming language and evaluated it on a mini-benchmark, whose results (in the attached PDF) show that MGD can guide LMs in generating code that adheres to richer properties.\n\nWe acknowledge that functional correctness specified through pre/post conditions and invariants may not be enforceable through MGD in the current form. We will make these limitations explicit in the paper, and thank the reviewer for pointing them out.\n\n> Experimental design, dataset construction and evaluation metrics appear to be specific to type-consistent identifiers. I understand that this is an instantiation of the MGD method, but my concern is that the MGD method only supports special instantiations from the methodological idea and experimental validation.\n\nWe clarify that the task in DotPrompts is to generate the complete method following a dereference location, with an average ground truth completion in the testset being about 12.7 lines of code. Hence, the task is long form code generation.\n\nCompilability is a necessary characteristic of any code and a general evaluation metric. In this paper, we demonstrate that LMs across parameter scale struggle with generating compilable code. While Next-Identifier Match (NIM) and Identifier Sequence Match (ISM) capture the specific scenario of type-consistency for MGD, Prefix Match (PM) and Compilation Rate (CR) are method-level metrics, that are used commonly in the literature, and are not specific to MGD.\n\nWe show that generation of type-consistent dereferences plays a significant role in improving the compilability of code generated by LMs (relative improvement  of 22.67% over SantaCoder and 22.81% over text-davinci-003 with MGD). While MGD is used in this paper to target a specific problem (generating type-valid dereferences), it is shown to improve match with ground truth (PM) as well. In the common response, we have discussed generality of MGD along several dimensions.\n\n> In this work, the MGD utilizes the \".\" as a trigger to update the state, is it possible to find a single unique trigger for all coding scenarios\n\nAs correctly pointed out by the reviewer, for different coding scenarios, the triggers may be different; however, the same MGD framework can support all of these triggers. For example, for object instantiation in Java, the trigger could be `[\"new\", \" \"]`, and for named `enum` values in switch statements, the trigger could be `[\"case\", \" \"]` (details in the common response). In both cases, the function pre in the formalism can be modified to handle these triggers.\n\n> Can MGD support nested programming languages? ... require setting up special triggers, and complex state management.\n\nWe believe the reviewer is referring to nested functions or nested classes (kindly correct us if our interpretation is wrong). Languages with advanced type systems, that support these features can be leveraged by MGD. For example, Java and C# have such advanced features, and their Language Servers are able to provide type consistent identifiers even with nesting. Further, our datasets PragmaticCode and DotPrompts indeed contain such evaluation scenarios. The monitor instantiated in the paper already handles this scenario, including nested function calls and definitions. We will highlight these interesting cases with concrete examples in the Appendix."}, {"review_id": "mxp2Qif1Mi", "reviewer": "Reviewer_LEhe", "summary": "This paper proposes to use the output of a code static analysis tool, of the sort used in IDE code completion tools, to constrain the output of an LLM to improve code generation. Concretely, the paper focuses on type-consistency in object dereference for Java code. When an LLM is called to generate a use of an identifier from a class (i.e. using the '.' operator, in Java), the static analysis tool is invoked to only allow the LLM to generate identifiers accessible in that class which are consistent with the current typed context. This is implemented by masking the probabilities in the LLM to zero-out any tokens inconsistent with the set of valid identifiers returned by the analysis tool, and sampling from these constrained probabilities. The paper compiles a new dataset of open-source Java repositories with dependencies and development environments (to allow applying the static analysis tool), and a typed-dereference code completion task on this dataset. The method consistently improves the ability of black-box code LLMs (CodeGen and SantaCoder) to produce code that successfully compiles and matches the ground truth. ", "questions": "Q1) What static analysis tool is being used? Is it from some IDE (the discussion mentions Eclipse and Visual Studio), and if so which one?\n\nQ2) What code does the model need to generate in the DotPrompts task, after the post-dereference location? Is it the rest of the method? Since this is a new dataset, it would help to provide some stats on how long the ground-truth completions are, to get a sense for their difficulty.\n\nQ3) I didn't understand the classExprType approach from the explanation in line 202-205. What is being included in the prompt -- type definitions, or methods/expressions that have matching types, or something else?\n\nQ4) It felt odd to me to not apply either MGD or the baseline methods to the text-davinci-003 model if I understood right, could some explanation be given for this (API costs)?\n\n*Other comments* (not necessary to respond to in response)\n- Clarify what sampling strategy is being used (e.g. top-p with a temperature? what hyperparameters?)\n- I think the definition of pass@k in 225 is wrong -- it's the expected # of times that >=1 success occurs in the list of k candidates.\n- Clarify how FIM interacts with truncation to fit within the context window.\n- The description of the metrics in 211-221 was unclear to me, in particular how \"ordered set\" deals with repeated identifiers. \n- The plots in Figure 3 were hard to read, with all lines being red.\n- Figure 1 was a bit confusing --- do both text-davinci-003 and SantaCoder generate the same code (in red, with X)?\n- \"Basic concepts and notation\" would be clearer if it's tied to the motivating examples, e.g. give a concrete example of property (compilation?)\n- Lines 154-173 spend a lot of space describing the bookkeeping necessary due to vocabulary mismatches, but it's pretty wordy and I'm worried it might not make things clearer to someone who doesn't already have a sense of how this would need to be done. I think this could be moved to the appendix, or perhaps made more precise with an algorithm block (possibly in the appendix).\n", "limitations": "The paper should do a better job of indicating that the evaluation of the proposed monitor framework is pretty limited in the current work, evaluating only on a single task (and a single dataset). I'm worried that the token-level constrained decoding presented here might not work well when combined with more complex static analysis settings.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The approach is simple but well-motivated, and can be applied on top of black box code LLMs. I could easily see this approach being deployed effectively in IDEs, if it isn't already. As a type of constrained decoding technique, it should easily be compatible with approaches that e.g. augment the model prompt.\n\nWhile the experimentation was limited to a single task and dataset, the evaluation on it was fairly thorough, evaluating two different base LLM families (CodeGen and SantaCoder) with FIM and non-FIM modes for SantaCoder, and comparing to reasonable baselines. \n\nThe approach demonstrates solid improvements across metrics, with large improvements in compilation rate and next identifier match.\n", "weaknesses": "The contribution felt a bit thin to me for a NeurIPS paper. I think that the general idea of using monitors is well-motivated and potentially widely applicable, but here they are only demonstrated on a single task which IMO is particularly well-suited task to the approach --- dereference constraints should really narrow down the space of compilable identifiers that the LLM can choose from, and the largest improvements from the method are on compilation rates and single identifier match. I think the paper would be much stronger if it showed that the approach can also be effectively applied on another task which affords static analysis, ideally one that LLMs are a clearer fit for (i.e. generating longer-form code, such as classes, while ensuring the generated code can compile), or one of the ones mentioned in the Future Work. This might require more sophisticated methods of integrating the static analysis constraints into the LLM decoding procedure beyond the simple prefix-consistency filtering used here.\n\nI felt that the baselines could be improved. One missing baseline that I think is important would just choose randomly from the set of possible identifier candidates produced by the static analysis tool, without using an LLM at all. This seems like it could do pretty well on CR (although it's unclear to me how much code beyond the next identifier the task requires generating, see below), and potentially also on NIM (it would at least give a sense for how many identifiers are output by the tool). I was also pretty unclear on what the classExprTypes baseline is doing; see questions below.\n\nThe writing of the paper was somewhat unclear, and I had trouble resolving a number of important details, in particular about the experimental setup. The dataset and new task definitions could potentially be a contribution, but many details about them were unclear; see questions below.  \n\n--- Update after response ---\nThe author response largely addressed these concerns; see comment below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q2) ... is it rest of the method?\n\nYes, the task is to generate the rest of the method, starting from the dereference location. Since the methods in DotPrompts consist of at least 7 lines of source code, a typical completion consists of multiple lines (Appendix line 75). On average, the number of lines of code in the ground truth completion of a testcase is 12.7 and 75% quantile is as high as 15 lines of code.\n\n> Show applicability of MGD on task LLMs are a clearer fit for (generating longer-form code ... while ensuring generated code compiles)\n\nAmong the evaluation metrics, except NIM, all other metrics - Compilation Rate (CR), Identifier Sequence Match (ISM) and Prefix Match (PM) evaluate the complete method-level generation. The significant jump in compilation in our results demonstrate that MGD guidance consistently improves ground truth match for longer form code generation, while significantly improving the compilability of the generated code.\n\n> experiments limited to single task & dataset, evaluation thorough ... different LLM families with FIM and non-FIM ... reasonable baselines.\n\nWe thank the reviewer for recognizing the thoroughness of our evaluation. We would like to mention that DotPrompts was sourced from large number of real world repositories, with a long form code generation task. While metrics NIM and ISM capture the specific scenario of type-consistency, PM and CR are method-level metrics, used commonly in the literature, evaluating a broad range of real world coding scenarios and properties.\n\n> Paper would be stronger if showed approach can apply on another task that affords static analysis, or ones mentioned in Future Work\n\nThank you for this suggestion. In the common response section, we demonstrate a concrete instantiation of MGD on a different programming language (Rust), utilizing 2 different static analyses (typestate & session-types), both of which are mentioned as future work. We will include these scenarios in the paper. We also show scenario for doing joint monitoring by combining multiple static analyses together (also mentioned in future work) for ensuring generation of type-correct method calls with right number of arguments passed (please refer to the common response).\n\n> How many identifiers are output by the static analysis?\n\nOn average in DotPrompts, the static analysis returned 44.86 identifiers, with a median of 25 identifiers.\n\n> missing baseline: choose randomly from possible ... identifier candidates\n\nWe implement the proposed baseline over TD-3 and evaluate it on a subset of DotPrompts with 533 test cases across 77 repos. Results are reported in **Table 1 and Figure 1 (TD-3-Random) in the attachment**. We observed a significant decrease across all metrics, likely due to the random sampling's inability to capture the intent implicit in the context, which the LLM captures.\n## Questions\n> Q1)\n\nWe use Eclipse JDT.LS, a Language Server for Java. It is an IDE agnostic tool and used to provide Java support in several IDEs (Appendix lines 84-92). As our implementation builds upon the Language Server Protocol, instantiating MGD for other languages can be achieved by changing the Language Server from Eclipse JDT.LS to another Language Server, as we did for the Rust PL discussed in the common response.\n\n> Q3)\n\nGiven a testcase to complete method M, within class C - We mask out M from C, and run a static analysis in the remaining contents of C to identify all possible expressions in it (for example, dereference expressions, function calls, concatenation, etc.), and list the type of all such expressions. We then identify the files where each of these types are declared, and concatenate the content from those files, truncating to fit the context budget allocated.\n\n>  Q4)\n\nWe thank the reviewer for the suggestion and implement MGD for TD-3. Detailed results are in **Table 1 in the attachment** and discussed in the common response.\n### Other Questions\n> What sampling strategy is used ...\n\nWe use nucleus sampling, with a top-p value of 0.95, generating 6 samples, 1 each at temperature 0.2 and 0.4, and 2 each at temperature 0.6 and 0.8 (Appendix line 93).\n\n> How FIM interacts with truncation to fit the context window\n\nWe assign 20% of prompt token budget for ClassExprTypes, 50% for FIM, and truncate from left for standard and ClassExprTypes prompt, while truncating on the right for FIM prompt. In case of joint FIM and ClassExprTypes, we assign a budget of 20% for ClassExprTypes, and 40% for FIM (Appendix lines 95-102).\n\n> ... how \"ordered set\" deals with repeated identifiers\n\nWe thank the reviewer for suggesting an improvement in writing of the evaluation metrics. We are using “ordered sequences” and not “ordered set”. We will update it in the writeup. Since we are using sequences, repeated identifier names are also matched for. For example, if ground truth identifier sequence is [\"withIp\", \"withPort\", \"withIp\", \"withIp\"], and the model generates the sequence [\"withIp\", \"withPort\", \"withIp\"] or [\"withIp\", \"withPort\", \"withIp\", \"withPort\"], then it gets a score of 3/4, but if it generates [\"withIp\", \"withPort\", \"withIp\", \"withIp\"], then it gets a score of 4/4.\n\n> Fig 1 ... text-davinci-003 and SantaCoder generate the same code?\n\nYes, both text-davinci-003 and SantaCoder generate the same code present in the red box with X.\n\n> \"Basic concepts and notation\" ...\n\n> Definition of pass@k ...\n\n> The dataset and new task definitions could potentially be a contribution ...\n\nWe thank the reviewer for their suggestions and for highlighting the dataset as a contribution. We will incorporate your valuable feedback. We will:\n1. Adapt the discussion in section A of Appendix, to be clearer, and fit with the \"Basic concepts and notation\" section of the paper.\n2. Rectify the wording on pass@k metric.\n3. Rectify the dataset section to explain details related to the dataset, including relevant statistics like typical length of testcases as reported here and Appendix section B."}, {"review_id": "y6L32UciKc", "reviewer": "Reviewer_fL4k", "summary": "The paper proposes a framework that modifies the output logits of a language model using a monitor-guided decoding approach. The monitor consists of type-guided method invocations across the repository obtained via static analysis tools and heuristics to update the contents of the monitor along with when to trigger it. The authors show performance improvements when their approach is used in conjugation with different LMs and when augmented with different prompt augmentation techniques.", "questions": "- Line 223-224: \"For our experiments, we use n = 6 independent trials. For a budget of k ∈ [1, n] samples, we compute the aggregate score score@k..\". Does this mean that you perform n forward passes through the LLM or sample n times? If the former, is the input to the LLM same during these passes?\n\n- It is not clear to me as to what is the stopping criteria used for decoding. Can it generate multiple lines of code following the \".\" or is it a single line of code?\n\n- Line 172-173: \"The update function removes the members in s that are not prefixed by xn+1, and those prefixed by xn+1 are updated by pruning the prefix string xn+1.\" I didn't quite understand this. \n\n- Line 206: \" ..to adapt RLPG to DOCPROMPTS\". Does this mean that RLPG was trained on DOCPROMPTS? Can you describe the process of adaptation and how the repository contexts selected by RLPG were incorporated with SantaCoder? \n\n- Minor formatting issues: (a) The sentence is not clear at the end of the caption of Figure 1; (b) Extra spaces in between in the abstract.", "limitations": "Both limitations and potential negative societal impact are mentioned.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is in line with some recent works that combine the strengths of static analysis tools (that can help incorporate domain knowledge and that are well-studied) with LLMs (that have inbuilt world knowledge).\n\n- The paper is clear in writing and easy to understand.\n\n- The experimental results are convincing.", "weaknesses": "- My main concern with the paper is the scope of its applicability to other settings. Even though the authors describe their framework as a general framework in Section 2, they have shown results in a very narrow setting of generating identifiers informed by type constraints. In the paper, the authors have used a pre-condition corresponding to the occurrence of the token \".\" which only represents method invocation in Java. Also, I do not see how the monitor will work in its current form when this pre-condition is made general, for example, say when the user hits a new line. It is not clear to me how this framework can be adapted to a more general code completion setting that is free of the notion of a pre-condition or to applications other than code completion (such as bug-repair) or even to a different programming language where it is difficult to define this pre-condition or the state corresponding to the monitor. I would like a discussion of the generalization capability of MCD to complex settings with concrete examples and experiments.\n\n- I didn't see an analysis of the time and memory overhead due to MCD in the main paper. As per my understanding, this can be significant given that the static analysis tool is triggered at each invocation point and no form of caching is done to save time. I would like to know the time taken by MCD to generate an accurate prediction and see whether it is practical from the point of view of deploying it in conjugation with a real-time end-user application. This factor becomes increasingly important since the authors are performing repeated sampling to obtain gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Main concern: scope of applicability to other settings. Authors describe framework as general ... have shown results in narrow setting of type constraints\n\nAs discussed in the common response, MGD is generalizable across programming languages, coding scenarios as well as different static analyses. We emphasize that compilability is an important desirable characteristic of code, and we demonstrate that LMs across parameter scale struggle with generating compilable code. We show that generation of type-consistent dereferences plays a significant role in improving the compilability and match with ground truth of code generated by LMs.\n\n> \".\" as precondition ... only represents method invocation in Java. How monitor will work when pre-condition is made general. How framework can be adapted to general code completion setting free of notion of a pre-condition\n\nOur precondition check (pre) is syntax driven, which in general can be thought of as syntactic pattern matching. These can be made robust to whitespaces such as newlines. While we use `\".\"` as the precondition trigger for our monitor, other triggers are possible, for example, `\"->\"` and `\"::\"` in C++, `[\"new\", \" \"]` or `[\"case\", \" \"]` in Java, etc. These can be instantiated with a simple change to the precondition check (pre). It’s a language specific implementation which can be easily changed.\n\n> Application other than code completion (such as bug-repair) or a different PL ... Discuss generalization capability of MCD to complex settings\n\nMGD is agnostic to the downstream coding application, and can be used in all coding scenarios, where LMs are used generatively. For example, consider the development scenario of code refactoring, where the repositories are in a transient state. MGD can be useful to have the LM generate code using API names with the latest updates to the codebase. Further, we have several generalization aspects of MGD in the common response and will include experimental results on the same.\n\n> Time taken by MCD to generate ... is it practical for deploying in a real-time end-user application\n\nYes, MGD is practical for real-time IDE deployment. Our implementation is based on Language Server Protocol, which are highly optimized implementations designed specifically for real-time IDE usage, and we inherit those optimizations. Specifically, we find that the end-to-end time overhead of generation with MGD is a modest 1.83x on average compared to a model without MGD (appendix G).\n\n> Memory overhead of MGD\n\nLanguage Servers used for MGD are already a part of IDEs like VSCode and Sublime Text, and loaded when a file of a particular language is opened. The instantiated monitor is a very thin client bridging the LM and the language server, and not expected to have any additional memory overhead beyond the language servers.\n\n> Since the authors are performing repeated sampling to obtain gains\n\nWe clarify that we do not perform repeated sampling to obtain gains. MGD can enhance the quality of generation even with a single sample. Following the common practice of evaluating models with different sampling budgets, we allow each model to sample multiple generations and compare the models across different sampling budgets. Our results show that MGD helps across all budgets (in [1,6]) including a single sample for every model considered.\n\nCost of sampling a single generation from larger models is higher than that of smaller models. Thus, it might be feasible to sample multiple generations from a smaller model at lower/comparable cost than generating a single generation from a larger model. It is in this context that we highlight that selecting from best of 3 samples of a 1.1B model (SantaCoder) with prompting and MGD, on average, has better prefix match with ground truth than much larger 175B model (text-davinci-003). This is in addition to showing that every model improves with MGD given the same sampling budget, including for one sample.\n\nPlease let us know if further explanation is needed from our end about our setup.\n\n> Q1: Line 223-224 ...\n\nGiven a testcase from DotPrompts, we use nucleus sampling with top-p value of 0.95 to generate $n=6$ samples using the same prompt (Line 93-95 in Appendix).\n\n> Q2: Stopping Criteria ...\n\nYes, the LM is required to generate multiple lines of code following the dereference until end-of-method (On average, the number of lines of code to be completed in the ground truth completion of a testcase in DotPrompts is 12.7 and 75% quantile is as high as 15) with a budget of 512 tokens. The decoding terminates when a closing brace that matches the method’s opening brace is generated or the generation budget is exhausted.\n\n> Q3: Line 172-173 ...\n\nWe explain this by an example. Kindly refer to figure 4 in Appendix which shows the decoding steps for the motivating example. Consider the transition from state $s_1$ = `{\"withIp\", \"withPort\", \"newServernode\"}` to state $s_2$ = `{\"Ip\", \"Port\"}`. Following one step of Monitor-Guided Decoding at state $s_1$, the token `with` is sampled. Next, `update` function removes all the strings in $s_1$ that can’t be generated following `with`: so `newServerNode` is removed. For the remaining strings, `update` truncates the part that has been sampled, so `with` is removed from `withIp` and `withPort`, leaving $s_2$ = `{\"Ip\", \"Port\"}`. We will improve the description in the paper.\n\n> Q4: RLPG ...\n\nRLPG considers a single line completion task (and hence a single line hole), whereas DotPrompts has a multi-line method-level generation task (and hence we consider the method suffix following an object dereference as a hole). We make necessary changes to adapt the retriever for the change in hole granularity. As RLPG model was trained for Java, and RLPG paper shows performance generalization to new repos, we reuse their officially released model checkpoints.\n\nWe thank the reviewer for also suggesting formatting improvements and we will make the necessary changes."}, {"review_id": "VM5Jy4y4AM", "reviewer": "Reviewer_27c5", "summary": "Authors propose to use static analysis to guide LLM decoding process to improve generation of code that may not be available in the context or training data. Authors show that their approach improves identifier generation compared to baseline and in certain cases compared to larger models.", "questions": "- What is the cost/overhead of static analysis during decoding? How much does this slow down the result generation?\n- In Table 1, could you explain the row ordering? Why CG is mixed with CG-X-MGD, but for SC, all SC-X-MGD lines are at the end of the table?\n\nI have read the author’s rebuttal. \nAuthors have addressed my concerns by providing cost/overhead of static analysis and explaining the Table 1 row ordering.", "limitations": "It would be good to have a longer discussion of static analysis limitations. What static analyses can be applied and how costly they could be? As implemented MGD can only be applied for identifier completion. It would be good to know what other static analyses could be used and for what tasks.\n\nI have read the author’s rebuttal. \nAuthors have addressed my concerns by further discussing static analysis techniques.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Authors propose a static analysis driven LLM decoding, which improves code generation (identifier generation) in situations when relevant code is not part of the context or training data\n- Using static analysis to drive decoding rather than adding additional code information to the prompt/context, allows to preserve context for other needed information while improving the identifier generation\n- Authors also create a PRAGMATICCODE dataset and DOTPROMPTS testset\n- Monitor guided decoding shows significant improvement vs the same model without MGD.\n- Authors also show the MGD is beneficial even when prompt augmentation is used.", "weaknesses": "- Unlike prompt/context augmentation, MGD approach can only improve output in specific cases where static analysis provides additional information. Although the paper shows significant improvement in the authors'  PRAGMATICCODE dataset and DOTPROMPTS testset, this testset is limited and geared towards identifier completion. Authors do not show a comparison in a general test set where identifier completion may be small part of accuracy. In such testset (and in real life usage), the MGD approach may not contribute much improvement. This would have to be known for real-life implementations.\n\nI have read the author’s rebuttal. \nAuthors have addressed my concerns by explaining DOTPROMPTS testset and its metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> this testset is limited and geared towards identifier completion ... Authors do not show a comparison in a general test set where identifier completion may be small part of accuracy.\n\nWe would like to clarify that each testcase in DotPrompts is obtained by identifying a dereference location in a method having at least 7 lines of source code, and the task for the model is to generate the rest of the method after the dereference (line 70-75 in Appendix). In DotPrompts, on average, the number of lines of code in the ground truth completion of a testcase is 12.7 and 75% quantile being 15 lines of code.\n\nFurther, among the evaluation metrics, all metrics except NIM evaluate the complete method level generation by the model. These metrics are Compilation Rate (CR), Identifier Sequence Match (ISM) and Prefix Match (PM).\n\nAs shown in our results, SOTA LMs struggle with the problem of generating type-consistent code, and our technique generates type-consistency without affecting (rather improving) match with the ground truth as measured by ISM and PM. Unlike contest style benchmarks, for example, HumanEval and CodeContests, we target pragmatic scenario encountered by real world developers in IDEs, where they are in the middle of implementing a method, in the context of a repository, rather than solving a standalone algorithmic problem.\n\n> What is the cost/overhead of static analysis during decoding? How much does this slow down the result generation?\n\nWe acknowledge the importance of measuring the performance impact of MGD in real-time code generation. On average, the decoding time overhead of MGD compared to the same model without MGD is modest 1.83x (section G in Appendix). A tighter coupling with the Language Server Protocol architecture can further reduce this overhead.\n\n> It would be good to have a longer discussion of static analysis limitations ... As implemented MGD can only be applied for identifier completion. It would be good to know what other static analyses could be used and for what tasks.\n\nWe posit MGD to be a general framework, which inherits the strengths and weaknesses of the static analysis techniques. MGD can be applied to many coding scenarios, like usage of correct number of arguments to methods and generic types, correct use of named-constant values, valid class instantiations, or to enforce richer code constraints like typestate or session type validity. All of these requires use of different static analysis techniques, but can be captured by MGD formalism. We have discussed these in the common response and will include them in the paper, including the experimental results on MGD-for-Rust case studies.\n\nHowever, to extend MGD to more sophisticated properties, such as general preconditions and post-conditions, advanced constrained decoding methods (including backtracking) might be needed. This is an exciting direction for future work and we will mention this in the main paper.\n\n> In Table 1, could you explain the row ordering? Why CG is mixed with CG-X-MGD, but for SC, all SC-X-MGD lines are at the end of the table?\n\nWe thank the reviewer for highlighting this and realize that readability can be improved following your suggestion, and we will make the changes accordingly. At the time of writing, the rationale had been that each of the CG-X configurations pertained to different models (350M, 2B, etc.), whereas all of the SC-X configurations pertained to the same model, with augmentation in the space of FIM modality or prompting, and we resolved to keeping augmentation of the same model together, while separating different models. We will make the necessary changes."}]

Author rebuttal:
We appreciate the reviewers' constructive feedback and suggestions. We first answer some common questions and present individual responses later.

# Generalization and Applicability of MGD
We demonstrate MGD’s applicability to more coding scenarios, programming languages, and different static analyses & properties. We will include this discussion in the paper.
## Generalization Dimensions
### Programming languages
MGD can be applied to most programming languages (PL). For instantiation, it requires static analyses that help infer and enforce semantic constraints on the code under development. Such analyses are available in IDEs (e.g., clangd for C/C++, Jedi for Python & Rust Analyzer for Rust) through the standard Language Server Protocol (LSP). We build a monitor as a thin client around LSP (Appendix lines 84-92).

Supporting new PLs is easy and doesn’t necessitate changes to the monitor's static analysis interface, as it is a generic LSP client. *We were able to develop an MGD implementation for Rust using Rust Analyzer in just two hours following reviewer comments* (discussed below).

### Coding scenarios
MGD can be extended to several coding scenarios, including those requiring spanning token sequences as triggers. Example scenarios realizable with MGD formalism introduced in the paper are as follows:
1. **Class instantiation**: MGD can trigger on a 2-token span: `['new', ' ']` to ensure only valid classes are instantiated. The trigger invokes a static analysis that identifies instantiable classes from the local & global context at the trigger location.
2. **Method call and arguments**: As an example of joint monitoring based on multiple static analyses, consider a monitor for 2 properties, $M_a$) Type-consistent dereferences (from the paper) and $M_b$) Correct number of arguments to calls. $M_b$ triggers on the final state of $M_a$ when the last token contains `'('` (start of a call). A static analysis determines numbers of arguments that decoded method takes and $M_b$ states correspond to the number of arguments left to be decoded. `update` function transitions on arguments to the current function call, accounting for nested parentheses. $M_b$ prevents the generation of a token with `')'` (end of the call) till the right number of arguments have been decoded.
3. **`switch` over `enum`**: A switch case statement over `enum` uses named enum values in `case <val>` to match. A monitor with `pre` triggering on multi-token sequence `['case', ' ']` is used to generate valid named values.

### Static analysis techniques
We now provide examples of deeper semantic properties (& static analyses) that can be used with MGD (concretely instantiated for Rust as discussed next):
1. Typestates [1], often expressed as finite state machines (FSMs), define valid sequences of operations that can be invoked on objects of a given type. For example, a type representing a file handle would have typestate that disallows calling `read` after `close` has been called.
2. Session Types [2] ensure that messages between concurrent programs are sent and received in the expected order, following a specified protocol, and specified as communicating FSMs.

## Concrete instantiation of MGD for Rust with Typestate & Session Type consistency
To concretely show that MGD can be applied to other programming languages & use results from other static analyses, we instantiated MGD for Rust using Rust Analyzer [3]. A mini-benchmark with small code completion scenarios requiring conformance to typestate or session-type specifications for valid generation along with results is reported in **Table 2 of the attachment**. Despite using the strongest SantaCoder configuration from our paper, SC-FIM-classExprTypes, the generated code doesn’t satisfy the typestate and session-type properties (e.g., generating `stop()` which is not valid in the state `Stopped` of `MusicPlayer`). Use of MGD ensures generation of correct invocations. We will include these results in the paper.

# Additional Results
Following reviewer LEhe’s suggestion, we instantiate MGD over text-davinci-003 (TD-3), and a valid identifier random selection baseline. Detailed results are in **Table 1 & Figure 1 of the attachment**. Notably, TD-3-MGD achieves a compilation rate of **73.73% (22.81% improvement over TD-3)** and all other metrics see significant improvements as well. TD-3-Random suffers across all metrics, likely due to random sampling's inability to capture the intent implicit in the context, which the LLM captures.

# Performance Overhead
Language Servers are optimized for real-time use in IDEs. MGD integrates with Language Servers (which perform static analysis) and inherits the optimizations such as caching and reuse of pre-computed values. In our implementation, MGD's average decoding time overhead is a modest 1.83x (appendix G). In production setting, tight integration with Language Servers can reduce the overhead further.

# Generality of Dataset and Metrics
DotPrompts testset task requires generating the *complete method after the dereference location* spanning multiple lines. Test cases are derived from methods with $\geq7$ lines of code (Appendix, lines 70-75). Mean ground truth code length to be generated is 12.7 lines, with a 75% quantile of 15 lines, making DotPrompts a method-level code generation task. The LM is allowed to decode upto 512 tokens and MGD monitors every dereferenced identifier during decoding.

All evaluation metrics except NIM assess method-level generation: Compilation Rate, Identifier Sequence Match, and Prefix Match. To achieve a successful compilation, the model not only has to generate a type correct next identifier, but all subsequent method calls and field accesses must be valid as well.

[1] Strom et al., Typestate: A programming language concept for enhancing software reliability. IEEE TSE 1986

[2] Jespersen et al., Session Types for Rust. In WGP 2015 (pp. 13-22). ACM

[3] “Rust Analyzer.” (online)

Meta-review: In this paper, the authors aim to address the problem of code LLMs in a larger code context that contains use types or functionalities defined in another module or library. The authors introduced the concept of monitors, which use static analysis in the background to guide the decoding. The authors tested this approach in generating identifiers that can improve compilation rates and match with the ground truth. 

The reviewers appreciated the motivation and novelty of the approach and its effective way to limit output generation simply through masking. While this idea is not technically challenging, the overall approach is quite a simple and elegant way to constrain the output search space of code LLMs, showing clear performance gains in experiment results. There were concerns from reviewers about the generalization of this approach to other types of tasks rather than generating type-consistent identifiers. However, the authors' rebuttal has mostly addressed these concerns, and I highly encourage the authors to include more discussion in their final paper.
Decision: Accept (poster)