SlowGuess/NeurIPS_2023_OCR huggingface dataset:
next token, and the composition of  $L_{\theta}$  and  $M_{\varphi}$  is repeated to obtain modified logits, and finally we obtain the sampled token  $\mathfrak{I}\mathfrak{p}$ . Since  $\mathfrak{I}\mathfrak{p}$  is a prefix of a member in  $s_2$ , update  $(s_2, \mathfrak{I}\mathfrak{p})$  transitions to  $s_3$ , with  $s_3 = \{\epsilon\}$ , a singleton consisting of the empty string. Note that the token '('' is consistent with the suggestions in state  $s_3$ , since it matches the regular expression  $w \cdot E \cdot \Sigma^*$ , where  $w = \epsilon$ . Following the sampling of the token '('', the monitor transitions back to the wait state  $s_0$ .
B Data Set Curation Details
For the evaluation of pragmatic code generation, we require real-world repositories, with their complete environments and dependencies. Most public datasets for code generation do not meet this requirement, due to their focus on standalone code generation, except the recent CoderEval (Yu et al., 2023) and PyEnvs (Pei et al., 2023), both of which are not publicly available at the time of this writing. Further, CoderEval just evaluates 10 Java and 43 Python repositories. Since these datasets do not filter for repositories by their creation date, the test repositories could be a part of the training set for the evaluated LMs. Considering these, we describe the curation of PRAGMATICCODE, a dataset of real-world open-source Java projects complete with their development environments and dependencies. We ensure that the training repositories were released publicly only after the determined training dataset cutoff date (31 March 2022) for the CodeGen (Nijkamp et al., 2023), SantaCoder (Allal et al., 2023), and text-davinci-003 (GPT-3.5) (Ouyang et al., 2022) family of models. All the repositories in PRAGMATICCODE are buildable with their respective build systems. The development environment also provides support for analysis over templated code generated with systems like ProtoBuf (pro, 2008) and Lombok (lom, 2009). Further, we create DOTPROMPTS from PRAGMATICCODE for the evaluation of code generation in a pragmatic setting.
PRAGMATICCODE. We queried the GitHub API on 25 March 2023 to obtain a list of the top 1000-starred Java repositories on GitHub created after the determined cutoff date of 31 March 2022. We then attempt to download the latest snapshot of each of the top 1000-starred Java repositories and were able to download 731 repositories from GitHub. We create a build environment for Java projects consisting of Oracle Java Development Kit 17.0.6, Apache Ant 1.10.13, Apache Maven 3.8.7, Gradle 7.3.3, and GitHub CodeQL CLI 2.12.5. In this build environment, we invoke the CodeQL database create command<sup>1</sup> for every software repository obtained from GitHub. The CodeQL database creation process identifies the build system used in the repository and invokes the command for a clean build. The respective build systems use the project-level dependency information stored in configuration files like pom.xml and build.gradle to fetch the dependencies and store them locally. Next, we filter for repositories with permissive licenses, and filter out the repositories for which the CodeQL database creation failed, as that indicates that the repository either uses an unrecognized build system, some of its dependencies aren't satisfied, or the repository is in a transient state. We are left with 302 GitHub repositories after filtering for successful builds. We store the CodeQL database created for each of these repositories. Finally, we invoke the initialization of Eclipse JDT.LS<sup>2</sup> on each of the repositories, and filter for the repositories where JDT.LS could be successfully initialized. The final filtered list of 100 repositories, along with their CodeQL databases, comprise the PRAGMATICCODE dataset.
DOTPROMPTS. Yu et al. (2023) show that standalone functions account for less than  $30\%$  of open source projects among the top 100 most popular open source projects on GitHub, with most functions referencing third-party APIs or variables/constants defined in cross-file context. Hence, we create DOTPROMPTS for the evaluation of code generation in a pragmatic setting and aim to evaluate over real-world projects, not restricting to standalone function generation. Each task instance in DOTPROMPTS consists of a prompt up to a dereference location (dereference ‘’ operator in Java) within a target method, and the task is to complete the remainder of the method. Since dereference locations might be the points of occurrence for cross-file entities in source code, a model’s ability to use cross-file context can be evaluated using DOTPROMPTS. Curation Details. We identify non-test class files that aren’t auto-generated and have at least 1 cross-file dependency in the repository. From these files, we identify methods that aren’t class and object initializers and have  $\geq 2$  top-level statements spanning  $\geq 7$  lines of source code, to ensure sufficient complexity in target methods. The average number of lines of code in the ground truth completion in DOTPROMPTS is 12.7. We use the CodeQL Query listed in section I to identify target methods from each of the repositories in PRAGMATICCODE based on the above criteria. As shown by Shrivastava et al. (2022), repositories are
$^{1}$ https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/creating-codeql-databases  
 $^{2}$ https://github.com/eclipse/eclipse.jdt.ls
quite uneven in terms of their size, so to avoid individual repositories from dominating our evaluation, we limit to including up to 20 methods from each repository as identified by the CodeQL query. In order to simulate the real-world usage scenario, where a developer may invoke code completion at different points within a method, we identify up to 10 uniformly distributed dereference locations within each of the identified methods. Each such dereference location becomes a data point for DOTPROMPTS.
C Experimental Setup - Additional Details
Monitor Implementation. Language Server Protocol (LSP) (lsp), is an open industry standard of communication between IDEs and programming language specific tools like static analyzers and compilers, called language servers. Eclipse JDTLS (ecl, 2016) is a language server for Java, Rust Analyzer provides support for Rust, OmniSharp supports C# and JEDI supports Python. All language servers provide access to results of various static analyses over the same API. These can be accessed by implementing a Language Server Client. We implement an extensible and cross-platform language server client, multilspy with the aim to make it easy to setup and use these language servers in a language-agnostic manner. At the time of this writing, multilspy has been tested to work with language servers for Java, Python, Rust and C#, and we plan to extend the support to more programming languages and language servers.
Using multilspy, we specifically instantiate Eclipse JDT.LS as an engine that implements the static analysis  $A_{\varphi}$  to check for type-consistency of identifiers. Notably, Eclipse JDT.LS supports reasoning with build time artifacts like those obtained from Project Lombok (lom, 2009) and ProtoBuf (pro, 2008), thus allowing it to consider a broad view of the code repository in the static analysis results it provides. We implement our monitor  $M_{\varphi}$  as a thin layer around an LM, in accordance with Section 2, as a language server client that communicates with Eclipse JDT.LS. Since our implementation is based on LSP, and LSP is compatible with most major programming languages, our implementation can be easily ported to other languages besides Java, with the use of multilspy.
Hyperparameters. We use nucleus sampling (Holtzman et al., 2020) with a top-p value of 0.95 to generate 6 samples in total—1 each with temperature 0.2 and 0.4, and 2 each with temperature 0.6 and 0.8. We fix a prompt budget of (2048-512)=1536 tokens, and generation budget of 512 tokens, for a total context window size of 2048. If the text exceeds the prompt budget, we truncate it from the left for standard and classExprTypes prompts, and from the right for FIM. For classExprTypes augmentation with autoregressive decoding, we reserve a budget of  $20\%$  of total prompting budget for classExprTypes, and the remaining for autoregressive prompt. For FIM decoding without prompt augmentation, we reserve  $50\%$  of total prompting budget for the suffix. For FIM decoding with classExprTypes prompt augmentation, we reserve  $20\%$  of total prompting budget for classExprTypes,  $40\%$  for suffix, and the remaining for autoregressive prompt.
D Calculation of score@k
Let  $S = \{s_1, s_2, \ldots, s_n\}$  be a multiset representing metric scores obtained across n-independent trials. Without loss of generality, we order  $S$  in monotonic decreasing order as  $S_{\geq} = (s_1^{\geq}, s_2^{\geq}, \ldots, s_n^{\geq})$ . We calculate score@k, n as per the equation below:
$$
s c o r e @ k, n = \frac {1}{\binom {n} {k}} \sum_ {i = 1} ^ {n - k + 1} \binom {n - i} {k - 1} S _ {\geq} [ i ] = \frac {1}{\binom {n} {k}} \sum_ {T \in \binom {S} {k}} \max  (T) \tag {5}
$$
$$
\binom {S} {k} = \{V | V \subseteq S, | V | = k \} \tag {6}
$$
where  $1 \leq k \leq n$ , and  $\binom{S}{k}$  is the set of all subsets of  $S$  having cardinality  $k$ .
(c) ISM
(d) PM
Figure 5: score@k for models with MGD and FIM compared against base models
E Effect of MGD on Fill-in-the-middle (FIM) Decoding - Complete Results
Among the base models, SantaCoder supports the FIM modality. Figure 5 shows the results for SantaCoder with autoregressive and FIM decoding strategies and text-davinci-003, compared with respective configurations of SantaCoder with MGD.
Compilation Rate. Figure 5a shows that SantaCoder with MGD outperforms SantaCoder-FIM by a relative margin of  $7.04\%$ . We see significant improvement in compilation rate, when SantaCoder-FIM is augmented with MGD, leading it to outperform text-davinci-003 with a relative margin of  $27.97\%$ . SantaCoder-FIM with MGD relatively improves over SantaCoder-FIM by  $17.52\%$ .
Next Identifier Match. Figure 5b shows that FIM modality boosts next identifier match but still underperforms text-davinci-003 and correspondingly, SantaCoder with MGD. Augmenting SantaCoder-FIM with MGD leads to a relative improvement of  $5.07\%$ .
Identifier Sequence Match. On ISM, SantaCoder-FIM with MGD improves over SantaCoder-FIM by  $5.39\%$ , which closes the gap with text-davinci-003, underperforming it by just  $1.06\%$ .
Prefix Match. Figure 5d shows that SantaCoder-FIM improves over SantaCoder, but still underperforms text-davinci-003 by a relative margin of  $6.83\%$ . SantaCoder-FIM with MGD outperforms
SantaCoder-FIM by  $4.95\%$  showing continued improvements, while reducing the gap with text-davinci-003, underperforming it by just  $2.21\%$ .
Summary. Similar to our observations with prompt augmentation, while FIM modality leads to improvements across all metrics, we see continued improvement when using both FIM and MGD.
SC-FIM-classExprTypes-MGD. Motivated by the complementary nature of MGD, we further evaluated SC-FIM-classExprTypes-MGD, combining both prompt augmentation and FIM modality, and consistent with our findings, it leads to a further improvement over SC-FIM-classExprTypes, as seen in Figure 5. We use classExprTypes with FIM since RLPG also selects a subset of post-lines in prompt augmentation in a large number of cases.
F Effect of Identifier Complexity on Next Identifier Match - Complete Results
(a) Distribution of methods by most complex identifier
(c) Prompt augmentation
Figure 6: (NIM, score@6) across next-identifier complexity
(b) Standard prompt
(d) FIM
Identifier Complexity. Identifier names in code repositories can often get specific and long (Karampatsis et al., 2020). The vocabulary of LMs like CodeGen and SantaCoder is generally created by training a BPE tokenizer over the complete or a sample of the pretraining dataset (Sennrich et al., 2016). Due to this, while commonly used APIs may get tokenized into single tokens, identifiers specific to the context of individual repositories, especially in private settings, can span over multiple subtokens in the LM vocabulary, making their accurate generation difficult for the LM (both due to
Table 2: Statistics on inference time comparing CodeGen-6B and Codegen-6B with MGD. Time in seconds.
CodeGen-6B	CodeGen-6B-MGD
Mean	22.57	41.34
Mean slowdown	83.16%
Standard deviation	12.44	22.39
Min	0.94	1.14
First quartile	21.23	26.76
Median	25.48	44.56
Second quartile	25.86	58.20
Max	79.06	111.73
the increased number of decoding steps leading to a larger search space and also due to the relative rarity of the identifier name). We define the complexity of an identifier as the mean number of subtokens required to decode it, using the tokenizers of all the models under study (CG, SC, TD-3). About  $36.97\%$  of methods in DOTPROMPTS dataset have at least 1 identifier of complexity [4, 18) as shown in Figure 6a, and therefore, for a model to generate those methods correctly, it is important for the model to be able to generate the complex identifier name.
Next Identifier Match. Figure 6 shows the result for the NIM metric, across increasing complexity of the next ground-truth identifier. We note that all models show a sharp degradation in performance with an increase in identifier complexity. The same trend holds true for prompt augmentation as well as FIM modality. Augmenting base models with MGD leads to significant improvement in the model's ability to predict the next identifier for the most complex identifier case [4, 18) with a relative improvement in the range of  $21\% -28\%$ . The smallest model, CodeGen-350M with MGD achieves parity with the largest model, text-davinci-003 and outperforms the much larger CodeGen-6B with a relative margin of  $11.95\%$ . CodeGen-2B with MGD outperforms CodeGen-6B and text-davinci-003 by a relative margin of  $22.14\%$  and  $9.79\%$  respectively. SantaCoder with MGD improves over text-davinci-003 by a relative margin of  $11.53\%$ . We further observe that both prompt augmentation and FIM-modality with MGD lead to a diminishing rate of degradation, as can be seen in the curves for SC-FIM-MGD in Figure 6d. SantaCoder-FIM with MGD outperforms text-davinci-003 and SantaCoder-FIM by a relative margin of  $16.11\%$  and  $17.04\%$  respectively.
Summary. The ability of LMs to accurately predict the identifier name decreases sharply with an increase in identifier complexity. All models with MGD outperform their respective base models with a large relative improvement in the range of  $21\% - 28\%$ , with smaller LMs outperforming much larger LMs (CodeGen-350M-MGD achieves parity with text-davinci-003, and outperforms CodeGen-6B, CodeGen-2B-MGD outperforms CodeGen-6B). While prompt augmentation and FIM modality lead to improvement over baselines, they also suffer from a sharp decrease across identifier complexity, but augmenting them with MGD leads to similar large improvements as observed with base models (relative improvement in the range of  $15.92\% - 18.59\%$ ).
G Impact of MGD on Inference Time
To study the impact of adding MGD on inference time, we compare the code generation times by CodeGen-6B model and CodeGen-6B with MGD. We select 500 prompts from DOTPROMPTS and generate up to 512 tokens with CodeGen-6B as well as CodeGen-6B with MGD. The inferencing is performed with HuggingFace implementation of the model on machine configuration (1) as described in Section 6. We use the same decoding scheme as described in Section 3. After generation, we filter out the cases where the number of tokens generated by CodeGen-6B and Codegen-6B with MGD is not the same, in order to ensure parity both in the size of the input prompt and number of generated tokens. We are left with 161 instances, where the size of the input prompt, as well as the number of generated tokens, is the same for both models. Figure 2 shows statistics related to inference time for both the models in seconds. We observe a mean slowdown of  $83.16\%$  in decoding time for CodeGen-6B with MGD. We note that there might be many opportunities to optimize our implementation though we have not explored them yet.
H Generalizability Study: Microbenchmark Results
Table 3 presents results over MGDMicroBENCH, detailing the scenarios we evaluated for studying the generalizability of MGD.
H.1 Different Programming Languages
In MGDMICROBENCH, CI1, NA1, VD1, J2 are coding scenarios in Java, SE1, SE2 and J1 are coding scenarios in C#, and TS1, TS2 and ST1 are coding scenarios in Rust for which monitors under the MGD framework, targeting the specific languages are built. Section 4 provided detailed evaluation of a monitor for Java. Hence, collectively, section 4 and MGDMICROBENCH demonstrate that MGD is viable for Java, C# and Rust.
H.2 Different Coding Scenarios
Instantiation of valid classes. Scenario CI1 (abbreviation for "class instantiation") considers demo code for a university setting, where Person is declared as an abstract class, extended concretely by Student and Teacher classes. In the scenario, the prompt code is setup to declare an object p1 of type Person as follows: 'Person p1 = new'. The task for the LM is to generatively complete the code, assigning a concrete instantiation of Person. Following the keyword 'new', a constructor should be invoked. SC configuration, without MGD, invokes the non-existent constructor Person( . . ) (non-existent since it is an abstract class), whereas SC-MGD, with a monitor that differentiates between abstract and concrete classes invokes Student( . . ), which is valid as per the scenario.
switch over enum. Scenarios SE1, SE2 (abbreviation for "switch over enum") and J1 (abbreviation for "joint monitoring") are representative scenarios obtained from a real world emulation software written in C#. In scenarios SE1 and SE2, the prompt code has been setup to 'switch(. . .' over enum values of type 'AccessSize' and 'Intrinsic' respectively. The model configurations used are SC and SC-MGD. While SC generated the 'case' branch '1', which is a violation of the enum type, SC-MGD correctly generated the case branch 'AccessSize.Byte' which is type-correct while also being consistent with the ground truth. In SE2, while SC was able to get the type-name of the enum value correct, i.e., it generated 'Intrinsic', the enum value under the enum type it generated is non-existent, i.e., the symbol 'X86Comisdgt' does not exist, and hence invalid. SC-MGD generates a valid case branch, 'Intrinsic.X86Comisdlt', consistent with the ground truth. Scenario SE2 is not completely solved due to the invalid dereference of '.LessThan' by SC-MGD, due to the absence of monitoring for dereferences, which is resolved in scenario J1 below, in the discussion of joint monitoring.
Valid number of arguments to methods. Scenarios  $NA1$  (abbreviation for "number of arguments"),  $VD1$  (abbreviation for "valid dereferences") and  $J2$  are derived by modifying the underlying API presented in Figure 1a (in the main paper). The task is to use the ServerNode.Builder fluent API to create an instance of ServerNode. For these scenarios, we replaced individual methods with  $Ip(ip)$  and with  $Port(0)$  (as seen in Figure 1a) with a single method with  $IpPort(ip)$ ,  $port$  after which build() can be called to instantiate the ServerNode object. In scenario  $NA1$ , the prompt code has been setup with an open call to method with  $IpPort(),$  and the task for the LM is to generatively write code to pass the correct arguments to the method. SC configuration, without MGD generates a single argument, and closes the method call, thus violating the API of the with  $IpPort$  method. It further generates a call to the non-existent method with  $Port$ . SC-MGD configuration guided by a monitor for valid number of arguments, generates two arguments, corresponding to ip and port respectively, in-line with the contract of with  $IpPort$ . Further, it calls the method build(). In scenario  $VD1$ , the setup is similar to  $NA1$ , except that the call to the method with  $IpPort$  is not made, and instead, the task for the LM is to generate the call to the right method with  $IpPort$ , and then generate the arguments for it. Unlike  $NA1$ , in this scenario, we use only the dereferences monitor. While the base configuration, SC without MGD, generates a call to non-existent 'hostAddress', SC-MGD with monitor for dereferences calls the right method, 'with  $IpPort$ . However, without the monitor for right number of arguments, it just generates one argument, thus violating the API. It further calls with  $IpPort$  a second time, which is a type-correct dereference, however, not the expected response as per the ground truth, since it is a redundant call. Scenario  $VD1$  is not completely
Table 3: Results over MGDMICROBENCH
ID	DESCRIPTION	GENERALIZATION ASPECT	WITHOUT MGD	WITH MGD
Code scenario — Valid class instantiations
CII	University - Instantiation and assignment to abstract class	Code Scenario, Java	Person p1 = new Person("John", ...)	Person p1 = new Student("John", ...)
Code scenario — Valid enum constants within switch statements
SE1	Emulation - switch over AccessSize enum	Code Scenario, C#	case 1: ...	case AccessSize.Byte: ...
SE2	Emulation - switch over Intrinsic enum	Code Scenario, C#	case Intrinsic. X86Comisdtgt: ... X86Condition. GreaterThan	case Intrinsic. X86Comisdlt: ... X86Condition. LessThan
Code scenario — Correct number of arguments in method calls
NA1	Fluent API - Parse Server	Code Scenario, Java	arr[0]) .withPort(...)	arr[0].trim(), Integer.parseInt(arr[1].trim())) .build()
Code scenario — type-correct dereferences
VD1	Fluent API - Parse Server	Code Scenario, Java	... hostAddress (arr[0]) .port(Integer. parseInt(arr[1]))	... withIpPort (arr[0]) .withIpPort(arr[1])
Joint monitoring for valid enum constants in switch and type-correct dereferences
J1	Emulation - switch over Intrinsic enum	Code Scenario, C#	case Intrinsic. X86Comisdgt: ... X86Condition. GreaterThan	case Intrinsic. X86Comisdlt: ... X86Condition. Below
Joint monitoring for type-correct dereferences and number of arguments
J2	Fluent API - Parse Server	Code Scenario, Java	... hostAddress (arr[0]) .port(Integer. parseInt(arr[1]))	... withIpPort (arr[0].trim(), Integer.parseInt(arr[1].trim())) .build()
Static Analysis — Typestate API protocols
TS1	Android MediaPlayer	Typestate, Rust	stop();	reset();
TS2	GPIO Pins	Typestate, Rust	set_inputPull_up();	set_input_high_z();
Static Analysis — SessionType 2-party communication protocol
ST1	Deposit money to ATM	SessionTypes, Rust	recv();	send(0).recv();
solved due to the invalid number of arguments by SC-MGD, and will be resolved in scenario J2 below.
Joint monitoring for multiple properties. MGD framework can utilize results from multiple monitors simultaneously to guide generation of code to follow multiple properties. Scenarios  $J1$  and  $J2$  explore 2 different instances of joint-monitors operating simultaneously.  $J1$  is the same coding scenario as  $SE2$ . In  $SE2$ , the monitor only monitored for generation of type-valid 'case' branches, whereas in  $J1$ , the monitor for valid case branches and monitor for type-valid dereferences are used jointly to perform the decoding. The difference can be noted in generation of the correct dereference 'X86Condition.Below' in  $J1$  (highlighted in green) for SC-MGD, which earlier was 'X86Condition.LessThan' in  $SE2$ , resulting in a non-existent symbol error. In  $J1$ , SC-MGD with the joint monitor is able to generate both the correct case branch, as well as a valid dereference. Scenario  $J2$  is the same coding scenario as  $VD1$ . In  $VD1$ , only monitor for valid dereferences was used in SC-MGD, whereas in  $J2$ , the monitors for valid dereferences and monitor for correct number of arguments to methods, are both used jointly with SC-MGD. The difference can be noted in the correct number of arguments (2) generated by SC-MGD in  $J2$ . Further, unlike in  $VD1$ , SC-MGD in  $J2$  subsequently calls build(), which is in-line with the ground truth, likely due to the improved code context that generating the right number of arguments provided.
H.3 Richer Static Analyses and Constraints