SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Each of the different coding scenarios discussed in MGDMicroBENCH require a different static analysis to be performed, and hence, MGD is applicable with a variety of static analysis techniques. In this section, we focus on richer properties that can be enforced with MGD. Incorrect use of symbol names is a broad class of errors, and a root cause of many compile-time and run-time errors. Invalid use of defined symbol/method names can also lead to various compile-time and run-time errors, and hence, just having the symbols in context may not be helpful. For the following scenarios, we use the SC-FIM-classExprTypes and SC-FIM-classExprTypes-MGD configurations, which we found to be the strongest configurations due to the additional context, in the detailed evaluation in section 4. Consider the following scenarios:
Typestate Protocols. The Android MediaPlayer has complex constraints on the ordering of the API calls<sup>3</sup> which might be hard even for developers to reason about (Mishra et al., 2016). Runtime violation of such contracts lead to exceptions like 'IllegalStateException'. Mishra et al. (2016) proposes the use of typestate analysis (Strom & Yemini, 1986) to detect violations of such protocols, through static analysis techniques, and prevent the bugs at runtime. Duarte & Ravara (2021) show that typestate protocols can be enforced by the rich analysis supported in the Rust type system itself. We utilize this, and instantiate a monitor to guide LMs to generate typestate-valid API calls.
TS1: The task in scenario TS1 (abbreviation for "typestate") is to generatively complete the partially-written code that iteratively play songs in a playlist using the Android MediaPlayer API. The completion is invoked at a point where the MediaPlayer object is in the Stopped state. While the base configuration generates a call to stop();, which is in violation to the API protocol, since the MediaPlayer is already in the Stopped state at the point of completion, the same model configuration with MGD generates reset(); which is the only legal transition at the state as per the implemented protocol.
TS2: Crichton (2023) and Rust on Embedded Devices Working Group et al. (2018) describe the representation of GPIO pin states and the transitions between them as a typestate protocol. We use the Rust API provided in Rust on Embedded Devices Working Group et al. (2018) and task the LM to complete partially written code, that initializes a GPIO pin and transitions to the input_high_z state. While the base configuration generates a call to set_input_PULL_up(); which transitions to the input_PULL_high_state, the same model augmented with MGD calls set_input_high_z(); which leads to the desired state.
SessionType Protocols. Session types can be used to capture ordering constraints on two party communication (Crichton et al., 2019) as communicating finite state machines. Jespersen et al. (2015) propose that Rust type system can be used to enforce session type contract at compile-time. We utilize the proposed approach, and build a monitor, that guides LMs to generate method-calls in line with the contract.
<sup>3</sup>https://developer.android.com/reference/android/media/MediaPlayer
ST1: Interaction with an ATM machine is an oft used example for two-way communication protocol. We use the session type formalism for ATM machine, described in (Jespersen et al., 2015). The task for the LM is to complete partially written code using the session type API, to deposit money and print the new balance. On the client side, after authentication and selecting the action to deposit money (from the ATM menu), the base model generated a call to recv(); , which would wait for the ATM to send a value, whereas the protocol expects the client to "send" the deposit value. The base model augmented with MGD is able to correctly generate the call to send(0). recv(); , which first sends amount to be deposited, and then waits for the ATM to communicate the new balance, which will be reported to the user. We note that with MGD, the model was able to generate code that follows the protocol appropriately.
I CodeQL Query for Identifying Evaluation Target Methods
/**
* @id java/examples/find_target_methods
* @name find_target_methods
* @description Identify target methods from a Java repository along with classExprTypes information for DotPrompts dataset */
*/
import java
predicate filterClass(Class c) {
    not(c instanceof TestClass) and
    c来源于Source() and
    not c.getFile().getRelativePath().matches("%generated%") and
    not(c.getFile().getRelativePath().matches("%test%") ) and
    not(c.getFile().getRelativePath().matches("%target%") ) and
    not(c/File().getRelativePath().matches("%build%") ) and
    count(Method m1 | m1 = c.getAMethod() and filterMethod(m1) | m1) >= 1
}
predicate filterMethod(Method m) {
    m来源于Source() and
    not(m instanceof TestMethod) and
    not(m.getName("<clinit>")) and
    not(m.hasName("<obinit>")) and
    m.getBody().getNumStmt() >= 2 and
        (m.getBody(). Locations().getEndLine() -
            m.getBody(). Locations().getStartLine()) >= 7
}
predicate typeDeclaredInFile(Type t, File file) {
    (t.fromSource() and t.getFile() = file) or
    (t.getErasure().fromSource() and t.getErasure().getFile() = file)
}
// Find classExprTypes files such that they contain type definition of any expressions defined in any callable in the class except for target_m predicate filterClassExprTypeFile(Class c, Method target_m, File CLASSFile, File CLASSExprTypesFile){
    classExprTypesFile != classFile and (
        // classExprTypesFile contains type definition of a singly imported type
        exists(Type t, ImportType impt | impt.fromSource() and
        c.getFile() = impt.getFile() and
        t = impt.getImportedType() and
        typeDeclaredInFile(t, classExprTypesFile)
    ) or
// classExprTypesFile contains type definition of return type or param type of the target method exists(Type t | (t = target_m.getAParamType() or t = target_m.getReturnType()) and typedefInFile(t, classExprTypesFile) ) or // classExprTypesFile contains type definition of type of any expression within a callable (that is not the target method) in class c, or a return type of a callable or any of its parameters exists(Expr e, Type t, Callable m | m = c.getAC callable() and m != target_m and ( e.getAnEnclosingStmt() = m.Body().getASstmt() and e.TYPE() = t ) or t = m.getAParamType() or t = m.getReturnType() ) and typedefInFile(t, classExprTypesFile) ) or // classExprTypesFile contains type definition of any field type exists(Type t, Field f | c.getAField() = f and f.TYPE() = t and typedefInFile(t, classExprTypesFile) ) } predicate expressionOfTypeContainedInBlock(Expr e, Type t, BlockStmt b) { e.getAnEnclosingStmt() = b.getASstmt() and e.TYPE() = t } from File,classFile, File classExprTypesFile, Class c, Method m, BlockStmt b, int startLine, int startCol, int endLine, int endCol where // Bind variables m = c.getAMethod() and b = m.Body() and CLASSFile = c.getFile() and // Apply filters filterClass(c) and filterMethod(m) and filterClassExprTypesFile(c, m, CLASSFile, classExprTypesFile) and // Bind method boundary locations startLine = b.getLocation().getStartLine() and startCol = b.getLocation().getStartColumn() and endLine = b.getLocation().getEndLine() and endCol = b.getLocation().getEndColumn() select classFile.getAbsolutePath(), classifyFile.getRelativePath(), classExprTypesFile.getAbsolutePath(), classExprTypesFile.getRelativePath(), startLine,
J Examples of Code Generation with MGD
We present examples of code generation with MGD and compare them to generations without MGD augmentation. The appearance of red-markers below identifier names indicate that the identifier name is not type-consistent with the target object/class for the dereference operator. We highlight the correct identifiers generated with MGD augmentation in light green.
Figure 7: TD-3 and SC generate invalid identifier names: getName, getDesc for target object envTypeEnum and success for target class ListResult. Augmenting SC with MGD leads to the generation of correct identifiers: getDescription, of for the respective objects, leading to complete agreement with the ground truth.
startCol, endLine, endCol
public AccountNumber { Guard-guard(value).againstNullOrWhitespace(ValidationMessages.ACCOUNT_NUMBER_EMPTY); }
Figure 8: SC and TD-3 generate the identifier name: ACCOUNT_NOT_found, which is invalid for the target class. The snippet of code presented above is from one of the files present in the classExprTypes augmentation. Augmenting SC with classExprTypes leads the model to generate the identifier name ACCOUNT_NUMBER_empty, which is consistent with the type: ValidationMessages, and therefore compileable. However, it still does not match the ground truth. Augmenting both SC and SC-classExprTypes with MGD leads the model to generate the correct identifier name: ACCOUNT_NUMBER_NOT Exist, achieving agreement with the ground truth.
Figure 9: TD-3 and SC generate code that match the ground truth, except for the identifier names for target objects: advancedSettings, advancedSettingsComponent. Augmenting SC with MGD leads to generation of correct identifier names, and therefore agreement with ground truth.
Figure 10: TD-3 and SC both generate code with invalid identifier names. SC augmented with MGD is able to use the correct identifier name and therefore generates compileable code. However, the generated code does not make use of the defined exception class: ZolaServerConnectionFailedException for exception handling. Augmenting SC with Fill-in-the-middle provides the model with the necessary context to perform exception handling, however, the model still hallucinates the identifier name getName for the target object message. Hence, to get a correct generation, SC is augmented with both FIM and MGD, and this configuration is able to match the ground truth.
SC-FIM
SC-FIM-MGD
Figure 11: Example of generation with classExprTypes prompt augmentation.
SC-classExprTypes-MGD
Figure 12: Example of generation with RLPG prompt augmentation. Augmenting SC with RLPG leads to generation of toDtos as compared to toDTOList, where both are invalid identifier names for the target object. Augmeting both the configurations with MGD leads to generation of the correct identifier and match with ground truth.
Figure 13: TD-3 and SC generate identifier names: ATTR, GROUP_KEY respectively for the target class Constant, which are both invalid. MGD augmentation leads to generation of correct identifier: UPDATEABLEPRICE.