BOOTSTRAPPING PROGRAM SYNTHESIS LANGUAGE MODELS TO PERFORM REPAIRING

Certain aspects of the present disclosure provide techniques and apparatus for improved program synthesis using machine learning. An input indicating a programming task is accessed. A generated program is generated based on processing the input using a trained machine learning model. In response to determining that the generated program failed to satisfy the programming task, feedback is generated, and a revised program is generated based on processing the feedback using the trained machine learning model. In response to determining that the revised program satisfied the programming task, one or more parameters of the trained machine learning model are updated based on the revised program.

INTRODUCTION

Aspects of the present disclosure relate to machine learning.

A wide variety of machine learning model architectures have proliferated and have been used to provide solutions for a multitude of prediction problems. Language models (also referred to as large language models (LLMs)) are a class of models that learn to generate text, such as natural language. Language models have been used for a variety of solutions, including chat bots, program synthesis, and the like. In program synthesis tasks, the model attempts to learn to generate computer program code that can be executed (by a computer) to perform a desired task. In some conventional systems, neural-guided program synthesis approaches relied on using a domain-specific language (DSL) due to the extremely large search space of popular programming languages.

In some conventional approaches, language models for program synthesis have been developed to improve upon DSL-based approaches. These language models may be trained and evaluated on programming competition datasets (e.g., the Mostly Basic Python Programming (MBPP) dataset, or the Automated Programming Progress Standard (APPS) dataset). Such datasets generally contain prompts describing a computational task or problem that the generated program should solve or satisfy, as well as one or more samples of program code (e.g., authored by human programmers) that solve or perform the task. However, such datasets are substantially limited in size (with relatively few exemplars) and quality (where some of the purported solutions do not work, or do not solve the corresponding task). Problematically, many machine learning models (and large language models in particular) rely on vast amounts of highly accurate training data to learn to produce accurate output.

BRIEF SUMMARY

Certain aspects provide a processing system comprising: one or more memories comprising processor-executable instructions; and one or more processors configured to execute the processor-executable instructions and cause the processing system to: access a first input indicating a first programming task; generate a first generated program, wherein, to generate the first generated program, the one or more processors are configured to execute the processor-executable instructions to cause the processing system to process the first input using a trained machine learning model; determine that the first generated program failed to satisfy the first programming task; generate first feedback based on the determination that the first generated program failed to satisfy the first programming task; generate a first revised program, wherein, to generate the first revised program, the one or more processors are configured to execute the processor-executable instructions to cause the processing system to process the first feedback using the trained machine learning model; determine that the first revised program satisfied the first programming task; and update one or more parameters of the trained machine learning model based on the first revised program in response to the determination that the first revised program satisfied the first programming task.

Certain aspects provide a method, comprising: accessing a first input indicating a first programming task; generating a first generated program based on processing the first input using a trained machine learning model; in response to determining that the first generated program failed to satisfy the first programming task: generating first feedback based on the determination that the first generated program failed to satisfy the first programming task; and generating a first revised program based on processing the first feedback using the trained machine learning model; and in response to determining that the first revised program satisfied the first programming task, updating one or more parameters of the trained machine learning model based on the first revised program.

Certain aspects provide one or more non-transitory computer-readable media comprising computer-executable instructions that, when executed by one or more processors of one or more processing systems, cause the one or more processing systems to: access a first input indicating a first programming task; generate a first generated program, wherein, to generate the first generated program, the one or more non-transitory computer-readable media further comprise computer-executable instructions that, when executed by the one or more processors, cause the one or more processing systems to process the first input using a trained machine learning model; determine that the first generated program failed to satisfy the first programming task; generate first feedback based on the determination that the first generated program failed to satisfy the first programming task; generate a first revised program, wherein, to generate the first revised program, the one or more non-transitory computer-readable media further comprise computer-executable instructions that, when executed by the one or more processors, cause the one or more processing systems to process the first feedback using the trained machine learning model; determine that the first revised program satisfied the first programming task; and update one or more parameters of the trained machine learning model based on the first revised program in response to the determination that the first revised program satisfied the first programming task.

To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the drawings. It is contemplated that elements and features of one aspect may be beneficially incorporated in other aspects without further recitation.

DETAILED DESCRIPTION

Aspects of the present disclosure provide apparatuses, methods, processing systems, and computer-readable mediums for improved training and fine-tuning of machine learning models using bootstrapping and program repair.

In aspects of the present disclosure, machine learning models (e.g., large language models) can be trained, updated, and/or fine-tuned using a combination of bootstrapping and program repair. As used herein “fine-tuning” refers to updating the parameter(s) of a machine learning model that has already been at least partially trained (often referred to as a foundational model). For example, a model may be trained using one set of data, and fine-tuned for a specific task (e.g., program synthesis) based on a set of fine-tuning data. As used herein, “bootstrapping” generally refers to the use of the output of the model to refine the model itself. For example, the system may use correctly generated output (e.g., synthesized programs that accurately perform the desired task) as training data during a refinement or fine-tuning stage.

Additionally, as used herein, “repairing” refers to the process of iteratively attempting to produce improved program code. For example, if an initial synthesized program fails to solve or satisfy the task (e.g., because of compilation errors, runtime errors, or errors in the logic of the program), the system may use these errors to prompt the model to generate an updated program (e.g., to attempt to repair the first generated program). This use of bootstrapping with an iterative repairing process can enable improved fine-tuning of the models, resulting in substantially improved prediction accuracy (e.g., improved ability to generate program code that satisfies the desired task).

In some aspects, using a coding language model (also referred to as a program synthesis machine learning model), the machine learning system generates a code solution (also referred to as a generated program) for a programming problem or task. The machine learning system may then attempt to verify or validate the generated program (e.g., using a compiler, using test cases, and the like) to decide whether the generated code is accurate (e.g., whether the generated code can be compiled into a valid program that accurately solves or satisfies the desired task).

In some aspects, if the generated program fails validation, feedback indicating this failure (e.g., compiler/interpreter information) can be provided as input to the machine learning model, prompting the model to generate a new program (referred to in some aspects as a revised program). In some aspects, if the model produces a correct answer (e.g., code that validly compiles and solves the problem), this model output can be used during a fine-tuning phase. In some aspects, if the model fails to produce a correct solution (e.g., the revised code still fails), the system may use a ground-truth solution for the task during the fine-tuning phase.

In this way, during fine-tuning, the model can be substantially improved as compared to some conventional training and fine-tuning approaches that rely on the programming task datasets themselves. For example, using aspects of the present disclosure, the model is exposed to more diverse training data (potentially including information from the compiler/interpreter), which may be of higher quality than that of the ground-truth in the available dataset(s). Additionally, allowing the model to attempt to repair faulty programs can result in substantially improved output, and this iterative improvement process yields more helpful fine-tuning data.

Example Environment for Improved Updating of Program Synthesis Machine Learning Models

FIG.1illustrates an example environment100for improved updating of program synthesis machine learning models, according to various aspects of the present disclosure.

In the illustrated environment100, a set of task descriptions110are accessed by a fine-tuning system125. In some aspects, the fine-tuning system125corresponds to a computing system that fine-tunes (e.g., updates) trained machine learning models to improve (or attempt to improve) the inferencing accuracy of the models. In some aspects, the fine-tuning system125may additionally serve as a training system (e.g., the computing system that initially trains the model) and/or as an inferencing system (e.g., the computing system that uses the trained and/or fine-tuned models to generate inferences during runtime).

In the illustrated example, the fine-tuning system125further accesses a machine learning model105The fine-tuning system125uses the task descriptions110to update or revise the machine learning model105in order to generate a revised machine learning model130. As used herein, “accessing” data (e.g., parameters for a model) can generally include receiving, retrieving, requesting, collecting, obtaining, generating, or otherwise gaining access to the data (e.g., parameters for the model).

In some aspects, the machine learning model105is a pre-trained machine learning model, such as a large language model. In some aspects, the machine learning model105has been trained for program synthesis and is revised or fine-tuned (e.g., by the fine-tuning system125) to generate more accurate programs. As discussed above, program synthesis generally corresponds to generation of computer program code that can be compiled and executed to perform a desired task. In some aspects, the machine learning model105is initially trained based on relatively large corpora of code, allowing the machine learning model105to learn to synthesize new code. During the fine-tuning phase (also referred to as a refinement stage, an updating stage, or simply as a training stage), the fine-tuning system125uses the task descriptions110to teach the machine learning model (revised machine learning model130) how to generate program code that solves desired tasks.

In some aspects, the task descriptions110each generally include a description of a programming task (also referred to as a prompt), as well as one or more unit tests115(also referred to as test cases and/or asserts in some aspects) and one or more solutions120(also referred to as ground-truth or labels). For example, in some aspects, the task descriptions110may correspond to samples from programming competition datasets (e.g., the MBPP dataset and/or the APPS dataset).

In some aspects, the prompt portion of the task description110may include a natural language description (e.g., describing the task in natural language) and/or a logical description (e.g., describing the task in terms of Boolean logic or other operators). For example, the task description110may include a prompt such as “write a function to find the first repeated character in a given string.” Additional examples may include, without limitation, prompts such as “write a function to find the longest chain which can be formed from a given set of pairs,” or “write a function to get a lucid number smaller than or equal to n.” Generally, the prompts may be of any length, and may include unstructured natural language, as well as other information such as numerical data, structured data, and the like.

In some aspects, each task description110may include zero or more unit tests115. Generally, each unit test115specifies a test that can be used to judge the validity or accuracy of generated code. For example, continuing the above programming task to find repeated characters in a string, one unit test115may specify that the program should output the character “A” when given the string “ABCABC.” In some aspects, some or all of these unit tests115are provided to the machine learning model105as part of the input (along with the prompt), as discussed in more detail below. In some aspects, some or all of these unit tests115are hidden (e.g., not provided to the model) and are used to validate the resulting code.

In some aspects, the solution(s)120generally indicate one or more ground-truth solutions for the corresponding programming task. That is, the solutions120may generally include computer program code that (purportedly) solves the task, such as code that can be compiled and executed to find and output the first repeated character in a string. In some aspects, some or all of the solutions120are manually authored (e.g., by programmers) and/or are generated by other program synthesis models. Although referred to as “solutions” and “ground-truth” answers in some examples described herein, in some aspects, one or more of the solutions120may, in fact, contain erroneous answers. That is, the solution(s)120may themselves contain errors that result in invalid code (e.g., code that does not compile, code that returns incorrect answers, and the like). Such errors are unfortunately common in many available datasets. However, using aspects of the present disclosure to provide bootstrapping and program repairing, the resulting revised machine learning model130may nevertheless be substantially improved. For example, because the validity of the model output is determined based on criteria other than the solution120itself, and the fine-tuning system125can use labels other than the solution120itself, the model may be fine-tuned based on more accurate data.

In some aspects, the fine-tuning system125can process each task description110(e.g., natural language text describing a programming task) using the machine learning model105to generate a corresponding program. The fine-tuning system125may then validate (or attempt to validate) the generated program, such as by determining whether the program compiles successfully (as opposed to generating compilation or interpreter error(s)), whether the program runs successfully (as opposed to generating runtime error(s)), and/or whether the program processes the unit test(s)115correctly (e.g., whether the program returns correct results based on the tests). In some aspects, if the generated program is validated (e.g., the program successfully performs the programming task), the fine-tuning system125may use the generated program as a ground-truth label to fine-tune the machine learning model105(either immediately, or after a number of task descriptions110have been processed), as discussed below in more detail.

In some aspects, if the generated program fails validation (e.g., the program caused errors to be generated and/or returned inaccurate results), the fine-tuning system125may prompt the machine learning model105to attempt to repair the code and/or to generate new code. For example, the fine-tuning system125may provide feedback (such as the generated error(s)) as input, allowing the machine learning model105to generate revised code. As discussed below in more detail, this revised code may comprise the originally generated code with revisions or repair attempts (e.g., modifications to the program) and/or may comprise entirely new code (e.g., the model may attempt to write an entirely new program).

In some aspects, the fine-tuning system125may similarly validate (or attempt to validate) the revised program. If this revised program is validated successfully, in some aspects, the fine-tuning system125may use the revised program as a ground-truth label to fine-tune the machine learning model105(either immediately, or after a number of task descriptions110have been processed), as discussed below in more detail. In some aspects, if the revised program still fails validation, the fine-tuning system125may use the solution120of the task description110as the ground-truth to fine-tune the model, as discussed in more detail below

In some aspects, the fine-tuning system125(or another system) can deploy the revised machine learning model130for inferencing after fine-tuning. Generally, deploying the revised machine learning model130may include any operations or procedures to provide the revised machine learning model130for inferencing, including indicating or transmitting the updated parameter(s) to an inferencing system, instantiating or storing the updated model for local inferencing, and the like.

In some aspects, the fine-tuning system125may perform multiple iterations of fine-tuning. For example, after processing each task description110to generate fine-tuning exemplars and updating the machine learning model105using these exemplars (to generate the revised machine learning model130), the fine-tuning system125may then repeat this process to further fine-tune the revised machine learning model130. That is, the fine-tuning system125may re-process the task descriptions110using the revised machine learning model130to generate new fine-tuning exemplars and use these new exemplars to further fine-tune the revised machine learning model130. In this way, the fine-tuning system125can use bootstrapping and program repairing to substantially improve the prediction performance of the machine learning model, resulting in more accurate and reliable program synthesis.

Example Workflow for Improved Updating of Program Synthesis Machine Learning Models

FIG.2depicts an example workflow200for improved updating of program synthesis machine learning models, according to various aspects of the present disclosure. In some aspects, the workflow200is performed by a fine-tuning system, such as the fine-tuning system125ofFIG.1. That is, the depicted components (e.g., a verification component215and a training component220) may be components of a fine-tuning system. Though depicted as discrete components for conceptual clarity, the depicted components (and others not illustrated) may be combined or distributed across any number of devices and components, and may generally be implemented using hardware, software, or a combination of hardware and software.

In the illustrated workflow200, the fine-tuning system accesses a task description110and processes the task description110using a machine learning model205to generate a program210. In some aspects, as discussed above, the task description110comprises a natural language description of a programming task, such as: “write a program that finds the first repeated character in an input string.” In some aspects, as discussed above, the task description110further includes a unit test or assert, such as specifying an input and the intended output (e.g., indicating that the program should output “B” when the input is “ABCDBA”).

As discussed above, in some aspects, the task description110may contain or be associated with one or more unit tests (e.g., unit tests115ofFIG.1). In some aspects, the fine-tuning system may retain at least one unit test as a hidden test. That is, the fine-tuning system may optionally provide one or more unit tests in the prompt to the machine learning model205, but may ensure that at least one unit test is kept as hidden (e.g., to use for validation of the generated program).

In some aspects, the machine learning model205corresponds to a pre-trained model, such as the machine learning model105ofFIG.1. The program210generally comprises computer program code, generated by the machine learning model205based on the prompt (e.g., based on processing the task description110and/or unit test(s)) to (attempt to) solve the described programming task. Generally, the program210may comprise code in any programming language and using any suitable formatting (e.g., any formatting, depending on the particular language, that can be compiled and executed).

In the illustrated example, the generated program210is accessed by the verification component215for evaluation (also referred to as verification and/or validation). In some aspects, as discussed above, the verification component215may validate the program210at least in part by determining whether the program210causes any errors to be generated. For example, the verification component215may determine whether any compilation errors occur when the verification component215(or another component) attempts to compile or interpret the program210, and/or whether any runtime errors occur when the verification component215(or another component) attempts to process input data using the program210). In some aspects, if any errors are generated, the verification component215may determine that the program210has failed to solve (also referred to as failing to satisfy) the programming task.

In the illustrated workflow200, the verification component215may also access one or more unit tests115associated with the task description110. The verification component215may use the unit test(s) to validate the program210. For example, the verification component215may process the input portion of a given unit test115using the program210to generate output, and the verification component215may compare the program output with the intended output portion of the unit test115. In some aspects, if the verification component215determines that any unit tests115are failed (due to inaccurate output), the verification component215may determine that the program210has failed to solve the programming task.

In the illustrated workflow200, the verification component215indicates to the training component220the results of the validation. In some aspects, the verification component215may indicate that the program210has failed, without providing additional detail or explanation. This may be referred to as a “simple feedback” implementation, in some aspects. In some aspects, the verification component215may provide additional detail, such as error explanations indicating the cause of failure (e.g., indicating the particular error(s) generated and/or indicating that one or more unit tests115were failed). This may be referred to as “full feedback” in some implementations. In some aspects, if available, the training component220may additionally provide suggestions or more detailed input. For example, if the interpreter and/or compiler are configured to provide more detailed suggestions (e.g., indicating the potential source of the error or suggesting fixes), the verification component215may include this detail in the feedback.

In some aspects, the training component220may take a variety of actions depending on the feedback provided by the verification component215. For example, in the illustrated workflow200, if the training component220determines that the program210successfully solved the programming task (also referred to as satisfying the task) (e.g., because no errors were generated, and the unit tests115were accurately processed), the training component220may add the program210as a fine-tuning exemplar in a fine-tuning dataset235. In some aspects, to form the fine-tuning exemplar, the training component220may use the task description110(and, in some aspects, any unit tests115that were included in the prompt to the machine learning model205) as the input portion of the exemplar, and the training component220may use the generated program210as the target output. That is, rather than using a ground-truth solution (e.g., the solution120ofFIG.1) as the label, the training component220can use the output of the machine learning model205itself.

In some aspects, if the training component220determines that the program210has failed to successfully satisfy the programming task, the training component220may determine whether to attempt to repair the program210. For example, in some aspects, the fine-tuning system may use a defined or maximum number of repairing iterations (e.g., allowing the machine learning model205to repair the program a maximum number of times). In some aspects, the fine-tuning system may allow for a single repairing attempt. That is, if the first program210fails validation, the training component220may prompt the machine learning model205to repair the code once. If the revised program also fails, the training component220may determine to use the ground-truth solution, as discussed above and in more detail below.

In the illustrated workflow200, if the training component220determines to attempt to repair the program210, the training component220provides feedback225as input to the machine learning model205. In some aspects, as discussed above, the feedback can include simple feedback (e.g., text indicating failure, such as “That is incorrect. Please try again”) and/or more complex feedback (e.g., text indicating the particular error(s) that occurred). In some aspects, the feedback225generally comprises natural language text.

In some aspects, to use the feedback225as input, the training component220may additionally provide other data as input to the machine learning model205. For example, in some aspects, the training component220may provide the original task description110, the feedback225, and/or the original program210as input. In some aspects, the training component220may concatenate some or all of the task description110, the program210, and the feedback225, and use this concatenated text as the input prompt to the machine learning model205. Using this prompt, the machine learning model205may generate a revised program (which may be represented in the illustrated example as the program210). As discussed above, the revised program may correspond to an updated version of the original program210. Alternatively, the revised program may be an entirely new program with newly drafted code.

In the illustrated workflow200, the verification component215can similarly evaluate the revised program to attempt to validate or verify the revised code, as discussed above. The results of this verification can similarly be provided to the training component220, as discussed above.

As illustrated, the training component220can similarly evaluate the results of the verification for the revised program to determine whether the revised program satisfies the programming task. If so, as discussed above, the revised program may be used as a ground truth for fine-tuning the model. For example, the training component220may add the revised program as a fine-tuning exemplar in the fine-tuning dataset235. In some aspects, to form the fine-tuning exemplar based on the revised program, the training component220may use the (revised) prompt (e.g., the task description110, unit test(s), original program210, and feedback225) as the input portion of the exemplar and use the revised program as the target output. That is, the training component220may use the same string that was used to generate the revised program (e.g., the concatenated task, original (failed) program, and feedback) as the exemplar input, and the revised (e.g., repaired) program as the target output.

In the illustrated example, if the training component220determines that the revised program still fails validation, the training component220may again determine whether to attempt to repair the code. In some aspects, as discussed above, the training component220may use a defined or maximum number of repair iterations (e.g., allowing one attempt to repair). If this threshold has been reached, the training component220may use the solution(s)120as the ground-truth. In some aspects, if the training component220determines that the revised program still fails validation but for one or more different reasons, the number of repair iterations may not increase and/or may be reset.

That is, the training component220may add the solution(s)120as a fine-tuning exemplar in the fine-tuning dataset235. In some aspects, to form the fine-tuning exemplar based on this failure to repair the program, the training component220may use the (revised) prompt (e.g., the task description110, unit test(s), original program210, and feedback225) as the input portion of the exemplar and use the solution(s)120as the target output. That is, the training component220may use the same string that was used to generate the revised program (e.g., the concatenated task, original (failed) program, and feedback) as the exemplar input and the actual ground-truth solution as the target output.

In some aspects, the illustrated workflow200may be performed for each task description110in the set of examples that are available for fine-tuning the machine learning model205. Once each task description110has been processed to generate a corresponding fine-tuning exemplar in the fine-tuning dataset235, the fine-tuning system may use the fine-tuning dataset235to refine or update the parameters of the machine learning model205.

Generally, the particular operations used to update the machine learning model205may vary depending on the particular architecture and implementation. For example, if the machine learning model205is implemented as a neural network, the fine-tuning system may process the input portion of each exemplar in the fine-tuning dataset235using the machine learning model205to generate an output program (or may access the program that was previously generated for the exemplar during the workflow200, if the prior-generated program was saved for fine-tuning). The fine-tuning system may then compare the output portion of the exemplar (e.g., the correctly generated program210, the correctly repaired program, or the solution120) with the generated program in order to generate a loss. This loss can then be used to update the model parameter(s), such as using backpropagation.

Generally, the fine-tuning system may update the parameters of the machine learning model205using individual exemplars from the fine-tuning dataset235(e.g., using stochastic gradient descent) or using batches of exemplars (e.g., using batch gradient descent).

As discussed above, using the fine-tuning dataset235, the fine-tuning system can thereby generate a refined or updated version of the machine learning model205(e.g., the revised machine learning model130ofFIG.1). In some aspects, this updated model can then be deployed for inferencing. In some aspects, as discussed above, the fine-tuning system may optionally perform another round of fine-tuning. That is, the fine-tuning system may repeat the workflow200for each task description110using the updated machine learning model in place of the original machine learning model205.

This fine-tuning workflow can similarly be repeated any number of times, depending on the particular implementation. For example, the fine-tuning system may repeat the workflow200a defined number of times (e.g., a fixed number of fine-tuning iterations), until a desired accuracy or performance of the updated machine learning model is reached, until a defined amount of time and/or computational resources have been consumed during the fine-tuning, and the like.

Example Method for Improved Updating of Program Synthesis Machine Learning Models Using Bootstrapping and Program Repairing

FIG.3is a flow diagram depicting an example method300for improved updating of program synthesis machine learning models using bootstrapping and program repairing, according to various aspects of the present disclosure. In some aspects, the method300is performed by a fine-tuning system, such as the fine-tuning system125ofFIG.1. In some aspects, the method300provides additional detail for the workflow200ofFIG.2.

At block305, the fine-tuning system accesses a set of task definitions (e.g., the task descriptions110ofFIGS.1and2). As discussed above, the task descriptions generally describe or indicate programming tasks, such as using natural language to describe the desired functionality of a program. In some aspects, as discussed above, each task definition may include one or more unit tests (e.g., the unit tests115ofFIGS.1and2) that can be used to verify the functionality of code (e.g., to verify that the code satisfies the programming task). In some aspects, each task definition may further include one or more solutions (e.g., the solutions120ofFIGS.1and2) corresponding to program code that (purportedly) satisfies the programming task. For example, the solutions may be human-authored programs to perform the task. Generally, the fine-tuning system may access the task definitions from any suitable repository, including local storage, remote storage, publicly accessible exemplars (e.g., the MBPP and/or APPS datasets), and the like.

At block310, the fine-tuning system selects one of the task definitions. Generally, the fine-tuning system may use any suitable technique to select the task definition (including randomly or pseudo-randomly), as each task in the set will be selected for evaluation during the method300.

At block315, the fine-tuning system generates a program (e.g., the program210ofFIG.2) using a machine learning model (e.g., the machine learning model205ofFIG.2). For example, as discussed above, the fine-tuning system may use the natural language task description as input to the machine learning model, prompting the model to generate a synthesized program that attempts to satisfy the task. In some aspects, as discussed above, the fine-tuning system may further use one or more of the unit tests (if available) for the task as further input (e.g., concatenating the task description and the unit test, and using the concatenated text as input).

At block320, the fine-tuning system determines whether the programming task was satisfied by the generated program. That is, the fine-tuning system may determine whether the generated program satisfies or performs the desired programming task. In some aspects, as discussed above, the fine-tuning system may determine whether the task was satisfied based on determining whether the generated program generates or results in any errors (e.g., compilation or interpreter errors, runtime errors, and the like). In some aspects, the fine-tuning system may additionally or alternatively determine whether the task was satisfied based on determining whether the generated program accurately or successfully performs any (hidden) unit tests that are associated with the task description but that were not provided as input to the model.

If, at block320, the fine-tuning system determines that the task was satisfied, the method300continues to block330, discussed in more detail below. If the fine-tuning system determines that the task was not satisfied by the generated program, the method300continues to block325.

At block325, the fine-tuning system determines whether to attempt to repair the (failed) program. Generally, the fine-tuning system may use a variety of criteria to determine whether to repair the program, depending on the particular implementation. For example, in some aspects, the fine-tuning system may determine whether a maximum number of repair attempts have been performed (e.g., where the fine-tuning system attempts to repair the program once before moving on to the next exemplar). As another (non-limiting) example, the fine-tuning system may determine whether computational resources are available to repair the program. In some aspects, the fine-tuning system may evaluate the progress or improvement of the program repair. For example, the fine-tuning system may determine whether the most recently generated program is better than the previous version according to one or more metrics or key performance indicators (e.g., whether a revised version has fewer errors than the original version, solves more unit tests accurately, etc.). In some such aspects, if the program is improved, the fine-tuning system may determine to attempt another round of repair. If the program has not improved (or has worsened), the fine-tuning system may determine not to attempt any further repair. In some aspects, the fine-tuning system may revert the changes from the last round of repair.

If, at block325, the fine-tuning system determines to not to repair the program, the method300continues to block330, discussed in more detail below. If the fine-tuning system determines to repair the program (or attempt to repair the program), the method300continues to block335.

At block335, the fine-tuning system generates feedback based on the evaluation of the generated program. As discussed above, in some aspects, the feedback comprises simple feedback, such as noting that the program failed to satisfy the task (e.g., natural language text stating “That is incorrect. Please try again”) without providing additional context or explanation. In some aspects, as discussed above, the feedback may comprise more complex feedback (e.g., full feedback), such as text indicating the particular errors that occurred, text noting that the unit test(s) failed, and the like.

At block340, the fine-tuning system generates a revised program based on the feedback. For example, the fine-tuning system may process the feedback using the machine learning model to generate a revised program. In some aspects, as discussed above, the fine-tuning system may concatenate the feedback with other data to generate a new prompt for the model. For example, in some aspects, the fine-tuning system may concatenate the original prompt (e.g., the task description, which may include one or more unit tests), the original (or most recently generated) program (e.g., generated at block315, and/or at block340during a prior repair attempt), and the feedback. This concatenated string may then be used as input to the machine learning model to generate a revised program. The method300then returns to block320to validate the revised program.

Returning to block330, after the fine-tuning system determines that the program is correct (at block320) or that no additional repair attempts should be made (at block325), the fine-tuning system adds a new exemplar to the fine-tuning dataset (e.g., fine-tuning dataset235ofFIG.2) based on the selected task definition. In some aspects, as discussed above, the input portion of the new exemplar may correspond to the model input used to generate the most recent program (e.g., the task definition itself, or the task definition concatenated with incorrectly generated code and resulting feedback). The output portion of the exemplar may similarly vary depending on whether the program satisfied the task. For example, if the program (or revised program) accurately performed the task, the label or target output portion of the exemplar may include this generated program. If the program (or revised program) failed to satisfy the task, the target output may correspond to a ground-truth solution for the task (e.g., the solution120ofFIGS.1and2). One example method for adding the exemplar is discussed in more detail below with reference toFIG.4.

This fine-tuning exemplar may then be stored in the fine-tuning dataset for subsequent use in updating the model. The method300then continues to block345, where the fine-tuning system determines whether there is at least one additional task, remaining in the set, that has not yet been used to generate a fine-tuning exemplar during the current iteration of updating. If so, the method300returns to block310to select the next task definition. If not, the method300continues to block350. Although the illustrated example depicts an iterative process for conceptual clarity (e.g., selecting and processing each task in sequence), in some aspects, the fine-tuning system may process some or all of the tasks in parallel.

At block350, the fine-tuning system fine-tunes the machine learning model based on the fine-tuning dataset. For example, as discussed above, the fine-tuning system may determine or generate model output based on the input portion of each exemplar (e.g., by processing the string using the model), and compare this generated output with the target output portion of the exemplar (e.g., the accurately generated code, or the ground-truth program). The results of this comparison (e.g., a loss) may then be used to refine the model (e.g., using backpropagation) for each fine-tuning exemplar (e.g., using stochastic or batch gradient descent).

In the illustrated example, the method300terminates after block350. In some aspects, as discussed above, the fine-tuning system may perform the fine-tuning operation (e.g., perform the method300) repeatedly. For example, the fine-tuning system may use the updated version of the model (generated at block350) as the (new) base model and perform the method300again to generate another updated version. This fine-tuning can continue for any number of iterations, such as until one or more termination criteria are met (e.g., a maximum number of iterations, a desired model accuracy, and the like).

In some aspects, after the fine-tuning is complete (e.g., after processing all fine-tuning exemplars in the dataset during the final fine-tuning iteration), the fine-tuning system may deploy the updated machine learning model for inferencing. As discussed above, deploying the model may generally include deploying the model locally (e.g., for inferencing by the fine-tuning system) and/or deploying the model to one or more other systems (e.g., dedicated inferencing systems).

Example Method for Generating Fine-Tuning Data for Program Synthesis Machine Learning Models

FIG.4is a flow diagram depicting an example method400for generating fine-tuning data for program synthesis machine learning models, according to various aspects of the present disclosure. In some aspects, the method400is performed by a fine-tuning system, such as the fine-tuning system125ofFIG.1. In some aspects, the method400provides additional detail for block330ofFIG.3.

At block405, the fine-tuning system determines whether the first generated program, for the task description, was correct. That is, the fine-tuning system determines whether the machine learning model generated a correct program that satisfied the programming task during the first attempt (e.g., where no repairing was used). For example, the fine-tuning system may determine whether the program generated at block315ofFIG.3satisfied the task. If so, the method400continues to block410.

At block410, the fine-tuning system generates a fine-tuning record comprising the prompt used to generate the program, as well as the program generated by the model. For example, as discussed above, each fine-tuning record may comprise an input portion (e.g., the text string to be used as input to the model) and an output portion (e.g., the label to be used as ground-truth for the input portion during fine-tuning). At block410, the input portion may correspond to the original task description (and, in some cases, unit test(s)) used to generate the program. The output portion comprises the generated program itself, allowing the fine-tuning system to use bootstrapping (rather than human-authored solutions) during fine-tuning.

Returning to block405, if the fine-tuning system determines that the first program was not correct, the method400continues to block415, where the fine-tuning system concatenates the original prompt (e.g., the task description and/or unit test(s)), the first (failed) program, and the feedback that was generated based on the failed program. Although the illustrated example depicts concatenating the first program, in some aspects, the fine-tuning system may concatenate the most-recent or final failed program. That is, if multiple repair attempts are used, the fine-tuning system may concatenate the prompt and feedback with the last or most recent attempt (rather than the first attempt) at generating the program.

At block420, the fine-tuning system determines whether the (last) revised program was correct. That is, the fine-tuning system determines whether the machine learning model was able to generate a correct program based on the feedback, as discussed above. If so, the method400continues to block425, where the fine-tuning system generates a fine-tuning record comprising the concatenated text (generated at block415) and the revised program. For example, as discussed above, the concatenated text may correspond to the input portion of the record, and the revised program may correspond to the target output.

Returning to block420, if the fine-tuning system determines that the (final) revised program also failed to satisfy the task, the method400continues to block430. At block430, the fine-tuning system generates a fine-tuning record comprising the concatenated text (generated at block415) and a ground-truth program for the task. For example, as discussed above, the concatenated text may correspond to the input portion of the record, and the ground-truth program may correspond to the target output. In some aspects, as discussed above, the ground-truth program may be a human-authored solution for the programming task.

Although the illustrated example depicts generating a single fine-tuning record for each task description (e.g., at block410,425, or430, depending on whether the program or revised program(s) succeeded in solving the task), in some aspects, the fine-tuning system may generate multiple records for some (or all) of the tasks. For example, in some aspects, if the fine-tuning system determines that the first program or the revised program for a given task was successfully validated, the fine-tuning system may generate a first record including this generated program as the target (e.g., at block410or425), and generate a second record including the original (e.g., human-authored) ground-truth as the target.

Using the method400, the fine-tuning system is able to dynamically generate fine-tuning records based on the performance of the model itself, allowing the fine-tuning system to use bootstrapping and repairing to generate improved and more diverse fine-tuning data, while retaining use of the ground-truth labels where the model fails to generate accurate outputs. This can substantially improve the accuracy of the updated model, as discussed above.

Example Method for Updating Machine Learning Models for Program Synthesis

FIG.5is a flow diagram depicting an example method500for updating machine learning models for program synthesis, according to various aspects of the present disclosure. In some aspects, the method500is performed by a fine-tuning system, such as the fine-tuning system125ofFIG.1. In some aspects, the method500is performed by the verification component215, training component220, and/or using the machine learning model205, each ofFIG.2. In some aspects, the method500is performed by a processing system, such as the processing system ofFIG.6. In some aspects, the method500provides additional detail for the method300ofFIG.3and/or for the method400ofFIG.4.

At block505, a first input indicating a first programming task is accessed. In some aspects, the first input comprises a natural language description of the first programming task. In some aspects, the first input further comprises a first unit test to verify accuracy of programs with respect to the first programming task.

At block510, a first generated program is generated based on processing the first input using a trained machine learning model.

At block515, first feedback is generated in response to determining that the first generated program failed to satisfy the first programming task.

In some aspects, the first feedback comprises an indication that the first generated program is incorrect. In some aspects, the first feedback further comprises one or more error explanations for the first generated program.

In some aspects, determining that the first generated program failed to satisfy the first programming task comprises at least one of: (i) determining that the first generated program caused one or more compilation errors, (ii) determining that the first generated program caused one or more runtime errors, or (iii) determining that the first generated program failed to accurately process one or more unit tests associated with the first programming task.

At block520, a first revised program is generated based on processing the first feedback using the trained machine learning model.

In some aspects, generating the first revised program comprises generating a concatenated input based on concatenating the first input, the first generated program, and the first feedback and processing the concatenated input using the trained machine learning model.

At block525, one or more parameters of the trained machine learning model are updated based on the first revised program in response to determining that the first revised program satisfied the first programming task.

In some aspects, the method500further includes accessing a second input indicating a second programming task, generating a second generated program based on processing the second input using the trained machine learning model, and, in response to determining that the second generated program satisfied the second programming task, updating one or more parameters of the trained machine learning model based further on the second generated program.

In some aspects, the method500further includes accessing a second input indicating a second programming task, and generating a second generated program based on processing the second input using the trained machine learning model. In some aspects, the method500further includes, in response to determining that the second generated program failed to satisfy the second programming task, generating second feedback based on the determination that the second generated program failed to satisfy the second programming task and generating a second revised program based on processing the second feedback using the trained machine learning model. In some aspects, the method500further includes, in response to determining that the second revised program failed to satisfy the second programming task, updating one or more parameters of the trained machine learning model based further on a ground-truth program for the second programming task.

Example Processing System for Training Machine Learning Models

In some aspects, the workflows, techniques, and methods described with reference toFIGS.1-5may be implemented on one or more devices or systems.FIG.6depicts an example processing system600configured to perform various aspects of the present disclosure, including, for example, the techniques and methods described with respect toFIGS.1-5. In some aspects, the processing system600may correspond to a fine-tuning system, such as the fine-tuning system125ofFIG.1. Generally, the processing system600may correspond to any computing system that updates or fine-tunes machine learning models. Although depicted as a single system for conceptual clarity, in some aspects, as discussed above, the operations described below with respect to the processing system600may be distributed across any number of devices or systems.

The processing system600includes a central processing unit (CPU)602, which in some examples may be a multi-core CPU. Instructions executed at the CPU602may be loaded, for example, from a program memory associated with the CPU602or may be loaded from a memory partition (e.g., a partition of memory624).

The processing system600also includes additional processing components tailored to specific functions, such as a graphics processing unit (GPU)604, a digital signal processor (DSP)606, a neural processing unit (NPU)608, a multimedia component610(e.g., a multimedia processing unit), and a wireless connectivity component612.

An NPU, such as NPU608, is generally a specialized circuit configured for implementing the control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), and the like. An NPU may sometimes alternatively be referred to as a neural signal processor (NSP), tensor processing unit (TPU), neural network processor (NNP), intelligence processing unit (IPU), vision processing unit (VPU), or graph processing unit.

NPUs, such as the NPU608, are configured to accelerate the performance of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, a plurality of NPUs may be instantiated on a single chip, such as a system on a chip (SoC), while in other examples the NPUs may be part of a dedicated neural-network accelerator.

NPUs may be optimized for training or inference, or in some cases configured to balance performance between both. For NPUs that are capable of performing both training and inference, the two tasks may still generally be performed independently.

NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly compute-intensive operation that involves inputting an existing dataset (often labeled or tagged), iterating over the dataset, and then adjusting model parameters, such as weights and biases, in order to improve model performance. Generally, optimizing based on a wrong prediction involves propagating back through the layers of the model and determining gradients to reduce the prediction error.

NPUs designed to accelerate inference are generally configured to operate on complete models. Such NPUs may thus be configured to input a new piece of data and rapidly process this piece of data through an already trained model to generate a model output (e.g., an inference).

In some implementations, the NPU608is a part of one or more of the CPU602, the GPU604, and/or the DSP606.

In some examples, the wireless connectivity component612may include subcomponents, for example, for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G Long-Term Evolution (LTE)), fifth generation connectivity (e.g., 5G or New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and/or other wireless data transmission standards. The wireless connectivity component612is further coupled to one or more antennas614.

The processing system600may also include one or more sensor processing units616associated with any manner of sensor, one or more image signal processors (ISPs)618associated with any manner of image sensor, and/or a navigation processor620, which may include satellite-based positioning system components (e.g., GPS or GLONASS), as well as inertial positioning system components.

The processing system600may also include one or more input and/or output devices622, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, and the like.

In some examples, one or more of the processors of the processing system600may be based on an ARM or RISC-V instruction set.

The processing system600also includes the memory624, which is representative of one or more static and/or dynamic memories, such as a dynamic random access memory, a flash-based static memory, and the like. In this example, the memory624includes computer-executable components, which may be executed by one or more of the aforementioned processors of the processing system600.

In particular, in this example, the memory624includes a machine learning component624A, a verification component624B, and a training component624C. The memory624further includes model parameters624D for one or more models or portions thereof (e.g., parameters of a machine learning model, such as the machine learning model105ofFIG.1, the revised machine learning model130ofFIG.1, and/or the machine learning model205ofFIG.2). The memory624also includes a fine-tuning dataset624E (which may correspond to the fine-tuning dataset235ofFIG.2). Although not included in the illustrated example, in some aspects the memory624may also include other data, such as training data (e.g., the task descriptions110, unit tests115, and/or solutions120ofFIGS.1and2). Though depicted as discrete components for conceptual clarity inFIG.6, the illustrated components (and others not depicted) may be collectively or individually implemented in various aspects.

The processing system600further comprises a machine learning circuit626, a verification circuit627, and a training circuit628. The depicted circuits, and others not depicted, may be configured to perform various aspects of the techniques described herein.

For example, the machine learning component624A and/or the machine learning circuit626may be used to generate synthesized programs using machine learning models, as discussed above. For example, the machine learning component624A and/or the machine learning circuit626may process input text, such as task descriptions (e.g., the task descriptions110and/or the unit test(s)115ofFIGS.1and2) using machine learning models (e.g., the machine learning model105ofFIG.1and/or the machine learning model205ofFIG.2) generate synthesized programs (e.g., the program210ofFIG.2). In some aspects, the machine learning component624A and/or the machine learning circuit626may similarly generate updated prompts (e.g., by concatenating task descriptions (such as the task description110ofFIG.1), incorrect programs (e.g., the program210ofFIG.2), and/or feedback (e.g., the feedback225ofFIG.2). The machine learning component624A and/or the machine learning circuit626may then use these updated prompts to generate revised models, as discussed above.

The verification component624B and/or the verification circuit627(which may correspond to the verification component215ofFIG.2) may be used to verify whether generated programs (including revised or repaired programs) satisfy programming tasks, as discussed above. For example, the verification component624B and/or the verification circuit627may determine whether generated programs result in any compilation errors when compiled or interpreted, and/or any runtime errors when executed. As another example, the verification component624B and/or the verification circuit627may determine whether the generated program operates accurately or correctly, such as whether the program generates a correct output based on a given input (e.g., using unit tests).

The training component624C and/or the training circuit628(which may correspond to the training component220ofFIG.2) may be used to generate fine-tuning data (e.g., the fine-tuning dataset624E) and use this fine-tuning data to update parameters of machine learning models, as discussed above. For example, the training component624C and/or the training circuit628may generate exemplars including generated program code (when the model generates a program that satisfies the input programming task) and/or ground-truth code (when the model fails to generate a program that satisfies the task), and may use these exemplars to refine the model in order to improve its accuracy.

Though depicted as separate components and circuits for clarity inFIG.6, the machine learning circuit626, the verification circuit627, and the training circuit628may collectively or individually be implemented in other processing devices of the processing system600, such as within the CPU602, the GPU604, the DSP606, the NPU608, and the like.

Generally, the processing system600and/or components thereof may be configured to perform the methods described herein.

Notably, in other aspects, elements of the processing system600may be omitted, such as where the processing system600is a server computer or the like. For example, the multimedia component610, the wireless connectivity component612, the sensor processing units616, the ISPs618, and/or the navigation processor620may be omitted in other aspects. Further, aspects of the processing system600may be distributed between multiple devices.

Example Clauses

Clause 1: A method, comprising: accessing a second input indicating a second programming task; generating a second generated program based on processing the second input using the trained machine learning model; in response to determining that the second generated program failed to satisfy the second programming task: generating second feedback based on the determination that the second generated program failed to satisfy the second programming task; and generating a second revised program based on processing the second feedback using the trained machine learning model; and in response to determining that the second revised program failed to satisfy the second programming task, updating one or more parameters of the trained machine learning model based further on a ground-truth program for the second programming task.

Clause 2: A method according to Clause 1, further comprising: accessing a second input indicating a second programming task; generating a second generated program based on processing the second input using the trained machine learning model; and in response to determining that the second generated program satisfied the second programming task, updating one or more parameters of the trained machine learning model based further on the second generated program.

Clause 3: A method according to any of Clauses 1-2, further comprising: accessing a second input indicating a second programming task; generating a second generated program based on processing the second input using the trained machine learning model; in response to determining that the second generated program failed to satisfy the second programming task: generating second feedback based on the determination that the second generated program failed to satisfy the second programming task; and generating a second revised program based on processing the second feedback using the trained machine learning model; and in response to determining that the second revised program failed to satisfy the second programming task, updating one or more parameters of the trained machine learning model based further on a ground-truth program for the second programming task.

Clause 4: A method according to any of Clauses 1-3, wherein the first input comprises a natural language description of the first programming task.

Clause 5: A method according to Clause 4, wherein the first input further comprises a unit test to verify accuracy of programs with respect to the first programming task.

Clause 6: A method according to any of Clauses 1-5, wherein determining that the first generated program failed to satisfy the first programming task comprises at least one of: (i) determining that the first generated program caused one or more compilation errors, (ii) determining that the first generated program caused one or more runtime errors, or (iii) determining that the first generated program failed to accurately process one or more unit tests associated with the first programming task.

Clause 7: A method according to any of Clauses 1-6, wherein the first feedback comprises an indication that the first generated program is incorrect.

Clause 8: A method according to Clause 7, wherein the first feedback further comprises one or more error explanations for the first generated program.

Clause 9: A method according to any of Clauses 1-8, wherein generating the first revised program comprises: generating a concatenated input based on concatenating the first input, the first generated program, and the first feedback; and processing the concatenated input using the trained machine learning model.

Clause 10: A processing system comprising: a memory comprising computer-executable instructions; and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method in accordance with any of Clauses 1-9.

Clause 11: A processing system comprising means for performing a method in accordance with any of Clauses 1-9.

Clause 13: A non-transitory computer-readable medium encoding logic that, when executed by a processing system, causes the processing system to perform a method in accordance with any of Clauses 1-9.

Clause 14: An apparatus comprising logic circuitry configured to perform a method in accordance with any of Clauses 1-9.

Clause 15: A computer program product embodied on a computer-readable storage medium comprising code for performing a method in accordance with any of Clauses 1-9.

Additional Considerations