LARGE LANGUAGE MODEL CODE TRANSLATION ERROR DETECTION

Large language model code translation error detection include receiving a code portion of a first programming language, and converting the code portion to a second programming language. A first accuracy of the converting of the code portion to the second programming language is calculated. A difference between the first accuracy and an historical accuracy of a conversion from the first programming language to the second programming language is determined. A potential error in the code portion of the first programming language is indicated based on the difference between the first accuracy and the historical accuracy being greater than a predetermined value is indicated.

BACKGROUND

The present disclosure relates to methods, apparatus, and products for large language model code translation error detection. Migrating the functionality of legacy source code to a more modern programming language can increase the maintainability and readability of the source code as well as improve system performance. However, such a migration is an arduous task that can include writing, testing, validating, and debugging massive amounts of code. If the modernized source code is deployed with errors into a complex software system, expected operations may fail to complete, exhaustive retry logic may hog resources, transactions can begin to fail, and entire components of the stack may fail, leading to a potential outage. This can be disastrous, especially in core production systems.

SUMMARY

According to embodiments of the present disclosure, various methods, apparatus and products for large language model code translation error detection are described herein. In some aspects, a method for large language model code translation error detection includes receiving a code portion of a first programming language; converting the code portion to a second programming language; calculating a first accuracy of the converting of the code portion to the second programming language; determining a difference between the first accuracy and an historical accuracy of a conversion from the first programming language to the second programming language; and indicating a potential error in the code portion of the first programming language based on the difference between the first accuracy and the historical accuracy being greater than a predetermined value.

In an embodiment, calculating the first accuracy of the converting of the code portion to the second programming language further comprises: calculating a first code structure representation of the code portion of the first programming language; calculating a second code structure representation of the converted code portion of the second programming language; and calculating the first accuracy based on a comparison of the first code structure representation to the second code structure representation.

In an embodiment, calculating the first code structure representation of the code portion of the first programming language further comprises calculating the first code structure representation of the code portion of the first programming language based on one or more software metrics. In an embodiment, calculating the second code structure representation of the converted code portion of the second programming language further comprises calculating the second code structure representation of the converted code portion of the second programming language based on the one or more software metrics. In an embodiment, the one or more software metrics include code complexity metrics.

In an embodiment, calculating the first code structure representation of the code portion of the first programming language based on one or more software metrics further comprises calculating the first code structure representation of the code portion of the first programming language based on a weighted combination of a plurality of the one or more software metrics.

In an embodiment, the method includes determining the historical accuracy based on an accuracy of at least one previous conversion of another code portion from the first programming language to the second programming language.

In an embodiment, converting the code portion to a second programming language comprises converting the code portion to a second programming language using a generative artificial intelligence model. In an embodiment, the generative artificial intelligence model comprises a large language model.

In an embodiment, indicating the potential error in the code portion based on the difference between the first accuracy and the historical accuracy being greater than a predetermined value further comprises providing an indication of the potential error in the code portion to the generative artificial intelligence model.

In some aspects, an apparatus may include a processing device; and memory operatively coupled to the processing device, wherein the memory stores computer program instructions that, when executed, cause the processing device to perform this method. In some aspects, a computer program product comprising a computer readable storage medium may store computer program instructions that, when executed, perform this method.

DETAILED DESCRIPTION

The advent of large language models and other improvements to artificial intelligence (AI) technology have enabled AI to generate source code. Large language models (LLMs), for example, are trained on massive datasets of source code to provide generative AI that outputs source code based on some input or prompt. As will discussed in further detail below, the migration of an application from its original source code to new source code can be assisted by such AI. That is, the AI can be used to generate new source code based on an input of original source code. For example, an LLM may be given a prompt such as “Generate Java code that achieves the same objectives as the following COBOL code,” where the legacy COBOL source code is provided as an input. In response, the LLM may output, at least ideally, AI-generated Java source code that performs the same functions and produces the same output as the legacy. Of course, any new source code, whether human or AI generated, is susceptible to the introduction of bugs, execution errors, or other faults.

Embodiments in accordance with the present disclosure advantageously utilize historical accuracy data to detect potential bugs, execution errors, or other faults in the original source code in a first programming language (e.g., COBOL) that are subsequently translated into converted source code in a second programming language (e.g., Java). LLMs have significantly advanced the process of translating code from one programming language to another. However, challenges arise when the translated output code deviates from an acceptable standard. In such situations, it is difficult to determine whether errors in the translated source code arises from a translation error or an underlying error in the original source code. Various embodiments recognize that this discrepancy between accurate translation for a good input and suboptimal results for a bad input can function as an indicator that the original source code should be corrected before translating the source code again.

In a particular example, an LLM used to convert COBOL to Java produces a particular accuracy of translation that is acceptable within a standard deviation under an assumption that the COBOL is written correctly without bugs. However, should the COBOL contain bugs or errors, an accurate translation of the “bugged” COBOL will be “bugged” Java. As a result, the accuracy of the code translation may decrease drastically below the standard deviation, potentially indicating an error in the input COBOL. A drop in translation accuracy from a historical accuracy of translating COBOL to Java may give rise to an assumption of either that the LLM is translating code that it hasn't seen before or there is an error in the source code. In an example, a drop in accuracy of 20% from the expected accuracy gives rise to an assumption with 80% certainty that there is a bug in the COBOL code.

By identifying potential errors in the original source code, the original source code can be reviewed to identify the potential errors. Once the errors are corrected, the original source code can be translated again using the LLM to produce a more accurate converted code. The process can be iterated until an acceptable accuracy of translation is achieved.

Remote server 104 is any computer system that serves at least some data and/or functionality to computer 101. Remote server 104 may be controlled and used by the same entity that operates computer 101. Remote server 104 represents the machine(s) that collect and store helpful and useful data for use by other computers, such as computer 101.

For further explanation, FIG. 2 sets forth a flowchart of an example method for large language model code translation error detection in accordance with some embodiments of the present disclosure. The method of FIG. 2 may be performed, for example, by a code analysis module 201 such as the code analysis module 107 of FIG. 1. In some embodiments, the code analysis module 201 may be implemented as a process or service separate from an application or software implementing code analysis. For example, the code analysis module 201 may be implemented by an operating system or other software that monitors the behavior and execution of an application implementing code analysis. As another example, in some embodiments, the code analysis module 201 may be implemented as a process or service that applies modifications to applications implementing code analysis.

The method of FIG. 2 includes receiving 202 a code portion of a first programming language, and converting 204 the code portion to a second programming language. Although FIG. 2 describes execution of an application, it is understood that the approaches set forth herein may be applied to any software or module capable of implementing code analysis, including applications, operating systems, libraries, and the like. In an example, the code portion may include legacy source code written in an older programming language (e.g., COBOL) and the converted code portion may include code written in a modern programming language (e.g., Java). In a particular example, both the code portion and the converted code portion are intended to achieve the same objectives, provide the same interfaces, and produce the same output. In particular examples, the code portion may include an application, a subroutine, a function, or driver. In an embodiment, the code portion is translated from the first programming language to the second programming language using generative AI such as an LLM. In some examples, some or all of the converted code is generated utilizing an AI language model. For example, some or all of the original source code can be provided as an input to the AI language model with a request (e.g., a prompt) to generate source code in a different programming language that achieves the same objectives as the original source code.

In an embodiment, the code analysis module 201 calculates 206 a first accuracy of the converting of the code portion to the second programming language. In a particular embodiment, the first accuracy indicates a degree of accuracy of the translation from the original source code to the converted source code. In one or more embodiments, the first accuracy of the converting of the code portion to the second programming language is based on a comparison of the code structures of the original code portion and the converted code portion for similarity within a predetermined acceptable threshold. In a particular embodiment, a first code structure representation of the code portion of the first programming language and a second code structure representation of the converted code portion of the second programming language are calculated. The first code structure representation and the second code structure representation are compared to calculate the first accuracy.

In one or more embodiments, calculating the first code structure representation of the code portion of the first programming language and the second code structure representation of the converted code portion of the second programming language is based on one or more software metrics. In a particular embodiment, an abstract syntax tree (AST) is used to calculate the first accuracy of the converting of the code portion to the converted code portion. An AST is a data structure used to represent the structure of the code portion in a tree representation of the abstract syntactic structure of text of the programming language. Each node of the tree denotes a construct occurring in the text. The syntax is “abstract” in the sense that it does not represent every detail appearing in the real syntax, but rather just the structural or content-related details. Compared to the source code, the AST does not typically include inessential punctuation and delimiters. By comparing ASTs of the original code portion and the converted code portion, a measure of the accuracy of the converting is calculated.

In another embodiment, one or more code complexity metrics are used to calculate a respective code complexity for each of the original code portion and the converted code portion. Correctly translated source code is expected to exhibit a similar degree of complexity as the original source code. In one or more embodiments, comparing the code complexities of the original source code portion to the converted code portion is a potential indicator of a logical error, fault, bug or other programming error in the original source code portion.

In an example, a cyclomatic complexity metric is used to calculate the complexity of the original code portion and the converted code portion. The cyclomatic complexity metric represents the number of linearly independent paths through the source code. A path is linearly independent if there is a subset of one or more paths in which the symmetric difference of their edge sets is empty. For example, if the source code contains no control flow statements, the complexity is 1 since there is only a single path through the code. If the source code includes a single conditional statement, there are two paths through the code indicating a complexity of 2.

In another example, a Halstead complexity metric is used to calculate the complexity of the original code portion and the converted code portion. The Halstead complexity metric is computed statistically without program execution and takes into account factors such as the number of distinct operators, the number of distinct operands, the total number of operators, and the total number of operands. In another example, a Maintainability Index metric is used to calculate the complexity of the original code portion and the converted code portion. The Maintainability Index metric is calculated to represent the relative ease of maintaining the code portion and is calculated based on the number of statements in the code, the cyclomatic complexity, and the Halstead volume (computed as a function of the code length, number of distinct operators, and number of distinct operands).

In another embodiment, calculating the first code structure representation of the code portion of the first programming language and the second code structure representation of the converted code portion of the second programming language is based on a combination of the one or more software metrics. In a particular embodiment, calculating the first code structure representation of the code portion of the first programming language and the second code structure representation of the converted code portion of the second programming language is based on a weighted combination of a plurality of the one or more software metrics. For example, one or more of the software metrics may be weighted differently than other software metrics in calculating the respective code structure.

In an embodiment, the code analysis module 201 determines 208 a difference between the first accuracy and an historical accuracy of a conversion from the first programming language to the second programming language. In an embodiment, the historical accuracy is based on an accuracy of at least one previous conversion of another code portion from the first programming language to the second programming language. In a particular embodiment, accuracy determinations of previous conversions of code from the first programming language to the second programming language are used to determine an expected accuracy within a particular deviation of translation of the conversion of code programmed using the first programming language to converted code of the second programming language. In a particular embodiment, the previously determined accuracy information is used to construct an accuracy model of code translation from the first programming language to the second programming language.

In an embodiment, the code analysis module 201 indicates 210 a potential error in the code portion of the first programming language based on the difference between the first accuracy and the historical accuracy being greater than a predetermined value. In a particular example, the code analysis module 201 indicates 210 a potential error in the code portion of the first programming language if the difference between the first accuracy and the historical accuracy is greater than 20%. In an example, the indication of the potential error in the code portion is used to review the code portion for errors, correct the errors, and perform conversion of the code portion in the first programming language to the second programming language using the corrected code. In particular embodiments, the process is iteratively repeated until an acceptable accuracy is obtained. In another embodiment, the indication further includes a probability of an error in the code portion.

In another embodiment, the code analysis module 201 determines that the difference between the first accuracy and the historical accuracy is less than the predetermined value, determines that the converted code portion is unexecutable, and indicates that the converted code portion is unexecutable. In particular embodiments, determining that the converted code portion is unexecutable includes determining that the converted code portion is uncompilable or includes one or more errors.

For further explanation, FIG. 3 sets forth a flowchart of another example method for large language model code translation error detection in accordance with some embodiments of the present disclosure. The method of FIG. 3 extends the method of FIG. 2 in that calculating 206 the first accuracy of the converting of the code portion to the second programming language includes calculating 302 a first code structure representation of the code portion of the first programming language, and calculating 304 a second code structure representation of the converted code portion of the second programming language. The method of FIG. 3 further includes calculating 306 the first accuracy based on a comparison of the first code structure representation to the second code structure representation.

In an embodiment, calculating the first code structure representation of the code portion of the first programming language is based on one or more software metrics. In an embodiment, calculating the second code structure representation of the converted code portion of the second programming language is also based on the one or more software metrics. In one or more embodiments, the one or more software metrics include code complexity metrics. In an embodiment, calculating the first code structure representation of the code portion of the first programming language is based on a weighted combination of a plurality of the one or more software metrics.

For further explanation, FIG. 4 sets forth a flowchart of another example method for large language model code translation error detection in accordance with some embodiments of the present disclosure. The method of FIG. 4 extends the method of FIG. 2 in that determining 208 the difference between the first accuracy and an historical accuracy of a conversion from the first programming language to the second programming language includes determining 402 the historical accuracy based on an accuracy of at least one previous conversion of another code portion from the first programming language to the second programming language.

For further explanation, FIG. 5 sets forth a flowchart of another example method for large language model code translation error detection in accordance with some embodiments of the present disclosure. The method of FIG. 5 extends the method of FIG. 2 in that converting 204 the code portion to the second programming language includes converting 502 the code portion to a second programming language using a generative artificial intelligence model. In a particular embodiment, the generative artificial intelligence model includes a large language model.

The method of FIG. 5 further extends the method of FIG. 2 in that indicating 210 the potential error in the code portion of the first programming language based on the difference between the first accuracy and the historical accuracy being greater than a predetermined value includes providing 504 an indication of the potential error in the code portion to the generative artificial intelligence model. In a particular embodiment, the generative artificial intelligence model is further trained based on the indication of the potential error in the code portion of the first programming language. In a particular example, the generative artificial intelligence model disregards the code portion from being used in training based on the indicated potential of the error in the code portion.

The above-described examples are provided in the context of translation error detection when migrating an application from original source code to new source code. It will be appreciated that the term ‘original source code’ as used herein refers to the instance of the application against which the new source code is being validated for accuracy, and should not be construed as limiting the term to mean the earliest implementation of that application. While embodiments are useful in migrating or porting an application from one programming language to a different programming language, and from legacy code to a more modernized programming language, it will be further appreciated that in some examples the original source code and the new source code may be written in the same programming language.

In view of the foregoing, large language model code translation error detection in accordance with the present disclosure provides a number of advantages. Identification of errors within original source code provides for greater success in producing accurate conversions of the original source code to new source code of a different programming language. Such errors can be corrected to enable the conversion of the source original source code to new source code to increase accuracy of the new source code and prevent or reduce execution errors int the converted code.