TRANSLATION QUALITY ASSURANCE BASED ON LARGE LANGUAGE MODELS

A method, computer program product, and computer system are provided for assuring quality of machine translations based on large language models. Data corresponding to an input in a first language is received for machine translation. The received data is translated from the first language to a second language through a translation engine. A confidence value associated with the translated data is determined. The translation is revised with executable source code based on the confidence value being greater than a threshold value. Otherwise, the translation is revised based on sending a prompt to a large language model based on the confidence value being less than the threshold value.

FIELD

This disclosure relates generally to the field of machine learning, and more particularly to large language models.

BACKGROUND

Machine translation (MT) uses either rule-based or probabilistic machine learning approaches to translate text or speech from one language to another, including the contextual, idiomatic and pragmatic nuances of both languages. Traditionally, machine translation has used statistical methods but has recently adopted neural network-based approaches. Machine translation has been widely used to lower costs associated with translation of textual materials.

SUMMARY

Embodiments relate to a method, system, and computer program product for assuring quality of machine translations based on large language models. According to one aspect, a method for assuring quality of machine translations based on large language models is provided. The method may include receiving data corresponding to an input in a first language for machine translation. The received data is translated from the first language to a second language through a translation engine. A confidence value associated with the translated data is determined. The translation is revised with executable source code based on the confidence value being greater than a threshold value. Otherwise, the translation is revised based on sending a prompt to a large language model based on the confidence value being less than the threshold value.

According to another aspect, a computer system for assuring quality of machine translations based on large language models is provided. The computer system may include one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, whereby the computer system is capable of performing a method. The method may include receiving data corresponding to an input in a first language for machine translation. The received data is translated from the first language to a second language through a translation engine. A confidence value associated with the translated data is determined. The translation is revised with executable source code based on the confidence value being greater than a threshold value. Otherwise, the translation is revised based on sending a prompt to a large language model based on the confidence value being less than the threshold value.

According to yet another aspect, a computer program product for assuring quality of machine translations based on large language models is provided. The computer program product may include one or more computer-readable storage devices and program instructions stored on at least one of the one or more tangible storage devices, the program instructions executable by a processor. The program instructions are executable by a processor for performing a method that may accordingly include receiving data corresponding to an input in a first language for machine translation. The received data is translated from the first language to a second language through a translation engine. A confidence value associated with the translated data is determined. The translation is revised with executable source code based on the confidence value being greater than a threshold value. Otherwise, the translation is revised based on sending a prompt to a large language model based on the confidence value being less than the threshold value.

DETAILED DESCRIPTION

Detailed embodiments of the claimed structures and methods are disclosed herein; however, it can be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods that may be embodied in various forms. Those structures and methods may, however, be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope to those skilled in the art. In the description, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

Embodiments relate generally to the field of machine learning, and more particularly to large language models. The following described exemplary embodiments provide a system, method, and computer program product to, among other things, ensure the quality of machine translations confidence values using large language models based on confidence values associated with the translation. Therefore, some embodiments have the capacity to improve the field of computing by allowing for machine translation of speech and text while minimizing the need for human post-editing of such translations.

As previously described, machine translation (MT) uses either rule-based or probabilistic machine learning approaches to translate text or speech from one language to another, including the contextual, idiomatic and pragmatic nuances of both languages. Traditionally, machine translation has used statistical methods but has recently adopted neural network-based approaches. Machine translation has been widely used to lower costs associated with translation of textual materials.

However, although machine translation quality is much better now with newer technologies, there are still some scenarios and cases in which machine translation results are not perfect. These may include, among other things, term translation or inline tag related format issues. For machine translation quality issues, human post-editing is one way to achieve a better translation result. However, this is expensive and requires a longer turnaround time. In addition, similar issues will re-occur when new content is translated with same machine translation engine, which may further introduce a poor user experience and increased translation costs.

It may be advantageous, therefore, to use a Large Language Model (LLM) based self-learning and intelligent solution to help improve and guarantee the translation quality, especially for machine translation results. Such solution may allow for auto-collection of low quality translation cases from user tickets or human post-editing history and create groups for different types of translation issues. Self-learning prompts may be composed with pre-defined templates and collected samples, so that verification testing may be used to check and tune the learning effects. Learned knowledge may be converted into executable source code for groups with good learning effects in order to check and improve the source code maturity with regression test. The system may then intelligently select the proper choice for new cases to improve translation quality within reasonable response time.

Thus, the method, computer system, and computer program product disclosed herein may be used to achieve better translation results with less human intervention and lower translation cost. The disclosed method, computer system, and computer program product may handle many translation cases with auto-generated source code, reduce the calls to large language models, mitigate challenges of large language model response latency, and process heavy workloads with acceptable turnaround time. The disclosed method, computer system, and computer program product may support self-serve translation review/correction that may help improve user satisfaction rate and user experience.

Referring now to FIG. 2, a functional block diagram of a networked computer environment illustrating a digital translation assurance system 200 (hereinafter “system”) for assuring quality of machine translations based on large language models. It should be appreciated that FIG. 2 provides only an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements.

The system 200 may include a computer 202 and a server computer 214. The computer 202 may communicate with the server computer 214 via a communication network 210 (hereinafter “network”). The computer 202 may include a processor 204 and a software program 208 that is stored on a data storage device 206 and is enabled to interface with a user and communicate with the server computer 214. The computer 202 may be, for example, a mobile device, a telephone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing devices capable of running a program, accessing a network, and accessing a database.

The server computer 214, which may be used for assuring quality of machine translations based on large language models is enabled to run a Digital Translation Assistant Program 216 (hereinafter “program”) that may interact with a database 212. The Digital Translation Assistant Program is explained in more detail below with respect to FIG. 4. In one embodiment, the computer 202 may operate as an input device including a user interface while the program 216 may run primarily on server computer 214. In an alternative embodiment, the program 216 may run primarily on one or more computers 202 while the server computer 214 may be used for processing and storage of data used by the program 216. It should be noted that the program 216 may be a standalone program or may be integrated into a larger digital translation assistant program.

It should be noted, however, that processing for the program 216 may, in some instances be shared amongst the computers 202 and the server computers 214 in any ratio. In another embodiment, the program 216 may operate on more than one computer, server computer, or some combination of computers and server computers, for example, a plurality of computers 202 communicating across the network 210 with a single server computer 214. In another embodiment, for example, the program 216 may operate on a plurality of server computers 214 communicating across the network 210 with a plurality of client computers. Alternatively, the program may operate on a network server communicating across the network with a server and a plurality of client computers.

The network 210 may include wired connections, wireless connections, fiber optic connections, or some combination thereof. In general, the network 210 can be any combination of connections and protocols that will support communications between the computer 202 and the server computer 214. The network 210 may include various types of networks, such as, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, a telecommunication network such as the Public Switched Telephone Network (PSTN), a wireless network, a public switched network, a satellite network, a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a private network, an ad hoc network, an intranet, a fiber optic-based network, or the like, and/or a combination of these or other types of networks.

Referring now to FIG. 3, a block diagram of a translation assurance system 300 is depicted according to one or more embodiments. The translation assurance system 300 may include, among other things, a machine translation module 302, a collection module 304, a learning and verification module 306, and a source code generation and testing module 308.

The machine translation module 302 may be a translation engine that may translate input translation data 310 from a first language to another. The input translation data 310 may be text or audio data. The machine translation data may receive a translation request from a user of the software program 208 (FIG. 2) on the computer 202 (FIG. 2) or the Digital Translation Assistant program 216 (FIG. 2) on the server computer 214 (FIG. 2).

The collection module 304 may be, among other things, a large language model (LLM) or other machine learning module. The collection module 304 may automatically collect data corresponding to low quality translation cases. For example, the collection module 304 may gather data from user tickets or from post-editing history from human translations. The collection module 304 may, for example, extract an issue description from a user ticket, such as an incorrect word being used in place of a homograph. The collection module 304 may summarize a translation issue by identifying a given translation issue from a user ticket or post-editing history and extract examples including a sentence in a first translation, a reported poor translation, and a user-suggested correct translation based on script template data 312. The collection module 304 may group user tickets and post-edit history data into several categories based on the summarized translation issue.

The learning and verification module 306 may learn from the translation issue data gathered by the collection module 306 in order to determine correct translations for cases needed correction or for which corrected translations have not been provided. The learning and verification module 306 may use the information learned from previous cases to generate a correct translation. Because self-learning may require several iterations, the learning and verification module may determine how many samples may be needed for learning based on verification of the accuracy of the translations. The learning and verification module 306 may verify the accuracy of the translations based on comparing proposed translations to known correct translations in order to determine whether the test case has passed. Based on the testing result, a leaning confidence score may be recorded for each group to indicate if the machine translation module 302 may be able to successfully handle cases in each group. If the verification testing score is lower than a pre-defined threshold value, the learning and verification module may identify a need for more samples to be provided for further learning.

The source code generation and testing module 308 may generate and update source code for use in translation of the input translation data 310. The source code generation and testing module 308 may check corrected translations generated by the learning and verification module 306 and may summarize a generalized set of rules applicable to the corrected translations for generation as source code. The source code generation and testing module 308 may perform regression testing on the generated source code against one or more test cases. Such testing may include, among other things, whether each and every word is translated, appropriateness of punctuation, consistency of spaces, and preservation of special formatting. If the regression testing exceeds a threshold value, the source code is deemed acceptable for use in future machine translations. If the regression testing does not exceed the threshold value, then the source code generation and testing module 308 may be asked to re-generate source code with additional information corresponding to failed translation cases. A scored may be recorded for each group's source code.

Thus, by providing several translation examples for digital translation, the translation assurance system 300 may be able to get better machine translation results for new cases. Machine translation users may, therefore, no longer suffer from similar translation issues, and the user experience and satisfaction can be highly improved. For professional translators, the translation assurance system 300 may learn from ongoing post-editing results, auto-apply the learned knowledge, and revise the translations of following segments in order to improve the translation efficiency and consistency during the post-editing process.

Referring now to FIG. 4, an operational flowchart illustrating the steps of a method 400 carried out by a program that assures quality of machine translations based on large language models is depicted. The method 400 may be described with the aid of the exemplary embodiments of FIGS. 1-3.

At 402, the method 400 may include receiving data corresponding to an input in a first language for machine translation. The received data corresponds to textual or audio data. In operation, the machine translation module 302 (FIG. 3) may receive input translation data 310 (FIG. 3) from the software program 208 (FIG. 2) on the computer 202 (FIG. 2) or from the Digital Translation Assistant Program 216 (FIG. 2) on the server computer 214 (FIG. 2).

At 404, the method 400 may include translating the received data from the first language to a second language through a translation engine to generate a translation of the received data. The translation engine is updated in response to one or more user tickets corresponding to issues associated with translation of the received data. In operation, the machine translation module 302 (FIG. 3) may translate the input translation data 310 (FIG. 3) to a new language. The collection module 304 (FIG. 3) may gather cases of poor translation from the machine translation module 302 and may pass information on such cases, along with user tickets and post-editing history data, to the learning and verification module 306 (FIG. 3) in order to train the machine translation module 302.

At 406, the method 400 may include determining a confidence value associated with the translated data. The translation engine is trained based on comparing one or more translation cases identified for review with one or more known correct translations based on confidence values associated with the threshold value. Comparing the one or more translation cases identified for review with the one or more known correct translations corresponds to determining whether confidence values for the one or more translation cases identified for review and the one or more known correct translations are greater than the threshold value. In operation, the collection module 304 (FIG. 3) may gather cases of poor translation and group them into one or more groups. The learning and verification module 306 (FIG. 3) may compare the groups to known good translation cases in order to improve the translation quality of the machine translation module 302 (FIG. 3).

At 408, the method 400 may include revising the translation with executable source code based on the confidence value being greater than a threshold value. The source code is updated based on the trained translation engine. In operation, the source code generation and testing module 308 (FIG. 3) may revise the source code based on a confidence value associated with the quality of source code translation being less than a pre-determined threshold value. The source code generation and testing module 308 may pass this source code to the machine translation module 302 (FIG. 2) for improving quality of translation of the translation input data 310 (FIG. 3).

At 410, the method 400 may include revising the translation based on sending a prompt to a large language model based on the confidence value being less than the threshold value. A second confidence value associated with the revising of the translation based on sending the prompt to the large language model is determined. A need for human post-editing is identified based on the second confidence value being less than a second threshold value. In operation, the source code generation and testing module 308 (FIG. 3) may determine that the quality of even the source code translation may be insufficient and may pass the translation of the input translation data 310 (FIG. 3) to a large language model for improving the translation.

It may be appreciated that FIG. 4 provides only an illustration of one implementation and does not imply any limitations with regard to how different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements.

The descriptions of the various aspects and embodiments have been presented for purposes of illustration but are not intended to be exhaustive or limited to the embodiments disclosed. Even though combinations of features are recited in the claims and/or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and/or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.