Abstract:
Computer system, methods, mobile app, and media to guarantee the quality of a language translation of content using a computer network of translators and reviewers communicating in real-time. The accuracy, and hence quality, of the translation is ensured by incorporating “real-time” quality checks comprising randomly inserted errors into an original human translation and evaluating if a reviewer detects the errors. By using a plurality of reviewers for grading each translation, while detecting and eliminating reviewers that are not competent, the quality of a translation is guaranteed. The level of quality is also controlled by increasing (higher quality) and decreasing (lower quality) the type and amount of errors to be detected by the reviewers. Therefore, a computerized system is able to guarantee the quality of a translation into any language and without knowing that language, or the translators who translate it, or the reviewers who grade the translators.

Description:
COPYRIGHT NOTICE 
     A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or patent disclosure as it appears in the Patent and Trademark Office, patent file or records, but otherwise reserves all copyright rights whatsoever. 
     FIELD OF THE DISCLOSURE 
     The present disclosure relates to the field of ensuring the quality of human translations of content. More specifically the invention relates to the field of guaranteeing the quality of a human translation of content using a computer network of translation graders (i.e. “reviewers”) and automated methods of determining the reliability of each reviewer, and thus translator, in real-time by inserting textual errors into the translation. 
     BACKGROUND OF THE DISCLOSURE 
     In today&#39;s global economy, obtaining a high quality translation of content (e.g. text, graphic design, document layout, etc.) from an original source language to a target language becomes more and more important. While there are many machine, computer based, translation systems, such as Google® Translate, professional human translators are still required in order to produce accurate, high-quality, translation. Unfortunately, human translators often err and produce inadequate translation. The common solution today is to have a proofreader or editor read the translated content and correct it as needed. This solution is expensive, slow and inadequate. 
     Language translation of textual content is also a complicated process due to a variety of factors, such as syntax, semantics, and language ambiguity that occurs in various aspects in natural language processing, i.e. lexical ambiguity, case ambiguity and referential ambiguity. Therefore, to ensure a high quality of translation, a translator must translate into a language they are fluent in both written and oral form, and they must also have a sufficient knowledge of the field being translated to have a full understanding of the subject matter. It is no wonder then that translations by professional translators can often be of variable quality; and why machine translations are often riddled with errors. 
     A bad translation can cause significant damage; sometimes even a single word can drastically change the meaning of the entire paragraph. Machine translation solutions are not accurate enough and the existing methods for evaluating translation quality are cumbersome, slow and expensive. Usually a supervising proofreader checks the translation and corrects it if errors are found. A single proofreader may not locate all the errors in the translation especially if s/he is under time pressure. 
     The level of quality of a given translation is hard to determine as it is a very subjective matter. In essence, a translation is considered to be good if enough people with control of both the source, i.e. original, language and the target language consider it to be an accurate and succinct translation. But, with the existing methods, submitting a project for proofreading or review by more than one proofreader will result is unacceptable costs in terms of time and money. 
     Therefore, there is a need within the art of human language translations for an efficient, economical, reliable, and timely method of evaluating the quality and accuracy of the translation simultaneously by a group of professional translators. There is also a need for a system does not require the user (e.g. network translation service provider) to know: 1) the target language; 2) the reliability and accuracy of the human translator; and/or, 3) the reliability and accuracy of any one reviewer within a plurality of reviewers. 
     The present disclosure is based on a novel computer system, method, and media comprising the use of a plurality of translator reviewers (i.e. reviewers of the quality of a language translation) connected in real-time simultaneously via a computer network (e.g. the Internet) to review the translation and give their rating on its quality. Incompetent reviewers are easily and automatedly identified by the computer system and their quality rating removed when computing a consensus quality rating of the original translation. 
     SUMMARY OF THE DISCLOSURE 
     The present disclosure comprises a computer system, methods, and media to guarantee the quality of a language translation of content (e.g. text, images, etc.) using a computer network of translators and reviewers communicating in real-time. The accuracy, and hence quality, of the translation is ensured by incorporating “real-time” quality checks comprising randomly inserted errors into an original human translation and evaluating if a reviewer detects the errors. By using a plurality of reviewers for grading each translation, while detecting and eliminating reviewers that are not competent, the quality of a translation is guaranteed. The level of quality is also controlled by increasing the type (higher quality) and decreasing the type (lower quality) and the amount of errors to be detected by the reviewers. 
     One or more embodiments of the present disclosure comprise one or more computer systems, computer implemented methods, and non-transitory computer media to guarantee and/or ensure the quality and accuracy of a language translation of content (e.g. text) using a computer network of human translators and reviewers communicating in real-time with a translation service provider over a network, and/or via a mobile app. The accuracy, and hence the quality, of the human translation is ensured by incorporating the “real-time” quality checks as disclosed herein and comprising randomly inserting errors into a translator&#39;s original translation and evaluating if a reviewer detects the errors (e.g. see  FIG. 4A  and  FIG. 4B ). By using a plurality of reviewers for grading each translation, while detecting and eliminating reviewers that are not able to detect the translation errors, the accuracy and thus the quality of a translation is guaranteed. 
     The level of quality is also controlled by increasing (higher quality) and decreasing (lower quality) the type and amount of errors to be detected by the reviewers. Therefore, a translation service provider is able to guarantee the quality of a translation into any language and without knowing that language, or the translators who translate it, or the reviewers who grade the translators. 
     In one or more embodiments, the method comprises: 1) the computer system processor (i.e. “the processor”) receiving an “original” human translation (e.g. over a computer network and/or on mobile app) and randomly inserting errors into the translation; 2) transmitting the original, and the “error ridden”, translation to a plurality of human translation reviewers (i.e. “reviewers) and receiving back a “reviewer rating” on the quality of the translation for both the original and the error ridden translation; 3) calculating the number of errors detected, and/or missed, by each reviewer within the plurality of reviewers for the error ridden translation (or comparing the grade assigned by a particular reviewer to an average of the other reviewers), and assigning a “competency” rating to each reviewer; 4) when a reviewer&#39;s competency rating is below a threshold rating, then omitting this reviewer rating of the original translation; and, 5) calculating the “consensus rating” of the remaining competent plurality of reviewers on the quality of the original human translation (e.g. see  FIG. 4A ). Additionally, if the consensus rating is below the ensured quality level (a threshold rating), then the translation may be sent to another human translator and steps (1)-(5) repeated. 
     In another embodiment, the method step (1), is preceded by the computer system processor automatedly breaking the text that is to be translated into parts, and then transmitting it to a human translator (e.g. see  FIG. 4B ). 
     An aspect of the various embodiments disclosed herein is the fact that the system does not require the user (e.g. network translation service provider, content author, etc.) to know: 1) the target language; 2) the reliability and accuracy of the human translator; and/or, 3) the reliability and accuracy of any one reviewer within a plurality of reviewers. 
     Another aspect is the ability to designate the level of quality required for an original human translation by adjusting parameters or factors within the system. The level of quality is adjusted for a variety of factors, such as: amount billed to the customer for the translation service (i.e. higher the fee, higher the guaranteed quality of the translation); amount of time available to conduct the translation; importance of content (e.g. text) that is being translated and/or the customer (e.g. intended use of translation—e.g. legal contract versus advertisement); and so forth. Parameters adjusted comprise, by way of non-limiting examples: the number of reviewers used in rating the quality of the original human translation; altering methods of randomly inserting errors into the original translation (omitting strings; switching strings; etc.); altering the number of errors to be detected, and/or missed, by a reviewer (e.g. higher the number of errors, higher the quality rating); etc. 
     Another aspect is the ability to calculate and assign a competency rating to a reviewer by a variety of methods, such as (e.g. see  FIG. 5 ): 1) automatedly calculating the number of errors detected, and/or missed, by a reviewer and assigning a grade; 2) calculating the “reviewer relative rating” of the translation (e.g. “Good, OK, Bad”), and comparing this to the consensus, average, mean, median, or mode rating of the other reviewers; 3) calculating the competency rating using different types of errors and assigning a relevancy weight designating the importance of each type (e.g. reviewer detection of a missing string is weighed higher than detection of switched strings, or vice versa). 
     Another aspect of the present disclosure is to objectively determine the competence of a specific reviewer within a plurality of reviewers. 
     Another aspect is to objectively determine if the performance of a specific reviewer has increased or decreased over time so that appropriate action can be taken to remedy the situation. 
     The various embodiments may further comprise a computer program product tangibly embodied in a computer-readable storage device and comprising instructions that, when executed by a processor, perform one or more methods disclosed herein. For example, the computer program product may comprise a mobile application (i.e. a mobile app) with a translation module accessible on a customer&#39;s, a translator&#39;s and/or a reviewer&#39;s (client) electronic computing device comprising a non-transitory computer readable storage medium comprising instructions, the instructions being operable to enable the electronic computing device to perform all of or parts of the steps of the methods disclosed herein. The mobile app of the present disclosure that is used on a customer&#39;s electronic computing device has the ability to electronically send and receive communications for ordering a translation (e.g. about project notifications), and may further communicate directly with one or more translator&#39;s electronic computing device, and/or one or more reviewer&#39;s electronic computing device. 
     The mobile app of the present disclosure on a reviewer&#39;s electronic computing device can also: receive project notifications; view the original translation and the error-ridden translation and provide a quality rating for both and/or identify errors in the error-ridden translation; invite other reviewer&#39;s to rate a translation project; and/or electronically communicate directly with a translator and/or a customer device. 
     And the mobile app of the present disclosure on a translator&#39;s electronic computing device can: receive a project notification comprising content to translate; input and transmit the translation; fix existing translations; invite other translators to the project; and/or communicate directly with the customer and one or more reviewers of their project. 
     The mobile app may also be used, for example, by a content author who desires to ensure the quality of a translation of his content that was translated by him or another translator, by directly sending the translation to a plurality of reviewers after the mobile app (or networked computer) automatedly inserts errors into the translation. The mobile app will then collect the plurality of reviewers&#39; ratings of the original content and the error-ridden content, and automatedly compute a competency rating for each reviewer and a consensus rating for the translation by the designated competent reviewers. The mobile app will then display on the content author&#39;s electronic computing device whether the translation is of the requested level of quality, and if not, recommend that the author obtain another translation from a different source, such as another translator within the system&#39;s network of translators. Or, the mobile app can be setup to automatically order another translation from a different translator, such as one who specializes in the target language and subject matter of the content. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The invention and its different aspects and advantages will be better understood from the following detailed description of preferred embodiments of the invention with reference to the following drawing: 
         FIG. 1  is a block diagram representing an embodiment of the computer system comprising a translation service provide system, a translator computing device, a plurality of reviewers computing devices, and a customer computing device, all communicating over a network. 
         FIG. 2  is a block diagram representing one embodiment of the reviewer&#39;s electronic computing device that is used to check the translation project and provides a rating on the quality of the human translation. 
         FIG. 3  is a block diagram representing one embodiment of the computer system of the translation service provider, or the mobile app installed on a reviewer&#39;s device, comprising optional features. 
         FIG. 4A  is an exemplary logic flow and functional block diagram illustrating one embodiment of the method of the present disclosure. 
         FIG. 4B  is an exemplary logic flow and functional block diagram illustrating another embodiment of the method of the present disclosure comprising automatedly breaking the text into parts before it is translated. 
         FIG. 5  illustrates exemplary embodiments of alternative methods for assigning each reviewer a competency rating. 
     
    
    
     DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS 
     Glossary of Terms 
     As used herein, the term “Electronic Computing Device” refers to any electronic communications device comprising a central processing unit (i.e. processor) with the Internet connectivity, such as: laptops, desktops, tablets, iPads, smartphones, cell phones, personal digital assistant devices, and the like. 
     As used herein, the term “Content” refers to any form of digital media that can be translated from an original, source language to a target language. By way of non-limiting examples, content to be translated may comprise one or more of: a textual document, an audio recording, text within images, the textual and image layout of a document or webpage, etc., and any combination thereof. It is understood that one of skill in the art could readily modify the disclosure herein for textual translations to include all forms of content translations. 
     As used herein, the term “Software” refers to computer program instructions adapted for execution by a hardware element, such as a processor or CPU, wherein the instruction comprise commands that when executed cause the processor to perform a corresponding set of commands. The software may be written or coded using a programming language, and stored using any type of non-transitory computer-readable media or machine-readable media well known in the art. Examples of software in the present invention comprise any software components, programs, applications, computer programs, application programs, system programs, machine programs, and operating system software. 
     As used herein, the term “Module” refers to a portion of a computer program or software, such as a mobile app, that carries out a specific function and may be used alone or combined with other modules of the same program. The module may comprise a native application, a web application, or a widget type application to carry out the methods of detecting and electronically transmitting untranslated character strings. In a one embodiment, a native application is installed on the customer&#39;s, reviewer&#39;s and/or translator&#39;s electronic computing device, wherein it can be downloaded automatically from the Internet. It may be written in a language to run on a variety of different types of devices; or it may be written in a device-specific computer programming language for a specific type of device. In another embodiment, a web application resides on the system server and is accessed via the Internet. It performs basically all the same tasks as a native application, usually by downloading part of the application to the device for local processing each time it is used. The web application software is written as Web pages in HTML and CSS or other language serving the same purpose, with the interactive parts in JavaScript or other compatible language. Or the application can comprise a widget as a packaged/downloadable/installable web application; making it more like a traditional application than a web application; but like a web application it uses HTML/CSS/JavaScript and access to the Internet. 
     As used herein, the term “Application Program Interface (API)” refers to a set of computer programming instructions, for accessing a web based software application or web tool. 
     As used herein, the term “A System” may be used to claim all of, or parts of, the aspects of the present disclosure wherein it refers to the entire configuration, or parts of, the networked based human translation service, e.g. all hardware and software used in all scenarios. Preferred embodiments are illustrated in the system architectures of  FIGS. 1-3 . 
     As used herein, the terms “Processing,” “Computing,” “Calculating,” “Determining,” “Establishing”, “Analyzing”, “Checking”, or the like, may refer to operation(s) and/or process(es) of a computer, a computing platform, a computing system, a computer central processing unit (CPU), or other electronic computing device, that manipulate and/or transform data represented as physical (e.g., electronic) quantities within the computer&#39;s registers and/or memories into other data similarly represented as physical quantities within the computer&#39;s registers and/or memories or other information storage medium that may store instructions to perform operations and/or processes. 
     As used herein, the term “Original Human Translation” and “Original Translation” refers to a translation of content (e.g. text) performed by a human translator from a source language to a target language, and wherein the computer system processor has not inserted errors into the translation. 
     As used herein, the term “Error-ridden Translation” refers to an original (human) translation wherein the computer system processor has randomly inserted errors into it to test the competency of the reviewers. The types of errors may also be assigned a relevancy weight designating the importance of each type (e.g. translator detection of a missing string is weighed higher than detection of switched strings, or vice versa). As a result, higher weighted error types will therefore have a higher impact on the reviewer&#39;s competency rating. 
     System Architecture 
     The various embodiments disclosed herein may be used in conjunction with, or in lieu of, one or more methods, systems and media for rating human translators and reviewers as disclosed in U.S. Pat. No. 8,527,521 issued Sep. 3, 2013 to Ofer Shoshan entitled “System and Method for Evaluating the Quality of Human Translation Through the Use of a Group of Human Reviewers”, the entirety of which is herein incorporated by reference. 
       FIG. 1  is a block diagram representation of a computer implemented system  100  of one embodiment of the present disclosure. System  100  comprises a translation service provider computer system  300  communicating via a network  110  (e.g. the Internet) with: 1) a customer computer system and/or electronic computing device  140  for ordering a human translation from system  300 ; 2) a plurality of human translator electronic computing devices  160  to generate and transmit back to system  300  an “original human translation”; and, 3) a plurality of translator reviewers (i.e. “reviewer&#39;s”) electronic computing devices  200 ,  202 , etc. to assess the quality of the transmitted translation. In another embodiment, a mobile app is installed on the customer&#39;s device  140 , the translator&#39;s device  160 , and/or the reviewers&#39; devices  200 ,  202  to carry out some or all of the functions of computer system  300 . 
     In the various embodiments, the computing devices-systems  140 ,  160 ,  200 ,  202 , and/or  300  communicate via network  110 , which comprises a local area network (LAN) or a wide area network (WAN), or a network connection may be made to via an external computer (for example, through the Internet using an Internet Service Provider), and using wired, wireless, and terrestrial or satellite links. 
     The “quality rating” is assessed by the system  300  processor to determine if it is sufficient (e.g. meets a threshold rating—Excellent, Fair, Good, etc.). In an embodiment, when the processor decides that the quality rating is too low or is insufficient (e.g. does not comply with the customer&#39;s order), then the system  300  automatedly re-orders the translation from another human translator. And when the “quality rating” is sufficient, it may be transmitted to the customer device  140  to provide proof of compliance with guaranteeing the quality of the translation ordered. 
     Another system embodiment includes the use of a mobile app by a user-customer wishing to check the quality of a human translation without contracting with a translation service provider computer  300 . The customer device  140  would receive a translation from a translator device  160 . The mobile app installed on the customer&#39;s (i.e. user&#39;s) device  140  would automatedly insert errors into the translation (or receive an error ridden translation from the system  300 ), and then transmit the original translation and the error-ridden translation directly to a plurality of reviewers&#39; devices  200 ,  202 , etc. via the network  110 . Alternatively, a mobile app installed on the translator&#39;s device  160 , and/or installed on the reviewers&#39; devices  200 ,  202 , would automatedly insert errors into the original translation to generate the “error ridden translation”. 
     The user&#39;s device  140  would then receive back the reviewers&#39; consensus rating of the quality of the original translation. If the consensus rating on the quality of the translation did not meet the user&#39;s desire, then the user could manually, or the mobile app would automatically, order the translation from another human translator and repeat the process of distributing the translation to a plurality of reviewers. 
       FIG. 2  is a block diagram representation of the reviewer&#39;s electronic computing devices  200  and  202  in an exemplary embodiment, in which it is understood that there may be a plurality of reviewer devices numbering more than the exemplified  200 ,  202 . The reviewers&#39; device  200  and  202  each receive a project notification comprising the original human translation from the central computer system  300 , and/or directly from the translator device  160 , and/or directly from the customer device  140 , and display it to the reviewer. Devices  200  and  202  then receive the reviewer&#39;s input of the quality rating of the original translation, and the error-ridden translation (or the identification of the errors) and electronically transmit it back to the central system  300  and/or directly to the customer&#39;s device  140  and/or the translator&#39;s device  160 . 
     The reviewer electronic computing device  200  and  202  (e.g. laptop, tablet, desktop PC, smartphone, PDA, etc.) has one or more processing units (CPUs)  205 , one or more memories  210 , user input devices—keyboard &amp; mouse— 220 , graphical user interface (GUI)  230  to display the source text, the original human translation and the error-ridden translation, and a network connection  240 . It may further comprise a Presentation module  250  in the mobile app, or accessible via the network, to provide functionality to display side-by-side, and thus be able to compare, the source text to the original translation and/or the error ridden translation. 
     The reviewer device  200  and  202  also has installed on, or accessible via the network  110  from the translation provider computer system  300 , a “Quality Rating Module”  400 , e.g. within the mobile app. The Quality Rating Module  400  provides functionality to receive the reviewer&#39;s input for the identification of errors in the error-ridden translation, and to receive the input for the reviewer&#39;s quality rating of the original human translation. The Quality Rating module when installed on the system  300  (e.g.  FIG. 3 ) and/or the device  200 ,  202  (as a mobile app), may further comprise the ability to automatedly: 1) divide the translation into parts (e.g. sentences, paragraphs, strings) before or after the source text is translated; and, 2) insert errors into the parts that the reviewer then needs to detect. 
     As illustrated in  FIG. 3 , the translation provider computer system  300 , and/or the mobile app, further comprise main processing modules for selecting qualified translators and reviewers, contacting them via the network, collecting the reviewers&#39; ratings, calculating a competency rating of each reviewer based on their error-ridden test results, calculating a quality rating on the original translation from a consensus of the competent reviewers, and electronically transmitting the quality rating to the customer electronic computing device  160 . Computer system  300  has one or more processing units (CPUs)  305 , one or more memories  310 , user input devices—keyboard &amp; mouse— 320 , user display  330 , Network connection  340 , and storage database  350 , e.g. any server computer. The storage database  350  comprises a plurality of human translators&#39; and/or reviewers&#39; records comprising the identity of each translator&#39;s and reviewer&#39;s language and subject matter expertise, availability, contact information, etc. 
     The computer system  300 , and/or the mobile app running on a user&#39;s device, may further execute a Reviewer Select and Contact module  600 , which is used to select qualified reviewers based on their language and subject matter expertise, and to perform the review of the translation. It also identifies and electronically contacts a plurality of reviewers for a specific translation project. 
     In the various embodiments, the translation project is sent from the customer device  140  to computer system  300 , or directly to translator device  160 ; and then from the translator device  160  to the system  300  or directly to the reviewers&#39; devices  200 ,  202 . It is understood that any combination of transmitting the communications comprising the translation and rating is easily adaptable to the skilled artisan. The translation project is analyzed to collect project characteristics, such as source language, target language, type of translation, length, files types etc. The computer system  300 , and/or the mobile app, creates a project profile based on the collected project characteristics that is used to select the reviewers to perform the quality rating. Reviewers are selected based on the project profile of characteristics and on their individual profile stored in the reviewers&#39; data-base  350  connected to or stored on the computer system  300 , and or a user&#39;s device memory. An individual reviewer profile may comprise a reviewer&#39;s areas of expertise, preferences for types of projects, availability, etc. A reviewer is selected if his/her skills meet the project meta-data requirements, e.g. source language and target language, and other parameters like previous reviewer score or competency rating. Each reviewer is alerted of the new translation project by use of electronic communication, for example, email, mobile text message, web-page, Instant messenger and potentially other communication means. 
     Computer system  300 , or the mobile app, may further contain the Rating collection and Consensus rating calculation module  700 . This module may be used in conjunction with one or more other modules (e.g. the Quality Rating module  400 ) to collect ratings from competent individual reviewers and then calculate a quality rating. 
     Computer system  300 , or the mobile app, may further contain the Presentation and Action module  800  to display side-by-side, and thus be able to compare, the source text to the original translation and/or the error ridden translation. It may further provide functionality to send the results to the customer device  140  over the network  110  and take predefined action if the quality rating is too low, such as automatically ordering another translation from a different translator and repeating the process of having a plurality of reviewers grade the translation while identifying and eliminating the quality rating of incompetent reviewers. 
     Computer system  300 , or mobile app, may further contain the Reviewer Score module  900  that is used in conjunction with one or more other modules (e.g. the Quality Rating module  400 ). This module may be used to review the score, or competency rating, generated by the Quality Rating module  400 , and transmit and display a reviewer&#39;s results to his/her device  200 ,  202 . The score or rating is subsequently used when reviewers are selected to evaluate and rate the quality of future translation projects. 
     Non-Transitory Computer Readable Storage Device and/or Media: 
     The various embodiments disclosed herein may further comprise a non-transitory computer readable storage device or media, which is a physical storage device readable by a machine. The computer readable storage device can store data that is accessible by a computer, such as system  300  or device  140 ,  160 ,  200 ,  202 , etc. The computer readable storage device is preferably a non-removable hard drive, but may include a backup Universal Serial Bus (USB) flash drive that can be easily removed and physically transferred to a computer. Other examples include devices such as a magnetic cassette, a flash memory card, a digital video disk, a compact disc, an optical disc, a Bernoulli cartridge, a random access memory (RAM) card, read only memory (ROM) card, and other such storage devices. 
     The applications or algorithms or modules disclosed herein (e.g. “Quality Rating”, “Presentation” modules, etc.) can be stored on computer readable storage device or media (CD, DVD, hard drive, portable storage memory, etc.) of the computing device  200 ,  202 , and/or system  300  and executed by the them. 
     In general, another innovative aspect of the subject matter described in this specification can be implemented in a computer program product tangibly embodied in a computer readable storage device and comprising instructions that, when executed by a processor, perform one or more methods disclosed herein. 
     Quality Rating Methodology 
     Although the disclosure herein for the flowcharts of  FIGS. 4A, 4B and 5  cover textual translations, one skilled in the art could readily adapt the teachings to the translation of any type of content. It is further understood that the steps disclosed in  FIGS. 4A, 4B, and 5  may be executed by one or more of the modules in  FIGS. 2 and/or 3 . 
       FIGS. 4A and 4B  are flowcharts illustrating computer steps conducted by the computerized system  100  in an exemplary embodiment comprising the use of the Quality Rating Module  400  alone or with one or more other modules for  FIGS. 2 and/or 3 . In step  410 , a reviewer&#39;s computing device  200 , e.g. smartphone with the mobile apps of the present disclosure installed thereon, and/or the translation service provider computer system  300 , receives the original human translation from the translator computing device  160 , in response to the translation that was ordered from the customer computing device  140  (e.g. via system  300  or directly or directly via the mobile app). The reviewer&#39;s device  200  and/or the system  300  then randomly and automatedly inserts errors into the translated text to generate the “error ridden translation”. 
     The translated text may further be broken into “parts” before or after it is translated, the parts comprising one of more of: sentences, paragraphs, strings, etc. This may occur, for example: before the source text is sent to be translated, the mobile app on the customer&#39;s device  140 , and/or the system  300 , divides the text into parts (e.g. see  FIG. 4B , step  405 ). Or the text may be broken into parts after the original human translation is received: back at the customer&#39;s device  140 ; at the reviewers&#39; devices  200 ,  202 ; and/or at the system  300  (e.g. see  FIG. 4A , step  410 ). Or the text may be inherently broken into parts because of its source (e.g. translations of websites and mobile applications). 
     The customer&#39;s device  140  and/or the translator&#39;s device  200  with the mobile app installed thereon, and/or the system  300 , may also insert a wide variety of types of errors into the text to produce the “error-ridden” translation or text. By way of non-limiting examples: a machine translation may be substituted for a part; different parts may be swapped within the same translation; specific words may be substituted with known words that are commonly mistranslated; and so forth. The Quality Rating Module  400  may further comprise computer code designed to insert errors specific to: different types of translations, different levels of quality required, different types of customers, and so forth. 
     In step  420 , the customer&#39;s device  140 , and/or the system  300 , transmits the original human translation and the error-ridden translation (parts) to a plurality of reviewer&#39;s electronic computing device  200 ,  202 , etc. The reviewers are also instructed to rate the quality of the original human translation (e.g. percentage of 100%; grade A-F; Good/OK/Bad), and to find errors in the error-ridden translation. The number of reviewers that are tasked depends on a variety of factors, such as: level of quality needed for the original human translation (e.g. higher the qualities, then more reviewers are tasked); availability of reviewers for specific target language and subject matter expertise of text topic; time allowed to deliver the translation to the customer; etc. 
     In step  430 , customer&#39;s device  140  and/or the system  300  receives the reviewers&#39; quality rating for the original human translation, and their error identification input. Based on their test results for detecting the errors, each reviewer is assigned a Competency Rating (score) based on a variety of methods, which comprise by way of non-limiting examples (e.g. see  FIG. 5 ): 1) automatedly calculating the number of errors detected, and/or missed, by a reviewer and assigning a grade; 2) calculating the “reviewer relative rating” of the translation (e.g. “Good, OK, Bad”), and comparing this to the consensus rating (average, mean, median, mode, etc.) of the other reviewers; 3) calculating the competency rating using different types of errors and assigning a relevancy weight designating the importance of each type (e.g. reviewer detection of a missing string is weighed higher than reviewer detection of switched strings, or vice versa). 
     In step  450 , when a reviewer&#39;s competency rating is not at or above a designated threshold level as required for a particular translation project (e.g. falls below a guaranteed threshold standard), then the reviewer&#39;s quality rating of the original human translation is omitted from the record. 
     In step  460 , the system  300  processor calculates a Consensus Quality Rating (score) based on the ratings of the remaining competent reviewers. The calculation may comprise computations well known in the art, such as the mean, median, average, mode or it may comprise a customized algorithm that is encoded in the Quality Rating Module  400 . If the Quality Rating is below a required threshold level, then the system can re-order the translation (automatedly or via a system administrator or user). The source text is sent to a different translator and steps  410 - 460  are repeated as many times as required. The final Quality Rating may then, optionally, be transmitted via the network  110  to the customer electronic computing device  140  along with the original human translation in compliance with ensuring that the quality guaranteed has been met. 
     The customer&#39;s device  140  and/or the system  300  may also adjust the quality needed for the original human translation by adjusting parameters. Parameters adjusted comprise, by way of non-limiting examples: the number of reviewers used in rating the quality of the original human translation; altering methods of randomly inserting errors into the original translation (omitting strings; switching strings; etc.); altering the number of errors to be detected, and/or missed, by a reviewer (e.g. higher the number of errors, higher the quality); etc. 
     A quality level is designated automatedly by the customer&#39;s device  140  and/or the system  300  and/or manually by the system administrators, based on a variety of factors, such as: amount billed to the customer for the translation service (i.e. higher the fee, higher the guaranteed quality of the translation); amount of time available to conduct the translation; importance of text that is being translated and/or the customer (e.g. intended use of translation—e.g. legal contract versus advertisement), and so forth. 
     CONCLUSION 
     The results of the operation of the novel process described above is a fast and efficient way to order and receive a human translation while ensuring its quality, even if the user and/or system administrator is not proficient in the target language. The aforementioned flowchart and diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods, apparatuses, devices and media according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. 
     Likewise, the terms “plurality” may be used throughout the specification to describe two or more components, devices, elements, units, parameters, or the like. Unless explicitly stated, the method embodiments described herein are not constrained to a particular order or sequence. Additionally, some of the described method embodiments or elements thereof can occur or be performed at the same point in time. 
     Although various features of the invention may be described in the context of a single embodiment, the features may also be provided separately or in any suitable combination. Conversely, although the invention may be described herein in the context of separate embodiments for clarity, the invention may also be implemented in a single embodiment. 
     It is to be understood that the phraseology and terminology employed herein is not to be construed as limiting and are for descriptive purpose only. 
     It is to be understood that the details set forth herein do not construe a limitation to an application of the invention. 
     Furthermore, it is to be understood that the invention can be carried out or practiced in various ways and that the invention can be implemented in embodiments other than the ones outlined in the description above. 
     And lastly, the trademarks, website names, programming language names, etc. that are used in the present disclosure are the property of the respective owner companies and are used herein for illustrative purposes only. The inventor does not claim any rights on it.