Patent Description:
With the advancement of technology, especially mobile devices and cloud network communications, easier means of collecting the codes have been contemplated. Historically, website access to loyalty accounts have been available to consumers, and those sites have been accessible via mobile devices, where users can type in the codes. One process for simplifying submission of the codes has contemplated the use of imaging the codes using a built-in camera of a smart phone. However, a problem with imaging codes that exists is that smart phone imaging and image processing of the imaged codes have been unreliable due to a variety of factors, including size of characters, resolution of cameras, beverage stains covering portions of codes, font of codes, lighting and shadows when capturing codes, and many other factors. Due to the various factors that are problematic for imaging and image processing codes, especially codes printed with a pin code character format, the use of conventional optical character recognition (OCR) does not provide commercially reliable results. As a result, marketers have been unwilling to support a mobile imaging solution so as to not negatively impact consumer loyalty.

<CIT> discloses a computing device that recognizes codes, such as digit claim codes from a image of a gift card, by using image processing, computer vision, and/or machine learning algorithms.

According to the invention, there is provided a computer implemented method of reading a multi-character code, the method comprising: in response to receiving an image of the multi-character code, identifying regions in which respective characters of the code reside; applying the identified regions to a neural network to determine the respective characters in the identified regions; displaying the determined characters in an ordered sequence for a user to visually inspect to confirm that each of the determined characters are correct, characterized wherein displaying the determined characters includes displaying the determined characters with character recognition probability levels over a threshold probability level as determined by the neural network; displaying one or more characters of respective one or more characters determined to have character recognition probability levels below the threshold probability level in a format different from the characters determined to have character recognition probability levels over the threshold probability level so as to notify the user to supply one or more corrected characters of the code displayed in the different format; enabling the user to enter the one or more corrected characters for each of the characters in a different format; and responsive to receiving the one or more corrected characters from the user, applying the one or more corrected characters to a training set of characters to train the neural network.

The invention may thus provide imaging capabilities and image processing codes, including for those printed with a pin code character format, that are accurate enough for consumer usage and for marketers to support a brand loyalty program, and an image capture and image processing process that has commercially acceptable character recognition rates. The image processing includes the use of machine learning by a neural network, and may enable consumer participation for active learning with a production mode that supports a loyalty program.

In an embodiment, characters of the code may be printed with a pin code character format, which include dots that provide for low resolution characters, as understood in the art. It has been found that conventional optical character recognition (OCR) algorithms do not handle pin code, low-fidelity characters sufficiently accurately for commercial usage. To provide for an image processing capable of resolving pin code characters, an embodiment may use an image capture, character segmentation, and neural network character recognition modeling and matching process. After proper training, the process may provide for a high confidence levels (e.g., <NUM>%) that each character is accurately identified, where the training may further use active learning through the engagement of many consumers, thereby improving accuracy of character recognition in an expedited manner. In an embodiment, the user interface, such as a browser-based user interface or mobile app, may provide feedback of one or more individual characters that may have low confidence scores (e.g., below <NUM>%, <NUM>%, or otherwise) that are indicative of characters not being correctly recognized or recognized with a high enough degree of probability by image processing.

Illustrative embodiments of the present invention are described in detail below with reference to the attached drawing figures, wherein.

With regard to <FIG>, an illustration of an illustrative scene <NUM>, in this case a scene with a beverage bottle <NUM> inclusive of a beverage, such as a soft drink, is shown. The beverage bottle <NUM> includes a cap <NUM> on which a multi-character code <NUM> may be printed. As is common, the multi-character code <NUM> may be printed using a low-resolution pin code character format, and be positioned on an inside surface of the cap <NUM> so as to be available to a user after purchase and when opened. The code <NUM> may be submitted to a website, and a value associated with the code <NUM> as established by a marketer of a product, in this case a beverage, may be applied to an account of the user.

As further shown, a mobile device <NUM>, such as a smart phone, may be used to image the code using a standard camera feature that exists on the electronic device <NUM>. A user interface <NUM> may provide for displaying an image <NUM> of the code <NUM>, and a set of characters <NUM> determined by image processing the image <NUM> of the code <NUM> is displayed. As described further herein, the image processing uses a neural network so as to provide for pin code character recognition with commercially satisfactory rates that are commercially acceptable. As further shown, within the set of characters <NUM>, blank character spots, shown as underscores, that were not able to be interpreted or were interpreted or matched with low confidence or probability levels by a neural network, such as a convolutional neural network (CCN), are displayed so that a user is able to manually add and/or correct characters for those characters identified with low probability matching. In an embodiment, a user interface keyboard or audio transcription feature may be used on the mobile device <NUM>. The user interface <NUM> may also enable a user to change all the characters, but characters that cannot be interpreted by the image processing are displayed in another format, such as in the color red, bold, within a box, or otherwise.

With regard to <FIG>, an illustration of a set <NUM> of illustrative consumer packaging components, including bottle caps 202a-202f and cartons <NUM>-<NUM>, on which codes 204a-<NUM> may be respectively printed using a pin code character format is shown. Other print formats may also be utilized. As shown, the bottle caps 202a-202f each have different colors and each of the respective codes 204a-204f are printed with different colors. The colors of the bottle caps 202a-202f may range from light to dark colors, including white, gray, green, red, black, or any other color, which may be considered background colors relative to colors of codes printed thereon or displayed in front of the caps. The codes <NUM> may be printed in foreground colors that may, for example, range from white to black, as well. As understood in the art, a plastic or other substrate component that may be transparent or translucent and used for bottle sealing purposes may be disposed over the printed codes, and the plastic component may have a different color (e.g., light blue) from the color (e.g., black) of the code, as well. The codes may alternatively be printed on the plastic component. The plastic component tends to introduce some amount of image noise or image distortion when the printed code is imaged by a user, as further described herein. The cartons <NUM> and <NUM> may have codes <NUM> and <NUM> printed thereon, and the cartons <NUM> and <NUM> may have different texture, color, and print quality than those of the caps 202a-202f.

In an embodiment, the code <NUM> is printed on a single line, whereas the codes 204a-204f and <NUM> are printed on two lines. It should be understood that the codes may be printed in a variety of different formats and with different number of characters. In an embodiment, the codes may be encoded to include certain metadata, including product identifier, date of production, particular promotion, color of background and/or foreground colors, and/or any other metadata as desired by the marketer to support product promotion, manufacturing, distribution, and/or loyalty program.

With regard to <FIG>, an illustration of an illustrative network environment <NUM> configured to support a consumer product loyalty program that utilizes image processing as described herein is shown. The network environment <NUM> may include a mobile device <NUM> on which a user interface <NUM> is shown to include an imaged bottle cap <NUM> including an imaged code <NUM> printed thereon. It should be understood that a user may image other objects on which codes may be printed for promotional or other purposes by a marketer. A set of characters <NUM> that is determined from the imaged code <NUM> using image processing, as further described herein, is shown. Moreover, it should be understood that the multi-character code <NUM> (<FIG>) may be printed on a variety of different substrates, including plastic (e.g., underneath bottle caps), cardboard (e.g., inside of cartons that contain multiple beverage products), paper (e.g., scratch tickets, calling cards, etc.), and so on. Moreover, the multi-character code <NUM> may be in the form of structured sentences, words, or otherwise.

The mobile device <NUM> may include electronics <NUM> that include a processing unit <NUM> that executes software <NUM>. The software <NUM> may be configured to operate the mobile device, and may further be configured to participate with or perform image processing to determine characters of the imaged code <NUM>. The processing unit <NUM> may be in communication with a memory <NUM> that may store data, such as the imaged bottle cap <NUM> and code <NUM>, character matching data, add character matching data, and so forth, input/output (I/O) unit <NUM> configured to communicate over a communications network, and electronic display <NUM> (e.g., touch screen) for displaying a user interface with which the user may interface.

The mobile device <NUM> may be configured in a number of ways to provide for supporting image processing of the imaged code <NUM>. Illustrative configurations may include communicating the entire image, portion of the image, and/or determined code to a cloud server for registering with an account of a user of the mobile device. In an embodiment, a set of rewards data <NUM>, which may include user ID, rewards account ID, image data, determined code data, packing identification data, and/or location may be communicated from the mobile device <NUM> via a communications network <NUM> to a server <NUM>. Additional and/or alternative information may be communicated along with the rewards data <NUM> for utilization with a loyalty or other program.

The image data may include one or more image segments of the imaged code <NUM>. In an embodiment, the software <NUM> may be configured to determine sub-areas of the imaged code <NUM> in which respective characters are positioned. It should be understood that additional information may be included with the rewards data <NUM>, such as mobile ID, lighting information, color information, and/or any other information associated with the imaged code, cap, or otherwise that may be used in processing the imaged code <NUM>. Moreover, correction data and/or training data in the form of character(s) entered by a user may be communicated with the rewards data <NUM> so that a neural network executed by the server <NUM> is trained in an active learning mode to improve future image processing.

The server <NUM> may include a processing unit <NUM> that executes software <NUM>. The software <NUM> may be configured to operate and manage a neural network used for determining characters of the imaged code <NUM>. The processing unit <NUM> may be in communication with a memory <NUM> that may store data, such as the imaged bottle cap <NUM> and code <NUM>, input/output (I/O) unit <NUM> configured to communicate over the communications network <NUM>, and storage unit <NUM> on which one or more data repositories <NUM> may be operated. The one or more data repositories <NUM> may store accounts of users who participate in one or more rewards programs for consumer products. The software <NUM> may be configured to support a master or central neural network that is used to perform image processing on images including code characters, such as the imaged code <NUM> having a pin code character format. It should be understood that the neural network may be performed on a separate server from a server that manages user accounts for the loyalty or rewards program. In operation, the server hosting the neural network (or master copy thereof) may be in communication with one another such that the rewards program server, in response to receiving a communication with the image code <NUM> along with the rewards data <NUM>, may communicate the image code and/or rewards data <NUM> to the neural network server for processing thereby.

The neural network may initially be trained and thereafter updated based on active learning in response to user feedback and may be centrally maintained. The neural network may be downloaded to local or "edge" servers of the network <NUM> or to mobile devices of consumers who participate in the loyalty program for execution thereon, where the neural network or any operate in a mobile app or via a browser, as understood in the art. It should be understood that the image processing using a neural network, including training and E active learning, may be performed on a mobile device, or on a network, or combination thereof.

With regard to <FIG>, screenshots 400a-400d of an illustrative user interface <NUM> that may enable a user to capture a code are shown. The user interface <NUM> may initially provide a message for a user that requests that the user provide access to a camera of the mobile device on which the user interface is operating. In response to the user's acceptance to allow an app to use the camera, the user interface <NUM> may present the user with an imaging region <NUM> in which the user is to image a code, as shown in <FIG>. In one embodiment, the user may be able to scan a machine-readable code (e.g., barcode, QR code) or text (e.g., <NUM>-character code) or other characters. As shown in <FIG>, an app being executed by the mobile device may determine that the code is fully encompassed within the image in region <NUM> and automatically capture the image and display a message <NUM> on the user interface <NUM> that the code has been successfully captured. Alternatively, the user may manually capture the character(s) <NUM> and/or code. As shown in <FIG>, a sequence of characters <NUM> is displayed after being determined from an imaged code by a neural network, as further described herein.

The neural network may be executed by the mobile device, remote server, or a combination thereof. In response to determining that one or more characters have been determined with less certainty probability than a probability threshold level, characters in an alternative format (e.g., different color, such as red or other highlight), are displayed to indicate to the user to correct or verify those character(s). As indicated, characters 412a and 412b are determined to have lower certainty than the probability threshold, thereby necessitating that the user confirm or correct those characters 412a and 412b. Still yet, if a probability level of proper identification or recognition of a character is within a range, such as about <NUM>% and about <NUM>%, then a determined character may be displayed in a different format than characters determined with probabilities above the probability threshold level. If the certainty level is below a lower probability threshold level, such as about <NUM>%, then a blank character may be displayed in a different format than those characters determined with probability over the lower probability threshold level. In an embodiment, the neural network, which may be a convolutional neural network, may be operated by the mobile device, or one more images of image characters and/or text character(s) submitted by a user may be communicated to a remote server for determining each of the characters, and determined characters may be communicated back to the mobile device for display on the user interface <NUM>. The characters submitted by the user is used for active learning by the neural network for machine learning.

With regard to <FIG>, steps 500a-500d are shown to be used for training a neural network for image processing pin code characters of codes. In an embodiment, a training phase may be performed prior to the application going "live" to consumers (i.e., users of mobile devices who participate in a rewards program). During the training phase, a mobile app may be distributed to a group of suppliers dedicated to training, and who have also been given a set of bottle caps and paperboards to photograph and label via the training app. The image in step 500a is an illustrative representation of the dedicated training app. Training may continue via consumer input once a promotional app is live in production (i.e., consumer usable for a promotion program).

More particularly, the process may start at step 500a, where a user may use a mobile device <NUM> that executes a mobile app or user interface via a mobile browser and web-based application may be presented with a user interface <NUM> to capture an image of a bottle cap <NUM> on which an imaged code <NUM> is printed. At step 500b, image segments or image portions 508a-508n (collectively <NUM>) inclusive of respective imaged characters 510a-510n (collectively <NUM>) are shown. The image portions <NUM> may be segmented from the captured image by the mobile app being executed by the mobile device <NUM>. Alternatively, the imaged code <NUM> may be communicated to a server operating on a communications network, and the server may perform the segmentation of the characters in the imaged code <NUM>. By identifying image portions <NUM> of each of the characters <NUM>, image processing using a character recognition process by a neural network may be easier.

At step 500c, the image segments 512a-512n (collectively <NUM>), which may be the same as the image portions <NUM> along with characters 514a-514n (collectively <NUM>), which may be text representations of the imaged characters <NUM>, are shown. The characters <NUM> are identified using image processing and confirmed and/or corrected by a user. The training of the neural network <NUM> may be performed so as to improve performance of identification of pin code characters that have low resolution, and are printed and imaged in a variety of ways (e.g., printed on different colors and with different colors, having different colored translucent substrates placed thereon, printed on different substrates captured with different lighting, captured with different noise components, printed in different locations, and so forth). As users of mobile devices who participate in rewards programs may be quite large, further training of the neural network <NUM> through "active learning" while in production (see <FIG> and <FIG>) may be performed rapidly using a large set of additional training codes being submitted, thereby leading to a highly accurate neural network. At step 500d, a test of the accuracy of the model may be performed by using test pin code images, such as pin code image <NUM>, in which the neural network <NUM> may produce an output with a confidence percentage <NUM> that may be used to determine how confident the character recognition model is with regard to one or more characters of the imaged code.

With regard to <FIG> and <FIG>, a set of illustrative steps 600a-<NUM> that may be used in supporting a promotional rewards system and including operating a character recognition model for imaging and identifying codes in an image during production is shown. At step 600a, a mobile device <NUM> may be configured to display a user interface <NUM> that includes an image captured by a camera of the mobile device <NUM>. As shown on the user display <NUM>, an imaged bottle cap <NUM> includes a printed code <NUM>. In an embodiment, an image region <NUM> may be displayed with a graphical representation such that the user knows to position the code <NUM> within the image region <NUM> for automatic imaging. At step 600b, after the code <NUM> is imaged, the app being executed on the mobile device <NUM> may be configured to generate image segments that include regions 612a-612n (collectively <NUM>) in which imaged characters 614a-614n (collectively <NUM>) are included. Alternatively, the imaged bottle cap <NUM> may be sent to a server to be segmented by an image processing application. As shown in step 600c, the image segments are processed by a neural network <NUM> configured as a character recognition model for determining characters and providing match probabilities associated with each of the determined characters <NUM>. The neural network <NUM> may be executed on the mobile device or a remote server.

As shown, the determined characters and match probabilities <NUM> may be performed multiple times so that an overall probability score (e.g., average probability) for each character may be determined. Once the overall probability scores are determined, a determination as to whether those individual probability scores are above a minimum probability score threshold level is made. If a probability score is above the minimum probability score threshold level, then the associated character is determined to be known. If an overall probability score is below the minimum threshold probability value, then a determination that the associated character is unknown or has some level of uncertainty, then the app may display the associated character for a user to correct and/or confirm. A character sequence <NUM> may be output for use in providing the user with a value associated with the code as part of a loyalty program. Other uses of the code may additionally and/or alternatively be provided to the user.

At step 600e, an active learning process for recursive training may be supported, where the determined code <NUM> or pin code text may be validated by communicating the text <NUM> via a pin code API <NUM> for validation of the code <NUM>. In an embodiment, a checksum algorithm may be used to validate the code <NUM> without communicating to an API. At step 600f, a determination <NUM> may be made as to whether the pin code <NUM> is valid. If so, then no further processing may be performed at step <NUM>. Valid code predictions and related images may be stored on a mobile device and communicated back to a network server (not shown) for further training of a neural network. In an embodiment, a validated code or flag indicating receipt of a valid code may be sent to a rewards server along with a user ID/account ID such that a reward may be posted to the user's account, entered into a sweepstakes, or other action may be taken. Otherwise, the process may present the user interface <NUM> to the user of the electronic device <NUM> with text boxes 628a and 628b. In one embodiment, the user interface <NUM> may enable a user to selectively enter one or more characters of the pin code <NUM>. In an embodiment, the user may be requested to submit each of the characters of the code. At step <NUM>, the corrected pin code as entered into the text boxes 628a and 628b may be communicated back to a network server (not shown) for further training of a neural network <NUM>.

With regard to <FIG>, a screenshot of an illustrative user interface <NUM> in which characters <NUM> of a code may be displayed is shown. Characters 704a and 704b that have a probability less than a minimum probability threshold value, such as <NUM>%, may be displayed with a highlight (e.g., red, bold, etc.) or shown as blank characters or spaces with underscores, for example. The user confirms, adds, and/or replaces the characters of lower probability to correct the characters of the code. It should be understood that any of the characters <NUM> may be replaced, but certain characters 704a and 704b that have low probabilities of being correct are highlighted to draw attention of the user. Once complete, the user may select a "confirm" soft-button 706a. Alternatively, the user may proceed a "back" soft-button 706b. In an embodiment, the user interface <NUM> may prevent the user from continuing until the characters having low probabilities have been submitted by the user.

With regard to <FIG>, a flow diagram of an illustrative process <NUM> that provides additional detail for imaging and image processing an imaged code on a bottle cap, carton, or otherwise is shown. Generally, the process <NUM> may identify the type of media on which the code is printed, including a bottle cap or fridge pack, for example. Additionally, the process <NUM> may determine that the code is not identified so as to avoid failure of the process <NUM>. The process <NUM> may be performed as described in each of the flow paths. In particular, colors may be identified and/or normalized so as to improve the ability to identify and read pin code characters.

More specifically, the process <NUM> may include receiving in input image <NUM> that is captured from a mobile device, as previously described. At step <NUM>, a determination as to what type of object is detected may be made using three different flow-paths 805a-805c. As an example, the types of objects may include a bottle cap, fridge pack, or nothing. It should be understood that other types of objects may also be detected at step <NUM>, as well.

In one embodiment, in detecting whether the object is a bottle cap, the process <NUM> may perform image processing to detect circles at step <NUM>. In detecting whether any circles exist, a determination may be made as to whether any circles surround a code, which is indicative of a code being printed on the inside surface of the bottle cap, as shown in <FIG>. The circle detection step <NUM> may further include (i) resizing an image to a standard size (e.g., resizing the circle to be sized to be a common size as circles in other images), (ii) performing a grey-scaling, and (iii) removing noise. A detect Hough circle analysis may be performed to process the image, and the actual image may be cropped to isolate the bottle cap.

At step <NUM>, a color detection may be performed. In perform the color detection, a color feature computation may be performed by passing the colors through a trained TensorFlow MLP color detection model to detect color of the bottle cap. Images that are sent to rectangle detection may be converted to a lighter background and darker font.

In response to determining that the cap is black/gray, pixel values may be inverted prior to detecting a bottle cap rectangle at step <NUM>. Other colored caps may not have the pixel values inverted. At step <NUM>, in detecting a bottle cap rectangle, illumination in the grey image may be flattened. Additionally, to obtain a clear binarized image, a threshold value may be utilized so that light and dark colors may be distinguished (e.g., below the threshold brightness level is dark, and above the threshold brightness level is light). At step <NUM>, a text detection model may be performed. In performing the text detection model, the image may be normalized and resized. The adjusted image may be passed through a trained tensor flow CNN model to identify text from the cropped, binarized images. At step <NUM>, character predictions may be performed to predict a value of one or more characters being processed.

In determining whether the object is a fridge pack in flow path 805b, a paperboard rectangle detection may be performed at step at <NUM>. In performing the paperback rectangle detection at step <NUM>, the image may be resized to a standard size. The image may also be gray-scaled, and an adaptive threshold may be applied to produce a binarized image. Additionally, small islands and large blobs of noise may be removed to determine actual text area. The image may be cropped thereafter. At step <NUM>, a text detection model may be applied to the adjusted image, and predictions of characters in the image may be made of step <NUM>.

If no object is detected at step <NUM>, then the process <NUM> may follow flow path 805c and simply return a response at step <NUM> that no characters and/or codes were identified in the image.

With regard to <FIG>, a flow diagram of an illustrative process <NUM> for performing an end-to-end pin code recognition model training process is shown. The process <NUM> may provide for generation of labeled images at step <NUM>, training (loss minimization) at step <NUM>, and performing a validation of test images at step <NUM>.

More particularly, in generating labeled images at step <NUM>, a plain background image may be created at step <NUM>. At step <NUM>, random strings of text may be generated with randomly selected fonts on a plain image. At step <NUM>, the plain image with text may be pasted on a randomly selected background with varied rotation, and different types of noise may be applied to the image at step <NUM>. The different types of noise may be applied to improve robustness of the model. At step <NUM>, a threshold may be applied to binarize the entire image.

After generating labeled images at step <NUM>, the training (loss minimization) may be performed at step <NUM>. In performing the training, training parameters (e.g., learning rate, initial weights, batch size, etc.) may be declared at step <NUM>. At step <NUM>, the generated image batch may be read followed by resizing and normalization of the images. At step <NUM>, the images and associated pin code strings may be fed to the convolutional neural network (CNN) model for processing thereby. Weights of the neural network model may be updated after each loss minimization at step <NUM>, and at step <NUM>, the updated weights may be saved after a certain number of iterations. The updated weights after each loss minimization may be applied to a model architecture <NUM> that is used for the neural network, as understood in the art. In an embodiment, hyperframes, as understood in the art, that utilizes a number of iterations and loss function, may be utilized. The training set may also include images of real pin codes printed on consumer packaging. The images may be created by users during a training or active learning process.

The validation on test images process that step <NUM> may include the steps of tracking the neural network model performance through a character accuracy curve at step <NUM>, and reporting prediction accuracy on test images after a certain number of iterations at step <NUM>. The process <NUM> may also include extracting the model with embedded weights at step <NUM>, and quantizing the model to reduce file size at step <NUM>.

As will be appreciated by one of skill in the art, the steps in the foregoing embodiments may be performed in any order. Words such as "then," "next," etc. are not intended to limit the order of the steps; these words are simply used to guide the reader through the description of the methods. Although process flow diagrams may describe the operations as a sequential process, many of the operations may be performed in parallel or concurrently. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed here may be implemented as electronic hardware, computer software, or combinations of both.

Embodiments implemented in computer software may be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof. A code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to and/or in communication with another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc..

The actual software code or specialized control hardware used to implement these systems and methods is not limiting of the invention. Thus, the operation and behavior of the systems and methods were described without reference to the specific software code being understood that software and control hardware can be designed to implement the systems and methods based on the description here.

When implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable or processor-readable storage medium. The steps of a method or algorithm disclosed here may be embodied in a processor-executable software module which may reside on a computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another. A non-transitory processor-readable storage media may be any available media that may be accessed by a computer. By way of example, and not limitation, such non-transitory processor-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer or processor. Disk and disc, as used here, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a non-transitory processor-readable medium and/or computer-readable medium, which may be incorporated into a computer program product.

Claim 1:
A computer implemented method of reading a multi-character code, said method comprising
in response to receiving an image of the multi-character code, identifying regions in which respective characters of the code reside;
applying the identified regions to a neural network to determine the respective characters in the identified regions;
displaying the determined characters in an ordered sequence for a user to visually inspect to confirm that each of the determined characters are correct,
characterized wherein
displaying the determined characters includes displaying the determined characters with character recognition probability levels over a threshold probability level as determined by the neural network;
displaying one or more characters of respective one or more characters determined to have character recognition probability levels below the threshold probability level in a format different from the characters determined to have character recognition probability levels over the threshold probability level so as to notify the user to supply one or more corrected characters of the code displayed in the different format;
enabling the user to enter the one or more corrected characters for each of the characters in a different format; and
responsive to receiving the one or more corrected characters from the user, applying the one or more corrected characters to a training set of characters to train the neural network.