Methods, systems and processor-readable media for designing a license plate overlay decal having infrared annotation marks

Methods, systems and processor-readable media for providing a license plate overlay decal with an infrared readable annotation mark for an optical character recognition and segmentation. The annotation mark with respect to character image of a license plate can be designed by training an ALPR engine to improve automatic license plate recognition performance. A plate overlay decal can be rendered with the annotation mark and attached to a license plate. The annotation mark can also be directly placed on the license plate when the license plate is rendered. The annotation mark is visible when illuminated by an infrared light and the license plate appears normal in visible light. The annotation mark enables an ALPR imaging system to obtain more information for each character and utilize the information to improve conclusion accuracy.

FIELD OF THE INVENTION

Embodiments are generally related to ALPR (Automatic License Plate Recognition) systems and methods. Embodiments are also related to image processing systems and methods. Embodiments are additionally related to license plate overlay decals.

BACKGROUND

ALPR (Automatic License Plate Recognition) system often functions as the core module of “intelligent” transportation infrastructure applications. License plate recognition can be employed to identify a vehicle by automatically reading a license plate utilizing an image processing and character recognition technology. A license plate recognition operation can be performed by locating the license plate in an image, segmenting the characters in the plate, and performing an OCR (Optical Character Recognition) operation with respect to the characters identified. Performance requirements for an ALPR engine are increasing over time driven by users' desire to reduce the number of recognition errors. One common source of errors is a close-character sibling error where certain characters look very similar to each other and in the presence of image noises, lead to incorrect classification by the ALPR engine. Examples include 8/B, 0/D, 2/Z, 5/S, etc. These most often occur on the letter-number boundary where numbers look like letters and vice versa. If ‘O’ or ‘Q’ is part of the OCR dictionary, then the confusion is typically between ‘O’, ‘Q’, and ‘D’.

The OCR subsystem typically accepts individual character images as input and runs algorithms to classify the image as one of 32-36 (depending on jurisdiction) class labels such as A-Z and 0-9.FIGS. 1A and 1Billustrate the performance of a trained OCR engine when tested on an independent test set of characters. The font excludes ‘IOQU’ and thus there are only 32 possible classes. The ground truth labels are on the X-axis and the OCR conclusion is on the Y-axis. This matrix is called a “confusion matrix” since the characters that are confused with each other by the system become readily apparent.

For a perfect OCR engine, all off diagonal entries of the confusion matrix are zero. InFIGS. 1A and 1B, the two highlighted cases of OCR errors are ‘B’ incorrectly recognized as ‘8’ (‘B/8’) 15 times and ‘D’ incorrectly recognized as ‘0’ (‘D/0’) a total of 16 times. These are the expected dose character errors and the combinations change depending on the font. Of note is that the reverse combinations (‘8/B’) and (‘0/D’) have 2 and 1 errors respectfully indicating an asymmetry in confusion. Example character images extracted from the license plate images are shown inFIG. 2. Under poor imaging conditions, a blurry ‘D’ loses its top and bottom left corner distinctions and begins to look very similar to a ‘0’. Similarly, a blurry ‘B’ appears and is often incorrectly classified as an ‘8’ whereas a blurry ‘0’ or ‘8’ still looks like ‘0’ and an ‘8’ are seldom misclassified. This explains the asymmetry in error rates observed inFIGS. 1A and 1B.

Majority of prior art methods utilize a higher resolution camera and increased illumination to distinguish between the close-characters siblings. Such methods require a large amount of expert hand tuning of camera setup parameters in order to achieve and maintain image quality levels enabling the ALPR engine to distinguish between the dose character siblings. License plate image signatures or image hashes have also been employed to augment the ALPR engine to improve the accuracy in license plates that have close-character siblings. Such approach requires manual plate recognition the first time each plate is checked by the system, an accurate automatic tight cropping of each license plate, access to a continuously updated central database, and a completely separate processing pipeline.

Another prior art approach places a barcode on the license plate in order to facilitate automatic reading of the plate, however, this approach occupies more valuable space on the plate and can lead to less visually appealing plates. Additionally, the barcode occupies significant space in order to make the barcode robust to the imaging noises present in the license plate reading systems and in order to carry its data payload which needs to include the plate character sequence and the state information. The bar code may also be visually unappealing depending on the technology.

FIG. 3illustrates license plate image180illustrating character segmentation in determining the first and last characters as the spacing of characters is arbitrary. InFIG. 3, the plate cover to the left of the ‘N’ and to the right of the ‘H’ depicted by arrows190and195can be mistakenly segmented and classified as a ‘1’ adding either one or two characters to the plate code. Additionally, the location of the logo can vary in size, location, and shape from state-to-state and across different plate designs are erroneously identified as a valid segmented character or by “attaching” itself to one of its neighboring characters in the segmentation results.

Based on the foregoing, it is believed that a need exists for an improved method and system for providing a license plate overlay decal with an infrared annotation mark for optical character recognition and segmentation, as will be described in greater detail herein.

SUMMARY

It is, therefore, one aspect of the disclosed embodiments to provide for improved automatic license plate recognition methods, systems and processor-readable media.

It is another aspect of the disclosed embodiments to provide for methods, systems and processor-readable media for recognizing a license plate close-character sibling.

It is yet another aspect of the disclosed embodiments to provide for methods, systems and processor-readable media for providing a license plate overlay decal with an infrared annotation mark for optical character recognition and segmentation.

The aforementioned aspects and other objectives and advantages can now be achieved as described herein. Methods, systems and processor-readable media for providing a license plate overlay decal with an infrared readable annotation mark for an optical character recognition and segmentation are disclosed herein. The infrared readable annotation mark with respect to character image of a license plate can be designed by training an ALPR engine to improve automatic license plate recognition performance. A plate overlay decal can be rendered with the annotation mark and attached to a license plate. The annotation mark can also be directly placed on the license plate when the license plate is rendered. The annotation mark is visible when illuminated by an infrared light and the license plate appears normal in visible light. The infrared readable annotation mark enables an ALPR imaging system to obtain more information for each character and utilize the information to improve conclusion accuracy.

The ALPR engine can be trained and tested utilizing the character mages. The character images can be, for example, a labeled ground truth character image and/or a synthetic character image with modeled image noise imposed on the image. A confusion matrix can be generated in order to determine a dose character sibling. The IR-readable annotation mark with respect to the close-character siblings can be defined and the ALPR engine can be retrained utilizing the character images including the annotation mark. The annotation mark improves the accuracy of the ALPR engine so that the annotation mark fits in the ALPR training paradigm without special processing at run time.

The annotation mark enables the OCR engine to differentiate the dose character sibling even under poor imaging condition. The close-character sibling can be determined for a variety of font sets, plate designs, and image noises found under modeled conditions. The synthetic plate generation and synthetic noise generation process can be iterated to optimize the additional IR annotation mark resulting in very low off-diagonal terms in the confusion matrix. The noise can include blurring, geometric distortion, random noise, and common occlusions.

A boundary of the license plate character and a cut point can be determined utilizing the annotation mark such that each character can be coherently cropped out of the license plate image to improve the character segmentation accuracy. A logo on the license plate can be clearly identified utilizing the annotation marks to significantly improve the character segmentation accuracy. The presence of the overlay decal can be easily determined based on the presence or absence of the annotation mark. A warning can be issued by an electronic tolling system upon entry and/or exit of a toll road way if a user has not attached the overlay decal.

DETAILED DESCRIPTION

As will be appreciated by one skilled in the art, the present invention can be embodied as a method, data processing system, or computer program product. Accordingly, the present invention may take the form of an entire hardware embodiment, an entire software embodiment or an embodiment combining software and hardware aspects all generally referred to herein as a “circuit” or “module.” Furthermore, the present invention may take the form of a computer program product on a computer-usable storage medium having computer-usable program code embodied in the medium. Any suitable computer readable medium may be utilized including hard disks, USB Flash Drives, DVDs, CD-ROMs, optical storage devices, magnetic storage devices, etc.

Computer program code for carrying out operations of the present invention may be written in an object oriented programming language (e.g., Java, C++, etc.). The computer program code, however, for carrying out operations of the present invention may also be written in conventional procedural programming languages such as the “C” programming language or in a visually oriented programming environment such as, for example, Visual Basic.

The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer. In the latter scenario, the remote computer may be connected to a user's computer through a local area network (LAN) or a wide area network (WAN), wireless data network e.g., WiFi, Wimax, 802.xx, and cellular network or the connection may be made to an external computer via most third party supported networks (e.g., through the Internet using an Internet Service Provider).

As illustrated inFIG. 4, the disclosed embodiments may be implemented in the context of a data-processing system200that includes, for example, a central processor201, a main memory202, an input/output controller203, a keyboard204, an input device205(e.g., a pointing device such as a mouse, track ball, and pen device, etc.), a display device206, a mass storage207(e.g., a hard disk), an image capturing unit208, and a USB (Universal Serial Bus) peripheral connection. As illustrated, the various components of data-processing system200can communicate electronically through a system bus210or similar architecture. The system bus210may be, for example, a subsystem that transfers data between, for example, computer components within data-processing system200or to and from other data-processing devices, components, computers, etc.

FIG. 5illustrates a computer software system250for directing the operation of the data-processing system200depicted inFIG. 4. Software application254, stored in main memory202and on mass storage207, generally includes a kernel or operating system251and a shell or interface253. One or more application programs, such as software application254, may be “loaded” (i.e., transferred from mass storage207into the main memory202) for execution by the data-processing system200. The data-processing system200receives user commands and data from a user249through user interface253; these inputs may then be acted upon by the data-processing system200in accordance with instructions from operating system module252and/or software application254.

The following discussion is intended to provide a brief, general description of suitable computing environments in which the system and method may be implemented. Although not required, the disclosed embodiments will be described in the general context of computer-executable instructions such as program modules being executed by a single computer. In most instances, a “module” constitutes a software application.

The interface253, which is preferably a graphical user interface (GUI), also serves to display results, whereupon the user may supply additional inputs or terminate the session. In an embodiment, operating system251and interface253can be implemented in the context of a “Windows” system. It can be appreciated, of course, that other types of systems are possible. For example, rather than a traditional “Windows” system, other operation systems such as, for example, Linux may also be employed with respect to operating system251and interface253. The software application254can include a license plate overlay decal generating module252that includes an infrared readable annotation mark370to improve license plate recognition and segmentation by reducing confusion between similar characters (0 & D, 8 & B) and to improve character location and plate boundary. Software application254, on the other hand, can include instructions such as the various operations described herein with respect to the various components and modules described herein such as, for example, the methods700,800, and900depicted inFIGS. 10,11and12.

FIGS. 4-5are thus intended as examples and not as architectural limitations of disclosed embodiments. Additionally, such embodiments are not limited to any particular application or computing or data-processing environment. Instead, those skilled in the art will appreciate that the disclosed approach may be advantageously applied to a variety of systems and application software. Moreover, the disclosed embodiments can be embodied on a variety of different computing platforms including Macintosh, UNIX, LINUX, and the like.

FIG. 6illustrates a block diagram of a license plate overlay decal generating system300that includes the infrared readable annotation mark370, in accordance with the disclosed embodiments. Note that inFIGS. 4-12, identical or similar blocks are generally indicated by identical reference numerals. The license plate overlay decal generating system300generally includes an image-capturing unit315(e.g., camera) for capturing an image of a vehicle305within an effective field of view. The image-capturing unit315provides an image of a license plate320mounted on the rear of the vehicle305. Note that the image capturing unit315is capable of reading the license plate on the front of the vehicle305as well as the rear. The image capturing unit315can be operatively connected to an image processing unit325via a network330.

Note that the network330may employ any network topology, transmission medium, or network protocol. The network330may include connections such as wire, wireless communication links, or fiber optic cables. Network330can also be an Internet representing a worldwide collection of networks and gateways that use the Transmission Control Protocol/Internet Protocol (TCP/IP) suite of protocols to communicate with one another. At the heart of the Internet is a backbone of high-speed data communication lines between major nodes or host computers consisting of thousands of commercial, government, educational, and other computer systems that route data and messages.

The image capturing unit315can be operated as a hand-held device and/or a vehicle-mounted device. Note that the image capturing unit315described in greater detail herein is analogous or similar to the image capturing unit208of the data-processing system200, depicted inFIG. 4. The image capturing unit315may include built-in integrated functions such as image processing, data formatting, and data compression functions. Also, the unit includes imager-positioning, range-finding, and a flash bulb.

The license plate overlay decal generating system300further includes the image processing unit325to receive the captured digital image320from the image capturing unit315in order to process the image320. The image processing unit325is preferably a small, handheld computer device or palmtop computer as depicted inFIG. 4that provides portability and is adapted for easy mounting. The image processing unit325can be configured to include the license plate overlay decal generating module252to accurately recognize the license plate character of the vehicle305.

The license plate overlay decal generating module252includes an ALPR engine335that can be trained and tested utilizing character images340. The character images340can be, for example, a labeled ground truth character image355and/or a synthetic character image360with modeled image noise imposed on the image. The ALPR engine335generates a confusion matrix345in order to determine a close character sibling385. The annotation mark generation unit350generates the IR-readable annotation mark370with respect to the close-character siblings385.

The license plate overlay decal generation module252retrains the ALPR engine335utilizing the character images340including the annotation mark370. The annotation mark370improves the accuracy of the ALPR engine335so that the annotation mark370fits in the ALPR training paradigm without special processing at run time. The annotation mark370determines a boundary380of the license plate character and a cut point such that each character can be coherently cropped out of the license plate image320to improve accuracy of the license plate character segmentation module390. A logo375on the license plate320can be clearly identified utilizing the annotation marks370to significantly improve segmentation accuracy.

A high level of ALPR performance can be achieved by using the synthetic character image360with modeled image noise. Note that the noise can include blurring, geometric distortion, random noise, and common occlusions. In addition, the close-character siblings385can be determined for a variety of font set, plate design, and image noises found under modeled conditions. The license plate overlay decal generation module252can be iterated to optimize the additional IR annotation marks370resulting in very low off-diagonal terms in the confusion matrix345and highly accurate ALPR performance with respect to characters that can be confused with each other. The annotation mark370enables an OCR module395to differentiate the close character sibling385even under poor imaging condition.

The license plate overlay decal generation module252determines the presence of the overlay decal based on the presence or absence of the annotation mark370. A warning can be issued by an electronic tolling system upon entry and/or exit of a toll road way if the user has not attached the overlay decal. The infrared readable annotation mark370reduce confusion between similar characters (0 & D, 8 & B) and improves the segmentation of a character and a plate boundary. A plate overlay decal can be rendered with the annotation mark370and attached to the license plate320. The annotation mark370can also be directly placed on the license plate320when the license plate320is rendered. The annotation mark370is visible when illuminated by an infrared (IR) light and the license plates320appear normal in visible light. The annotation mark370in IR enables an ALPR imaging system to obtain more information for each character and utilize the information to improve conclusion accuracy.

FIG. 7illustrates similar character images400with annotation marks405and410extracted from a license plate image, in accordance with the disclosed embodiments. The character images400with the annotation marks405and410can be captured when illuminated by infrared light. Note that the embodiments discussed herein should not be construed in any limited sense. It can be appreciated that such embodiments reveal details of the structure of a preferred form necessary for a better understanding of the invention and may be subject to change by skilled persons within the scope of the invention without departing from the concept thereof.

For example, the annotation mark370to differentiate between 0 and D is to change ‘0’ into an upside down ‘Q’ and to differentiate8and B is to fill in the upper oval of the ‘8’. When viewed under visible light, the license plate characters look likeFIG. 2to enable police officers and users to recognize the plate and assist the ALPR image capturing unit315to be more effective in accurately recognizing the code. Note that the plate under visible light looks like it possesses standard characters, while the plate under infrared light has characters with the annotation marks370which aid the ALPR engine335to distinguish between close-character siblings385.

FIGS. 8-9illustrate a license plate image500and550with the annotation marks510,520, and530that can be employed to aid in the segmentation of individual characters, in accordance with the disclosed embodiments. The IR readable annotation mark510and520can be added to the boundary of the license plates500and550to differentiate license plate boundaries as depicted inFIG. 8. The IR readable annotation mark510,520, and530can be added to the license plates500and550to aid in the segmentation of individual characters as depicted inFIG. 9. Similarly, the logo375on the license plate can be clearly identified utilizing the IR readable marks370. The IR readable marks370clearly identify the locations of the logos375or other special symbols on the plate to significantly improve segmentation accuracy.

FIG. 10illustrates a high level flow chart of operations illustrating logical operational steps of a method700for providing annotation marks370for actual tolling images, in accordance with the disclosed embodiments. Note that the annotation marks370can simply be a table of additional line strokes depending on the character, or depending on the two close-character siblings385. The ALPR engine335can be trained utilizing labeled ground truth character images355, as shown at block710. The ALPR engine335can be tested utilizing the labeled ground truth character images355, as described at block720.

The confusion matrix345can be generated, as mentioned at block730. The close-character siblings385can be determined, as depicted at block740. The annotation marks370can be defined for the close-character siblings385, as indicated at block750. The ALPR engine335can be retrained utilizing labeled ground truth character images355including the annotation marks370, as shown at block760. The license plate overlay decals can be rendered with the annotation marks370, as mentioned at block770and the license plate overlay decals can be issued to patrons, as described at block780.

FIG. 11illustrates a high level flow chart of operations illustrating logical operational steps of a method800for providing the annotation marks370for synthetic plate images with appropriate modeled noises360imposed on the plate images, in accordance with the disclosed embodiments. The synthetic plate images360can be employed for new applications or even for new plate design with new fonts. The synthetic plate images360can be employed for plate characters that are rendered from the actual font. The ALPR engine335can be trained utilizing synthetic character images with modeled image noises360imposed on the images, as shown at block810.

The ALPR engine335can be tested utilizing synthetic character images360, as described at block820. The confusion matrix345can be generated, as mentioned at block830. The close-character siblings385can be determined, as depicted at block840. The annotation marks370can be defined for the close-character siblings385, as indicated at block850. The ALPR engine335can be retrained utilizing synthetic character images360with modeled image noises and the annotation marks370, as shown at block860. The license plate overlay decals can be rendered with the annotation marks370, as mentioned at block870and the license plate overlay decals can be issued to patrons, as described at block880.

FIG. 12illustrates a high level flow chart of operations illustrating logical operational steps of a method900for improving accuracy of an ALPR system utilizing the augmented IR readable annotation marks370, in accordance with the disclosed embodiments. The synthetic plate generation and synthetic noise generation process can be iterated to optimize the additional IR readable strokes resulting in a system with very low off-diagonal terms in the confusion matrix345and highly accurate ALPR performance with respect to characters that can be confused with each other. The ALPR engine335can be trained utilizing synthetic character images with modeled image noises360imposed on the images, as mentioned at block910. The ALPR engine335can be tested utilizing synthetic character images360, as described at block920. The confusion matrix345can be generated, as mentioned at block930.

A determination can be made whether the off-diagonal elements are large, as described at block940. The close-character siblings385can be determined if there exists large off-diagonal elements, as depicted at block950. Thereafter, the annotation marks370can be defined and adjusted for the close-character siblings385, as indicated at block960. The ALPR engine335can be retrained utilizing the synthetic character images with modeled image noises360and the annotation marks370, as shown at block970and the ALPR engine335can be tested utilizing the synthetic character images360, as described at block920.

If off-diagonal elements are small, the license plate overlay decals can be rendered with the annotation marks370, as mentioned at block980. The license plate overlay decals can be issued to patrons, as described at block990. The system300can be employed for license plate design and printing and to define the overlay decal that is attached to an already printed license plate that is in the possession of a patron, for example. Note that the annotation marks370can be employed with license plate fonts of any language.