Evaluating form designs for optical character recognition (1994)

The National Institute of Standards and Technology, under the sponsorship of the Internal Revenue Service, has conducted an extensive study of three different redesigned tax forms.The NIST Model Recognition System was used in conjunction with the NIST Scoring Package to generate performance measures at the form, field, and character levels.The analyses of these measures conclude that factors introduced onto forms by the writer are the primary cause of segmentation errors, which are the major source of errors within the recognition system.One configuration of the recognition system achieved a 10% character error rate across 13,316 fields containing money amounts.Of these errors, 83% are attributed to segmentation errors (deleted and inserted characters).Analysis shows that 97% of these segmentation errors can be attributed to factors introduced by the writer.Anomalous behavior referred to as human factors include such things as leaving a field blank that requires a value, completing a field with an incorrect value, and crossing out previously written characters.The recognition system achieved a 2.8% character error rate when the fields containing these human factors were removed firom the performance analysis.This paper cites three ways in which these types of human factors can be handled so as to increase recognition performance.First, the algorithms and tech- niques deployed within the system can be improved.One configuration of the recognition system initially achieved a 31% character error rate with a 33% field error rate when reading count fields and Social Security Number fields.A new spatial normalization technique was developed, and when integrated, the system achieved a 24% character error rate with a 26% field error rate, for a gain of 7%.Second, the instances of human factors leading to system errors can be detected.Third, writers can be influenced by the design of the form including the layout and structure of the fields.One configuration of the recognition system achieved a 20% character error rate with a 20% field error rate on 14,336 money fields in which there are no inter-character markings on the form to denote proper character spacing.The same recognition system achieved an 11% character error rate for a gain of 9% with a 12% field error rate on 13,3 16 money fields in which the position of each character within the field is denoted by a separately spaced bounding box.The best performance achieved on alphabetic fields was a 45% character error rate with a 43% field error rate.By applying a combination of these three approaches, human factors can be dealt with, and the errors made by a form processing system can be effectively reduced to classification errors.compounded reducing the throughput <rf automated processing and inareasing the Miount of manual labor required.In addition, complex forms will frustrate an already unmotivated tax filer.Legal records require thoroughness.However, if too much informaticHi is contained on a form, then the writer wiU be cramped for space and the quality of his writing win degrade, increasing OCR errors.Field separation will also become ambiguous.Automated recognition of handprint has been the topic of much lesearch.^^"^^hiMay of 1992, the First Cen- sus Optical Character Recognition Systems (COCR) Conference sponswed by the Bureau of the Census was run by NIST.^^IheConference compared the results from 45 different systems submitted by 26 participants r^resenting organizations from the private sector, academia, and government ftoperly sejpnented images of individual hand- printed characters were recognized and tiie results reported.It was demonstrated that error rates as low as 3% could be achieved on large samples of digits without rejecting any classificatiais.firor rates as low as 5% to 6% were demonstrated on uppercase letters; errca* rates of 10% to 15% were demonstrated for lowercase letters.The results from the COCR Conference show Optical Character Recognition (OCR) of handprinted information to be an economically viable technology.Unfortunately, few real applications can be reduced to only recognizing well segmented and isolated characters.Many OCR applications requhe elements of document understanding and form processing.This paper addresses die latter, the processing of field information entered onto forms.In this domain, complex and intelligent processing is required to get to the point of classifying isolated character images.Steps includ- ing form identification, form registration, form removal, field isolation, and field segmentation must be conducted pricnr to classifying the characters in each field.Each one of these steps adds complesdty and the potential for error to a form processing system, hi theory, die results demonstrated in the COCR Conference are achievable, but in practice, automated form processing systems will not deliver error rates this low.The study presented in this paper documents three approaches that permit m automated fcHm processing system to achieve a level of performance similar to the COCR Conference results.First, the algraithms and techniques deployed within the reception system cm be improved.For example, neural network-based classifiers can be retrained to improve accuracy, and new filtering techniques can be developed to increase system tolermce to image noise and writing vteiatians.Chie configuration of the recognition system initiaUy achieved a 3 1% character error rate with a 33% field error rate when reading count fields md Social Security Number fields.This system configuration utilizes a segmentor based on cutting the chmacters printed withm a field along inter-character spaces defined by field marldngs on the form.Upem closer inspection, it was determined ^t pi^es of neighboring characters were bemg included in each segmented character image, and these extraneous pieces where causing severe image distortions when the characters were spatially noim^ized.A new spatial normalization technique was developed that essentially i^ores these extraneous character fra^ents.Whm integrated, the system achieved a 24% ch^acter error rate with a 26% field error rate.In this case, the 7% gain in performance is substantial.One configuration of die recognition system achieved a 10% character error rate across 13,3 16 fields contain- ing money mnoimts from P3 forms.Of these errors, approximately 83% are attributed to segmentation errors (deleted and inserted characters).TTie analysis in Section 5 shows that 97% of these segmentation errcffs can be attributed to anomalous behavim exhibited by the writer.Ihese anomalies are referred to as human factors and are shown in Figure 6.Another hmnan faaca: not shown in the figure is a writo' leaving a field blank when it requires a value.These results suggest that factors inttoduced onto forms by the writer are tiie primary cause of segmentation errors, which are the major source of errors within the recognition system.Therefore, it is expected that the performance of this system cmfiguration can be dramatically improved by hnprovk^the segmentation dgorithms used.Unfortunately, the impact of an algorithmic improvement decreases as the overall performmee of the system increases, and improvements as large as those seen with the new spatial normalizer are unhkely.A robust segmentation solution can be seen as an n-dimensional problem in which the solution sp^encompasses as many writer and char- acter variations as possible.These variations are unbounded, so unique solutims are developed that encompass only portions erf this multi-dimensional space based on algorithm constraints and Umitatiems ttiat attempt to cluster similar variations together.To improve upon an existing solution implies encompassmg new portions of the solution space.This results in a huge incremental change in the volume of coverage.Machine learning techniques are very useful in solving n-dimensional problems.UnfOTtunately, these techniques must define fids incremental change in volume through examples cxjntained in a training set The solution becomes intractable because, as the volume of coverage increases, the frequency witii which examples occur within this volume decreases.Other challenges to the recognition system are human factors that basically have no solution.If a writer leaves a field blank, enters the wrong information, ot crosses out a previously written field value, there is very little the recognition system can do to compensate for these events apart from applying some type of external context.It is con- ceivable that certain types of human factors, which are a major contributor to system errors, can be detected.This is the second approach to increasing recognition system performance.Fields containing detected instances of human fac- tors can be routed to a human operator for appropriate action so that system errors are reduced.This detection approach was simulated in the analysis in Section 4.2.The first money field.Line 7 under the Income column on the front page, was examined across every P3 form.Of 169 fields, 40 were determined to contain combinations of the human factors shown in Figure 6.When these fields were removed from the performance analysis, the recognition system achieved a 2.8% character error rate.The same recognition system achieved a 10% character error rate across all 13,3 16 money fields on the P3 forms.A 7.2% improvement in character error rate is demonstrated by simulating human factor detec- tion.A third way to increase the performance of an automated fonn processing system is to reduce the complexity of the form itself. Making a form more readable to a computer usually implies maximizing the space within fields so as not to cramp the writer, maximizing the space between fields so that the fields can be isolated easily, printing large registration marks on the form for deskewing the image, etc.The amount of information on the form is traded off for the machine readability of the form.To design a form properly, a compromise must be found between what amount of complexity the current technology is able to rehably handle and what amount of information is reasonable to include on a single form.AU this compromise must be made without negatively impacting the person filling out the form.The 1040T forms contains various types of fields structures.There are fields demarcated by a single horizon- tal baseline; other fields contain inter-character vertical tick marks along a baseline.Characters in Social Security Numbers (SSNs) and Employer Identification Numbers (EINs) are grouped by bounding boxes sharing neighboring sides with a vertical dashed line, and mark-sense fields are signified by circles.The three versions of the 1040T forms vary in how money fields are represented.On PI forms, money fields are signified by a single boundiug box that is to contain aU characters handprinted in tbe field.Punctuation marks such as commas and decimal points are provided on the form.The position of each character in a money field on a P2 form is demarcated by a separately space bounding box.The sides of neighboring boxes are not shared.P3 money fields are similar to P2 money fields, only each character box contains two vertically stacked ovals intended to guide the writer's shaping of characters.One configuration of the recognition system achieved a 20% character error rate with a 20% field error rate on the 14,336 money fields from the PI forms.The same recognition system achieved an 11% character error rate for a gain of 9% with a 12% field error rate on 13,316 money fields frcm P3 forms.In addition, the recognition system achieved only a 25% character error rate with a 25% field error rate across numeric PI fields comprised of baselines, baselines with vertical ticks, and SSN-type fields.These results clearly show that superior OCR results are obtained from fields in which the position of each diaracter within the field is denoted by a separately spaced bounding box.The character boxes used for SSNs and EINs do not sufficiently influence the writer.To effectively influence the writer, there must be noticeable spacing between the character boxes.This observation is supported by the performance results on P2 forms as well.In this case, the recognition system achieved a 12% character error rate with a 13% field error rate.This study shows that segmentaticHi errors plague the performance of form processing systems, and that human factors are the primary cause of segmentation errors.By applying a combination of these three approaches: improving algorithms and techniques, detecting human factors, and carefully redesigning forms, the errors made by a form processing system can be effectively reduced to classification errors, making the results from the COCR Confer- ence obtainable.The remainder of this report documents the details of the evaluation.Section 2 describes the database of 1040T forms and presents the performance assessment methods applied.Section 3 defines the six different config- urations of the Model Recognition System.Section 4 presents system configuration results across the three versions of forms in Section 4.1, and results for a select number of individual fields are reported in Section 4.2.Section 5 contains an analysis of segmentation errors, and conclusions are summarized in Section 6.This paper also contains a mnnber of appendices.Appendix A contains color copies of the three versions of 1O40T forms.Appendix B lists two sets of field values requested to be entered on the forms.Appendix C presents issues related to form-based scoring and eval- ': ,'.''' A. V-..