Federal Register Document Image Database

NIST Special Database 25 - Volume 1

NISTIR 6245

Michael D. Garris, Stanley A. Janet, and William W. Klein

National Institute of Standards and Technology

Building 225, Room A216

Gaithersburg, MD 20899
ACKNOWLEDGEMENTS

We would like to acknowledge the Department of Defense who provided funding and resources

in conjunction with NIST to support the development of this document image database.
i

TABLE OF CONTENTS

ABSTRACT..........................................................................................................................................................1

1. INTRODUCTION.......................................................................................................................................1

2. CREATING THE DOCUMENT IMAGE DATABASE.............................................................................4

2.1 THE FEDERAL REGISTER .............................................................................................................................4

2.2 CREATING THE DOCUMENT IMAGE DATABASE............................................................................................5

2.2.1 Scanning and Validating the Federal Register..................................................................................7

2.2.2 Running OCR on the FR Page Images..............................................................................................7

2.2.3 Parsing the GPO Typesetting Files ..................................................................................................8

2.2.4 Extracting Ground Truth by Matching OCR and GPO Text ..............................................................8

2.2.5 Uncertainty Issues and Quality Control..........................................................................................11

3. DISTRIBUTION HIERARCHY AND CONTENT..................................................................................13

3.1 PUBLIC DOMAIN SOFTWARE ....................................................................................................................14

3.2 DOCUMENT IMAGE DATA.........................................................................................................................15

3.2.1 GPO-Related Data Files................................................................................................................18

3.2.1.1 GPO Microcomp Files ..............................................................................................................................18

3.2.1.2 GPO SGML Files .....................................................................................................................................18

3.2.1.3 Master Word Index Files...........................................................................................................................21

3.2.2 Book-Related Data Files................................................................................................................23

3.2.2.1 Page Image Files.......................................................................................................................................23

3.2.2.2 OCR XDOC Files.....................................................................................................................................24

3.2.2.3 Header Cut Files.......................................................................................................................................25

3.2.2.4 Image Quality Assessment Files................................................................................................................25

3.2.2.5 OCR SGML Files .....................................................................................................................................26

3.2.2.6 OCR Word Index Files .............................................................................................................................27

3.2.2.7 Rejected Hits Plot .....................................................................................................................................28

3.2.2.8 Ground Truth Word Index Files ................................................................................................................29

3.2.2.9 Ground Truth Error Estimate Files ............................................................................................................29

3.2.2.10 Ground Truth SGML Files...................................................................................................................30

3.2.2.11 OCR Scoring Files...............................................................................................................................30

3.2.2.12 Ground Truth Error Estimate Plot.........................................................................................................32

4. SUMMARY ...............................................................................................................................................32

5. REFERENCES..........................................................................................................................................35

APPENDIX A. SOFTWARE REFERENCE GUIDE......................................................................................36

A.1 ORDER OF SOFTWARE INVOCATION ............................................................................................................36

A.2 MANUAL PAGES.........................................................................................................................................38

APPENDIX B. IHEAD IMAGE FILE FORMAT...........................................................................................76

APPENDIX C. IMAGE EXAMPLES..............................................................................................................80
ii

LIST OF TABLES AND FIGURES

TABLE 1. DISTRIBUTION DATA FILES. .....................................................................................................................17

TABLE 2. TAGS INCLUDED IN GPO SGML FILES. ...................................................................................................19

TABLE 3. ORDER IN WHICH GPO MICROCOMP FILES ARE PROCESSED......................................................................36

TABLE 4. ORDER IN WHICH OCR RESULT FILES ARE GENERATED. ...........................................................................37

TABLE 5. ORDER IN WHICH GPO FILES AND OCR RESULTS ARE PROCESSED TOGETHER. ..........................................37

FIGURE 1. OVERVIEW OF HOW THE DOCUMENT IMAGE DATABASE WAS CREATED.......................................................6

FIGURE 2. MATCHING THE OCR PAGE TEXT TO THE GPO BOOK TEXT. ......................................................................9

FIGURE 3. EXTRACTING A CHUNK OF GPO TEXT.......................................................................................................9

FIGURE 4. TRIMMING THE CHUNK OF GPO TEXT.....................................................................................................10

FIGURE 5. PLOT OF ACCEPTED/REJECTED HITS FOR THE FR BOOK PUBLISHED JANUARY 3, 1994. ..............................12

FIGURE 6. PLOT OF GROUND TRUTH ERROR ESTIMATES FOR FR PAGES ON JANUARY 3, 1994. ...................................13

FIGURE 7. TOP-LEVEL DISTRIBUTION HIERARCHY. ..................................................................................................14

FIGURE 8. DATA DISTRIBUTION HIERARCHY. ..........................................................................................................16

FIGURE 9. LOCATION OF DATA FILES. .....................................................................................................................17

FIGURE 10. FREQUENCY DISTRIBUTION OF TAGS (ACCUMULATED BY DAY) FOR JANUARY, 1994. .............................20

FIGURE 11. PORTION OF A GPO SGML FILE. .........................................................................................................21

FIGURE 12. PORTION OF A MASTER WORD INDEX FILE. ............................................................................................22

FIGURE 13. PORTION OF AN XDOC FILE.................................................................................................................24

FIGURE 14. PORTION OF AN OCR SGML FILE. .......................................................................................................27

FIGURE 15. PORTION OF AN OCR WORD INDEX FILE. ..............................................................................................28

FIGURE 16. PORTION OF A GROUND TRUTH WORD INDEX FILE..................................................................................29

FIGURE 17. PORTION OF A GROUND TRUTH SGML FILE. .........................................................................................31

FIGURE 18. EXAMPLE WORD SCORING REPORT. ......................................................................................................32

FIGURE 19. GROUND TRUTH ERROR ESTIMATES FOR ALL OF JANUARY. ....................................................................33

FIGURE 20. AN ILLUSTRATION OF THE IHEAD RASTER FILE FORMAT. .......................................................................76

FIGURE 21. IHEAD C LANGUAGE DEFINITION..........................................................................................................77

FIGURE 22. HARD COVER PAGE (DATA/01/03/BOOK01/IMG/A0000000.PCT). ............................................................80

FIGURE 23. INSIDE SOFT COVER PAGE (DATA/01/03/BOOK01/IMG/A0000001.PCT)....................................................81

FIGURE 24. PREFIX CONTENT PAGE (DATA/01/03/BOOK01/IMG/A0000003.PCT). ......................................................82

FIGURE 25. SECTION PAGE (DATA/01/03/BOOK01/IMG/B0000109.PCT). ...................................................................83

FIGURE 26. BODY DETAIL PAGE (DATA/01/03/BOOK01/IMG/B0000100.PCT).............................................................84

FIGURE 27. APPENDIX PAGE (DATA/01/03/BOOK01/IMG/C0000002.PCT)..................................................................85
1

Federal Register Document Image Database

NIST Special Database 25 (Vol. 1)

Michael D. Garris (mgarris@nist.gov),

Stanley A. Janet, and William W. Klein

ABSTRACT

A new, fully-automated process has been developed at NIST to derive ground truth for document

images. The method involves matching optical character recognition (OCR) results from a page

with typesetting files for an entire book. Public domain software used to derive the ground truth

is provided in the form of Perl scripts and C source code, and includes new, more efficient string

alignment technology and a word-level scoring package. With this ground truthing technology,

it is now feasible to produce much larger data sets, at much lower cost, than was ever possible

with previous labor-intensive, manual data collection projects. Using this method, NIST has

produced a new document image database for evaluating Document Analysis and Recognition

technologies and Information Retrieval systems. The database produced with this method

contains scanned images, SGML-tagged ground truth text, commercial OCR results, and image

quality assessment results for pages published in the 1994 Federal Register. These data files are

useful in a wide variety of experiments and research. There were roughly 250 issues, comprised

of nearly 69,000 pages, published in the Federal Register in 1994. This volume of the database

contains the pages of 20 books published in January of that year. In all, there are 4711 page

images provided, with 4519 of them having corresponding ground truth. This volume is

distributed on two ISO-9660 CD-ROMs. Future volumes may be released, depending on the

level of interest.

Keywords: CD-ROM, database, document, ground truth, image, information retrieval, OCR,

public domain software

1. INTRODUCTION

In 1997, the National Institute of Standards and Technology (NIST) in conjunction with the

Department of Defense (DoD), began a project known as the Metadata Text Retrieval

Conference (METTREC). The purpose of the project was to develop and evaluate the interface

between Document Analysis and Recognition (DAR) technology with Information Retrieval (IR)

technology. Specifically, the use of non-text items in a document (called metadata) was to be

explored and its impact on retrieval performance evaluated. A need for technology development

and evaluation in this area is much needed as the demand for large, automatically built, digital

libraries continues to increase. Unfortunately, the project was discontinued due to a lack of

interest and capability within the DAR and IR communities. This is discussed further in

Reference [1].

In order to conduct DAR/IR integrated experiments and technology evaluations, a large database

of prepared materials for training and testing is required. This paper documents the data that was

collected in preparation for the first round of METTREC evaluations. Even though there is no

longer a formal evaluation pending, the data produced for the project is valuable both to the

DAR and IR research communities. To this end, the data preparation was completed and

published on CD-ROM as a NIST Special Database 25.
2

This database fundamentally contains images of pages and their corresponding text from the

1994 Federal Register (FR). The United States Government Printing Office (GPO) prints the FR

each work day to record the transactions of the government. The FR is the official daily

publication for Rules, Proposed Rules, and Notices of Federal agencies and organizations, as

well as Executive Orders and other Presidential Documents. Each issue is printed and bound

into one or (on rare occasions) more books. Nearly 69,000 pages were published in the FR in

1994.

There are 20 daily FR issues (totaling 4711 pages) included in this volume of the database.

These pages were taken from the FR books published for the entire month of January in 1994.

Each image was scanned binary at 15.75 pixels per millimeter (ppm), equaling 400 pixels per

inch (ppi). With each image is a file that contains the textual content of the page. The images

and associated text can be used to evaluate alternative DAR technologies, and they can be used

in IR experiments where retrieval performance can be compared between retrieving the pages of

text provided with the database and retrieving the corresponding text produced by DAR. In this

way, the impact of recognition errors on retrieval performance can be evaluated (a key goal of

METTREC).

The text of each FR page in this database is formatted in Standard Generalized Markup

Language (SGML) [2]. This representation is commonly used in IR experiments and has been

used extensively in the TREC conferences [3]. Included in the ground truth text files is a set of

tags that represent a small collection of logical structures contained in the FR books. These

include such things as notices, tables, and footnotes. An SGML Document Type Definition

(DTD) is provided that defines these structures. Recipients of the database may find it useful to

augment these tags to suite there own needs, as this is a reasonable way to encode and utilize

metadata.

The database is 1.27 gigabytes in size and contains much more than just images and their

corresponding text. Optical character recognition (OCR) results generated from Xerox’s

ScanWorX API1

 are provided for each FR page in the database. These files are representative of

the performance of commercially available OCR technology. In addition to the text recognized

on each page, these files contain (among other things) rotation estimates of the page, word and

character confidence values, and word bounding boxes. All of these are useful for a variety of

experiments. The OCR application program used to generate this information is included in the

distribution for use by those who have ScanWorX API.

The results of an image analysis process developed at Los Alamos Laboratory [4] are also

provided for each image in the database. The quality of an image is automatically assessed

based on several factors, and these factors are used to predict an OCR word error rate. The

measured factors and the predicted OCR error rate are reported for each image in the database.

A comprehensive discussion of all the various types of data provided with this distribution is

provided in Section 3.2.

A document image database of this magnitude is every expensive to produce. Approximately 2.5

person-years were invested. Labor to scan in the entire 1994 FR was contracted to a scanning

 

1

 Specific hardware and software products identified in this paper were used in order to adequately support the

development of the technology described in this document. In no case does such identification imply

recommendation or endorsement by NIST, nor does it imply that the equipment identified is necessarily the best

available for the purpose.
3

service bureau. Each of the resulting images had to be validated (at NIST) as being a complete

image of reasonable dimensions (not blank or truncated) and verified to contain its

corresponding page. This machine-assisted validation process was conducted at NIST and is

documented in Reference [5].

While validating the images was necessary, the bulk of the labor was invested in creating an

automated method for generating the ground truth text for each FR page image. Original GPO

typesetting files, used to print each book of the 1994 FR, were obtained and are included in the

database in their entirety. Portions of these files have been used in previous TREC evaluations

[3]. The GPO files contain the text printed in each FR book along with binary typesetting codes.

No page boundaries are provided in these files; therefore, a method for locating and extracting a

specific FR page of text within the GPO files was developed. This method involves matching

noisy OCR results from the corresponding page image to the GPO-provided data. This process

is discussed in detail in Section 2.

Not only is the data produced by this method provided in this distribution, but a public domain

software implementation of this process is also provided in the form of Perl scripts and C

programs. This software precisely specifies how the ground truth text was derived, it enables the

method to be applied to other document collections, and it makes it possible for the recipient to

improve upon the method. With this new ground truthing technology, it is now feasible to

produce much larger data sets, at much lower cost, than was ever possible with previous laborintensive, manual data collection projects.

The organization of the software distribution and instructions for installing it are provided in

Section 3.1. Manual pages for the various programs and utilities are included in the reference

guide found in Appendix A.

At the heart of the ground truth production process is a robust and efficient string alignment

technology. Alternative alignment methods were explored, including those based on the

Levenstein distance [6] and an alignment algorithm developed by the University of Washington

[7]. These algorithms are implemented using dynamic programming and in general execute very

slowly and consume large amounts of memory. The University of Washington algorithm

apparently attempts to find an optimal alignment in a way that requires significantly more time to

compute as the discrepancies between the two strings increase. As a result, the use of these types

of algorithms becomes prohibitive when analyzing the recognition performance on low quality

documents. To compensate, NIST developed a new alignment technology based on a greedy

implementation of a recursive maximum substring matching algorithm. This approach is

designed to produce reasonable alignments between two different versions of a full page of text

in reasonably fast and relatively constant time regardless of the level of discrepancy.

This public domain alignment technology is provided in the distribution and is bundled within a

word-level OCR scoring package. Word-level scores are provided for each FR page image in the

database and were computed by matching the ScanWorX results of each page image with the

database’s corresponding derived ground truth text.

Quality assessment and control was key to automatically generating ground truth text. This was

extremely important because there was noise inherent throughout the process. The OCR text

contains varied degrees of errors depending on the page’s composition and image quality, and

the GPO files presented certain challenges onto themselves. To account for this, two levels of

quality assessment were imposed in the production of the ground truth text. First, a located
4

portion of GPO text (hypothesized to belong to a specific page image) was rejected if the

location didn’t fit trends derived from its neighboring pages. This was a binary accept/reject

decision.

Once the body of a page’s text was located in the GPO files, the page’s starting and ending

position had to be determined. The string alignment technology was applied and points where

alignment sufficiently degraded (at both front and end) were determined to be the page

boundaries. The accuracy of these selected page boundaries varies depending on the quality of

the OCR and the condition of the text in the GPO files at or near the actual page boundaries.

Due to the level of uncertainty involved in both the OCR and GPO texts, a method of estimating

the amount of error in the derived ground truth text was developed. An error estimate is

provided for each ground truth text file in the database. This is a conservative estimate that

allows researchers to select files for experimentation based on their own criteria and tolerance.

Both quality assessment steps are described further in Section 2.2.5.

Volume 1 of the Federal Register Document Image Database, NIST Special Database 25, is

distributed on 2 CD-ROMs in ISO-9660[8] data format. (This format is widely supported on

UNIX, VMS, DOS and Windows-based computers.) Information on how to order this database

is given in Section 4.

2. CREATING THE DOCUMENT IMAGE DATABASE

This section provides an overview of how the FR document image database was created. The

composition of the FR is first discussed, and then the process of generating the database is

presented. Example FR page images are shown in Appendix C.

2.1 The Federal Register

This database contains pages scanned from the 1994 FR. The GPO prints the FR each work day

to record the transactions of the government. Typically, the GPO publishes one book per day,

and each book contains three major sections:

Prefix Pages: Labeled in the database as ‘a’ files, these pages consist of a hard cover page, a

soft cover page, and content pages. Figure 22 illustrates a hard cover page that contains the date,

the volume number, an address label area, and a postal class identification; it is printed on high

grade paper to minimize or prevent damage incurred by postal processing and handling. (All

other pages are printed on recycled newspaper-quality paper.) As illustrated in Figure 23, a soft

cover page identifies the date, the volume number, and the page numbering sequence for the

body pages contained within the book. A content page, illustrated in Figure 24, contains a page

heading that includes an upper case Roman numeral page number. The first content page

contains general information with regard to the FR and its usage. Each subsequent page

identifies the contents of the specific book. There are typically less than 10 prefix pages printed

in each FR issue.

Body Pages: Labeled in the database as ‘b’ files, these pages represent the bulk of each FR issue

and provide a record of the meeting notices, proposals, and transactions of the United States

government for the day. There are two types of body pages: section pages and detail pages. A

section page, illustrated in Figure 25, is similar in appearance to a soft cover page and is used to

divide the FR into distinct parts. A section page contains the name of the issuing agency, the
5

Code of Federal Record (CFR) title and part(s) affected, and a brief description of the section’s

subject; this type of page does not contain a page number field. A detail page, illustrated in

Figure 26, elaborates Presidential and Executive Order(s), Rules and Regulations, Proposed

Rules, and Sunshine Act Meeting Notices. Each page contains a page heading that includes an

Arabic page number. Page numbers run consecutively across FR books. Typically, 200-300

pages are printed in each FR issue.

Appendix Pages: Labeled in the database as ‘c’ files, these pages provide reader aids which

allow a reader of the FR to access information and to index specific information contained within

previous issues. An appendix page contains a page heading that includes a lower case Roman

numeral page number. Typically, there are less than 10 appendix pages printed with each FR

issue. An example of an appendix page is illustrated in Figure 27.

With the exception of cover and section pages, each page of the FR is printed with a page

heading that includes a text banner printed above two horizontal lines. The text banner contains

information that identifies the document, the volume, the date, the topic, and a page number.

Blank pages are not included in this database.

The FR for 1994 was chosen to be the source for this database because it is:

1. a complete set of documents within the public domain;

2. a large collection containing over 250 issues consisting of nearly 69,000 pages of

information;

3. a structured document set;

4. a collection of pages containing significant variations in print and image quality;

5. a set of documents for which the text for the entire collection is stored within electronic

files that can be used as ground truth for evaluating OCR and IR systems.

2.2 Creating the Document Image Database

Figure 1 illustrates the steps taken to generate the images and ground truth text files included in

this database. The original GPO typeset files were obtained through a data provider, and all the

FR books issued in 1994 were obtained from the NIST Reference Library as the books were

being removed from circulation. Having the books enabled us to get images of each page, and

having the GPO files provided us with the text printed in each book. The goal was to create a

process whereby OCR results from each page could be matched against the GPO files, and the

text corresponding to a specific page be located and extracted and used as ground truth.

The bindings of the books were cut and nearly 69,000 pages were scanned by a service bureau at

15.75 ppm (400 ppi) binary. Upon scanning, each image was validated at NIST according to

Reference [5]. Scanning and validating the images is discussed further in Section 2.2.1.

After the images were validated, they were processed by ScanWorX API1

, and OCR results were

recorded. OCR text is provided with every page in the database (for which ScanWorX

terminated successfully). The quality of this text varies significantly from page to page,

depending on a page’s composition and image quality. The impact of quality on OCR

performance within the FR images is discussed in Reference [9]. The process of conducting

OCR is discussed further in Section 2.2.2.
6

Figure 1. Overview of how the document image database was created.

On the left side of Figure 1, we have the GPO files that contain the true text printed in the books.

While we have an entire book’s text, we don’t know what text was printed on which page;

moreover, the GPO files are riddled with binary typesetting codes (a format named Microcomp).

These codes represent physical markup for the corresponding book. A parser was written at

NIST to strip the typesetting codes from the actual text. Then, based on the physical markup

seen, logical structures were inferred and recorded. The parsing of the GPO files is discussed

further in Section 2.2.3.

Multiple GPO files were used to typeset each FR book, and the ordering of the files and the order

of the text within the files were not always clear, which compounded the challenge of locating

the text associated with any specific page. As a result, the text parsed form each GPO file was

simply concatenated together to represent the text for an entire FR book. This ordering was

based on the alphabetical order of the GPO file names.

We now have in the middle of Figure 1 a book’s text concatenated together in an arbitrary order

on the left, and pages of OCR text on the right. A process was engineered at NIST to match a

page’s noisy OCR text against the large list of true text for the book and extract ground truth text

for the page. This fully automated process of extracting ground truth is discussed further in

Section 2.2.4.

This section has provided an overall description of how the FR document image database was

created. Details of each step are provided in subsequent sections. Recipients of this database

receive all the tools developed to prepare this data with the hope that they may prove useful in

similar data collection efforts.

GPO FR

FILES

FR BOOKS

PAGE

IMAGES

PAGES OF

OCR TEXT

FR TEXT

IN BOOK

PAGES OF

FR TEXT

TYPESETTING SCANNING

OCR PARSING

PAGE

 MATCHING
7

2.2.1 Scanning and Validating the Federal Register

As stated above, the scanning of the 1994 FR books was contracted to a service bureau. While

the per-page scanning rate was very reasonable, significant additional cost was invested by NIST

to verify the quality and content of the images. Pages of the FR were scanned by the service

bureau using a high-speed Kodak 923 scanner1

 at a resolution of 15.75 ppm (400 ppi). The

images were scanned as binary and stored in compressed Tagged Image File Format (TIFF[10]).

Each FR book had its binding cut, and its pages were placed in a hopper, feeding a high-speed

paper transport. The pages were scanned with a sequential index assigned to each resulting

image file. Upon scanning, the sequentially assigned index file names were cross-referenced

automatically to ranges of physical pages in the book. This method of batch scanning is

efficient, but prone to errors. For example, a significant number of blank pages were detected

and their image files removed. In addition, a series of image quality checks was performed to

detect truncated or otherwise corrupted bitmaps. Images that did not conform to the following

criteria were rescanned using a Fujitsu 3096G scanner1

.

1. Resolution = 15.75 ppm

2. Compressed image file size >= 30 Kbytes

3. Width < 4000 pixels

4. 4200 pixels < Height <= 4900 pixels

5. Rotational skew < 5°

These tolerances were selected in order to minimize the amount of rescanning. Nonetheless,

nearly 3% of the pages required rescanning. Apparently one shift operator scanned all his

images at the wrong resolution, which accounted for 2% of the total pages rescanned.

Next to having the wrong image resolution, the most common error occurred when pages were

scanned out of order, or two pages stuck together so that only one image was captured. OCR

was used to catch out-of-order or missing pages. NIST utilized technology from its public

domain OCR system [11] to create a machine print recognition application that located the page

number on the top of FR pages. A subimage containing the page number was then segmented

and classified. If the OCR result matched its hypothesized page number, then the image was

determined to contain valid contents. If the OCR results did not match, then the image had to be

reviewed by a human. Using this approach, 83% of the images were automatically verified

without human adjudication [5].

Upon completion of this step, all FR page images were verified to be of a reasonable dimension,

containing a properly encoded bitmap, and containing its proper page.

2.2.2 Running OCR on the FR Page Images

Once validated, each FR page image was processed using Xerox’s ScanWorX API 3.11

. NIST

developed a ScanWorX application program to process the FR pages, the source code for which

is provided in this distribution. ScanWorX successfully completed execution on nearly all the

FR images, providing results in XDOC format [12]. An XDOC file contains much more

information than just the recognized text. Section 3.2.2.2 discusses the contents of the XDOC

files in greater detail.
8

The majority of pages in a FR book (Figure 26 for example) contain a running header that

identifies the book’s volume, date, section, and page number. The GPO typesetting files do not

contain page-level boundaries and information, so these headers are specified once at the

beginning of major sections. As a result, the page images have headers whereas the parsed GPO

text being matched against does not. To facilitate accurate matching, the OCR application

locates the header and strips it off, so that its text is not included in the XDOC files. This also

implies that header information is not provided in the database ground truth text files (a caveat

introduced by the nature of the GPO files).

2.2.3 Parsing the GPO Typesetting Files

The files used by the GPO to print the FR are in Microcomp format. The GPO provided NIST

with a collection of code tables. These binary codes are physical markup used by the typesetting

system to choose fonts, format lines, construct tables, etc. The Microcomp documentation we

received was incomplete, so significant effort was invested at NIST to reverse engineer a

functional parser. Approximately one person year of labor was spent on this task. The resulting

GPO parser (written in Perl) is provided in this distribution.

The GPO parser does more than strip the Microcomp codes from the text in the typesetting files.

Additionally, it tracks the sequence of codes and infers logical context and structures from the

physical markup. As a result, the parser is capable of producing SGML text tagged with

sections, tables, footnotes, etc. Currently, the parser infers a rather limited number of logical

structures defined in a provided DTD file (Section 3.2.1.2). If one desires, the parser may be

augmented (with significant effort) to identify a broader set of logical entities.

Once all the GPO files for a FR book have been parsed, the resulting tagged text is concatenated

together into one long list. At this point in the process, we have the textual content of an entire

FR book in an uncertain order on one hand, and we have OCR text from each scanned page on

the other. The next step is to map the OCR page text onto the GPO book text, extracting pages

of text as ground truth.

2.2.4 Extracting Ground Truth by Matching OCR and GPO Text

The first sequence of steps involved in this stage of processing is illustrated in Figure 2. The

concatenated text from the parsed GPO files (representing an entire book) is depicted on the left

and the OCR page text is on the right. Both texts are treated as separate vectors of words. The

book word vector is considerably longer than the page word vector. A FR book may contain

250,000 words whereas a page may contain 2000.

The immediate neighbors of each word are grouped into triples called trigrams. The locations of

all the unique trigrams within the book’s word vector are recorded. All the trigrams are

computed from the OCR page word vector, and then matched against the unique trigrams from

the book. Matching trigrams form a cluster, and the median of the cluster is used to anchor the

page of OCR text to an area within the GPO text. The median of the cluster is called a “hit”, and

the area of GPO text matched is assumed to belong to the page from which the OCR was

produced.
9

Figure 2. Matching the OCR page text to the GPO book text.

Figure 3. Extracting a chunk of GPO text.

PAGE OF

OCR TEXT

GPO TEXT

IN BOOK

GPO WORD VECTOR

OCR WORD VECTOR

HIT

UNIQUE TRIGRAM MATCHES

HIT

-n +n

GPO BOOK VECTOR GPO CHUNK VECTOR
10

Figure 4. Trimming the chunk of GPO text.

Once the area of the page (or hit) has been located, the beginning and ending of the page must be

determined from within the GPO text. String alignment is used to map the noisy OCR page text

to the area surrounding the hit point within the GPO book vector. The FR pages vary in terms of

the number of words on a page. Some pages with small fonts and dense text have more than

2000 words on a page. To localize the matching, a “chunk” of the GPO text is extracted,

centered on the hit point. The radius of the chunk is conservatively set at (n=1500) words.

Figure 3 illustrates extracting the chunk of GPO text from the book’s word vector.

The chunk is assumed to contain the words for the desired page, but the chunk needs to be

trimmed to correspond to the beginning and ending of the page. To narrow the chunk to the

page’s boundaries, string alignment technology developed at NIST was used to match the OCR

page text to the GPO chunk. This is illustrated in Figure 4. String alignment was performed

starting at the hit point in the chunk vector, working backward to the front of the chunk and

forward to the end of the chunk. Working backward, the point at which the alignment

sufficiently degraded was determined to be the beginning of the page, and working forward, the

point at which the alignment sufficiently degraded was determined to be the end of the page.

This string alignment technology is included in the distribution.

The detection of page boundaries is very difficult because the quality of the OCR text may be

very poor at the beginning and end of the page and the OCR order of words may be different

from the GPO typesetting order of words in these areas. Nonetheless, heuristics were developed

based on the quality of alignment to hypothesize where page boundaries occur. The C program

used to make these decisions is provided in the distribution. The reader is referred to the source

code for documentation of these heuristics.

PAGE OF

FR TEXT

HIT

NOISY STRING

ALIGNMENT

GPO CHUNK VECTOR

OCR PAGE VECTOR
11

Based on the determined page boundaries, the GPO chunk of text was trimmed, and the

remaining text was stored as the page’s ground truth. An internal representation called a “word

index file” was developed to support the steps described in this section. This representation

(described in Section 3.2.1.3) enables SGML text files to be represented as word vectors while

preserving the tagged context of the text. Using this representation, the trimmed GPO text chunk

was reformatted into an SGML ground truth text file.

2.2.5 Uncertainty Issues and Quality Control

In the previous section, it was mentioned that not only does the performance of OCR impact the

quality of the extracted ground truth, but that there were also order issues. The text contained in

the GPO Microcomp files is in an order prescribed for the typesetting process, whereas the order

derived by the OCR may be quite different. This difference occurs frequently in tables. The

order that the cells are populated by the typesetter is typically row-oriented, whereas ScanWorX

typically captures text in a column-oriented fashion. Differences in order also occur due to

footnotes. OCR will locate footnotes typically at the bottom of a column of text, whereas there

is no notion of bottom of page within the GPO files themselves. As a result, a footnote’s text

occurs sometime after the footnote’s reference is encountered. But the footnote text may occur

within a few sentences or a few paragraphs of its reference. To compound these ordering issues

further, the OCR will at times incorrectly decolumnate a page, merging paragraphs across

columns and vertically splitting paragraphs within a single column. This causes uncertainty in

the OCR-derived order of the words on the page.

The good news is that the OCR order and GPO typesetting order generally follow the same

reasonable reading order. Unfortunately, the order of the words at times may be quite different

between the two sources. This, along with the potential of dismal OCR performance on lowquality images, introduces uncertainty in the derived ground truth text.

Quality control steps were developed to help reduce and manage this uncertainty. The process of

clustering matched unique trigrams shown in Figure 2 may fail to produce a hit. This could

happen, for example, if the OCR performance was so poor as to corrupt all the relevant trigrams

on the page. This condition is easily detected. More difficult is detecting when an incorrect hit

is generated. A hit may be bad due to 1) the ordering issues already discussed, 2) ambiguous

word trigrams caused by OCR errors, or 3) text appearing on the FR page that is not explicitly

contained in the GPO files.

The GPO Microcomp format uses external references to import data from external graphics files.

These external references typically refer to items such as graphs, maps, mathematical formulae

and some tables. When the OCR engine processes the contents of these pages, it recognizes text

and symbols that are not present in the GPO parsed text for the book. This can throw the

clustering of matching unique trigrams off, resulting in hits that do not point to their page’s

proper location.

An automated method of rejection was developed to detect incorrect hits. This process analyzes

all the hits computed for all the pages in a book. Each hit references a location within the book’s

word vector as shown in Figure 4. FR pages on average contain 1000-2000 words each. Ideally,

the hit values on contiguous pages will be evenly spaced at intervals of 1000-2000 words. The

graph in Figure 5 plots all the hits calculated for each page in the FR book for January 3, 1994.

Notice that the majority of points are members of linear segments. Those that do not follow the
12

linear trend of their neighbors are likely to be incorrect. These are automatically detected and

flagged as bad, and no ground truth is extracted for these pages. It has been observed that the

erratic points in the graph frequently correspond to pages comprised of large tables, which are

prone to all the problem-causing issues described above.

The hit rejection algorithm is included in the distribution. There is a plot, similar to the one

shown in Figure 5, provided for each FR book in the database. Inspection of these graphs gives

the recipient of the database a visual representation of how well hits were detected within each

FR book.

Ground truth files are provided for each accepted hit file. The same problem-causing issues of

text order, OCR quality, and missing text also contributed directly to how accurately page

boundaries could be determined. A method of rejecting ground truth was explored, but there was

no way found that yielded reasonable results. Imposing such a mechanism would have required

assumptions about the acceptable tolerances of experiments to be run on the data.

Rather than impose rigid and arbitrary procedures, a method was developed whereby the amount

of error within a ground truth text file could be estimated. Giving a conservative estimate of

error permits the researcher to determine what level of quality within the ground truth is required

for a particular experiment. Once the ground truth text was extracted for a page, it was realigned

with the OCR text using the NIST string alignment technology. A secondary step was invoked

after initial alignment in order to match up blocks of text out of order between the ground truth

(in typesetting order) and the OCR text.

Figure 5. Plot of accepted/rejected hits for the FR book published January 3, 1994.
13

Figure 6. Plot of ground truth error estimates for FR pages on January 3, 1994.

After out-of-order blocks of text were matched, the remaining inserted and deleted words

represent how much of the OCR text was not covered by the ground truth and how much of the

ground truth was missing from the OCR text. The number of deleted and inserted words were

used to compute a conservative error estimate as described in Section 3.2.2.9. An error estimate

is provided with each ground truth text file in the database. These error estimates are plotted on

a log scale in sorted y-order for each FR book in the database. An example of a plot for January

3, 1994 is shown in Figure 6. This graph provides the recipient of the database with a visual

representation of the estimated quality of the ground truth for a given book.

3. DISTRIBUTION HIERARCHY AND CONTENT

This section describes the organization and contents of this database distribution. Figure 7

illustrates the top level directory structure in the distribution. The src, include, bin, and lib

directories contain the source code and scripts used to produce the document image database as

described in Section 2.2. The man directory contains on-line manual pages. A software

reference guide is provided in Appendix A.

The doc directory contains PostScript formatted documentation (including this paper and other

NIST reports referenced in this paper). All the files in doc have been GZIP compressed (see the

compression discussion at the beginning of Section 3.2). The numbers used in these file names

correspond to their reference number in this document. The file corresponding to this document

is db.ps.
14

Figure 7. Top-level distribution hierarchy.

The dtd directory is a special directory containing two SGML Document Type Definition (DTD)

files; one for the ground truth text files, and one for the SGML formatted OCR text files.

Finally, the data directory holds all the database files for the distribution, including the FR page

images and ground truth text.

3.1 Public Domain Software

All the software used to produce the data in this distribution has been provided. This is public

domain code, and any portion of it may be used without restrictions because it was created with

U.S. government funding.2

Three C programs have been provided. Their main routines are stored under the src/bin

subdirectory, and they link subroutines archived in a C library, the source code for which is

stored under the src/lib subdirectory. The C source code has been developed and tested to run

on most common UNIX platforms, including LINUX, SunOS, Solaris, IRIX, AIX and HP-UX.1

Compilation is controlled using the UNIX make facility through the use of a system of

hierarchical makefile scripts. The makefile scripts are configured to use the GNU gcc compiler

[13]. Porting the source code to other platforms and other compilers is the sole responsibility of

the recipient.

To compile the C programs, you must mount the first of the two ISO-9660[8] CD-ROMs on your

system. You may need the assistance of your system administrator. Once mounted, the

directories src, include, bin, and lib must be copied completely to a read-writable disk partition.

The directory into which the software is copied is referred to as the distribution’s “installation

directory” or install_dir. Once copied, all directories should be set to read, write, and executable

and all files should be set to readable and writable (you may choose to keep the “.c” files readonly).

Variables are provided in the top-level makefile script (makefile.mak) that may be edited if you

are using a different compiler than gcc. These variables allow you to define your specific

compiler and the options it requires to have passed. It still may be necessary to edit the low-level

makefile scripts to account for your specific compiler’s options.

 

2

 This software was produced by NIST, an agency of the U.S. government, and by statute is not subject to copyright

in the United States. Recipients of this software assume all responsibilities associated with its operation,

modification, and maintenance.

bin

(C-compiled programs

& Perl scripts)

doc

(documentation

& reports)

include

(C language

include files)

makefile.mak src

(C language & gzip

source files)

data

(FR image

& text files)

dtd

(SGML document

type definitions)

lib

(C-compiled library

& Perl include files)

man

(manual pages)
15

The majority of utilities in this distribution are written in the portable scripting language

Perl[14]. Some supporting Perl files are located under the top-level distribution lib directory,

with all main scripts located under the top-level bin directory. One of these scripts, catalog.pl,

automatically compiles C source code catalogs when the C library and programs are compiled.

In order to access this script, the full path name to the top-level bin directory in install_dir must

be added to your shell’s environment path variable.

In order to execute these scripts, Perl must be installed on your computer [14]. The Perl scripts

provided in this distribution are set up assuming that the Perl interpreter has been installed in

/usr/local/bin. If Perl is not installed in this directory on your computer, then you will need to

edit the top header line of each Perl script in the top-level bin directory. If, for example, the Perl

interpreter is installed in /apps/perl/bin, then the header will need to be changed from:

	



to:





To start compilation, it is recommended that you invoke the top-level makefile in your

install_dir. You will likely need to edit the top-level makefile so that the variable PROJDIR is

assigned to your install_dir. The makefile is set up by default with PROJDIR set to

/usr/local/fr94.

Now you are ready for compilation. The first step is to direct the compiler to compute source

code dependencies. If your compiler does not have this capability, then edit the low-level

makefiles (in the program directories under src/bin and in the library src/lib/db), adding the

general dependencies provided in the corresponding depend.mak files located in their same

directories. Dependencies are created with gcc with the following command:







The next step is to compile and install the libraries and binaries. This is accomplished with the

following command:








Manual pages for all the C programs and Perl scripts are provided on-line in the man directory

and are included in Appendix A.

3.2 Document Image Data

This section describes the organization and content of data files in the database. Each data file

distributed under the top-level directory data has been compressed. Image files with extension

pct are compressed using CCITT Group 4 [16]. All other files have been compressed using the

public domain GNU zip (GZIP) utility [13]. The GZIP-1.2.4 source code distribution (requiring

installation and compilation) is provide in the tar file gzip.tar under the top-level src directory.

Typically, files compressed using GZIP have an added extension of “.gz”. This CD-ROM

distribution is formatted in ISO-9660, requiring that all file names conform to the 8.3 naming

convention. This means file names may be up to 8 characters long followed by a single

extensions of up to 3 characters. This prohibits the use of a second extension on compressed

data files. Therefore, all data files (other than image files) are GZIP compressed although their

file names do not indicate it with an extra extension.
16

Figure 8. Data distribution hierarchy.

For example, the file data/01/03/book01/txt/b0000100.xdc is an XDOC file for page 100,

published on January 3rd in the FR, and has been GZIP compressed. To access the file’s

contents, one can run the GZIP-provided utility gzcat with the command:

	


	

The resulting decompressed data can be stored to a new file by redirecting the standard output of

the previous command to a new file on a read-writable disk partition on your computer system.

Figure 8 illustrates the hierarchy in which the data files are distributed. Starting at the top level,

the data is organized by month (MM). The files in this distribution are for the month of January

(01). The data is sorted by day (DD) within each month. The FR is published each working day

of the year, and there are 20 days worth of data provided for the month of January, 1994.

The overall size of the distribution is 1.27 gigabytes, requiring 2 CD-ROMs. The data files have

been split by day over the two discs. The first CD-ROM contains the entire top-level hierarchy

illustrated in Figure 7 and part of the data hierarchy illustrated in Figure 9, including day 01

through day 13 (excluding day 05, which is included on the second disc due to space limitations).

The second CD-ROM contains day 05 and the remaining data files for days 14 through 31, with

only a single top-level directory data. The first disc holds about 614 megabytes of data files,

whereas the second disc holds 649 megabytes of data files. The data partitions are designed to

be merged by simply copying the contents of their CD-ROM’s top-level data directory into the

same read-writable directory.

Within each day, the data files are divided into two parts. The first directory, gpo, contains all

the FR files obtained from GPO and derivatives of these files. These files contain the true text

for all the FR books published for a particular day. The original Microcomp typesetting files are

stored in a subdirectory, gpo/org, with the subdirectory gpo/sgm containing corresponding

parsed SGML files.

data

org

(typesetting files)

sgm

(parsed SGML files)

img

(page-related

 image files)

txt

(page-related

text files)

MM

(month-related files) .. ..

gpo

(GPO-related files)

bookBB

(book-related files) .. ..

DD

(day-related files) .. ..
17

Figure 9. Location of data files.

Table 1. Distribution data files.

data

01

gpo book01

org sgm txt .wdx img

03 31 .

.oml

rejhits.ps

.odx

.gdx

.est

.gml

.xdc

.qua

.hdy

.scr

errest.ps

.xxx

.nnn

.sgm .pct

*

 TDDmmttt T - alphabetic file type; DD - numeric day; mm - alphabetic month; ttt - alphanumeric identifier

 MMDD MM - numeric month; DD - numeric day

 t9999999 t - alphabetic file type [a,b,c]; 9999999 - numeric page number

EXT ROOTNAME* DESCRIPTION

xxx

nnn

TDDmmttt Original GPO Microcomp typesetting files

sgm TDDmmttt SGML files parsed from Microcomp files

wdx MMDD Master word index for the day

pct t9999999 IHead page image

xdc t9999999 Xerox XDOC file containing OCR results

hdy MMDD Vertical scanline at which page headers were detected and cut

qua MMDD Los Alamos image quality analysis and OCR error rate predictions

oml t9999999 OCR SGML file parsed from corresponding XDOC file

odx t9999999 OCR word index file parsed from corresponding OML file

ps rejhits Quality assurance plot of accepted and rejected trigram hits

gdx t9999999 Ground truth word index file extracted from master word index

est MMDD Estimates of error in corresponding GDX files

gml t9999999 Ground truth SGML file created from corresponding GDX file

scr t9999999 Word-level score between corresponding GDX and ODX files

ps errest Quality assurance plot of sorted EST values for the entire book
18

The remaining directories, bookBB, contain files associated with each book published for a day.

The gpo directory contains files for all the books, whereas the bookBB directories contain files

for each page in a book. There was only one FR book published for each day in January 1994,

so each day (DD) directory contains only one directory (book01). (If a day were to have two

books published, it would have two book directories present, book01 and book02.)

The data files within a bookBB directory are divided into two subdirectories. The first

subdirectory, bookBB/img, contains the page images for the book, whereas the subdirectory,

bookBB/txt, contains all the text-based files related to each page in the book (including the

ground truth text files). All the text-based data files in this distribution are encoded using the

ISO 8859-1[15] character set, which represent Western European characters. (Actually, many

special characters are represented as SGML entities that map to the ISO 8859-1 character set.

These entities are defined in the files located in lib/sgml.)

Table 1 lists all the various types of data files included in this database. File types are primarily

identified by their extension, which are listed in the first column. The second column lists

associated root file names followed by a short description. The location of these files in the data

distribution hierarchy is illustrated in Figure 9.

A more detailed description of these files is provided below in the order prescribed in the table,

starting first with the GPO-related files, and then continuing with the book-related page files. To

help explain these files, examples have been taken from a specific FR page in this distribution.

This page, shown in Figure 26, is body page 100 published on January 3, 1994.

3.2.1 GPO-Related Data Files

These files include GPO-provided Microcomp files, parsed SMGL files, and a master word

index file. The GPO used the Microcomp files to typeset and print each FR book. The NIST

SGML files contain FR text as well as a set of logical formatting tags. The master word index

file provides cross-referencing for ground truth extraction.

3.2.1.1 GPO Microcomp Files

Each daily issue of the FR is represented by a set of Microcomp files. These files, used to

typeset the FR, consist of ASCII text with binary encoded markup and typesetting instructions.

The ordering within a Microcomp file does not always match the physical ordering of pages in a

FR book. Additionally, a Microcomp file does not contain any information that denotes page

boundaries on which the text is printed. Therefore, it is a difficult task to extract the exact

content of a printed page from a Microcomp file. Nonetheless, an automated process (described

in Section 2.2) was created to extract an hypothesized page-level ground truth.

All the GPO Microcomp files for a particular day are stored in the subdirectory gpo/org and

have either the extension xxx or nnn. In our example, the text on FR page 100 is located in the

Microcomp file data/01/03/gpo/org/r03ja3.xxx. A portion of this file is not illustrated here due

to its binary encoding.

3.2.1.2 GPO SGML Files

A parser was developed at NIST that converts Microcomp files into text-based SGML files. The

parser filters the physical markup codes and infers certain logical structures such as paragraphs

of text, figures, tables, and footnotes. These logical structures are then represented by SGML
19

tags. The complete set of tags detected and stored is documented in the SGML DTD file

dtd/fr94gpo. dtd. Table 2 lists those tags that are commonly detected. The middle column lists

the total instances that a structure was detected in the FR for the entire month of January 1994.

The last column provides a brief description of each tag. As can be seen, footnotes, dates, and

tables occur quite frequently.

Figure 10 plots the frequency of each tag accumulated by day. The total in each category equals

the value listed in Table 2. Each daily issue contributes to these categories in a relatively

proportionate way. The majority of text is labeled with a generic tag, TEXT, whose total

magnitude is off the chart.

TAG TOTAL DESCRIPTION

DOCNO 160 GPO SGML file base name

FN 2379 footnote body including identifier and text

FNID 2218 footnote identifier (ex. number used in reference)

FNTEXT 2378 footnote text

FR 2551 footnote reference (ex. superscripted numbers)

NOTICE 2063 major notice item submitted by agency

NOTICE-ACTION 1252 action field in notice

NOTICE-AGENCY 1218 agency field in notice

NOTICE-CONTACT 1277 contact information for notice

NOTICE-DATE 2431 date of notice

NOTICE-LOCATION 140 location of notice

NOTICE-MATTERS 48 matters field in notice

NOTICE-STATUS 80 status of notice

NOTICE-SUMMARY 1216 summary field in notice

NOTICE-TITLE 108 title of notice

TABLE 851 table

TEXT 226055 character sequence of unknown (generic) type

Table 2. Tags included in GPO SGML files.
20

Figure 10. Frequency distribution of tags (accumulated by day) for January, 1994.

0

500

1000

1500

2000

2500

3000

 DOCNO

 FN

 FNID

 FNTEXT

 FR

 NOTICE

 NOTICE-ACTION

 NOTICE-AGENCY

 NOTICE-CONTACT

 NOTICE-DATE

 NOTICE-LOCATION

 NOTICE-MATTERS

 NOTICE-STATUS

 NOTICE-SUMMARY

 NOTICE-TITLE

 TABLE

 TEXT

01/31

01/28

01/27

01/26

01/25

01/24

01/21

01/20

01/19

01/18

01/14

01/13

01/12

01/11

01/10

01/07

01/06

01/05

01/04

01/03
21

Figure 11. Portion of a GPO SGML file.

All the GPO SGML files for a particular day are stored in the subdirectory gpo/sgm and have the

extension sgm. There is one SGML file for each Microcomp file, and they share the same root

file name. Figure 11 lists a portion of the GPO SGML for body page 100. This text corresponds

mostly to the top-middle column on the page. Note that there are no newline characters

contained on the end of the lines that wrap in the figure. This data is stored in the file

data/01/03/gpo/sgm/r03ja3.sgm.

3.2.1.3 Master Word Index Files

A GPO master word index file contains, in typesetting order, a record for each word that was

printed in an issue of the FR. The GPO SGML files are concatenated together based on the

alphabetic order of their file names, and the concatenated text is converted to a word-based

representation. This representation is critical to the ground truth extraction process described in

Section 2.2.





	























!"


#
$%&'(')
""

&'&*)+&,-.
/(012

*

! 34%#

%
5%



&67221

*! 34%#







*



%



8
".

!"


#


*



9:%21,-.
/1&'('"
7&,-.
/1',-.
/21/+50(;

*



<3==3!> %6(16,-.
/'7,-.
/

*

*! 3

! 3



!#3 !#==#< ::=#3 !< #:%

*



#..
3	"





8



:

<


*



(?0170@A@0A

B	


..
	"







.








.	"




":







."



B	






-

	"

.

"
..


"


..






.









!"=
:
"<


-




7722&



.


1'7221

*

22

Figure 12. Portion of a master word index file.

A portion of the master word index for January 3, 1994 is listed in Figure 12. Each row in the

index corresponds to a successive word from the GPO SGML files.

A row is comprised of the following:

1. global word index

2. local word index

3. word

4. SGML file

5. variable length list of all open logical document tags

The global word index is a unique index incremented for each word across all the GPO SGML

files for a given day. The local word index is reset at the beginning of each GPO SGML file and

incremented for each word in the file. The third field is the word string, and the fourth field is

the SGML file from which the word was extracted. The remaining variable number of fields



&&7(26 12+1+ %
5 
'1C1 9% 5+;9! 35+';9! 34%#57'(;

&&7(2+ 12+12 %



 
'1C1 9% 5+;9! 35+';9! 34%#57'(;

&&7(22 12+0' &6 
'1C1 9% 5+;9! 35+';9! 34%#57'(;

&&7)'' 12+07 7221 
'1C1 9% 5+;9! 35+';9! 34%#57'(;

&&7)'7 12+0& 
 
'1C1 9% 5+;9! 35+';957160(;

&&7)'& 12+01  
'1C1 9% 5+;9! 35+';957160(;

&&7)'1 12+00 
 
'1C1 9% 5+;9! 35+';957160(;

&&7)'0 12+0( %


 
'1C1 9% 5+;9! 35+';957160);

&&7)'( 12+0) 
 
'1C1 9% 5+;9! 35+';957160);

&&7)') 12+06  
'1C1 9% 5+;9! 35+';957160);

&&7)'6 12+0+ 8
" 
'1C1 9% 5+;9! 35+';957160);

&&7)'+ 12+02 .

 
'1C1 9% 5+;9! 35+';957160);

&&7)'2 12+(' !" 
'1C1 9% 5+;9! 35+';957160);

&&7)7' 12+(7 
 
'1C1 9% 5+;9! 35+';957160);

&&7)77 12+(& 
 
'1C1 9% 5+;9! 35+';957160);

&&7)7& 12+(1 
 
'1C1 9% 5+;9! 35+';957160);

&&7)71 12+(0 #
 
'1C1 9% 5+;9! 35+';957160);

&&7)70 12+(( 9: 
'1C1 9% 5+;9! 35+';9571606;

&&7)7( 12+() % 
'1C1 9% 5+;9! 35+';9571606;

&&7)7) 12+(6 21,-.
/1&'(''1C1 9% 5+;9! 35+';9571606;

&&7)76 12+(+ "
 
'1C1 9% 5+;9! 35+';9571606;

&&7)7+ 12+(2 7&,-.
/1',-.
/21/'1C19% 5+;9! 35+';9571606;

&&7)72 12+)' +50( 
'1C1 9% 5+;9! 35+';9571606;

&&7)&' 12+)7 ; 
'1C1 9% 5+;9! 35+';9571606;

&&7)&7 12+)& <3==3!> 
'1C1 9% 5+;9! 35+';957160+;

&&7)&& 12+)1  % 
'1C1 9% 5+;9! 35+';957160+;

&&7)&1 12+)0 6(16,-.
/'7,-.
/'1C19% 5+;9! 35+';957160+;

&&7)&0 12+)( !#3 !#= 
'1C1 9% 5+;9! 35+7;9571602;

&&7)&( 12+)) =#< : 
'1C1 9% 5+;9! 35+7;9571602;

&&7)&) 12+)6 :=#3 ! 
'1C1 9% 5+;9! 35+7;9571602;

&&7)&6 12+)+ < #:% 
'1C1 9% 5+;9! 35+7;9571602;

&&7)&+ 12+)2 #..
 
'1C1 9% 5+;9! 35+7;95716(';

&&7)&2 12+6'  
'1C1 9% 5+;9! 35+7;95716(';

&&7)1' 12+67 3	" 
'1C1 9% 5+;9! 35+7;95716(';

&&7)17 12+6&  
'1C1 9% 5+;9! 35+7;95716(';

&&7)1& 12+61 


 
'1C1 9% 5+;9! 35+7;95716(';

&&7)11 12+60  
'1C1 9% 5+;9! 35+7;95716(';

&&7)10 12+6( 


 
'1C1 9% 5+;9! 35+7;95716(';

&&7)1( 12+6)  
'1C1 9% 5+;9! 35+7;95716(';

&&7)1) 12+66 8



 
'1C1 9% 5+;9! 35+7;95716(';

&&7)16 12+6+ :

 
'1C1 9% 5+;9! 35+7;95716(';

&&7)1+ 12+62 <
 
'1C1 9% 5+;9! 35+7;95716(';

23

store the SGML context of the word. In other words, all the SGML tags that are currently open

when the word was encountered are recorded here.

There is one master word index file per day, and it is stored directly within the gpo subdirectory.

A master word index file has the extension wdx and a root name formed from the concatenation

of the numeric month and day strings (MMDD). For example the data listed in Figure 12 is

stored in the file data/01/03/gpo/0103.wdx.

The words listed in this example correspond to the top-middle column of body page 100 (shown

in Figure 26). The list starts with the line “Dated: December 27, 1993.” and ends within the start

of a new notice section for the National Labor Relations Board. Note the words on the date line

are tagged with NOTICE-DATE followed by the number 105. This number is an instance of that

particular tag type occurring in the current daily issue. The instance indices are zero-oriented, so

that this date belongs to the 81st notice section in the issue. (Locate the preceding tag “NOTICE:

80” for the words on the date line.)

Now locate the sequence of words: “NATIONAL LABOR RELATIONS BOARD.” This starts

a new notice section which is represented by the change of the instance index on the NOTICE

tag for the word “NATIONAL.” The instance increments from 80 to 81 denoting the start of a

new section.

This word index format allows a passage of SGML tagged text to be represented as a word list,

preserving the logical context of every word. This format is the same used for the page-related

OCR and ground truth word index files. (The ground truth word index files are actually

extracted sequences from the master word index file.)

3.2.2 Book-Related Data Files

This section describes the data files contained within the distribution’s bookBB directories. All

the daily issues of the FR in January 1994 were printed as single books, so all day (DD)

directories contain a single directory (book01). Two classes of page-related files are contained

within each bookBB directory: page image files and page-related text files. The text files may

be further distinguished as being either OCR results or derived ground truth files. The bulk of

this database contains these page-related files.

Many different experiments may be performed on these files. For example, the images may be

input to other OCR systems to evaluate and benchmark accuracy using the existing ground truth

files and the provided scoring package. In this way, differences between OCR systems may be

compared and understood. OCR results are also provided for every image file in the distribution

and contain font detection, word segmentations, and character classifications. Thus, a large

collection of commercial OCR results is provided that can be studied and modeled for various

applications. Since the images vary in image quality, experiments can be setup to measure the

effect of image quality on OCR errors and, subsequently, OCR errors on retrieval performance.

3.2.2.1 Page Image Files

Before scanning each FR book, its pages were cut from its bookbinding. This resulted in paper

pages that were approximately 20 cm (8") by 28 cm (11") in size. Image scanning was

performed using a high speed Kodak 923 scanner1

. For the month of January 1994, 4771 pages

were scanned. The pages were scanned binary at 15.75 ppm (400 ppi) and converted to 2-

dimensionally compressed IHead image files (described in Appendix B).
24

IHead is a public domain, image file interchange format that has been designed with an extensive

set of attributes and can be used to represent both binary and gray level images. It has been

implemented across heterogeneous computer architectures and environments. Software is

provided in the CD-ROM distribution that allows applications to read, decompress, and write

IHead images.

All FR page image files for a given book are stored under the subdirectory book01/img with

extension pct. They have a root file name format t9999999, where t represents either ‘a’, ‘b’, or

‘c’, corresponding to prefix pages, body pages, or appendix pages respectively. The characters

9999999 correspond to the page number within the FR book. The majority of page-related files

use this same root file name format. For example, the file containing the image for body page

100 (shown in Figure 26) is stored as data/01/03/book01/img/b0000100.pct.

3.2.2.2 OCR XDOC Files

OCR results recorded in the database were generated using Xerox’s ScanWorX API 3.11

. Each

page of OCR results is stored in a text-based XDOC format [12]. A portion of an XDOC file is

shown in Figure 13. Although this format is text-based, it is not easily deciphered by a human.

Some of the types of information contained in an XDOC file include:

1. global estimation of image skew

2. word bounding boxes

3. font identification

4. character classifications and associated confidences

5. word confidences

6. line formatting information

Figure 13. Portion of an XDOC file.



9/(/666/'/6'6/./+;9/+');9/11/0&/00;9/661/)+0/21&/6'2;9/+;9B/+(+;!9B/216;#9B/2&+;9B/+27;

39B/212; 9B/21';!9B/207;#9B/+');=9/21)/77;9/611;9/11;9/20&/)+(/7'02/6'2;9B/611;=9B/2(2;#9

B/+)+;<9B/207; 9B/+6&;:9/7'(1/77;9/60&;9/11;9/7')'/)+(/7&1)/67';9B/+)2;:9B/+)+;9B/60&;=9

B/60&;#9B/21);9B/+67;39B/207; 9B/+)2;!9B/212;9-/76(2/(&'/6'6/7/;

9/(/666/7/607/./+;9/17';9/11;9/660/67+/++(/60&;9B/0&&;<9B/1)1; 9B/17';#9B/1));:9B/160;%9-

/76(2/+67/607/7/D;

9/(/666/'/+'&/./6;9/+'';9/11;9/66&/662/262/+'+;9/6;9B/21';#9B/+(0;.9B/21';.9B/+27;9B/+6

1;9B/+2&;9B/217;9B/27+;9B/+27;
9B/2'7;9B/21);9B/+'';9/2+0/77;9/6)0;9/11;9/2+2/662/

7'76/+'1;9B/+)';9B/6)0;9/7'&&/77;9/12);9/11;9/7'&6/6+'/77+)/+'(;9B/+2);39B/+(7;9B/27(;

9B/+6(;9B/2&1;9B/++&;9B/2&0;9B/6'';	9B/)26;9B/12);"9B/+01;9/7727/7';9/+)(;9/11;9/7

72)/6+&/7&&1/+'(;9B/+)(;9B/277;9/7&&+/77;9/6+(;9/11;9/7&10/6+'/717)/+'(;9B/216;9B/2&&;


9B/6+(;
9B/6+(;9B/2&0;
9-/76(2/01+/+'&/7/;

9/(/666/'/+1(/./6;9/+01;9/11;9/66&/+7+/+'(/+1);9B/++&;9B/+01;9/+7'/77;9/7;9/1&;9/+7

)/+7&/202/+12;9B/00(;9B/0'6;
9B/6&0;99B/122;9/2(0/77;9/+)7;9/11;9/2)'/+71/2+6/+1);9B/

2'6;9B/+)7;9/22&/77;9/6));9/11;9/22+/+71/77+0/+1+;9B/21(;89B/210;
9B/6));
9B/6));9B/+2

+;9B/+'2;
9B/+');9B/2&';9B/+0';9B/++2;9B/217;
9/77+2/7&;9/+((;9/11;9/772)/+70/7&22/+

1+;9B/27);:9B/2&0;
9B/276;9B/+((;9B/2&0;
9B/++';9-/76(2/0((/+1(/7/;

9/(/666/7/+)2/./);9/17';9/11;9/661/+0)/+66/+6';9/);9B/17';<9B/1++;9B/0'6;9B/17';
9B/1+

+;9B/0(7;9-/76(2/+66/+)2/7/D;

25

This distribution includes a public domain XDOC parser written at NIST in Perl. The parser can

be used to filter the different types of information from XDOC files. For example, it can

separate the recognized text from non-text items and store the results as tagged SGML.

An XDOC file is provided for every image in the database (with a few exceptions caused by

ScanWorX failures). All XDOC files for a given FR book are stored in a subdirectory

book01/txt with extension xdc. ). For example the data listed in Figure 13 is from body page

100 and is stored in the file data/01/03/book01/txt/b0000100.xdc. The results listed in the

figure are for a few lines from the top-middle column of the page, starting with the line

“NATIONAL LABOR RELATIONS BOARD.”

3.2.2.3 Header Cut Files

With the exception of cover and section pages, each page of the FR is printed with a page

heading that includes a text banner printed above two horizontal lines. The text banner line

contains information that identifies the volume, date, topic, and page number. An example of a

FR page header is seen in Figure 26. The GPO typesetting files do not contain page-level

boundaries and information, so these headers are specified once at the beginning of major

sections. As a result, the page images have headers whereas the parsed GPO text being matched

against does not. To facilitate accurate matching, the OCR application locates the header and

strips it off, so that its text is not included in the XDOC files.

The OCR application program locates the bottom of the heading’s two horizontal lines and

erases the header from the page image prior to recognition. The distance (in terms of pixels or

scan lines) from the top of the image to the detected bottom of the header is recorded. If no

header is detected, a value of “-1” is recorded.

The provided utilities detect the header location for each XDOC file in the database. Each of

these locations is stored to a separate file. These small files have been concatenated together into

a single compressed file for distribution. All the detected header locations for the pages of a

given FR book are stored in a subdirectory book01/txt in a single file named MMDD.hdy. For

example, the detected header location for body page 100 is recorded in the file

data/01/03/book01/txt/0103.hdy as:

In this case, the bottom of the header was detected 380 pixels from the top of the image and

removed.

3.2.2.4 Image Quality Assessment Files

Software developed at Los Alamos Laboratories was used to assess the image quality of each FR

page image in the database [4]. This software is designed to predict an OCR word error rate

based on a set of computed image quality descriptors. These descriptors include measurement

of:

1. Small Speckle Factor

measures fine background speckle in the document image.

2. Large Speckle Factor

measures larger chunks of background speckle.

 
26

3. White Speckle Factor

measures the small white enclosed areas in characters. These shrink in size as characters bloat.

4. New White Speckle Factor

same as above, but possibly more meaningful to the eye.

5. Touching Character Factor

measures how much neighboring characters touch each other.

6. Broken Character Factor

measures how broken the text characters are. This factor is also affected by small chunks of noise

that may lie on a line of text.

7. Font Size Factor

indicates the size of the font, normalized from one (small) to zero (large).

These image quality descriptors may be used as a selection criteria for designing experiments by

obtaining different image sample profiles based on different levels of image quality. The results

recorded for each FR page image contain a vector of 8 floating point numbers, representing in

order the 7 image quality descriptors listed above, followed by a predicted OCR word error rate.

The Los Alamos software was trained to predict the error of the Caere OmniPagePro recognition

engine1

. (The image quality assessment utility is the only software used to produce elements of

the database that is not included in this distribution.)

The image quality results for each page of a given FR book are stored together in a single file

named MMDD.qua in a subdirectory book01/txt. Each line in an image quality assessment file

contains the results for a specific page. A line begins with the page’s root file name followed by

the computed image quality descriptors and a predicted OCR word error rate. For example, the

results for body page 100 are recorded in the file data/01/03/book01/txt/0103.qua as:

In this case, the predicted word error rate is 33.4%.

3.2.2.5 OCR SGML Files

Results from parsing each XDOC file are stored within an SGML file that contains the

recognized words and includes line break information. The files contain noisy text generated by

the commercial OCR process and can be used in IR experiments. The OCR system reports

results one isolated line of text at a time without any additional logical context. As a result, the

OCR text is simply tagged at the line level. A DTD for these OCR SGML files is provided in

dtd/fr94ocr.dtd.

An OCR SGML file is provided for every XDOC file in the database. All OCR SGML files for

a given FR book are stored in a subdirectory book01/txt with extension oml. An example of a

portion of one these files is listed in Figure 14. This figure lists the OCR results corresponding

mostly to the top-middle column of body page 100. This data is stored as the file

data/01/03/book01/txt/b0000100.oml.

Notice that the OCR system merged the first three lines of the right-most column with the first

three lines of the middle column. This OCR text can be compared to the passage of GPO SGML

text listed in Figure 11. Notice that the OCR SGML is tagged line by line, whereas the GPO

SGML is tagged paragraph by paragraph.

	
			




27

Figure 14. Portion of an OCR SGML file.

3.2.2.6 OCR Word Index Files

Each OCR SGML file is formatted into a word index file (the same format used to construct the

master word index file in Section 3.2.1.3). This representation facilitates matching a page’s

OCR results to a book’s text parsed from the GPO typesetting files. It also allows for the crossreferencing of words while maintaining each word’s logical context. An OCR word index file

contains an entry for each recognized word from a page image in an OCR-derived order. This

order does not always match the GPO’s typesetting order as seen in Figure 14.



=3!

	










*=3!

=3!








*=3!

=3!



	



5!?=#::>?=# :

*=3!

=3!

 

!"


 33 !

*=3!

=3!

#
$E%&'(')
""9%F
!(E&)7;

*=3!

=3!

&'&*)+&,/(012

*=3!

=3!

%
5%



&67221

*=3!

=3!





*=3!

=3!

%



8
".

!"

*=3!

=3!




#


*=3!

=3!

9:%21,/1&'('"
7&,/1@E21/+50(;

*=3!

=3!

<3=?!> G6116E7E

*=3!

=3!

!#3 !#==#< ::=#3 !

*=3!

=3!

< #:%

*=3!

=3!

#..
3	"




*=3!

=3!


E8



:



*=3!

=3!

<


*=3!

28

Figure 15. Portion of an OCR word index file.

An OCR word index file is provided for every OCR SGML file in the database. All OCR word

index files for a given FR book are stored in a subdirectory book01/txt with extension odx.

Figure 15 lists a portion of the OCR word index file for body page 100. This data is stored in the

file data/01/03/book01/txt/b0000100.odx.

3.2.2.7 Rejected Hits Plot

As part of the ground truth extraction process described in Section 2.2.4, a page’s OCR text is

matched against the GPO text for an entire FR book, pinpointing the page’s location within the

GPO text. This location is referred to as a hit. Due to OCR errors and anomalies in the GPO

typesetting text, a computed page hit is not always accurate. An automated method of rejecting

bad hits was developed, and the results after imposing this rejection criteria are plotted. An

example of one of these plots is shown in Figure 5.

There is one rejected hits plot computed and stored for each FR book processed, and each point

in the plot represents a page whose hit was either accepted or rejected. Each of these plot files



1)( 1)( %
5 ''''7''" 9% 5';9=3!5)2;

1)) 1)) %



 ''''7''" 9% 5';9=3!5)2;

1)6 1)6 &6 ''''7''" 9% 5';9=3!5)2;

1)+ 1)+ 7221 ''''7''" 9% 5';9=3!5)2;

1)2 1)2 
 ''''7''" 9% 5';9=3!56';

16' 16'  ''''7''" 9% 5';9=3!56';

167 167 
 ''''7''" 9% 5';9=3!56';

16& 16& %


 ''''7''" 9% 5';9=3!567;

161 161 
 ''''7''" 9% 5';9=3!567;

160 160  ''''7''" 9% 5';9=3!567;

16( 16( 8
" ''''7''" 9% 5';9=3!567;

16) 16) .

 ''''7''" 9% 5';9=3!567;

166 166 !" ''''7''" 9% 5';9=3!567;

16+ 16+ 
 ''''7''" 9% 5';9=3!56&;

162 162 
 ''''7''" 9% 5';9=3!56&;

1+' 1+' 
 ''''7''" 9% 5';9=3!56&;

1+7 1+7 #
 ''''7''" 9% 5';9=3!56&;

1+& 1+& 9: ''''7''" 9% 5';9=3!561;

1+1 1+1 % ''''7''" 9% 5';9=3!561;

1+0 1+0 21,/1&'('''''7''" 9% 5';9=3!561;

1+( 1+( "
 ''''7''" 9% 5';9=3!561;

1+) 1+) 7&,/1@E21/''''7''" 9% 5';9=3!561;

1+6 1+6 +50( ''''7''" 9% 5';9=3!561;

1++ 1++ ; ''''7''" 9% 5';9=3!561;

1+2 1+2 <3=?!> ''''7''" 9% 5';9=3!560;

12' 12'  G ''''7''" 9% 5';9=3!560;

127 127 6116E7E ''''7''" 9% 5';9=3!560;

12& 12& !#3 !#= ''''7''" 9% 5';9=3!56(;

121 121 =#< : ''''7''" 9% 5';9=3!56(;

120 120 :=#3 ! ''''7''" 9% 5';9=3!56(;

12( 12( < #:% ''''7''" 9% 5';9=3!56);

12) 12) #..
 ''''7''" 9% 5';9=3!566;

126 126  ''''7''" 9% 5';9=3!566;

12+ 12+ 3	" ''''7''" 9% 5';9=3!566;

122 122  ''''7''" 9% 5';9=3!566;

0'' 0'' 


 ''''7''" 9% 5';9=3!566;

0'7 0'7  ''''7''" 9% 5';9=3!56+;

0'& 0'& 
E ''''7''" 9% 5';9=3!56+;

0'1 0'1  ''''7''" 9% 5';9=3!56+;

0'0 0'0 8



 ''''7''" 9% 5';9=3!56+;

0'( 0'( :

 ''''7''" 9% 5';9=3!56+;

0') 0') <
 ''''7''" 9% 5';9=3!562;

29

contain a PostScript encoded graph. There is one rejhits.ps file provided in each book01/txt

subdirectory in the database. The plot in Figure 5 is stored in the file

data/01/03/book01/txt/rejhits.ps.

3.2.2.8 Ground Truth Word Index Files

A ground truth word index file contains all words extracted from the GPO master word index

determined to appear on a specific FR page image. A ground truth word index file is provided

for every OCR XDOC file in the database (excluding those pages with rejected hits).

All ground truth word index files for a given FR book are stored in a subdirectory book01/txt

with extension gdx. Figure 16 lists the beginning and ending of the ground truth file derived for

body page 100. This data is stored in the file data/01/03/book01/txt/b0000100.gdx. Notice that

in this example the ground truth text was accurately trimmed to the beginning and ending of the

page.

Figure 16. Portion of a ground truth word index file.

3.2.2.9 Ground Truth Error Estimate Files

Locating page boundaries when matching a page’s OCR text to the GPO typesetting text is a

difficult, inexact process. Page boundaries may be obscured by a number of factors including

quality of the OCR text and word order discrepancies between the OCR and GPO texts. To help

manage this uncertainty, an error estimate is computed for each ground truth word index file

&&7&1+ 12062 ,-.
/
H'1C1 9% 5+;9! 3562;95716&2;

&&7&12 120+' :
.
 
'1C1 9% 5+;9! 3562;95716&2;

&&7&0' 120+7 ,-.
/

'1C1 9% 5+;9! 3562;957161';

&&7&07 120+& :
.
 
'1C1 9% 5+;9! 3562;957161';

&&7&0& 120+1 ,-.
/$
.4	.*#	"'1C19% 5+;9! 3562;9571617;

&&7&01 120+0 
	"
 
'1C1 9% 5+;9! 3562;9571617;

&&7&00 120+(  
'1C1 9% 5+;9! 3562;9571617;

&&7&0( 120+) 

 
'1C1 9% 5+;9! 3562;9571617;

&&7&0) 120+6 

 
'1C1 9% 5+;9! 3562;9571617;

&&7&06 120++ 3 
'1C1 9% 5+;9! 3562;957161&;

&&7&0+ 120+2  
'1C1 9% 5+;9! 3562;957161&;

&&7&02 1202' .


 
'1C1 9% 5+;9! 3562;957161&;

&&7&(' 12027  
'1C1 9% 5+;9! 3562;957161&;

&&7&(7 1202& 
 
'1C1 9% 5+;9! 3562;957161&;

&&7&(& 12021 

 
'1C1 9% 5+;9! 3562;957161&;

&&7&(1 12020 
 
'1C1 9% 5+;9! 3562;957161&;



&&&766 0'07+ 	"" 
'1C1 9% 5+;9! 35+&;9571662;

&&&76+ 0'072 .

 
'1C1 9% 5+;9! 35+&;9571662;

&&&762 0'0&' .

 
'1C1 9% 5+;9! 35+&;9571662;

&&&7+' 0'0&7 

 
'1C1 9% 5+;9! 35+&;9571662;

&&&7+7 0'0&&  
'1C1 9% 5+;9! 35+&;9571662;

&&&7+& 0'0&1 

	
" 
'1C1 9% 5+;9! 35+&;9571662;

&&&7+1 0'0&0 	
 
'1C1 9% 5+;9! 35+&;9571662;

&&&7+0 0'0&( 
 
'1C1 9% 5+;9! 35+&;9571662;

&&&7+( 0'0&) " 
'1C1 9% 5+;9! 35+&;9571662;

&&&7+) 0'0&6 .

 
'1C1 9% 5+;9! 35+&;9571662;

&&&7+6 0'0&+ .- 
'1C1 9% 5+;9! 35+&;9571662;

&&&7++ 0'0&2 
 
'1C1 9% 5+;9! 35+&;9571662;

&&&7+2 0'01' 


"
 
'1C1 9% 5+;9! 35+&;9571662;

&&&72' 0'017  
'1C1 9% 5+;9! 35+&;9571662;

&&&727 0'01& .
.

 
'1C1 9% 5+;9! 35+&;9571662;

&&&72& 0'011 	
 
'1C1 9% 5+;9! 35+&;9571662;
30

produced. This is a conservative estimate computed by aligning a page’s OCR text with its

corresponding ground truth text file. Blocks of out-of-order text are accounted for, and the

remaining number of deleted and inserted words are used to represent how poorly the derived

ground truth matches the page.

The error is estimated using the following formula:

nwords

ins del

error

( + ) = (1)

where ins is the number of inserted words after alignment and reordering, del is the number of

deleted words after alignment and reordering, and nwords is the number of words recognized on

the page by OCR. This error estimate loosely represents the reliability of the associated derived

ground truth. Using these estimates, experiments can be designed by selecting pages that have

an "acceptable" level of potential error.

The provided utilities compute an error estimate for every ground truth word index file in the

database. Each of these error estimates is stored to a separate file. These small files have been

concatenated together into a single compressed file for distribution. All the ground truth error

estimates for the pages of a given FR book are stored in a subdirectory book01/txt in a single file

named MMDD.est. For example, the error estimate for body page 100 is recorded in the file

data/01/03/book01/txt/0103.est as:

In this case, the amount of word errors contained in the derived ground truth is estimated to be

1.4%, which is quite low.

3.2.2.10 Ground Truth SGML Files

The extracted ground truth word index files contain the hypothesized words on a page and their

logical context. Given the context, the word index files are converted into an SGML format

where tags (listed in Table 2) are provided to identify paragraphs of text, figures, tables,

footnotes, and some hierarchical labeling of sections within the FR. The full set of tags is

defined in the DTD file dtd/fr94gpo.dtd.

Ground truth SGML files can be used as baseline text for indexing and retrieval experiments.

One of these files is provided for each ground truth word index file in the database. All ground

truth SGML files for a given FR book are stored in a subdirectory book01/txt with extension

gml. Figure 17 lists the derived SGML ground truth for the beginning and ending of body page

100 (shown in Figure 26). This data is stored in the file data/01/03/book01/txt/b0000100.gml.

3.2.2.11 OCR Scoring Files

The string alignment technology used in ground truth extraction has been bundled into a wordlevel scoring package which is also included in this distribution. This package is designed to

take an OCR word index file and reconcile it against a corresponding ground truth word index

file, accumulating various word-level statistics.


31

Figure 17. Portion of a ground truth SGML file.

% 

% ! 

''''7''"

*% ! 

! 3



,-.
/
H:
.


*



,-.
/

:
.


*



,-.
/$
.4	.*#	"
	"






*



3.







"



	"

.



F
-.
.

*

! 34%#

%
5%



&+7221

*! 34%#



-	""

*



#
-







*



9:%21,-.
/1&')+"
7&,-.
/1',-.
/21/+50(;

*



<3==3!> %6(7',-.
/'7,-.
/

*

*! 3

! 3



!#3 !#= ?!%#3 !!D#:#!%DD?#!33

*






.
.

I
.	"""
.

-
"



B	


7':.
('

#..
I
3J&

"


D<:	""-
I






-."

<-"



!


&77221	.."


!


&&7221
"



B	




I
.



B	


7':.
('#..
I		"


I



D<::""


-8"7221
"


."
	

	""4.
.
I


"

	
"""

.



K!


1'7221
"




B	


I
.






	

.


I



	

"


	"


	
	
	"
I



D<:


-8"	


	




	
-	
	"
	







	


.
.

"-"".


D<:





B	


		"


I



D<:


-8"D



"


.
.


.




I


"

	
"""


"	




	

	"
!


1'7221
"




B	


"-



.

7221	"
I



!


1'7221


F
&77220




B	



I





-
"



	

-	"
	"


.
.
!


1'7221

	"

"







-	


	

	.


	
"	
7(





	"".

.





	
"	

".

.-





"
.
.

	


*

*! 3

*% 
32

Figure 18. Example word scoring report.

Scoring results are recorded for each FR page in an OCR scoring file. An example of a scoring

report is shown in Figure 18. The scoring package accumulates errors after applying string

alignment alone. No accounting for out-of-order blocks of text is currently done; however, such

algorithms are provided in the distribution and could be incorporated. The word error rate is

computed as:

cor wrn del ins

wrn del ins word error

+ + +

+ + = ( ) _ (2)

where wrn is the number of wrong words, del is the number of deleted words, ins is the number

of inserted words, and cor is the number of correct words.

The scoring files can be used to assess and compare OCR accuracy. One of these files is

provided for each pair of OCR and ground truth word index files in the database. All word

scoring files for a given FR book are stored in a subdirectory book01/txt with extension scr.

The example shown is for body page 100 and is stored in the file

data/01/03/book01/txt/b0000100.scr.

3.2.2.12 Ground Truth Error Estimate Plot

A PostScript encoded plot file of all the ground truth error estimates is provided for each FR

book in the database. An example of one of these plots is shown in Figure 6. All the error

estimates for a book are sorted in increasing order and then plotted on a log scale. This gives the

recipient a snapshot of the quality of the ground truth for a given FR book.

There is one errest.ps file provided in each book01/txt subdirectory in the database. The plot in

Figure 6 is stored in the file data/01/03/book01/txt/errest.ps.

4. SUMMARY

NIST has produced a document image database designed to evaluate both DAR and IR

technologies. The database is comprised of scanned images and ground truthed pages taken

from the 1994 Federal Register for the month of January. In total, there are 4711 full page

	#	&6'251)5'1722+

*72*

*
20***




*72*

*
20***'7*'1*F'7*
F*''''7''I

*72*

*
20***'7*'1*F'7*
F*''''7''I

D8"
5*72*

*
20***'7*'1*F'7*
F*''''7''I

:"
5*72*

*
20***'7*'1*F'7*
F*''''7''I

$ :% :3!>#335

L:




$
M2((

LD-.
$
M202

L


$
M+(&

L$
$
M26

L%
"

$
M)

L3


$
M6

$ :%:: ::#M7701N
33

images, comprising 20 FR books. An automated truthing process generated 4519 ground truth

files (96% of the total number of images.) If there is sufficient interest, future volumes of other

FR months may be released.

The ground truth files were created by matching noisy OCR from each page to typesetting files

containing the text for an entire book. A page’s location in the typesetting text was located, its

boundaries detected, and its corresponding text extracted. This process is very difficult due to

OCR errors and caveats of the typesetting files. To help manage uncertainty, an error estimate is

provided with each derived ground truth file that conservatively represents the percentage of

words missing and/or inserted in each ground truth file.

Figure 19 plots the total distribution of ground truth error estimates for this database (the entire

month of January). The pie chart is divided into ranges of estimate values, starting with the

largest wedge on the right and working clockwise: [0%-1%], [1%-5%], [5%-10%], and so on.

The percentage of files represented by each wedge is provided in parentheses below each range

label. For example, 53% of the ground truth files have an error estimate below 1%, whereas

17% of the ground truth files range between 1% and 5%. The number displayed on top of each

wedge is the total number of pages counted in each range.

Figure 19. Ground truth error estimates for all of January.

These error estimates are provided so that experiments can be designed by selecting pages that

have an “acceptable” level of potential error. As can be seen in the chart, 70% of the ground

truth files have an error estimate below 5%, and 77% of the files are below 10%.

A host of data files are provided in addition to page images and their ground truth files.

Commercial OCR results are provided on each page image, corresponding word error rates are

DISTRIBUTION OF GROUND TRUTH ERROR ESTIMATES

0-1

(53%)

1-5

(17%)

5-10

(7%)

10-20

(7%)

20-50

(7%)

50-100

(5%)

100-

(4%)

2434

201

209

298

297

323

757
34

reported, image quality analysis results are included, and more. These files may be used to

support a wide variety of experiments.

The software (in the form of Perl scripts and C programs) that was used to derive the ground

truth files is also included in this distribution. These utilities serve as documentation and may be

augmented and improved to produce similar data collections. With this new ground truthing

technology, it is now feasible to produce much larger data sets, at much lower cost, than was

ever possible with previous labor-intensive, manual data collection projects.

This is public domain code, and any portion of it, may be used without restrictions because it

was created with U.S. government funding. This software was produced by NIST, an agency of

the U.S. government, and by statute is not subject to copyright in the United States. Recipients

of this software assume all responsibilities associated with its operation, modification, and

maintenance.

In all, the distribution is 1.27 gigabytes in size, requiring two CD-ROMs. The first disc (614

megabytes in size) contains all the software and documentation for the distribution. The data

files are divided across the CD-ROMs. The first disc contains the data files for FR days 01

through 13 (excluding day 05, which is included on the second disc due to space limitations).

The second disc contains day 05 and days 14 through 31. The second CD-ROM (containing

only data) is 649 megabytes in size. All the data files are compressed: image files are

compressed using CCITT Group 4[16] and all other data files are compressed using GNU

GZIP[13]. The GZIP-1.2.4 distribution is provided with the database.

Volume 1 of the Federal Register Document Image Database, NIST Special Database 25, is

distributed by the Standard Reference Data Group at NIST. To get pricing information and/or to

order the database on CD-ROM, please contact:

Standard Reference Data Program

National Institute of Standards and Technology

Bldg. 820, Rm. 113

Gaithersburg, MD 20899-0001

(301)975-2208 (voice)

(301)926-0416 (FAX)

srdata@nist.gov (e-mail)

This document references numerous NIST Internal reports. These papers are included in

PostScript format in the distribution. They may also be accessed via our group’s Web page,

http://www.itl.nist.gov/div894/894.03 . Hardcopies may be requested from:

ITL Publications

National Institute of Standards and Technology

Bldg. 225, Rm. A216

Gaithersburg, MD 20899-0001

(301)975-2832 (voice)

(301)926-3696 (FAX)

itlab@nist.gov (e-mail)
35

5. REFERENCES

[1] M.D. Garris, “Document Image Recognition and Retrieval: Where Are We?,” NIST Internal

Report 6231, October 1998.

[2] ISO-8879, Information Processing – Text and Office Systems - Standard Generalized

Markup Language (SGML), Standard by the International Organization for Standardization,

1986.

[3] E.M. Voorhees and D.K. Harman Eds., “The Fifth Text REtrieval Conference (TREC-5),”

NIST Special Publication 500-238, November 1997.

[4] M. Cannon and P. Kelly, “An Automated System for Numerically Rating Document Image

Quality,” Proceedings of Document Recognition IV, Vol. 3027, SPIE, San Jose, CA, February

1997.

[5] M.D. Garris, W.W. Klein, “Creating and Validating a Large Image Database for

METTREC,” NIST Internal Report 6090, December 1997.

[6] H.G. Zwakenberg, “Inexact Alphanumeric Comparison,” The C Users Journal, pp. 127-131,

May 1991.

[7] “UW-II English/Japanese Document Image Database,” CD ROM, University of Washington,

1995.

[8] ISO-9660, Information Processing – Volume and File Structure of CD-ROM for Information

Interchange, Standard by the International Organization for Standardization, 1988.

[9] M.D. Garris, S. Janet, and W.W. Klein, “Impact of Image Quality on Machine Print Optical

Character Recognition,” NIST Internal Report 6101, December 1997.

[10] “TIFF Revision 6.0,” Aldus Corporation, Seattle, WA, June 1992, off the Web at

ftp://ftp.sgi.com/graphics/tiff/TIFF6.ps.Z .

[11] M.D. Garris, J.L. Blue, G.T. Candela, P.J. Grother, S.A. Janet, and C.L. Wilson, “NIST

Form-Based Handprint Recognition System (Release 2.0),” NIST Internal Report and CD-ROM

5959, January 1997.

[12] “XDOC Data Format: Technical Specification,” Part Number 00-08137-00, Xerox

Corporation, March 1995.

[13] “GNU Project,” Free Software Foundation, 59 Temple Place – Suite 330, Boston, MA,

02111, http://www.gnu.org .

[14] L. Wall, et al, “Practical Extraction and Report Language (Perl),” Ver. 5.004,

Comprehensive Perl Archive Network (CPAN), http://www.perl.com .

[15] ISO/IEC 8859-1:1998, Information Technology – 8-bit single-byte coded graphic

character sets – Part 1: Latin alphabet No. 1, Standard by the International Organization for

Standardization, 1998.

[16] CCITT, "Facsimile Coding Schemes and Coding Control Functions for Group 4

Facsimile Apparatus, Fascicle VII.3 - Rec. T.6," 1984.
36

APPENDIX A. SOFTWARE REFERENCE GUIDE

This appendix provides a reference guide to the software distributed with the Federal Register

Document Image Database. All the software tools used to produce the data have been provided

in this distribution.3

 The software is written as a collection of Perl scripts and C programs and is

intended to document the data production process and enable recipients to augment and improve

the process if desired.

This reference guide is comprised of two major items. The first is a sequence of tables that

document the order in which the software utilities were invoked to produce the data. These

tables are then followed by a series of manual pages that give an overview of each script or

program and document their purpose and how they are invoked.

A.1 Order of Software Invocation

The illustration in Figure 1 depicts the processing of two sources of data (GPO-provided files on

the left and OCR-generated files on the right) with the results of their matching in the middle at

the bottom. Using this illustration as a reference, the software utilities are grouped into three

categories: 1) utilities that process GPO Microcomp files; 2) utilities that generate OCR results;

and 3) utilities that conduct matching between the GPO files and OCR results to generate pages

of ground truth text. Each of these stages is represented by a separate table below. The middle

column of these tables lists the file name of the utility; the left column lists the types of input to

the utility by file extension; the right column lists the types of output (primarily by extension).

The first group listed in Table 3 are those scripts used to process the GPO Microcomp files. The

top-level script, gpomonth.pl, processes all the Microcomp files for a month. The middle-level

script, gpoday.pl, processes all the Microcomp files in each daily issue. The low-level scripts

parse the Microcomp files for each day into SGML (gpoparse.pl), subsequently generating a

master word index file (mkwdindx.pl) and corresponding word trigram files (mktrigrm.pl and

mkunqtri.pl).

INPUT SCRIPT / PROGRAM OUTPUT

gpomonth.pl

gpoday.pl

xxx

nnn gpoparse.pl sgm

sgm mkwdindx.pl wdx

wdx mktrigrm.pl wd3

wd3 mkunqtri.pl wf1

Table 3. Order in which GPO Microcomp files are processed.

 

3

 All software has been provided except for the image quality assessment program developed at Los Alamos

Laboratories [4]. This program was invoked in the Perl script doimqual.pl.
37

The second group is listed in Table 4 and contains the scripts and programs used to generate

OCR results and predict OCR error. The top-level script, ocrmonth.pl, processes all the FR page

images in a month. The middle-level script, ocrbook.pl, processes the FR page images scanned

from a single book. The low-level program , xreadpg, runs ScanWorX on each page image; and

the script, doimgqual.pl3

, assesses the image quality of the page image and computes a predicted

word error rate.

INPUT SCRIPT / PROGRAM OUTPUT

ocrmonth.pl

ocrbook.pl

pct xreadpg xdc

pct doimqual.pl3 qua

Table 4. Order in which OCR result files are generated.

The last group of utilities is listed in Table 5 and contains the scripts and programs used to match

OCR results to the GPO files in order to generate page-level ground truth files. The top-level

script, trumonth.pl, processes all the GPO files and OCR results in a month. The middle-level

script, trubook.pl, generates all the ground truth files for all the pages of a FR book.

INPUT SCRIPT / PROGRAM OUTPUT

trumonth.pl

trubook.pl

xdc doxdcprs.pl oml

oml mkwdindx.pl odx

odx mktrigrm.pl od3

od3

wf1 doqryhit.pl hit

hit rejhits.pl

hitg

hitb

rejhits.ps

hitg

wdx cutchunk.pl cdx

odx

cdx

hitg

pgfrchnk gdx

est

gdx wdx2sgml.pl gml

odx

gdx wrdscore scr

est ploterr.pl errest.ps

Table 5. Order in which GPO files and OCR results are processed together.
38

The low-level utilities in this group begin with the script, doxdcprs.pl. This script parses the

XDOC-formatted OCR results into SGML. Mkwdindx.pl converts the OCR SGML files into

word index files. Mktrigrm.pl computes trigrams from each OCR word index file. Doqryhit.pl

matches the OCR trigrams against the unique trigrams from the issue’s GPO files and computes

a hit for each page (the median of the clustered matches). Rejhits.pl rejects bad hits and then

plots the results. (Accepted or “good” hits are labeled in Table 5 as hitg, and rejected or “bad”

hits are labeled as hitb.) Cutchunk.pl extracts a chunk of GPO text centered around the hit

location. Pgfrchnk trims the extracted chunk by aligning the OCR text to the GPO text, creating

a ground truth word index file. Wdx2sgml.pl converts the ground truth word index file to SGML.

Wrdscore reconciles the OCR word index file with the ground truth word index file,

accumulating errors and computing a word error rate. Finally, ploterr.pl plots the ground truth

error estimates computed by the program, pgfrchnk.

A.2 Manual Pages

This section contains manual pages for the software utilities included in this distribution. These

manual pages are provided in plain text and in UNIX nroff format, both under the top-level

distribution directory man. The utilities are listed here alphabetically. Refer to the previous

section for the order of their execution.
39

	

























	


NAME








 
!"
#$% &
%

"!
%
'""(

)






#$
%*#+
"
"% 
%
# ! 
%&#

SYNOPSIS

 cutchunk.pl

hit_file master_word_index

DESCRIPTION






Cutchunk.pl
 
%
	

 

%

%#

#

)#%#+






"% "

"#+
 
%
%&#(
 
! #
%+
#$%
%
"#






,% ,#-
%&#

"!
%
'""(
)

#$


#

#$%#+







 
###+
"
#
 
,%#
%+
 
%*#+
"
"% 






#
# #
"#
"!
#
"#"+ &
%&#
#

#$%#+






%%&#
"!
#$
 
 #+
"
standard output

OPTIONS

 hit_file











#
 
! #
!"
#
%&#
#-
'# &
"##+

 master_word_index











#
*%#
."+
 +#$
! #
"%  &
%
#
)

#$











!"
#
"#"+ &
'""

EXAMPLES

 % cutchunk.pl $DAYDIR/book01/wrk/b0000100.hit \

 $DAYDIR/gpo/wrk/0103.wdx > $DAYDIR/book01/wrk/b0000100.cdx






.##
/01
2
33"%3!453+%%36367

FILES

 input











.hit









! #

. 

 " 

"% "

"!


%&#

























'#.##

%+
)
#$











.wdx









)
*%#
."+
 +#$
! #
!"
%
'""

 output











.cdx









#$%#+
)

."+
 +#$
! #

SEE ALSO

 trubook.pl(1), doqryhit.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;


















40

<01	

























<01	


NAME






+"=- 

 
!"
 "  &
#
"% "
"!
%

%&#






. 
#
)", +#+
#$
!"
%
# #
'""

SYNOPSIS






doqryhit.pl

OCR_trigrams GPO_unique_trigrams

DESCRIPTION






Doqryhit.pl
 
%
	
 
%
"%#
#
%#%
.  

%






'""(

)
#$
%
"+
#
"#
"!
%
# ! 
%&#






 
 
%"* #+
'-
*% &
."+
 &%*
!"*
%
%&#(








#$

"

#
 &%*
" &
 =#-
 
#
'""(






)
#$
#
"% "
"!
#%
*%
 

#"+#+>

%+

#






*#+ %
*% &
 &%*
 
###+
%
#
hit
#
"% "






"!
#
 
 " 
#
%&#
.  
#

'""(

#$


1






 
.%->
#

#
%#
#+
"
?=#-?
#
)
#$






#
"% "
"!
#
*#+ %
*% &
 &%*
.  
#








%+

)
#$
%#
 #+
"
standard output
%"&
. 
#






"%
*'#
"!
*% &
 &%*

OPTIONS

 OCR_trigrams











#

."+
 &%*
! #
#+
"
=#-
#
'""(

)











#$

 GPO_unique_trigrams











#
! #
"%  &
%
."+
 &%*

%+
# 

"%











 "

%
"
#$%-
"#
%"
#
)
#$
!"











%
& ,#
'""

EXAMPLES

 % doqryhit.pl $DAYDIR/book01/wrk/b0000100.od3 \

 $DAYDIR/gpo/wrk/0103.wf1 > $DAYDIR/book01/wrk/b0000100.hit






.##
/01
2
33"%3!453+%%36367

FILES

 input











.od3










."+
 &%*
! #











.wf1









)
 =#
."+
 &%*
! #

 output











.hit









! #

. 

 " 

"% "

"!

%&#

























'#.##

%+
)
#$

SEE ALSO

 trubook.pl(1), mkunqtri.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















:
41

@	

























@	


NAME






+"$+

 
!"
% &

#
!"
%
%&#
!"*






%
@
! #

SYNOPSIS






doxdcprs.pl A



output flags



B


output_rootname






XDOC_file

DESCRIPTION






Doxdcprs.pl
 
%
	
 
%
 
#+
"
#$%
,% "






-#
"!
 !"*% "
"#+
 
%
@
! #
@
! #
"






% 
#

#

&##%#+

'-

@#"$(

%C"@

1






##

%#

%

*'#
"!
!%&
. 
+##* #
.%
-#
"!






 !"*% "
%#
"
'#
%#+
%+
#$%#+
1!

"

!%&

 






# ! #+>
#
+#!%
 
"
#$%
#
#"& D#+
#$
%+






"
 
 
)	
!"*%
1!
*"#
%
"#

!%&

 

# 






! #+>

#

"#
"
! #
 
#%#+
!"
#%
!%&
# 






! #+
#
"*##
#
"!
"
!%&
 
+#! #+
'#".

OPTIONS







output flags












-sgml









)	
#$

+#!%











-txtfmt







% 
#$

1
;;E43
. 
%#

!"


























*% &











-txtunfmt





% 
#$


1

;;E43

. "

%#


























!"*% &











-wdcon








."+
"! +##
,%#











-chcon








%%#
"! +##
,%#











-wdbox








."+
'"+ &
'"$#

 output_rootname











#
""
! #
%*#
"
'#
#+
!"
"
! #

 XDOC_file











#
@
! #
"
'#
%#+

EXAMPLES

 % doxdcprs.pl -sgml $DAYDIR/book01/img/b0000100 > \

 $DAYDIR/book01/img/b0000100.xdc






.##
/01
2
33"%3!453+%%36367

FILES

 input











.xdc










@
! #

 output











.oml










)	
! #











.fmt









% 
#$
! #
. 
%#
!"*% &











.txt









% 
#$
! #
. "
%#
!"*% &


" #+

42

@	

























@	


 .wcn









."+
"! +##
! #











.ccn









%%#
"! +##
! #











.wbx









."+
'"+ &
'"$
! #

SEE ALSO

 trubook.pl(1), xreadpg(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















7
43

)0	





























)0	


NAME






&"+%-

 
!"
"# &
%
)", +#+
! #

 






%
+%-

SYNOPSIS






gpoday.pl




install_dir





src_GPO_dir





dst_SGML_dir






dst_WRK_dir

DESCRIPTION






Gpoday.pl
 
%
	

 

%

"##

%

#

)






", +#+

! #

!"

%

 &#

+% -

 #
"!
#
Federal

 Register






#
9

gpoparse.pl
%#
)
 ""*

! #

 "

)	






#$
! #






#
:9

mkwdindx.pl
#%#
%
*%#
."+
 +#$

! #

!"*






#
)	
! #






#
79

mktrigrm.pl
&##%#
%
! #
"!
."+
 &%*

!"*






#
*%#
."+
 +#$
! #






#
59

mkunqtri.pl
#%#
%
 
"!
%

 =#

 &%*






!"*
#
."+
 &%*
! #

OPTIONS

 install_dir











#
!
%%*#
"
#
+ #"-
 
. 

#

+%%











'%#
+  ' "
.%
 %#+

 src_GPO_dir











#
%%*#
"
#

"#

+ #"-

"%  &

)











 ""*

! #

#+
"
-##
#
Federal Register












install_dir3+%%3MM3DD3&"3"&

 dst_SGML_dir











#
%%*#
"
#
+# % "
+ #"-
.##
#











 &



)	



#$



! #



%#


"


'#


"#+












install_dir3+%%3MM3DD3&"3&*

 dst_WRK_dir











#
%%*#
"
#
+# % "
+ #"-
.##
#











 &

."+

 +#$

%+

 &%*

! #
%#
"
'#
"#+












install_dir3+%%3MM3DD3&"3.

EXAMPLES

 % gpoday.pl /usr/local/fr94 $DAYDIR/gpo/org $DAYDIR/gpo/sgm \

 $DAYDIR/gpo/wrk






.##
/01
2
33"%3!453+%%36367


" #+

44

)0	





























)0	


FILES

SEE ALSO

 gpoparse.pl(1), mkwdindx.pl(1), mktrigrm.pl(1),

 mkunqtri.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















5
45

)	

























)	


NAME






&"*"

*%#
 
!"
"# &
%

)", +#+






! #
 
%
*"

SYNOPSIS






gpomonth.pl 
install_dir month

DESCRIPTION






Gpomonth.pl
 
%
	
 
%

"##

%

#

)






", +#+
! #
!"
#%
 #
"!
#
Federal Register
!"
%






# #
*"
8"

#%

 #>

 

*%#

 

%






gpoday.pl

OPTIONS

 install_dir











#
!
%%*#
"
#
+ #"-
 
. 

#

+%%











'%#
+  ' "
.%
 %#+

 month











#
*"
 
#
%&#
A6:B

%

 

"

'#

"











##+

EXAMPLES

 % gpomonth.pl /usr/local/fr94 01

FILES

SEE ALSO

 gpoday.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















E
46

)	

























)	


NAME






&"%#

",#
 ""*

8#+#%

#& #

+%%

"






)	
"
#%+%'#
#$

SYNOPSIS






gpoparse.pl
A
options
B
Microcomp-file


DESCRIPTION






Gpoparse.pl
 
%
#
"&%*
%
",#
 ""*
! #

"!






Federal Register

+%%

"

)	


#
+#!%
"
#%+%'#






#$







)	
"
! #
"% 
11
#$
"!"* &

"

#






"*#

-#

#!   "




%

 

##

 
#






8#+#%
#& #
"*#
1*%&#
%%'%#
+  ' "


#






! #
"% 
#$>
%
.#
%
%'#
"!
%
##*#







%
%&#-
%*#
%+
!"""#

#
#$

 &

"






 

"!

%+%+

)	

%%#
%+
*%-
"!
#
# %






%%#

+#! #+

 

 $

1


1#% "%

%+%+






&% D% "
%%#
# -
#>

%
)##
-*'"






#
)	
"
"+#+
'-
"# &
#%

 ""*

+%%






! #
!"*
#
445
Federal Register
%
'##
,# ! #+
'-
#






)	
%#
nsgmls
 &
#
"**%+9











F
&*




sgml-dir

dtd-dir3!45&"++

sgml-file






.##
sgml-dir
 
#
"% "
"!
#
%%#

# -

#






! #
%+
#
 ,#"-
! #
catalog.dat>
%+
dtd-dir
 
#






"% "
"!
#

! #

"#
! #
%#
 #+

 

#






8#+#%

#& #
"*#
1*%&#
%%'%#
+  ' "

#






"&%*
nsgmls
 
%
"!

#

!##

"!.%#

%%&#

sp>






. 
 
%,% %'#
%
#
	9











933...G%"*33 +#$*






H-
+#!%>
#
"&%*
#%+
%+
"##

#,#-

 #

"!






#,#-

 ""*

! #
# ! #+
"
#
"**%+
 #

1!
"






! #
%#
# ! #+>
#
%+%+
 
 
"##+






"
 * !-
"# &>
%
 #
 
#

#

 ""*






! #

%#
#%+
!"

H#%#
#$
 
#
 ""*
!"






*%
 
%&#-
!##!"*%>
#

#

! #(

 #

%#






#

"%#%#+
 "
%
 &#
 #

 
'#%, "
%
'#






#+
"!!
. 
#
-o nocombine
" "






#$>
%
 #
 
 
 "
#,#%
 #

#%

'#&  &






. 

#

#=##

\007F

"

\007S>

. 

"#"+
"






Federal Register

# "


#&

 #



## &







 

'#%, "

%

'#

#+

"!!

. 

#

-o nosplit






" "
"#
 #
%
%#
%&#
%
E
'

%#

!#






 
 "
 ##
"!
%"$ *%#-
%
#&
"
#>
#%


" #+

47

)	

























)	







'#&  &
. 
#

#=##

\007I>

. 

"#"+

"






"*%

Federal Register
#$


 
'#%, "
%
'#
#+






"!!
. 
#
-o noresplit
" "
#
E
'
 D#
.%

"#






*% -

'#%#

 
%#%#+
"
'#
*"#
#!!  #
%
:6
'






"
66
'
.#
+" &
%#
"#% "
"

#

 #


#+ 






%+

 >
'
%"
'#%#
##
%#%#+
"
'#
%
'&
 






#(
#&%

#$# "

"#% "

.#

%+ &

*






%&#

#&
*%-
*#&%'-#
#=##






#
"# &
"!
#%
 #
#
'#& 
. 

#

!"". &






#+  &
#9










I

&%'%&#
#=##


%

J:EKALBALBJ:EM

%#













#*",#+










I

%%#
%
%,#
"
 *%

"
".
 *%
"













!"*% &




#&

% %&#

#>
%'>
 +#>













%%#
\000

%#
#*",#+










I

#=##
%
%,#
"
 *%

"
".
 *%

"













!"*% &

#&
#*-
%'#
##*#>
  #
&%













 
%#
#*",#+










I

%%#
%+
#=##
%

##*

"

%!!#

"-













% &
%#
*%#+
"
'%










I

#%
#=##
"!
* #
%+G%#
'%
 
=##D#+













"
%
 &#
'%







%
" >
 #
"  &
"-
"!
D#"
"
*"#

'%






%#
+ %+#+






%
 #
 
#
'"#
 "
 
"*"#

%

 *#






#$>

%'#

'#& 

%+
#+
"+#>
%'#
##*#>
!"""#






#$>
%+
"**#

 
 
+"#
 &
%
"* %#+

#&






%
#$# "
*%#
%
 
#+
"
*%
!"*
#%
".






#=##
'#&  &
"+#

%-
\007

!"".#+

'-

"#

"






*"#

%%'# 

%%#


">
'
"
 + &>
#






#$
#=##
'#&  &
"+#






%
"*"#
 

#

"##+

'%#+

"

#

#=##






'#&  &
"+#

8"
#$%*#>
#
#=##9











J66M16
J66M:!"

!#

 !"*% "

"%9J66M











N%-
%!
%

:6:
:44
J7MM64
4K76






 
! 
%#+
"
#
'" #
%
%+#

\007

"+#>






#

"

#

'" #

%

%+#

\007 I
"+#>
%+






! %-
"
#
'" #
%
%+#
\007 I10
"+#

%






'" #

#"& D#

#

#=##

%

"#

"!

#
*%-






#=##
!"
% 
#$>
%+
%#
#

#*%  &

%%


" #+

48

)	

























)	







#

 

#

#=##

"

#

'" #

%>
!"
#%






#=##+
"
!"*%>
",#
 
"
#

"#

 &






%+

. #

 
"
#
%" %#
"
#%*

"
 
#






%#
"!
#
#$%*#
#=##>
#
%%#
'#.##
\007 T2






%+

\007 T1
"+#
%#
% % D#+>
%+
#
\377 09
"+#
 






",##+
"
??
!"
#$
"
%+3"

?‐?

!"

)	






"

OPTIONS






-b


#
'%#
%*#
"!
#%
"
! #
. 
'#
#
'%#
%*#











"!

#

"#"+ &

 

! #
>
!"".#+
'-
#











%" %#
#$# "
!"
%
"

-#


1!

"











# &

#

%+%+
 
.#
 
" "
"
-r
%











'##
###+>
"
! #
. 
'#

%*#+

%

 !

#











'%#

%*#

"!
#
 
! #%*#
.%
stdin


#
'%#











%*#
"!
%
! #
 
#
%
#*%  &
%!#
#*", &
%











%%#
"&
#
! %
?3?>
 !
%-
%#
##






-r


#
'%#
%*#
"!
#%
"
! #
. 
'#
#
'%#
%*#











""

"!

#

"#"+ &
 
! #
>
!"".#+
'-











#
%" %#
#$# "
!"
%
"

-#



#











'%#

%*#
""
 
#
'%#
%*#

">
'
"
 +











 &>
#
! %
??>
 !
%-
%#
##


1!

"#

%#











##>

#

'%#

%*#

""
 
#
%*#
%
#
'%#











%*#






-D
outdir












! #
"+
'#
%#+
 
#
+ #"-

outdir











#
+#!%
 
#
#
+ #"-






-I
incdir











##+
#
+ #"-
incdir
"
#

#

%%-

@INC>











#+
"
"%#
#= #+
 '%-
pl
! #






-L
libdir











#
#
 '%-
+ #"->
#+
!"
"% &

%$  %-











#$
! #






-S
sgmldir











#
#
)	
+ #"->
#+
!"
"% &
)	

%%











#
# -
#
! #






-R
rootdir











#
#
+  ' "
""
+ #"->
#+
.#

%-

"!











#
%'",#
##
%#
"
#






-s


"
"
. #
)	
! #


" #+

49

)	

























)	







-u


C #
!"*%#+
#$
! #






-t


C #
% 
#$
! #






-w
#
#
#
"
 #
. +>
#+
.#

.  &

"

#$











%+
!"*%#+
! #

#
+#!%
 
;6
%%#






-l
#
1#%+
"!
"# &
%
 #
"!
%
# ! #+
! #>











"
#$# "
%!#
#
# ! #+
*'#
"!
 #






-U


"

"

. #

+%#


"


"&#


"


#* %












/dev/tty


#

+%#

" 

"!
#
*'#
%+











##%&#
"!
! #
%+

 #

%

%,#

'##

"











##+






-o
option[,option...]











#
* #%#"
"&%*#
" "

"
# !-
* 











#

"&%*#

" ">
#
"**%+
 #
"+
%,#
%











 &#
-o
!"".#+
'-
%
 &
"* #+
"!
#
+# #+











"&%*#
" "
#%%#+
'-
"**%


% &
* 











#
-o
 &
"
#
"**%+
 #
#
 
+#! #+











'#%, "
#
"&%*#
" "
%#9











nocombine




"
"
"*' #
%
 #
 
%

! #

 "

























"#

 #


 
- %-
%
#
* "

























#!!#
%

#$

#=##

 

",#

























* #

 #
%#
"##+
#%%#->

























. 
%
 #
"#
%
#
! 
'# &

























#%#+
%
&## 
#$

8"
#$%*#
#

























 #9



































J66M165
 &
%%&#>



































& #
%+
"##
 #"%#>



































 %!
# ! % "
#, #

























. 
'#
"##+

. 

#

! 

 #

























'# &
%#+
"
#
! "
%
%+#

























%+
 
\007 I04

#=##

%+

#

























"#

 #
'# &
%#+
"
#
! "

























%
 *-
 
&## 
#$











nosplit






1!
"*'  &
 #>

+"

"

#

 

























 "

# "

'#&  &

. 

J66MA8B

























'-#
#=##>
. 
%#

# ! 

'""

























# "

%
 #

## &











noresplit




1!
  &
 #>
 



,#-

"&

























# "
 "
%"$ *%#-
E
'
 ##











fmt










C #
!"*%#+
#$
! #



" #+

50

)	

























)	












footrefs





C #
!"""#
#!###
! #












footnotes




C #
!"""#
#$
! #












contents





C #

! #

. 

**% #

"!


 

























! #(
"#











sections





C #
# "
"&
! #












tables







C #
%'#
! #












comments





C #
"**#
! #












seqlog







C #
#=##
"&
! #












wordlog






C #
."+
"&
! #












linelog






C #
 #
"&
! #







-h


 
 
#
*#%&#
"
#
%+%+
"
%+
#$ 











. 
%
"!
D#"

FILES

 Input files






#
 ""*
! #
. 
'%#
%*#
*% &

#

!"". &






%#
- %-
"% 
+%%
%
+# '#+9











r*











#$
!"
#
"*%
8#+#%
#& #
#

























 "











[po]*








#$
!"
# +# %
"*#











adv*









# 
#$











a*

"#
%
"#
! #
*% &
adv*

























#$
!"
#%+#
 +
# "











u*











8#+#%
#& #
#$

%

45

"!

##

























! #

"* #

#

",#*'#

5>

445

























8#+#%
#& #
'""

 Output files






#
"&%*
%
"+#
#,#%
-#
"!
"
! #

C#






# #

#
-b
"
#
-r
" ">
%#
# ! #+>
#
,% 






"
"
! #%*#
%#
! $#+

!"

#%

-#

"!

"






#=##+>

"#

! #

 
#%#+
"%  &
%
"
!"






%
 
! #

#
"
-#

+##* #

#

! #%*#






#! $>

%+

#

!! $
 
.out

#
! #%*#
%#
 #+






'#".


" #+

51

)	

























)	







C#
# #
#
-b
"
#
-r
" "
 
# ! #+>
#

,% 






"

"

! #%*#

%#
&##%#+
!"*
#
'%#
%*#
"






'%#
%*#
""
"!
#
 
! #>
. 
#$# "
%#+#+






'%#+
"
#
"
-#

"#
#$# "
%#
# ! #+
 






#
%###
 
#
 
'#".9











sgml.out

.sgm

























)	
"
! #











unfmt.out

.unfmt

























!"*%#+
#$
"
! #

"

#!!"

























 
*%+#
"
*%#
#
"
""
 #
!"

























*%#+
8#+#%
#& #
%&#

##

 

























"
 +# &>
"&
 #
%#
"
.%#+>

























#

 
! #
. 

"% 

%%#

























 

#

1

;;E43
%%#
#
'#-"+

























G
"#
 
#
 %'#
11

%&#

























8"

#$%*#>

#
'-#
,%#
#$%+# *%

























6$H>
.#
 #+>
+ %-
"
*"
#

























* %
%
 &#%%# D#+
?3:?











tables.out

.tables

























	 
"!
#%
%'#(
!"*% &
"+#

%+

























%'#
#%+ &
%+
##*#











comments.out

.comments

























	 
"!
"**#
#$
!"+
 
#

 "

























"*
"+#>
%"&
. 
#
"% "

! #

























%+
 #
*'#











seq.log

.seq.log

























	"&
! #
"!
 ""*
! #
#=##











word.log

.word.log

























	"&
! #
"!
 ""*
! #
#$
."+











line.log

.line.log

























	"&
! #
"!
 ""*
! #
 #






"+#
"
%+#
#
"+ "

"!

#

!"". &

! #

 






##

 

gpoparse.pl

".#,#
# 
"#
%#
 "*






##>
# #
'#%#
%, &
#
"
.%
"

%

 &

%






 " -
%
)	
">
"
+#
"
 *#
"% 











fmt.out

.fmt

























8"*%#+
#$
"
! #

"*#

#!!"

























 
*%+#
"
*%#
#
"
""
 #
!"

























*%#+
8#+#%
#& #
%&#

##

 

























 +# &>
"&
 #
%#
.%#+>
#


" #+

52

)	

























)	












text.out

.text

























% 
#$
"
! #

%%#
"#

























%

#
 %'#
11
#
%#
 #+

























 
#
"#
" '#
###% "
"

























"

 #+

%

%


8"
#$%*#>
#

























!% "
"#%!
."+
'#
 #+
%
#

























##
%%#
#=##
?3:?











contents.out

.contents

























	 
"!
##*#
!"+
 

#

 ""*


























%'#>

&% >

!"""#>
"**#>

























#











sections.out

.sections

























	"&
! #
"!
8#+#%

#& #

'""

#

























 "











footnotes.out

.footnotes

























	"&
! #

"!

!"""#

#$

%+

# 

























"% "

! #
%+
 #
*'#











footrefs.out

.footrefs

























	"&
! #
"!
!"""#
#!###
%+
# 

























"% "

! #
%+
 #
*'#











dictionary.out

.dictionary

























	"&
! #
"!
."+
!"+
 
#

 ""*

























"+#

 Auxiliary files






#
"&%*
gpoparse.pl
#= #
!"
 '%-
! #>
%

"!






. 

%#

##

 

#
8#+#%
#& #
"*#
1*%&#






%%'%#
+  ' "9











octal.pl





8 "
!"
  &
 &
. 

#%

























"11
%%#
#%#+
'-
?AJnnnB?>

























.##
nnn
 
#
%%#(
"%
,%#











timestmp.pl


8 "
"
#

%

 &

# !- &

























#

#

 *#>
*#%
"
'#
##+#+

























"
#
 #
. #
"
"&

! #


#

























!"*%
"!
#
 &
 9






























<MONTH> <DAY> <HOUR>:<MIN>:<SEC>











sdata.pl





"+#
"
#%+
)	

%%#

# -

#

























! #


#

! #

"% 

*% &
"!

























# %
%%#
%*#
"

 &


" #+

53

)	

























)	


























8"
#$%*#>
#
*% &
!"
#
!% "

























"#%!
 
# ! #+
%9






























OP10
!%:

?A!%:B?Q











sgmlpars.pl


#% "
"
*% % 
%

%

"!

"#

























)	

%&

#
%
 
#+>
!"
#$%*

























#>
%
#
#+
"!
"# &
%
! #>

"

























"#

%

)	

%&
 
"#
%
%

























" 






 
+  ' "
", +#
 $

)	

%%#

# -

#






! #


1

%++  "

"

'# &

##+#+
!"
#
)	
%#






nsgmls
"
,# !-
)	
">
#
! #
%#
#%+
+ #-
'-






gpoparse.pl


#-

"% 

*% &
%
%#
"#+
 
%






%" % ,#
%%->
%+
,%#
!"*
%
%%-
%#
. #
"






)	


"


! #


%


%" %#


 

 * ! #






gpoparse.pl>
 #
 
+"#
"
##+

"

"% 

%+"+#+






)	# ! 

 !"*% "


#
%%#
# -
#
! #






%#9











isolat1.dat


)	
%%#
# -
#
! #
!"

	% 

























%%#


%
#
&%,#%##+
%>

























####+
'-
?à?











isogrk1.dat


)	
%%#
# -
#
! #
!"

)##

























%%#



%
beta>
####+
'-

























?&bgr;?











isonum.dat



)	

%%#

# -

#


! #


!"

























*# 

%+

# %

&% 
%%#




























%


#



"* 



 &>

























####+
'-
?±?











isopub.dat



)	
%%#
# -
#
! #

!"

'

























  &

%%#


%
#
!% "

























"# +>
####+
'-
?⅓?











isotech.dat


)	
%%#
# -
#
! #

!"

&#

























#%

# %

%%#

%+

-*'"


























%
#
&#%#%"#=%"
-*

























'">
####+
'-
?≥?











isodia.dat



)	

%%#

# -

#


! #


!"

























+ %  %
*%>

%
?˜?






 
+  ' "
%"
", +#
."
! #
##+#+
'-
#
)	






%#
nsgmls
%"&
. 
#
)	
%%#
# -
#
! #






"
,# !-
#
 #& -
"!
)	
"
! #9


" #+

54

)	

























)	












fr94gpo.dtd



"*#
-#
#!   "
!"

#

)	

























! #
&##%#+
'-
gpoparse.pl











catalog.dat



%%"&
"!
#

)	

%%#

# -

























#
! #

ENVIRONMENT






#
"&%*
#"& D#
#,#%
#, "*#
,% %'#


%






"!

#

,% %'#

%
%
"#"+ &
"**%+
 #
" "






#
"**%+
 #
" "
",# +#
#
#, "*#
,% %'#

 GPOPARSE_LIBDIR

























#

! #

 

 

+ #"-


"% 

























8#+#%


#& #

# "

#%+#




























#%+#
"% 
#
#$
%
#
'#&  &

























"!

#% 

'""

# "



%
#

























*%#%+

##
 
"
+#!%
,%#

!"

























 

+ #"->

"

 !
#
+ #"-
 

























"
# ! #+
"
#
"**%+
 #
, %
#

























-L
!%&>
# #
 
#, "*#
,% %'#

























*
'#
+#! #+
"
##
#

""

+ #

























"-
*
'#
# ! #+

 GPOPARSE_SGMLDIR

























#
"&%*
#%+
#
 $
)	

%%#

























# -
#
! #
!"*
 
)	
+ #"-

























##
 
"
+#!%
,%#
!"
 
+ #

























"->

"

 !
#
+ #"-
 
"
# 

























! #+
"
#
"**%+
 #
, %
#
-S
!%&>

























# #

 
#, "*#
,% %'#
*
'#

























+#! #+
"
##
#
""

+ #"-

*

























'#
# ! #+

 GPOPARSE_INCDIR

























1!
"
 +#
+ #"-
 
# ! #+

, %

























#

-I

"**%+

 #

!%&>
%+
 !
 

























,% %'#
 
#>

#

#

,%#

 

%

























 &

%

 
##+#+
"
#
 
"!

























+ #" #
%
#
#%#
.#
""

























 &

!"

 '%-

! #
# ! #+
'-
#

























require
"

H-
+#!%>
 
 

























 

#(
@INC
%%->
#
,%#
"!
. 

























 
.%#,#
.%
"* #+
 "

#

.#

























 
.%
 %#+

























#
!"
 '%-
! #

%

gpoparse.pl

























#= #
%#
%
"!
#
8#+#%
#& #

























"*#
1*%&#
%%'%#
+  ' "

%+

























%#
+# '#+
 
#
$  %-
8 #
#

























 "
"!
 
*%%
%&#


" #+

55

)	

























)	


 GPOPARSE_ROOTDIR

























1!
%-
"!
#
 +#>

 '%-

"

)	

























+ #" #
%#
"
# ! #+

, %
# #

























#
"**%+
 #
" "
"
#

#, "

























*#
,% %'#>
#-
%
'#
# ! #+
%

























"#
. 
 
,% %'#
 !

#-

%#

 

























#
%*#
+ #"-

EXAMPLES






#
!"". &
#
"**%+

#

 

#

! #

sgml.out






'# &
. #9











% gpoparse.pl -L/usr/local/fr94/lib/gpoparse \

 -S/usr/local/fr94/lib/sgml \

 -I/usr/local/fr94/lib \

 r03ja0.xxx >& log








#$%*#
%*#
%
#
#(
#

 

csh

"

"#






%
#"& D#
 
-%$






##
#
"**%+
%"
#
 
#
! #
sgml.out

'# &






. #9

 % setenv GPOPARSE_LIBDIR /usr/local/fr94/lib/gpoparse

 % setenv GPOPARSE_SGMLDIR /usr/local/fr94/lib/sgml

 % setenv GPOPARSE_INCDIR /usr/local/fr94/lib

 % gpoparse.pl r03ja[023].xxx >& log






"#
%
"-
"#
"
! #
 
. #
#,#
"&

##






%#
* #
 
! #






*#
%
#
%'",#
#, "*#
,% %'#

%#

 

#






!"

%

!#

#$%*#


#

#

!"". &

"**%+






#
 
%
)	
"
! #
'# &
. #
!"
#%

 






! #9

 % gpoparse.pl -D/tmp -r r03ja[023].xxx >& log






#
"
! #
. 
'#
%*#+

r03ja0.sgm>

r03ja2.sgm

%+






r03ja3.sgm>
%+
#-
. 
'#
#%#+
 
#
+ #"-
/tmp






 * %->
#
!"". &
"**%+

"!!
)	

"

%+







"
#
!"*%#+
"

#
"
! #
. 
%,#






#
#$# "
.unfmt

 % gpoparse.pl -D/tmp -r -s -u r03ja[023].xxx >& log

DIAGNOSTICS






1!
%-
!%%
"
# "
#"
">
#
"&%*
#$ 
. 






%

"D#"

%


#. #

 

#$ 
. 
%
%
"!






D#"
"# "
#"
%

">

!"

#$%*#>

+#

"






-"&% %
#"
##
 
#
 ""*
! #


" #+

56

)	

























)	


BUGS






 ""*
#=##
%
%,#
"
'##

 +# ! #+


%-






 !#=#-
" &
#=##
%#
- %-
 &"#+
+






 &
"# &






- %
*%

%
"*#
!"""#

#!###

%#

"






%'##+
 
)	
"
%
%- &
"#
%
#&%
#$






 ""*
#=##
%
%,#
'##
 +# ! #+
%
%, &

"#






*#% &

*%-

%,#

+ !!##
*#% &
 
"#
"#$
%+






##!"#
* &
'#
%+#+
 "#-

845
%%'%#





	%
%	
:
&
44;

















K
57

1)	

























1)	


NAME






* &*

 
!"
&##% &

."+

 &%*

!"*

%






."+
 +#$
! #

SYNOPSIS






mktrigrm.pl

word_index_file

DESCRIPTION






Mktrigrm.pl 
%
	
 
%
 
%
."+

 +#$

! #






%+

#%#

."+

 &%*>
  &
#
# &
 &%*






 
"
standard output

OPTIONS

 word_index_file











#
."+
 +#$
! #
!"*
. 
#
 &%*
%#

"

'#











&##%#+

EXAMPLES

 % mktrigrm.pl $DAYDIR/gpo/wrk/0103.wdx > \

 $DAYDIR/gpo/wrk/0103.wd3

 % mktrigrm.pl $DAYDIR/book01/wrk/b0000100.odx > \

 $DAYDIR/book01/wrk/b0000100.od3






.##
/01
2
33"%3!453+%%36367

FILES

 input-1











.wdx









)
*%#
."+
 +#$
! #

 output-1











.wd3









)
."+
 &%*
! #

 input-2











.odx










."+
 +#$
! #

 output-2











.od3










."+
 &%*
! #

SEE ALSO

 gpoday.pl(1), trubook.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















M
58

<1	

























<1	


NAME






*= 

 
!"
"% &
#
 =#
 &%*

 

%






."+
 &%*
! #

SYNOPSIS






mkunqtri.pl

trigram_file

DESCRIPTION






Mkunqtri.pl
 
%
	
 
%
%#
%
."+
 &%*

! #






%+

"%#

%

 &%*

%

"

#$%-
"#
%






 =#
 &%*
 
 #+
"
standard output
%"&
. 

 






"% "
.  
#
 
! #

OPTIONS

 trigram_file











#
! #
.  
. 
 =#
."+
 &%*

%#

"

'#











#%#+

EXAMPLES

 % mkunqtri.pl $DAYDIR/gpo/wrk/0103.wd3 > \

 $DAYDIR/gpo/wrk/0103.wf1






.##
/01
2
33"%3!453+%%36367

FILES

 input











.wd3









)
."+
 &%*
! #

 output











.wf1









)
 =#
 &%*
! #

SEE ALSO

 gpoday.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















;
59

C1@	

























C1@	


NAME






*.+ +$

 
!"
#% &
%
."+
 +#$
! #

!"*

%






 
"!
)	
! #

SYNOPSIS






mkwdindx.pl

A
-sgml2iso
B


SGML files


DESCRIPTION






Mkwdindx.pl
 
%
	
 
%
"##
%
 

"!

)	






! #

%+
"%#%#
#*
"&##>
#!"*% &
#
#$






%+
%&
 "
%
."+
 +#$
! #
#
# &
."+
 +#$
 






 #+
"
standard output






%
".
 
#
."+
 +#$
! #
"#"+
"
#%

#






 ,#
."+
 
#
)	
! #

".
%
#
!"". &
! #+9






8 #+
9
)"'%
."+

 +#$



 #*##+

!"

#%

."+






%"
%
)	
! #






8 #+
:9
	"%
."+
 +#$

##
!"

#%

)	

! #

%+






 #*##+
!"
#%
."+
.  
#
! #






8 #+
79
C"+
 &

#
#
."+






8 #+
59
)	
! #
%*#


! #

!"*

. 

#

."+

.%






#$%#+






8 #+
E9
)	
%&

"#
%&
"#
.#

#

."+

.%






#"##+






 
###% "
 
  %
 
% &
#$

"&"






#

&"+



#$% "

"#

1
", +#
"






#!## &
%+
##,#
#
"& %
"#$
!"
#%
."+






#
Federal Register
+"*#
 *%&#
+%%'%#
#
)	
# 






 #

"

###

1

	% 
%%#
# !- &
#






" "%
-sgml2iso
!%&
%#
##

)	

#  #

"

'#






",##+
"
#
1
;;E43
;' 
%%#
#

OPTIONS

 -sgml2iso











",#
1
	% 
)	
%%#
#  #

"

# 











#= ,%#
1
;;E43
;' 
,%#

 ... SGML files ...











#
"
*"#
)	
!"*%#+
#$
! #
 
#

"+#

"











'#
",##+
%+
"%#%#+
 "
%
."+
 +#$
! #


" #+

60

C1@	

























C1@	


EXAMPLES

 % mkwdindx.pl $DAYDIR/gpo/sgm/*.sgm > \

 $DAYDIR/gpo/wrk/0103.wdx

 % mkwdindx.pl $DAYDIR/book01/wrk/b0000100.oml > \

 $DAYDIR/book01/wrk/b0000100.odx






.##
/01
2
33"%3!453+%%36367

FILES

 input-1











.sgm









)
)	
! #

 output-1











.wdx









)
*%#
."+
 +#$
! #

 input-2











.oml










)	
! #

 output-2











.odx










."+
 +#$
! #

SEE ALSO

 gpoday.pl(1), trubook.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















4
61

H	



























H	


NAME






"'""

 
!"
"+ &

"
%
#
%&#
'






 #+
 
%
'""

SYNOPSIS






ocrbook.pl

server install_dir src_image_dir dst_XDOC_dir

DESCRIPTION






Ocrbook.pl
 
%
	
 
%
 ,"#

%

"**# %








%%&#

"
#"& D#
#%
%&#
 *%&#
%#+
!"*
%
Federal






Register
'""
 
 
%
#

% % "
"&%*

 xreadpg

OPTIONS

 server











#
+# &%#+

#,#

 install_dir











#
!
%%*#
"
#
+ #"-
 
. 

#

+%%











'%#
+  ' "
.%
 %#+

 src_image_dir











#
%%*#
"
#

"#

+ #"-

"%  &

#











'""(
 *%&#
! #

install_dir3+%%3MM3DD3'""BB3 *&

 dst_XDOC_dir











#
%%*#
"
#
+# % "
+ #"-
.##
#











 &









#




%#




"



'#



"#+












install_dir3+%%3MM3DD3'""BB3.

EXAMPLES

 % ocrbook.pl hostname /usr/local/fr94 $DAYDIR/book01/img \

 $DAYDIR/book01/wrk






.##
/01
2
33"%3!453+%%36367

FILES

SEE ALSO

 xreadpg(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















6
62

	

























	


NAME






"*"

*%#
 
!"
"+ &


"

%

#






%'""
' #+
 
%
*"

SYNOPSIS






ocrmonth.pl 
install_dir month

DESCRIPTION






Ocrmonth.pl
 
%
	
 

%



%

"**# %








%%&#

%"

%
%&#
"!
#
Federal Register
' #+






 
%
*"
8"
#%
%#+
'"">
 
*%#
 

%






ocrbook.pl

OPTIONS

 install_dir











#
!
%%*#
"
#
+ #"-
 
. 

#

+%%











'%#
+  ' "
.%
 %#+

 month











#
*"
 
#
%&#
A6:B

%

 

"

'#

"











##+

EXAMPLES

 % ocrmonth.pl /usr/local/fr94 01

FILES

SEE ALSO

 ocrbook.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;


















63

)8































)8


NAME






&!

"&%*
!"
 ** &
%

"!
%
'""(
)
#$






"
'#
%
%&#(
&"+

#$

SYNOPSIS






pgfrchnk
A
-v

B

OCR_word_index chunk_word_index hit_file

 output_file

DESCRIPTION






Pgfrchnk
 
%

"&%*
%
 &
% &
%
%&#(

#$






. 

%

#$%#+

"!
)
#$
#
% &*#
'#.##






#
."
%%&#

"!

#$

%

%

#

 

" 

%+






"##+

'%.%+
%+
!".%+
%&#
'"+% #
%#
+##






* #+
"
"
.##
#

% &*#

!!  #-

+#&%+#






#

 
#
 **#+
%
#
%&#
'"+% #
%+
%*#+






"
'#
#
&"+

!"
#
%&#






"
 
#

#$
%+
%"*% #
 
#
)
#$
*%#
 






,#-

+ !! 

"
%%#-
+##* #
%&#
'"+% #
"






#
*%%&#
 
#% ->

%

#"

# *%#

!"

#






#$%#+


&"+



 

"*#+

#



#$

 






#% &#+
. 
#
#$%#+
&"+

#$>
%+

""!






"+#
'"
"!
#$
%#
*%#+

#
*'#
"!
#*%  &






+###+
%+
 ##+
."+
%#
%*%#+

%+

+ , +#+

'-






#

"%

*'#

"!

."+
#"& D#+
"
#
%&#
'-







 
% "
#,#
%
%
"#,% ,#
 + %"

"!

".

.#






#

%#&#+

&"+

#$
*%#
#
#
"#
"!






#
%&#

#
"&%*
 
 
#"
# *%#

" stan-

 dard output

OPTIONS






-v



"
,#'"#
*"+#
"
%
"&#
"!

#

"&%*











*%-
'#
%#+

 OCR_word_index











#

."+
 +#$
! #
!"
#
%&#
'# &
"##+

 chunk_word_index











#
)



."+

 +#$

! #

#$%#+

!"*

#











"#"+ &
'""(
)
#$

 hit_file











#
 "  &
"% "
"!
%
%&#
'#.##
 


%+











)
#$

 output_file











#
#$%#+
&"+

."+
 +#$
! #


" #+

64

)8































)8


EXAMPLES

 % pgfrchnk $DAYDIR/book01/wrk/b0000100.{odx,cdx,hit,gdx} > \

 $DAYDIR/book01/wrk/b0000100.est






.##
/01
2
33"%3!453+%%36367

FILES

 input











.odx










."+
 +#$
! #











.cdx









)

."+
 +#$
! #











.hit









! #

. 

 " 

"% "

"!


%&#

























'#.##

%+
)
#$

 output











.gdx









&"+

."+
 +#$
! #











.est









&"+

#"
# *%#
! #

SEE ALSO

 trubook.pl(1), doqryhit.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















:
65

		



























		


NAME






"#

 
!"

" &

#

&"+



#"






# *%#
!"
%
# #
'""

SYNOPSIS






ploterr.pl

install_dir error_est_dir

DESCRIPTION






Ploterr.pl
 
%
	
 
%
%#
%
#

#"

# 






*%#>

"#

!"

#%

+# ,#+
&"+

! #>
%+
"






#*
 
"#+
"+#
"
%
"&
%#
#
# &

"

 






"#+

 

%

" 

!"*%#+

! #

 
", +#
%






, %
###% "
"!
#
=% -
"!
&"+

!"
%






#
%&#
 
%
'""
H#%#
#
" 
%#
"#+
"
%
"&






%#>
#"
# *%#
#$%-
#=%
"

6

%#

"#+

%






66
 
#
&%






WARNING:
 
 
"+
"-
'#
#+
 !
#
"

-#*






%


#

"&%*

gnuplot

 %#+

%+

 

 

 

#






#, "*#(
#$# "
%

OPTIONS

 install_dir











#
!
%%*#
"
#
+ #"-
 
. 

#

+%%











'%#
+  ' "
.%
 %#+

 error_est_dir











#
%%*#
"

#

+ #"-

"%  &

#

#"











# *%#

! #

!"
%
#
&"+

! #
+# ,#+











!"






#






%&#






"!






%







'""












 %R+ 3+%%333'""HH3.

EXAMPLES

 % ploterr.pl /usr/local/fr94 $DAYDIR/book01/wrk






.##
/01
2
33"%3!453+%%36367

FILES

 input











.est









&"+

#"
# *%#
! #

 output











errest.pts



"#+
" 
! #
"%  &

%

#"

























# *%#
!"
%
'""











ploterr.ps



"
"!
%

# #

'""(

&"+



























#"
# *%#

SEE ALSO

 trubook.pl(1), pgfrchnk(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















7
66

N1	



























N1	


NAME






#G 

 
!"
#G# &
 
%

!% 

"

%






%#-

 " 

#

"% "

"!
%
%&#
. 
%
'""(
)






#$

SYNOPSIS






rejhits.pl

A
-gnuplot
install_dir
B

hit_dir

DESCRIPTION






Rejhits.pl
 
%
	
 
%
%"*% %-
+##* #
 !






%

+# ,#+

 
'#.##
%
%&#(

#$
%+
 
'""(
)






#$
 
"
'#
%*#+
%%#
"
"


 
! #

"% 






7

,%#

#

! 

,%#

 
#
"% "
"!
#
*#+ %






*% &
."+
 &%*
 
#
%&#(



#$

#

#"+






,%#

 
#
"% "
"!
#
*#+ %
*% &
 &%*
 
#






"#"+ &
'""(
)
#$
#
 +
,%#
 
#

"%






*'#
"!
*% &
 &%*







 
 
#G##+
 !
 
)#%#+
 &%*

"% "


#






#"+

,%#
+"#
"
!"".
#
 #%
#+
#%' #+






'-
 
 **#+ %#
# &'"

%&#
. 
#G##+
 
%#






#*",#+

!"*

!#

"# &

%+
%,#
"
&"+







#$
#$%#+
#G##+
 
! #
%#
#%*#+

. 

#$#






 "
bad






#
" "%
!%&
-gnuplot
+ #
#
 
"
&##%#

%






"

"!

%
%##+
%+
#G##+
 
%!#
"# &
%






# #
'""
%
" 
 
#
"
 
#
)#%#+
"%






 "


#
#"+
,%#
 
%
%&#(
 
! #
#
# &






"
 
"#+
 
%
" 
!"*%#+
! #






WARNING9
#
"
" "
"+
"-
'#

#+

 !

#

"






-#*

%

gnuplot
 %#+>
%+
 
#$#%'#
 
 
#






#, "*#(
#$# "
%

1!
#

"

!%&

 

#+>






#
#
install_dir
%&*#
*
%"
'#
", +#+

OPTIONS

 -gnuplot











 #
#
 
"
&##%#
%
&"
%!#
"#











 &
%
#
%&#
 
%
'""

 install_dir











#
!
%%*#
"
#
+ #"-
 
. 

#

+%%











'%#

+  ' "
.%
 %#+


'#
# ! #+
 !











#
"
!%&
 
#+

 hit_dir











#
%%*#
"
#
+ #"-
"%  &

%

#

 











! #
!"
%
'""


install_dir3+%%3MM3DD3'""BB3.


" #+

67

N1	



























N1	


EXAMPLES

 % rejhits.pl -gnuplot /usr/local/fr94 $DAYDIR/book01/wrk






.##
/01
2
33"%3!453+%%36367

FILES

 input











.hit












! #
. 
 " 

"% "

"!

%&#




























'#.##

%+
)
#$

 output











.bad












#G##+
 
! #











goodhits.pts




" 
! #

"%  &

%##+

 




























,%#











badhits.pts





" 
! #

"%  &

#G##+

 




























,%#











rejhits.ps






"
"!
%
'""(
%##+
%+
#G##+




























 

SEE ALSO

 trubook.pl(1), doqryhit.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















5
68

H	



























H	


NAME






'""

 
!"

*% &

%&#
#

"

)






", +#+

! #
%+
#$% &
&"+

#$
!"
%
#






%&#
' #+
 
%
'""

SYNOPSIS






trubook.pl

install_dir book_wrk_dir

DESCRIPTION






Trubook.pl
 
%
	
 
%
*%#

%&#
#
"






 

"#"+ &

'""(
)
%#+
#$>
"+ &
&"+







#$
!"
#%
%&#
 
%
Federal Register
'""






#
9

doxdcprs.pl
%#

#
!"*
%

%&#(

@






! #
 "
%
)	
#$
! #






#
:9

mkwdindx.pl
",#
#

)	
! #
 "
%

."+






 +#$
! #






#
79

mktrigrm.pl
"*#
."+
 &%*

!"*



."+






 +#$
! #






#
59

doqryhit.pl
*%
#
%&#(

."+
 &%*

"






 
'""(
)
 =#
."+
 &%*>
"% &
%
 






#
E9

rejhits.pl
#G#
 %%#
 
%+
"

#G#






 "
#






#
K9

cutchunk.pl
 !
 
 
%##+>
#$%



"!






)
#$
###+
"
#
 






#
M9

pgfrchnk
 &
% &

%&#

#$

"

#

)








#$>
+##* #
%&#
'"+% #
 
#
>
#$%






&"+

#$
!"*
#
>
"#
#$%#+
#$
"

%






."+
 +#$
! #>
%+
"*#
%
#"
# *%#






#
;9

wdx2sgml.pl
",#
&"+

."+

 +#$

! #






"
%
)	
#$
! #






#
49

wrdscore
"*#
."+
#"

%#

'#.##

%&#(








."+
 +#$
! #
%+
 
+# ,#+
&"+

."+
 +#$






! #






#
69

ploterr.pl
"
&"+

#"
# *%#

!"






%
#
%&#
 
%
'""
 
"#+
"+#
"
%
"&
%#


" #+

69

H	



























H	


OPTIONS

 install_dir











#
!
%%*#
"
#
+ #"-
 
. 

#

+%%











'%#
+  ' "
.%
 %#+

 book_wrk_dir











#
!
%%*#
"
#
+ #"-
"%  &

%

#














#


!"


%


#


%&#


 


%

'""












install_dir3+%%3MM3DD3'""BB3.

EXAMPLES

 % trubook.pl /usr/local/fr94 $DAYDIR/book01/wrk






.##
/01
2
33"%3!453+%%36367

FILES

SEE ALSO

 doxdcprs.pl(1), mkwdindx.pl(1), mktrigrm.pl(1),

 doqryhit.pl(1), rejhits.pl(1), cutchunk.pl(1), pgfrchnk(1),

 wdx2sgml.pl(1), wrdscore(1), ploterr.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















E
70

	

























	


NAME






*"

*%#
 
!"
*% &

%&#
#
"






)", +#+
! #
%+
#$% &
&"+

#$
!"
#%






%&#
 
%
*"

SYNOPSIS






trumonth.pl 
install_dir month

DESCRIPTION






Trumonth.pl
 
%
	
 
%
*%#


%&#

#






"

 

"#"+ &

'""(

)

%#+

#$>
"+ &






&"+

#$
!"
#%
Federal Register

%&#

' #+






 

%

*"

8"

#%

' #+
'"">
 
*%#
 






%
trubook.pl

OPTIONS

 install_dir











#
!
%%*#
"
#
+ #"-
 
. 

#

+%%











'%#
+  ' "
.%
 %#+

 month











#
*"
 
#
%&#
A6:B

%

 

"

'#

"











##+

EXAMPLES

 % trumonth.pl /usr/local/fr94 01

FILES

SEE ALSO

 trubook.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















K
71

C@:)		

























C@:)		


NAME






.+$:&*

 
!"
",# &
%
."+
 +#$

! #

 "






%
)	
#$
! #

SYNOPSIS






wdx2sgml.pl

word_index_file sgml_file

DESCRIPTION

 Wdx2sgml.pl
 
%
	
 
%
%#
%

."+

 +#$

! #






%+

",#
 
"#
 "
%
)	
!"*%#+
#$
! #






 
 
#
 ,##
"#
"
mkwdindx.pl

OPTIONS

 word_index_file











#
."+
 +#$
! #
%
 
"
'#
",##+

 sgml_file











#
# &
)	
#$
! #

EXAMPLES

 % wdx2sgml.pl $DAYDIR/book01/wrk/b0000100.gdx \

 $DAYDIR/book01/wrk/b0000100.gml






.##
/01
2
33"%3!453+%%36367

FILES

 input











.gdx









&"+

."+
 +#$
! #

 output











.gml









&"+

)	
! #

SEE ALSO

 trubook.pl(1), mkwdindx.pl(1), wrdscore.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















M
72

C































C


NAME






.+"#

"&%*
!"
%# &
."+#,#

#!"*%#

SYNOPSIS






wrdscore

A

-v

B

A

-sm

B


A


-na


B


OCR_word_index

 truth_word_index

DESCRIPTION






Wrdscore
 
%

"&%*
%

#" #

%



&##%#+






%%&#
"!
#$
. 
 
"#"+ &
&"+
>
%*






% &
."+#,#
#"
%  
#
 
! #
%#
."+






 +#$
! #






#
! 
" "%
!%&
#%'#
%
,#'"#
*"+#
!"

% &






#

"&%*(
"&#
#
#$
."
!%&
%#
#
*#"+






 
. 
#
#$
%#
% &#+

%+

"#+

#

! 

"!






##
" ">
-sm

%+
!"
? 
%+
*#&#?>
+ #






#
% &*#
%&" *
"
%"

!"

 

%+

*#&#






.#

%# &

"#

#
#"+
" ">
-na

%+
!"






?"
%" &?>
+ #
#
% &*#
%&" *
"

"

+"






%

   %

%" &
"!
#
'#&  &
"!
 #
'#.##
#







%+
&"+


#$

1#%+>

#

"&%*

 *-






% &
#
# #
#$
%%&#
%
."
"&
 &






#
 &
% &*#
#""&-
#+
 
 
" &
%%&#






.%

+# &#+

"
% &
%&#
%%&#
"!
#$
%
%#
!"






*%#+
 
##"&##"
 
8"
#$%*#

 

#

Federal

 Register

+"*#

 *%&#

+%%'%#>

#



#

%#






#"#+

 #'- #
.##%
#
&"+

! #
%#
 






&##%
!"*%#+
%%&%'-%%&%






"
%"* 
 >
#
'#&  &
"!

 #

%#

*%#+






"

#

'#&  &
"!
&"+

%%&%
##
*%#






%#
+# &%#+
%
?%"?
#
 #,# &

%+

&"+








#$

'#.##

%"

%#

#

#%#+
%
 &#






 &
%+
% &#+

1
&##%>
%  " &
#
% &*#






 *",#
#$# "
#!!  #-

## %-
"
".
=% -







#>
%+
#
%"
%
%
-" D% "

" 

"






%

#

% &*#

"#

*%-
#",#
!"*
% %-






""







H-
+#!%>
#
"&%*
. 

"

%"

!"

 

%+






*#&#

%+

. 

+# ,#

%

"-

#
"!
%"
'#!"#






!#
 &
% &*#

#
"&%*

"+#

%

" &






#"
. 
 
 #+
"
standard output
 
"&%*
.%






#+
"
"#
#,#-
%&#(

."+
 +#$
! #
 
#

+%%






'%#
. 
 
"#"+ &
&"+

."+
 +#$
! #


" #+

73

C































C


OPTIONS






-v



"
,#'"#
*"+#
"
%
"&#
"!

#

"&%*











*%-
'#
%#+






-sm

 #
#
% &*#
%&" *
"
%"

!"

 











%+
*#&#
.#
%*% &
#"
%  






-na

 #
#
% &*#
%&" *
"
"

*%

%">











'

+ #-

% &
#
."
%%&#
"!
#$
%
%
% 











"!
"&
 &

 OCR_word_index











#

."+
 +#$
! #
!"
#
%&#
'# &
"##+

 truth_word_index











#
&"+

."+
 +#$
! #

!"

#

%&#

'# &











"##+

EXAMPLES

 % wrdscore $DAYDIR/book01/wrk/b0000100.odx \

 $DAYDIR/book01/wrk/b0000100.gdx > $DAYDIR/book01/wrk/b0000100.scr






.##
/01
2
33"%3!453+%%36367

FILES

 input











.odx










."+
 +#$
! #











.gdx









&"+

."+
 +#$
! #

 output











.scr









" &
#"
! #

SEE ALSO

 trubook.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















;
74

@)

































@)


NAME






$#%+&

"&%*
!"
 &
@#"$(
%C"@
1
"
%
%&#






 *%&#
! #

SYNOPSIS






xreadpg
A
-fr94hdr
B
A
-
i
S
f
S
l
S
x
B
server image_file

 output_file
A
lexicon_file
B

DESCRIPTION






Xreadpg
 
%
% % "
"&%*
. #
 

%

 ,"#






@#"$(

%C"@

1

"
%
# ! #+
 *%&#
! #
#
 *%&#






! #
 
#$##+
"
"% 
%
!%&#
 *%&#
%#+
' %-






%+
"#+
 
#
1
1#%+
!"*%






##
%#
%
*'#
"!
" "
"" &
 

"&%*

1!






#

" "%

!%&

-fr94hdr

 

#+>

#

#

"&%*






#%#
#
 *%&#
!"
%
# ! 
%&#

#%+#

 +#+

"






*"
Federal Register
%&#
1!
#
#%+#
 
"%#+>
 
 






#*",#+
!"*
#
 *%&#
 "
"
#"&  "
#
"% "
"!






#

+###+

#%+#

 
"#+
 
%
#%+#

! #
1!
"






#%+#
 
"%#+>
%
"% "
,%#
"!
??
 
"#+






#
#$
#
"!
" "%
!%&
-(i | f | l | x)
+ %#
.%






!"*%

#

#
%#
"
'#
"#+
 
##
!%&
%#






+#! #+
'#".
% 
#$>
#
! 
" ">
 
#
+#!%






 
"&%*
.%
#+
"
"#
%
#
%&#
 *%&#
 

#






Federal Register

+"*#
 *%&#
+%%'%#>
. 
#
#






"#+
%
@
! #

Doxdcprs.pl
 
%
	
 
%

 






%%'#
"!
% &
"
*%-
"!
#
,% "
-#
"!
 !"*%






 "

 + &
#"& D#+
#$
"#+
 
@
! #

OPTIONS

 -fr94hdr











 #
#
"&%*

"

"%#

%+

#%#

%

Federal











Register

%&#

#%+#

!"*
#
 
 *%&#
'#!"#












"# &
">
" &
#
+###+
"% "

"

%











#%+#



! #

. 

#
%*#
""
! #
%*#
%
#











# ! #+
output_file
. 
#$# "
hdy






-
i
S
f
S
l
S
x











 
!%&
+# &%#
#
"
! #
!"*%9











-i





% 
1
;;E43
#$

+#!%











-f





@#"$
8
!"*%











-l





@	1
!"*%











-x





!
@
!"*%

 server











#
+# &%#+
%C"@

#,#


" #+

75

@)

































@)


 image_file











#
1#%+
' %-
%&#
 *%&#
"
'#
#"& D#+

 output_file











#

#
"
! #

 lexicon_file











 "%
#$ "
! #
"
#
 *",#
#"&  "

#











!"*%#

EXAMPLES

 % xreadpg -fr94hdr -x hostname $DAYDIR/book01/img/b0000100.pct \

 $DAYDIR/book01/wrk/b0000100.xdc






.##
/01
2
33"%3!453+%%36367

FILES

 input











.pct









1#%+
 *%&#
! #

 output











.xdc









@

#
! #











.hdy









#%+#

! #

SEE ALSO

 ocrbook.pl(1), doxdcprs.pl(1)

DIAGNOSTICS

BUGS

845
%%'%#





	%
%	
:
&
44;

















4
76

APPENDIX B. IHEAD IMAGE FILE FORMAT

Image file formats and effective data compression and decompression are critical to the

usefulness of image archives. In this application, a raster image is a digital encoding of light

reflected from discrete points on a scanned document page. The 2-dimensional area of the form

is divided into discrete locations according to the resolution of a specified grid. Each cell of this

grid, which is called a pixel, is represented by a single bit value 0 or 1; 0 represents a pixel

predominately white, 1 represents a pixel predominately black. This 2-dimensional sampling

grid is then stored as a 1-dimensional vector of pixel values in raster order, left to right, top to

bottom. Successive scan lines (top to bottom) contain the values of a single row of pixels from

the grid.

Certain attributes of a raster image are required to interpret the 1-dimensional pixel data as a 2-

dimensional image. Examples of such attributes are the pixel width and pixel height of the

image. These attributes are stored in a machine readable header prefixed to the raster bit stream.

A program that is used to manipulate the raster data must first read the header to determine the

proper interpretation of the data which follows it.

Figure 20. An illustration of the IHead raster file format.

Numerous image formats exist, but most image formats are proprietary. Some are widely

supported on small personal computers and others are supported on larger workstations. A

header format named IHead has been developed for use as a general purpose image interchange

format. The IHead header is an open image format which can be universally implemented across

heterogeneous computer architectures and environments. Both documentation and source code

for the IHead format are publicly available and included with this database. IHead has been

designed with an extensive set of attributes in order to adequately represent both binary and gray

level images, to represent images captured from different scanners and cameras, and to satisfy

the image requirements of diversified applications including, but not limited to, image

Header Length

ASCII Format Image Header

Binary Raster Stream

0000000010000010000011111110 ...

• Representing the digital scan across the page

left to right, top to bottom.

• ‘0’ - Represents a white pixel

• ‘1’ - Represents a black pixel

• 8 Pixels are packed into a single byte of

memory.
77

archival/retrieval, character recognition, and fingerprint classification. Figure 20 illustrates the

IHead format.

Since the header is represented by the ASCII character set, IHead has been successfully ported

and tested on several systems including UNIX workstations and servers, DOS personal

computers, and VMS mainframes.1

 All attribute fields in the IHead structure are of a fixed

length with all multiple character fields null-terminated, allowing the fields to be loaded into

main memory in two distinct ways. The IHead attribute fields can be parsed as individual

characters and null-terminated strings, an input/output format common in the C programming

language, or the header can be read into main memory using record-oriented input/output. A

fixed-length field containing the size in bytes of the header is prefixed to the front of an IHead

image file as shown in Figure 20.

Figure 21. IHead C language definition.

The IHead structure definition written in the C programming language is listed in Figure 21.

Referencing the structure members listed in this figure, the first attribute field of IHead is the

identification field, id. This field uniquely identifies the image file, typically by a file name.

The attribute field, created, is the date on which the image was captured or digitized. The next

three fields hold the image’s pixel width, height, and depth. A binary image has a pixel depth

of 1 whereas a grayscale image containing 256 possible shades of gray has a pixel depth of 8.

*OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO

"
!
53D


8F
5!33

"3
D




#	
5
"%>



%
5&*'+*2'

OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO*

*O%

	
-
3D

		

O*

L

3D%:P3Q &++ *O"



@"-

-
AO*

L

D :PD#:+ *OL#33

.



O*

L

<?3Q +' *O
	"	

K
O*

L

%#=! &) *O


"

O*

-.


	
R


9<?3Q;/ *O
*

"O*





9%#=!;/ *O



O*


9D :PD#:;/ *O.I
"
O*



9D :PD#:;/ *O.I
"

O*



.9D :PD#:;/ *O.

.I
"O*



-9D :PD#:;/ *O.I
".

O*


.

9D :PD#:;/ *O.


O*


."
9D :PD#:;/ *O.


"
O*


"9D :PD#:;/ *O"
	"."
5+S7)S1&O*


	K
9D :PD#:;/ *OK



-	O*


/ *O'4
S74"O*


-
P


/ *O'4"S74"O*


.IP
9D :PD#:;/ *O.I
""	
O*



.I9D :PD#:;/ *O
-
.I
"O*



/ *O'4	
S74
O*


P/ *O'4
CS74"	CO*


P/ *O'4.&S74&.O*


"
P
"/ *O'4"
&
S74
&"
O*


.

9<?3Q;/ *O.


"
O*


.
PI9D :PD#:;/ *O
I.I
".

O*


.
P-9D :PD#:;/ *O
-.I
".

O*

T3D#%/
78

The attribute field, density, contains the scan resolution of the image; in the case of this

database, 15.75ppm (400ppi). The next two fields deal with compression.

In the IHead format, images may be compressed with virtually any algorithm. The IHead header

is always uncompressed, even if the image data is compressed. This enables header

interpretation and manipulation without the overhead of decompression. The compress field is

an integer flag which signifies which compression technique, if any, has been applied to the

raster image data that follows the header. If the compression code is zero, then the image data is

not compressed, and the data dimensions: width, height, and depth, are sufficient to load the

image into main memory. However, if the compression code is nonzero, then the complen field

must be used in addition to the image’s pixel dimensions. In order to load the compressed image

data into main memory, the value in complen is used to load the compressed block of data into

main memory. Once the compressed image data has been loaded into memory, the appropriate

decompression technique can be used to produce an image which has the pixel dimensions

consistent with those stored in its header. The images in this database have been 2-

dimensionally compressed using CCITT Group 4 compression [16].

The attribute field, align, stores the alignment boundary to which scan lines of pixels are padded.

Pixel values of binary images are stored 8 pixels (or bits) to a byte. Most images, however, are

not an even multiple of 8 pixels in width. In order to minimize the overhead of ending a

previous scan line and beginning the next scan line within the same byte, a number of padded

pixels are provided in order to extend the previous scan line to an even byte boundary. Some

digitizers extend this padding of pixels out to an even multiple of 8 pixels, other digitizers extend

this padding of pixels out to an even multiple of 16 pixels. This field stores the image’s pixel

alignment value used in padding out the ends of raster scan lines.

The next three attribute fields identify binary interchanging issues among heterogeneous

computer architectures and displays. The unitsize field specifies how many contiguous pixel

values are bundled into a single unit by the digitizer. The sigbit field specifies the order in

which bits of significance are stored within each unit; most significant bit first or least significant

bit first. The last of these three fields is the byte_order field. If unitsize is a multiple of bytes,

then this field specifies the order in which bytes occur within the unit. Given these three

attributes, binary incompatibilities across computer hardware and binary format assumptions

within application software can be identified and effectively dealt with.

The pix_offset attribute defines a pixel displacement from the left edge of the raster image data

to where a particular image’s significant image information begins. The whitepix attribute

defines the value assigned to the color white. For example, the binary images in this database

are black text on a white background and the value of the white pixels is 0. This field is

particularly useful to image display routines. The issigned field is required to specify whether

the units of an image are signed or unsigned. This attribute determines whether an image with a

pixel depth of 8 should have pixel values interpreted in the range of -128 to +127, or 0 to 255.

The orientation of the raster scan may also vary among different digitizers. The attribute field,

rm_cm, specifies whether the digitizer captured the image in row-major order or column-major

order. Whether the scan lines of an image were accumulated from top to bottom, or bottom to

top, is specified by the field, tb_bt, and whether left to right, or right to left, is specified by the

field, rl_lr.
79

The final attributes in IHead provide a single historical link from the current image to its parent

image; the one from which the current image was derived or extracted. The parent field

typically contains the full path name to the image from which the current image was extracted.

The par_x and par_y fields contain the origin point (upper left hand corner pixel coordinate)

from where the extraction took place from the parent image. These fields provide a historical

thread through successive generations of images and subimages. If the image has no parent,

these three fields contain a null string. The IHead image format contains the minimal amount of

ancillary information required to successfully manage binary and gray scale images.

C source code modules are provided in this distribution to decompress, read, and write IHead

images.
80

APPENDIX C. IMAGE EXAMPLES

Figure 22. Hard cover page (data/01/03/book01/img/a0000000.pct).
81

Figure 23. Inside soft cover page (data/01/03/book01/img/a0000001.pct).
82

Figure 24. Prefix content page (data/01/03/book01/img/a0000003.pct).
83

Figure 25. Section page (data/01/03/book01/img/b0000109.pct).
84

Figure 26. Body detail page (data/01/03/book01/img/b0000100.pct).
85

Figure 27. Appendix page (data/01/03/book01/img/c0000002.pct).