Method and apparatus for creating, indexing and viewing abstracted documents

Method and apparatus for storing document images, for creating retrieval index by which the document images may be retrieved, and for displaying the retrieved document images. When a document image is obtained, the document image is subjected to rule-based block selection techniques whereby individual regions within the document region are identified, and the types of regions are also identified, such as title-type regions, text-type regions, line art-type regions, halftone-type regions and color image-type regions. The identification is used to create structural information and both the document image and the structural information is stored. A word-based retrieval index is created based on title-type regions and/or text-type regions, the retrieval index being used in conjunction with a search query so as to be able to retrieve documents which match the search query. The retrieved documents are displayed in either a full image mode or a rapid browsing mode. In the rapid browsing mode, the full image of the document is not displayed, but rather only an abstract structural view of the document image based on the stored structural information. The level of abstraction may be specified by the operator in connection with the identified structural regions of the document, whereby, for example, only a structural view is displayed, or only title-type regions are displayed mixed with remaining structure.

BACKGROUND OF THE INVENTION 
1. Field of the Invention 
The present invention relates to a method and apparatus for creating a 
collection of indexed document images whereby the document images may be 
retrieved through the index, and to a method and apparatus for rapidly 
browsing through document images by viewing abstract structural views of 
the document images rather than the document images themselves. 
2. Description of the Related Art 
Recently, as increasingly larger storage devices have become available, and 
it has become possible to store documents not simply as ASCII text but 
also as a full facsimile image of the document. More specifically, it is 
now commonplace to convert a document into a computer-readable bit map 
image of the document and to store the bit map image of the document. 
Accordingly, whereas ASCII text storage permitted storage and display of 
only text portions of documents, it is now possible to store a document in 
computer readable form and to display not only the text but also pictures, 
line art, graphs, tables and other non-text objects in the document. 
Likewise, it is also possible to store and display documents such that 
text attributes, such as size, font, position, etc., are preserved. 
Despite these advances, however, it is still difficult to retrieve the 
document images into computer memory quickly, and then to browse quickly 
through computer-displayed document images, for example, in a situation 
where a computer operator retrieves many document images and searches 
through those document images to find a particular document. These 
difficulties can be attributed to at least two limitations. First, current 
limitations on bandwidth of the computer systems limit of the speed at 
which documents may be retrieved from storage and displayed. For example, 
at 300 dots-per-inch resolution, an ordinary 81/2 by 11 inch black and 
white document requires approximately 8.4 million bits to store a full 
document image. Adding halftone (grey levels) or color to the image, or 
increasing the resolution at which the image is stored, can easily 
increase storage requirements to many tens of millions of bits. The time 
required to retrieve those bits from storage and to create and display the 
resulting image is significant, even with current high speed computing 
equipment. The time is lengthened even further in situations where the 
document image is retrieved from storage in a first computer and 
electronically transmitted, for example, by modem, to a second computer 
for display on the second computer. 
Second, even when a full document image is displayed to an operator, there 
is ordinarily too much information for an average operator to comprehend 
quickly. Much of the information displayed to the operator is not relevant 
to the operator's query and much time is therefore wasted in displaying 
the non-relevant information. And the presence of such non-relevant 
information can slow the operator in his attempt to locate and understand 
document information that is relevant to the query. 
Moreover, simply retrieving appropriate documents for presentation to an 
operator from a large collection of documents can prove difficult because 
of the large amount of information that must be searched. Conventional 
document retrieval systems ordinarily rely on the creation of a text index 
by which text documents may be retrieved. With document images (as opposed 
to text documents), it has been proposed to subject the document images to 
optical character recognition processing ("OCR processing") and to index 
the resulting text. Systems such as those proposed in U.S. Pat. No. 
5,109,439 to Froessl suggest that is only necessary to OCR-process 
specific areas of the document to simplify the indexing process, but it 
has nevertheless heretofore proved difficult to create an adequate index 
for retrieval of document images. 
SUMMARY OF THE INVENTION 
It is an object of the present invention to address the foregoing 
difficulties. 
In one aspect, the invention provides a system for rapidly browsing through 
document images by displaying not the full document image, but rather only 
an abstract structural view of the document image. More specifically, a 
structural view of a document image includes labelled objects such as 
"title", "text", "table", "line art", "half-tone" and similar objects that 
are ordinarily found in documents, and those objects are displayed for 
presentation to the operator in an abstract structured layout that is 
representative of the full document image. The level of abstraction and 
how the document is rendered on the display may be defined by the 
operator. Thus, it is possible for the operator to specify a high level of 
abstraction where only labelled objects are displayed in the appropriate 
layout; likewise, it is also possible for the operator to specify a lower 
level of abstraction where the original document image for only specific 
types of objects, such as for titles, is displayed. It is also possible 
for the operator to specify that text type objects should be displayed as 
ASCII equivalent characters rather than the original image of the text. By 
displaying only abstract structural views of document images rather than 
the full document image, and by permitting the operating to specify the 
level of abstraction, it is possible to retrieve and display document 
images quickly and to present the information to the operator in a quickly 
comprehendible format. 
According to this aspect, the invention provides a method and apparatus for 
storing document images in which a document image is obtained, for 
example, by scanning an original document, the document image is processed 
to determine the structure of the document, and the structure of the 
document is stored together with the document image. Preferably, the 
structure of the document is determined by conventional block selection 
techniques which utilize a rule-based knowledge system for identifying 
specific areas in a document and for determining the content of the image 
within those areas such that the document image is decomposed into a 
general set of objects. Suitable block selection techniques are described 
in application Ser. No. 07/873,012 filed Apr. 24, 1992, now U.S. Pat. No. 
5,680, 479 entitled "Method and Apparatus for Character Recognition". 
According to this aspect, the invention further provides a method for 
displaying document images in which, given an image of a document and its 
corresponding structure, a representation of the structure of the document 
is displayed and selected structural areas of the display are selectively 
replaced with corresponding areas from the full document image. The 
abstraction level at which the structure is displayed may be designated, 
whereby the structural representation may be displayed with mixed areas of 
structure and full document image (or ASCII text equivalent) in accordance 
with the designated abstraction level. If desired, the display may be 
either a visual display or a verbal display in which the structure and 
content of the document is enunciated through text-to-speech techniques 
which convert the structural and contextual information into spoken words 
for aural communication. 
In a further aspect of the invention, a retrieval index by which images of 
documents may be retrieved is created. According to this aspect a document 
is scanned to form a document image, and the document image is processed, 
for example, by block selection techniques, to identify areas on the 
document of a first type such as title areas, and areas on the document of 
other types. The document image in the first type areas are converted into 
text, for example, by optical character recognition ("OCR") techniques. 
The converted text is indexed so as to form a retrieval index, and the 
document image is stored together with the indexed text such that the 
stored document index may be retrieved with reference to the indexed text. 
Advantageously, the invention creates a retrieval index only from 
first-type areas of the document, such as from title areas of the document 
or from the first paragraph of text areas of the document. 
This brief summary has been provided so that the nature of the invention 
may be understood quickly. A more complete understanding of the invention 
can be obtained by reference to the following detailed description of the 
preferred embodiment thereof in connection with the attached drawings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
FIGS. 1 and 2 show an apparatus according to the present invention. 
As shown in these figures, reference numeral 10 designates personal 
computing equipment such as an IBM PC or PC-compatible computer. Computing 
equipment includes a CPU 11 such as an 80386 processor which executes 
stored program instructions such as operator selected applications 
programs that are stored in RAM 12 or specialized functions such as 
start-up programs or BIOS which are stored in ROM 14. Computing equipment 
10 further includes a local area network interface 15 which provides 
interface to a local area network 16 whereby the computing equipment 10 
can access files such as document files on a remote file server or send 
files for remote printing or have remote machines access document images 
on equipment 10 or otherwise interact with a local area network in 
accordance with known techniques such as by file exchange or by sending or 
receiving electronic mail. 
Computing equipment 10 further includes a monitor 17 for displaying graphic 
images and a keyboard/mouse 19 for allowing operator designation of areas 
on monitor 17 and inputting information. 
Mass storage memory 20, such as a fixed disk or a floppy disk drive, is 
connected for access by CPU 11. Mass storage 20 typically includes stored 
program instruction sequences such as an instruction sequence for 
scanning, indexing, retrieving and displaying documents according to the 
invention, as well as other stored program instruction sequences for 
executing application programs such as word processing application 
programs, optical character recognition programs, block selection 
applications programs, spreadsheet application programs, and other 
information and data processing programs. Mass storage memory 20 further 
includes document index tables which contain index information by which 
documents may be retrieved as well as bit map images of documents, 
document structures, and ASCII text for text areas of the documents. Other 
data may be stored on mass storage memory 20 as desired by the operator. 
A modem 21, a facsimile interface 22, and a voice telephone interface 24 
are provided so that CPU can interface to an ordinary telephone line 25. 
Each of the modem 21, facsimile interface 22, and voice telephone 
interface 24 are given access to the telephone line 25 via a telephone 
line switch 26 which is activated under control by CPU 11 so as to connect 
telephone line 25 to one of the modem 21, the facsimile 22, or the voice 
telephone interface 24, as appropriate to the data being sent and received 
on the telephone line. Thus, CPU 11 can send and receive binary data such 
as ASCII text files or document images files via modem 21 and it can be 
controlled by a remote computer via modem 21, it can send and receive 
facsimile messages via facsimile interface 22, and it can interact on an 
ordinary voice telephone line via voice telephone interface 24. In this 
regard, voice telephone interface 24 is provided with a DTMF decoder 24A 
so as to decode tones on the voice telephone line 25 which correspond to 
operator depressions of a telephone keypad. In accordance with stored 
program instruction sequences in mass storage memory 20, the decoded tones 
are interpreted by CPU 11 into operator commands, and those operator 
commands are executed so as to take predesignated actions in accordance 
with operator depressions of the telephone keypad. 
A conventional text-to-speech convertor 27 is connector to the CPU 11. The 
text-to-speech convertor 27 interprets text strings that are sent to it 
and converts those text strings to audio speech information. The 
text-to-speech convertor 27 provides audio speech information either to a 
speaker 27 for enunciation to a local computer operator, or provides audio 
speech information to the voice telephone interface 24 for enunciation 
over ordinary voice telephone lines. 
MIDI ("Musical Instrument Digital Interface") synthesizer 30 is also 
connected to CPU 11 and interprets MIDI music commands from CPU 11 so as 
to convert those MIDI music commands to audio wave forms. The audio wave 
forms are, in turn, played out over speaker 28 or provided to voice 
telephone interface 24 for play out over ordinary voice telephone lines. 
Scanner 31 operates to scan original documents printed on a sheet of paper, 
and to convert the information of those original documents into a 
bit-by-bit computer readable representation of that document. Scanner 31 
may be a simple black and white scanner, but more preferably scanner 31 
includes at least half-tone (grey scale) processing capabilities and/or 
color processing capabilities. 
Printer 32 is provided to form images of documents under the control of CPU 
11. Printer 32 may be an ordinary black and white printer, but, more 
preferably, printer 32 includes half-tone and/or color capabilities. 
A CD ROM 34, such as an optical disk, is connected for access by CPU 11. CD 
ROM 34 operates to supplement the storage in mass storage memory 20 and 
contains additional information concerning document indexes and document 
images and document structure. It is also possible to provide a 
write-once-read-many ("WORM") optical device or an ordinary read/write 
optical device so as to further supplement the storage capabilities. In 
addition, the local area network 16, CPU 11 can access document indexes 
and document images and document structure stored at remote file server 
locations, and via modem 21, CPU 11 can access document indexes and 
document images stored at centralized data base locations over ordinary 
voice telephone lines. 
FIG. 3 is a flow diagram showing overall operation of the FIGS. 1 and 2 
apparatus. The process steps shown in FIG. 3 are executed by CPU 11 in 
accordance with stored program instruction sequences stored in mass 
storage 20 and executed out of RAM 14. 
In step S301, CPU 11 displays a main selection menu on monitor 17 and 
permits operator selection of one of the menu items. In step S302, if the 
operator has selected to input documents, then flow advances to step S303 
in which document image processing is executed, after which flow returns 
to step S301. If the operator does not select document input but instead 
selects to retrieve documents (step S304) then flow advances to step S305 
in which document retrieval processing is executed, after which flow 
returns to step S301. If the operator does not select document retrieval 
but instead selects to display documents (step S306), then flow advances 
to step S307 in which document display processing is executed, after which 
flow returns to step S301. Each of these processes is discussed in greater 
detail below. 
Although the flow processing illustrated in FIG. 3 appears to be executed 
serially and sequentially, it is to be understood that the processes may 
be executed at significantly different times, in different order, and 
indeed may be executed by different machines. For example, it is possible 
for a first machine to input documents and to store those documents for 
retrieval and display by a different machine. Likewise, it is possible for 
one machine to retrieve documents and transmit them to another machine, 
for example, via modem 21, for display on that other machine. 
FIG. 4 is a detailed flow diagram illustrating document input processing in 
accordance with step S303 of FIG. 3. 
In step S401 CPU 11 causes scanner 31 to scan an original document fed into 
scanner 31 so as to convert the original document into a computer readable 
bit map image of that document. The bit map image may be temporarily 
stored in RAM 14 for further processing. 
In step S402, CPU 11 derives structural information concerning the layout 
of objects within the document. More specifically, CPU 11 subjects the bit 
map document image to block selection techniques such as those described 
in the above mentioned U.S. application Ser. No. 07/873,012, now U.S. Pat. 
No. 5,680,479 so as to decompose the document image into a general set of 
objects. Using rule-based decisions, such block selection techniques are 
able to identify related regions or areas within the document, and are 
able to determine the type of images within those regions. For example, 
using the block selection techniques described in the aforementioned U.S. 
application Ser. No. 07/873,012, CPU 11 processes the scanned-in bit map 
image stored in RAM-14 to identify related regions in the image. Then, CPU 
11 identifies the type of region, for example, whether a region is a title 
region, a text region, a paragraph region, a table region, a line art 
image region such as a graph, a half-tone image region, a color image 
region, etc. 
FIG. 5 illustrates regions identified on a document image in accordance 
with processing according to the process steps in Step 402. FIG. 5 is a 
representative document image consisting of page 40 from a magazine 
article. As shown in FIG. 5, page 40 includes title regions such as 
regions 41, 42 and 47, as well as text regions such as regions 43, 44 and 
48. Likewise, page 40 includes line art image regions 45, table regions 49 
and half-tone object regions 46. Other types of regions are possible. In 
accordance with processing in Step S402, CPU 11 identifies the related 
regions in the document and determines the type of image in each of the 
regions. In FIG. 5, each region has been indicated diagrammatically with a 
circumscribing rectangle. 
Reverting to FIG. 4, flow then advances to Step S403 in which CPU 11 
identifies the document type. More specifically, many documents have 
relatively standard structures; for instance, a business letter virtually 
always includes a corporate letterhead region (which will be identified in 
Step S402 as a line art image area), a date and an addressee region (which 
will be identified in Step S402 as separate but adjacent text regions), 
salutation, body and closing regions (which will be identified as text 
regions in Step S402), and a signature (which will be identified as a line 
art image region). CPU 11 stores templates of these standard document 
types, and in Step S403 compares the structural information derived in 
Step S402 with those stored templates. If a match is found between the 
stored template and the structural information derived in Step S402, then 
CPU 11 assigns the associated document type to the document image stored 
in RAM 14. 
In Step S404, CPU 11 correlates related regions within the document. More 
specifically, there are often regions within a document that should be 
considered together, such as the text caption for a figure or the footnote 
for a text block. For example, referring to FIG. 5, text region 48 is 
related to title 47, and table region 49 is related to title 50. Using 
rule-based decision processes, CPU 11 correlates related regions. 
In Step S405, CPU 11 subjects designated document regions to OCR processing 
in preparation for creating a retrieval index for the document. More 
specifically, so as to be able to retrieve the document image based on a 
word-based retrieval system, CPU 11 creates a retrieval index from text 
regions within the document. It is possible to subject all text-type 
regions, such as title and text regions, which were identified in Step 
S402 to OCR processing and to form the index based on the resulting text, 
but more preferably CPU 11 offers the operator a selectable level of 
indexes. More specifically, CPU 11 offers the operator the option of 
selecting the index level in accordance with the following levels: 
Level 1: Title Regions Only 
Level 2: First Line Of Text Regions 
Level 3: Text Captions Related To Graphics 
Level 4: Full Text 
If the operator selects Level 1 ("Titles Only"), then CPU 11 subjects only 
title regions to OCR processing. The resulting text from the title regions 
is provided for a retrieval index to allow the document to be retrieved 
when words in the title match a specified search query. Likewise, if the 
operator specifies Index Level 2 ("First Line of Text Regions"), then CPU 
11 subjects both title regions as well as only the first line of each text 
regions to OCR processing, and creates a retrieval index based on the 
resulting text. If Index Level 3 is selected ("Text Captions Related To 
Graphics"), then CPU 11 subjects the items in Levels 1 and 2 to OCR 
processing as well as text regions related to graphics as specified in the 
correlations obtained in step S404. Finally, if the operator specifies 
Index Level 4 ("Full Text"), then both title regions and full text regions 
wherever located are subjected to OCR processing and the resulting text 
provided to form the retrieval index. It is possible to include other 
indexing levels. 
In step S406, the ASCII text characters obtained from OCR processing are 
indexed and the document retrieval index is updated appropriately. In 
addition to indexing the ASCII text words, a flag is also indexed which 
indicates the type of object from which the word was obtained, that is, 
title, text, caption, etc. The flag may be used as part of the retrieval 
process so as to retrieve documents based on the occurrence of words in 
particular regions of documents. 
In Step S407, CPU 11 stores the structural information that was derived in 
Step S402 and that was correlated in Step S406. FIG. 6 is a representative 
view of how the structural information is stored. As shown in FIG. 6, for 
each document the structural information includes a document identifier 51 
which is the same identifier as that assigned to the full document image 
and by which it is possible to retrieve the full document image. In area 
52, the document type derived in Step S402 is stored. At 53, all of the 
structural information for the document, and its layout within the 
document, is stored. As shown at 53, for each region identified in Step 
S402, CPU 11 stores a region identifier, a region type, rectangular 
coordinates that specify the position of the region on the page (here the 
upper left corner ("ULC") coordinates and the lower right corner ("LRC") 
coordinates), and all related regions identified in Step S406. In FIG. 6, 
region 1 corresponds to region 41 in FIG. 5 and, as shown in FIG. 6, 
includes a "title"type, upper left hand coordinates of &lt;0,0&gt;, lower right 
hand coordinates of &lt;5,40&gt;, and no related regions. The remaining regions 
illustrated in FIG. 6 follow in like fashion. 
Region attributes 54 may also be stored, for example, font selection and 
size as illustrated in FIG. 6. Other attributes may also be stored, such 
as number of text lines, number of paragraphs, etc. 
In Step S408, the document image is stored, preferably with OCR-processed 
document text. The document text is stored in the event that a quick 
display of ASCII text characters is desired rather than the slower display 
of images of text regions. Accordingly, at the end of Step S408, CPU 11 
has completed document input processing and has stored a text-searchable 
index for the document, a full image for the document, the structure of 
the document, and, if desired, OCR-processed text of text regions of the 
document. 
Reverting to FIG. 3, if an operator selects document retrieval (Step S304), 
then flow advances to Step S305 for document retrieval processing. 
Document retrieval processing used in the preferred embodiment of the 
invention is a word-based retrieval and search system based on an 
operator-specified search query. Such a query may be specified by a 
logical sequence that must be found in each document (a "Boolean" search), 
or may be specified by a natural language based query system. The query 
may be made directly by a local operator, or remotely by an operator with 
voice or DTMF phone access, or from commands issued by a remote computer 
via modem or LAN. 
The operator may also specify that documents are retrieved based on the 
occurrence of words in particular document regions, such as title regions 
or captions. Documents matching the query are retrieved and are assigned 
scores based on the degree that the document matches the query. Higher 
scores may be assigned for occurrences of words in particular regions of 
documents, the score being scaled the same as the above indexing level 
selection. Thus, for example, a higher score may be assigned for 
occurrences of words in title regions than for occurrences of words in 
text regions. When plural documents are retrieved, they are displayed in 
order of their score. 
When an operator selects document display processing (Step S306), then flow 
advances to Step S307 for CPU 11 to execute document display processing. 
FIG. 7 is a detail flow diagram illustrating document display processing. 
In Step S701, CPU 11 allows the operator to set display options by which 
retrieved documents will be displayed. Specifically, the operator selects 
one of two modes by which retrieved documents will be displayed: full 
document image mode or browse mode. If the full document image mode is 
selected, then full document images of retrieved documents will be 
displayed. If the browse mode is selected, then only an abstract 
structural view of each document image is displayed, thereby permitting an 
operator to browse rapidly through all retrieved documents. In the browse 
mode, three further parameters may be set by the operator. First, the 
operator may select whether to display ASCII text characters of 
OCR-processed text regions or document images of text regions. Second, the 
operator may select the abstraction level at which the structural view is 
displayed. Nine abstraction levels are possible in this embodiment of the 
invention, though other arrangements are possible: 
Level 1: Structure Only 
Level 2: Titles 
Level 3: Retrieval Parameters 
Level 4: First Line of Text 
Level 5: Full Text 
Level 6: Tables 
Level 7: Line Art 
Level 8: Half-tone 
Level 9: Color 
In accordance with the selected abstraction level, CPU 11 will display 
structural views of the document mixed with regions of full document 
images. If abstraction Level 1 is selected, then only structure is 
displayed. But if abstraction Level 2 or above is selected, then 
structural information is displayed mixed with regions of full document 
images. For example, at abstraction Level 2, structural information is 
displayed mixed with images (or ASCII text, if selected) of title regions 
of the document. The needed image regions are obtained from the full 
document image in accordance with the ULC and LRC coordinates stored with 
the structure. If abstraction Level 3 is selected, then structural 
information is displayed mixed with images of titles (or ASCII text, if 
selected) and the retrieval parameters by which the document was 
retrieved. At abstraction Level 4, structural information is displayed 
mixed with images of title regions, retrieval parameters, and the first 
line of text images in each text block. As before, if ASCII text display 
is selected, then ASCII text is displayed rather than images of text. 
Higher abstraction levels cumulatively display more and more of the full 
document image in accordance with the selected abstraction level. 
In addition to the abstraction level parameter, in the browse mode the 
operator can also select whether to allow image enhancement manually or 
automatically. Specifically, in manual image enhancement, when the 
structure of a retrieved document is displayed, the operator may target a 
particular region of the document, for example with mouse 19, and request 
for CPU 11 to replace the displayed region with the full document image. 
If automatic enhancement is selected, then CPU 11 automatically replaces 
the structural information that it is displaying with full document 
images. Automatic enhancement operates during quiescent periods of CPU 
operation, that is, when the CPU has sufficient processing time so as to 
enhance the image. Enhancement progresses in the order of the abstraction 
levels given above. Thus, during quiescent processing periods, CPU begins 
to replace, first, title regions with full images of the title regions, 
second, retrieval parameter regions with full images of the retrieval 
parameters, third, the first lines of the text regions with full images of 
the first lines of text, etc. 
After display options have been set in Step S701, flow advances to Step 
S702 in which CPU 11 determines whether the full image mode has been 
selected. If the full image mode has been selected, then flow branches to 
Step S703 in which full images are displayed. 
On the other hand, if the browse mode has been selected, then flow advances 
to Step S704 in which the structure of the document is displayed at the 
selected abstraction level. A representative display of structural 
information is shown in FIG. 8. In FIG. 8, the operator has selected the 
browse mode, and has selected ASCII text display at abstraction Level 2, 
that is "Titles". Thus, as shown in FIG. 8, an abstracted structural view 
of the document is presented to the operator, with ASCII text equivalents 
of title regions. The remaining regions are simply labeled with the region 
type, as specified in the region type area of FIG. 6. Labelling may be by 
color coding of the region, or, as shown in FIG. 8, by applying a text 
label. Each of the regions is circumscribed by a rectangular shape whose 
coordinates are specified by the "ULC" and "LRC" coordinates of FIG. 6. 
ASCII text equivalents of title regions are displayed at the corresponding 
ULC and LRC coordinates. Thus, the operator is presented with an abstract 
structural view of the document with a selectable abstraction level. This 
permits the operator to browse quickly through retrieved documents so as 
to find particular documents of interest. 
Flow then advances to step S705 in which CPU 11 determines if the manual 
enhancement parameter has been selected. If the manual enhancement 
parameter has not been selected, then flow branches to step S706 in which 
CPU 11 automatically replaces structurally displayed regions of the 
document with full images (or ASCII text, if selected) of those regions 
during quiescent operating times of the CPU. Thus, assuming that 
sufficient quiescent operating time is available, if the automatic 
enhancement mode has been selected, then the full document will eventually 
be presented to the operator as the CPU replaces the structurally 
represented regions of that document. At any time during this process the 
operator may select for the next retrieved document to be displayed (Step 
S710) whereupon flow returns to Step S702 so as to display the next 
retrieved document. 
If the manual enhancement mode has been selected, then flow advances to 
Step S707 in which CPU 11 determines whether a document region has been 
targeted by the operator. If no document region has been targeted by the 
operator, then CPU 11 does nothing, or more preferably, begins to replace 
structural regions of the document with full document images as shown at 
Step S708. Again, this process may be interrupted at any time by operator 
selection of the next document (Step S710). 
On the other hand, if in Step S707 the operator targets a document region 
for replacement, then flow advances to Step S709 in which CPU 11 retrieves 
the corresponding document image form the document image file and replaces 
the structurally represented region of the displayed document with the 
full document image. Thus, as shown in FIG. 9, the operator has targeted 
area 54 which is a text region. CPU 11 thereupon retrieves the document 
image corresponding to the rectangular area circumscribed by the ULC and 
LRC coordinates, or the ASCII text equivalent if that option has been 
selected, and replaces the area in FIG. 8 with the corresponding document 
image as shown in FIG. 9. Flow then returns to Step S707 to await further 
designations by the operator. 
In FIG. 9, the ASCII text equivalent for region 54 does not fit into the 
allotted area. In this situation, CPU 11 only displays the amount of text 
that will fit, but, so as to allow the operator to view all the text, CPU 
11 permits scrolling within the area. 
As further shown in FIG. 9, the operator has requested for the first line 
of text from region 44 to be displayed, and the ASCII text equivalent of 
that text is displayed at 55. Of course, if image mode was selected, then 
the image of the first line of text would have been displayed. 
In Step S709, if an operator targets a document region which has a related 
region, such as region 42 or 43, then not only does CPU 11 replace the 
targeted region, but CPU 11 also replaces the related region as well. 
Accordingly, if the operator targets region 42, which is a title region, 
CPU 11 replaces title region 42 with the full image (or text) of region 
42, and also replaces text region 43 with the full image (or text) of text 
region 43. By virtue of this feature, the operator is provided with quick 
access to information that is desired without requiring the operator to 
execute additional keystrokes, etc. 
FIG. 10 shows a situation in which the operator has selected for certain 
regions to be displayed as images rather than as ASCII text. More 
particularly, the operator has requested for full images of regions 46 and 
48, and those regions have been provided as shown at 56 and 57, 
respectively.