Method and apparatus for automatic image segmentation using template matching filters

Abstract: The present invention is a method and apparatus for automatic image segmentation using template matching filters (30,40). The invention generally segments differing binary textures or structures (i.e. text 24; images 26) within an input image (10) by passing one or more structures while removing other structures. More particularly, a first set of templates (136a-f; Fig.3) in a first LUT (32) are used for a first iteration in first filter circuit 30; and a second set of templates (138; Fig.3) in a second LUT (42) are used for a second iteration in a second filter circuit (40). XOR circuits (46), (50) are used to produce output images (48), (50) with text-only and image-only, respectively.

Description: This invention relates generally to a method and apparatus for automatic image segmentation using template matching filters, and more particularly to a method and apparatus for segmenting regions of differing texture or structure within a stored binary image using a template matching filter that is designed to pass at least one texture while removing one or more other textures. The present invention is a novel approach to separating text, halftones, or other image structures in composite images using template-based filtering methods. A key application of the present invention is the segmentation of text regions from halftone regions. In the reproduction of an original document from video image data created, for example, by electronic raster input scanning from an original document, one is faced with the limited resolution capabilities of the reproducing system and the fact that output devices remain predominantly binary. This is particularly evident when attempting to reproduce halftones, lines and continuous tone images. Of course, an image data processing system may be tailored so as to offset the limited resolution capabilities of the reproducing apparatus used, but this is difficult due to the divergent processing needs required by the different image types that may be encountered. In this respect, it should be understood that the image content of the original document may consist entirely of high frequency halftones, low frequency halftones, continuous tones, text or line copy, or a combination, in some unknown degree, of some or all of the above. Optimizing the image processing system for one image type in an effort to offset the limitations in the resolution capability of the reproducing apparatus used, may not be possible, requiring a compromise choice that may not produce acceptable results. Thus, for example, where one optimizes the system for low frequency halftones, it is often at the expense of degraded reproduction of high frequency halftones, or of text or line copy, and vice versa. Beyond the issue of accurate reproduction, segmentation of different image types is key to the successful application of recognition algorithms (e.g., character recognition and glyph recognition) and efficient application of image compression techniques. As one example of the problems encountered, reproduction of halftoned images with screening tends to introduce moire effects, caused by the interaction of the original screen frequency and applied screen frequency. Although the use of high frequency line screens can reduce the problem, the artifact can still occur in some images. In a networked environment particularly, it is desirable that the image processing device (e.g., raster input scanner) detect the halftone, and low-pass filter the document image into a continuous tone for subsequent halftone reproduction by printers in the network in accordance with their particular capabilities. Heretofore, a number of applications, patents and publications have disclosed techniques for segmentation of digital image data, the relevant portions of which may be briefly summarized as follows: US-A-5,245,445 discloses an image processing apparatus includes an image processing part for carrying out an image processing for an input image having pixels with multilevel bits supplied by an input unit, the image processing being carried out at a processing level among a plurality of predetermined processing levels, a first detecting part for detecting an edge quantity of the input image, a second detecting part for detecting a screened dot quantity of the input image, and a control part for determining a control index proportional to the detected edge quantity and the detected screened dot quantity, and for supplying the determined control index to the image processing part, so that the image processing part selects a processing level appropriate for the input image from the processing levels in response to the control index, and carries out an image processing for the input image at the selected processing level. In accordance with the present invention, there is provided a method for automatic image segmentation using template matching filters, for processing a document image to determine image structures present therein, the steps comprising: receiving, from an image source, a document image having a plurality of pixels therein, each pixel represented by a binary image signal, and storing at least a portion thereof representing a region of the document image in a data buffer; retrieving, from the data buffer, the binary image signals for the document image; determining, using template matching filters, image structures present in the region of the document image, characterised in that the step of determining image structures comprises the steps of: applying a first stage template matching filter to the binary image signals for the document image to produce a first filtered binary output image; applying a second stage template matching filter to the first filtered binary output image to produce a binary error image; and XORing the binary error image and the first filtered binary output image to mask from the first filtered binary output image any binary image signals from segments of the document image not comprised of a first image structure to produce a first binary output image, wherein the first binary output image contains only first image structure segments therein. In accordance with another aspect of the present invention, there is provided an apparatus for processing binary image pixels in an image represented by a plurality of rasters of binary image pixels, to identify regions exhibiting a particular structure therein, comprising: an image source for producing a document image having a plurality of binary image pixels therein, each pixel represented by a binary image signal; memory for storing at least a portion of the binary image signals representing a region of the document image in a data buffer; and processing means employing template-matching filters to identify the presence of the particular structure in the region of the image stored in said memory, characterised by the processing means being adapted to apply a first stage template matching filter to the binary image signals for the document image to produce a first filtered binary output image; to apply a second stage template matching filter to the first filtered binary output image to produce a binary error image; and to XOR the binary error image and the first filtered binary output image to mask from the first filtered binary output image any binary signals from segments of the document image not comprised of a first image structure to produce a first binary output image, wherein the first output image contains only first image structure segments therein. Preferably, the processing means comprises a segmentation circuit which may include a multiple-stage template-based filter. Preferably, the first stage and/or the second stage of said multiple-stage filter includes a look-up table preprogrammed with a plurality of entries, each of said entries corresponding to a pixel pattern determined to represent a segment of a training image having only textual content. The particular structure may be a halftone or may be text. The apparatus may further comprise a logic filter for removing the particular structure from the image region stored in said memory to produce a binary output image substantially void of the particular structure. The apparatus may further comprise a logic filter for preferentially passing the particular structure for the image region stored in said memory to produce a binary output image substantially comprised of regions containing the particular structure. The apparatus may further include a logic circuit for XORing the output signals stored in said output memory with corresponding signals in said image memory to substantially remove regions exhibiting the first structure from the binary output image. The processing means may comprise a suitably programmed programmable image processing apparatus. One aspect of the invention is based on the discovery that templates may be employed to recognize one binary structure within one or more textures. More specifically, template-based filters may be used to recognize regions of an image that contain text and line art. This discovery further avoids problems that arise in techniques that attempt to cover a broad range of document types, as the present invention further enables the "customization" of the template-based filters used therein in response to training documents that are representative of documents commonly encountered by the image processing system. This aspect is further based on the discovery of techniques that generate statistical representations of the patterns found in text and halftone regions of documents. The technique described herein is advantageous because it is inexpensive compared to other approaches and is flexible, in that it can be adapted to any of a number of input document types exhibiting a wide range of possible patterns. As a result of the invention, a low-cost image segmentation system may be accomplished. Figure 1 is a general block diagram showing an embodiment of the automatic image segmentation apparatus in accordance with the present invention, where the invention is employed in a document reproduction system; Figure 2 is a data flow diagram illustrating a two-stage image segmentation process in accordance with the present invention; Figure 3 is a flowchart illustrating the various steps in a serial process used to apply a template-based segmentation filter to the input image of Figure 1; Figures 4A and 4B are pictorial representations of the operation of the template-based segmentation filter on a region of an image in accordance with the present invention; Figure 5 is a simplified electrical schematic illustrating the implementation of a parallel processing scheme for a template-based segmentation circuit in accordance with one aspect of the present invention; and Figures 6 and 7 are data flow diagrams illustrating the various stages in the process used to design the template-based segmentation filters of the present invention. For a general understanding of the present invention, reference is made to the drawings. In the drawings, like reference numerals have been used throughout to designate identical elements. In describing the present invention, the following term(s) have been used in the description. An "image" is generally a pattern of physical light. An image may include characters, words, and text as well as other features such as graphics. Text may be included in a set of one or more images. An image may be divided or segmented into "segments" or "regions," each of which is itself an image. The "structure" of an image segment or region is generally determined by the primary content of the region including, for example, text, halftone or graphics structures. A segment of an image may be of any size up to and including the whole image. An image may also refer to a two-dimensional data array that represents a pattern of physical light. A "document," which may exist in either hardcopy (written or printed) or electrical (data array) form, is a representation of one or more images and/or text. A document may include multiple pages. An item of data "defines" an image when the item of data includes sufficient information to produce the image. For example, a two-dimensional array can define all or any part of an image, with each item of data in the array providing a value indicating the color and/or intensity of a respective location of the image. If a two-dimensional array or other item of data defines an image that includes a character, the array or other data also defines the character. Each location in an image may be called a "pixel." A "pixel" is the smallest segment into which an image is divided in a given system. In an array defining an image in which each item of data provides a value, each value indicating the color and/or intensity of a location may be called a "pixel value". Each pixel value in a binary image is an electrical signal in a "binary form", a gray-scale value in a "gray-scale form" of an image, or a set of color space coordinates in a "color coordinate form" of an image, the binary form, gray-scale form, and color coordinate form each being a two-dimensional array defining an image. Hence, the term pixel may also refer to the electrical (or optical) signal representing the measurable optical properties of a physically definable region on a display medium. A plurality of physically definable regions for either situation represents the measurable properties of the entire image to be rendered by either a material marking device, electrical or magnetic marking device, or optical display device. Lastly, the term pixel may refer to an electrical (or optical) signal representing physical optical property data generated from a photosensitive element when scanning a physical image, so as to convert the physical optical properties of the image to an electronic or electrical representation. In other words, in this situation, a pixel is an electrical (or optical) representation of the optical properties of an image measured at a definable area by an optical sensor. An item of data "relates to" part of an image, such as a pixel or a larger segment of the image, when the item of data has a relationship of any kind to the part of the image. For example, the item of data could define the part of the image, as a pixel value defines a pixel; the item of data could be obtained from data defining the part of the image; the item of data could indicate a location of the part of the image; or the item of data could be part of a data array such that, when the data array is mapped onto the image, the item of data maps onto the part of the image. An operation performs "image processing" when it operates on an item of data that relates to part of an image. Pixels are "neighbors" or "neighboring" within an image when there are no other pixels between them or when they meet an appropriate criterion for neighboring, such as falling within a positioned observation window. If the pixels are rectangular and appear in rows and columns, each pixel may have 4 or 8 connected neighboring pixels, depending on the criterion used. An "image input device" is a device that can receive an image and provide an item of data defining a representation of the image. A "scanner" is an image input device that receives an image by a scanning operation, such as by scanning a hardcopy document. For example, the Xerox 7650 Pro Imager scanner and Xerox® 7009 Facsimile Terminal are devices which receive hardcopy documents to produce data defining an image. Other image input devices include data processing systems suitable for generating digital documents in response to instructions from a user. An "image output device" (IOT) is a device that can receive an item of data defining an image and provide the image as output. A "display" is an image output device that provides the output image in human viewable form. The visible pattern presented by a display is a "displayed image" or simply "image." A "printer" or "marking engine" is an image output device capable of rendering the output image in human readable form on a removable medium. Turning now to Figure 1, which shows an embodiment of the automatic image segmentation apparatus, employed in an image reprographic setting in accordance with the present invention, the general components of digital printer 12 are depicted. More specifically, an input image 10 would be presented to digital printer 12 to produce a printed output 20. Within digital printer 12 a segmentation filter 14 transforms the input image in accordance with the present invention into at least two segmented images, in a simplified case, to segment text and halftone regions thereof. The segmented image bitmaps are, in turn, passed to an image processing / recombination circuit 15. As will be further described, the image processing circuit 15 processes the segmented images to produce an output image 16, that is optimized for the given marking process. Alternatively, while in segmented form, the specific image segments may be isolated and grouped into regions using techniques, such as morphological opening or closing. Once a segment or region is isolated, each pixel may be tagged by setting the state of an associated tag bit in accordance with the image type (e.g., text, halftone, other). The tagged sections may then be recombined into a single bit-map with tags. When passed on to subsequent operations, the individual pixels within a region are treated in a manner that is optimized for the particular region. Subsequently, output image 16 may be passed to marking engine 18 for exposure and development, as is well-known, to produce output print 20. Referring now to Figure 2, the data flow diagram illustrates a two-stage image segmentation process that is carried out within segmentation filter 14 of Figure 1. Although depicted as a preferable two-stage filter, it is understood that a single-stage iteration could have been employed or that additional stages or iterations could be used to further reduce the error in classification of text and halftone regions of the image. In the figure, input document 10, including regions of both text 24 and halftone 26, is passed to a first iteration filter circuit 30. Input document 10 is preferably a plurality of binary data signals representing, for example, the text, halftone and graphic image regions that make up the document. The input document image may be produced as a digitized representation of a hardcopy document scanned on a scanner. In the first filter circuit 30, the document image is filtered by comparing segments thereof with predefined patterns, referred to herein as templates, stored as LUT 1 . As is further illustrated in the flowchart of Figure 3, and associated examples of Figures 4A and 4B, once the input document is obtained, step 100, a target pixel "X" is identified, and a set of surrounding pixels are treated as a window, step 102. Figure 4A illustrates a portion of a digitized image 130 having an upper portion 130a containing text, and a lower portion 130b containing a halftone region. Magnified portions 132 and 134, of image 130, are shown in Figure 4B, where the individual pixels comprising the image are discernible. The values of the pixel signals within the window are then compared to the templates stored in memory, for instance, the templates 136a-136f and 138 illustrated in Figure 4B. Preferably, the template filter is implemented as a look-up table (LUT). When a matching template is found, step 106, the target pixel is identified as a text pixel, step 120, and allowed to pass unchanged through the filter, step 122. Otherwise, the process continues at step 108 where the presence of further templates is tested. If further templates are available for comparison with the window, the process continues at step 110. Otherwise, when no further templates are available, the pixel is identified as being representative of a halftone segment or background region of the input image, step 112, and an output of "0" or an "OFF" pixel signal is produced, step 114. In a preferred multiple iteration embodiment, following the first iteration as described above, the filter attempts to identify error pixels and further remove them from, say, the text image. Because errors pixels tend to be much sparser than the identified text pixels, a different class of filters could be used for successive iterations. For example an order-statistic filter could be used, where if less than a predetermined number of pixels are active within a neighborhood window, the target pixel will be considered an error pixel. Alternatively, a similar neighborhood checking could be performed with morphological filters. Note that the serial method described above with respect to the flowchart of Figure 3, which is appropriate for software implementations of the LUT, may also be accomplished using electrical circuitry. Higher speed performance many be obtained using a hardware implementation where the LUT's would be implemented using an Application Specific Integrated Circuit (ASIC) or Programmable Logic Array (PLA). As a simplified example of such an embodiment, the electrical circuit schematic of Figure 5 is provided. In filter circuit 200, the document image is filtered by comparing segments thereof with predefined patterns, referred to herein as templates, stored as look-up tables (32 or 42). As is illustrated in Figure 5, once the input document is obtained a target pixel X is identified, and a set of surrounding pixels are treated as a window, 202. The values of the pixel signals within the window are then transferred to a register 204, or similar memory location suitable for holding data signals representative of the pixels within window 202. Using a plurality of logic gates 206 (which may be a combination of AND or NOR gates depending upon binary value in any particular template position), or similar logic operations, the signals stored in register 204 are compared to the templates stored as a series of signals in LUT memory; where the set of stored signals for each template represents a unique pixel pattern to be detected within the window. For example, LUTs 32 or 42 would be populated with templates similar to those of 136a-136f and 138 as illustrated in Figure 4. It is further noted that, while illustrated for simplicity as single elements 206, implementation of the parallel comparison operation described would require a plurality of logic gates for each template or look-up table entry, as would be apparent to one skilled in the art. As is further apparent to one skilled in the art, logic minimization techniques may be employed to enable the rapid, parallel comparison of the LUT templates with the values stored in register 204. After the parallel comparison step, accomplished by comparing the signals in register 204 with the associated signals in each entry of LUT 32, 42, any match between the signal sets would result in a positive logic signal being passed into the logical OR array, represented by reference numeral 208. As previously described with respect to the text structure template-based filter, the output of the OR gate array would identify target pixel X as a text pixel, where it may be allowed to pass unchanged. Otherwise, the target pixel could be identified as a non-text region and flagged to produce an output of "0" or an "OFF" pixel signal. Once processed in accordance with the steps of Figure 3 or by the circuitry of Figure 5, both being preferably implemented with a look-up table 32, the first filter output image 34 results. Subsequently, a second iteration of a template-based filter 40 is accomplished in a manner similar to that previously described with respect to Figures 3, 4A-B and 5. More specifically, referring again to Figure 2, the first filter output is then used as the input and compared against the templates stored in LUT 2 (42), to generate the error image 44. Subsequently, error image 44 is XORed (46) with the first output image to produce the text-only output image 48. Alternatively, the text-only output image 48 may be further XORed (50) with the input document to produce halftone-only output image 52. Thus, segmented binary images 48 and 52 may be output, said segmented images primarily comprising marks representing one of the structures passed by the filters. For example, segmented images 48 and 52 are, respectively, images having only text or halftone image segments therein. It is further understood that the output of the filtering process, as depicted in Figures 3, 4A-B and 5, may also be a purely binary signal indicating whether a match was located for each particular target pixel considered. In this manner, the output of the template-based filters would be binary in nature, and would not necessarily allow the passage or masking of the image segments without performing further logic operations on the input image. Referring again to Figure 1, once the segmented images are produced by segmentation filter 14, they may be subsequently operated on by a image processing and/or recombination operation, represented as block 15. In particular, the image processing operations may employ filters and other well-known techniques specifically designed to process the segmented image structures isolated by segmentation filter 14. Furthermore, once processed, the segmented images may be recombined to form output image 16 in a form suitable for improved rendition by marking engine 18. Turning now to Figures 6 and 7, data flow diagrams illustrating the various stages in the process used to design the template-based segmentation filters of the present invention will now be described. As illustrated in Figure 6, LUT 1 is produced by using a pair of training documents wherein the first training document 150 is a digital representation of an electronic document containing both text and halftone regions. The second training document, document 152, is identical to the first training document, except that it has been edited to remove the halftone regions therein. To produce the templates to be stored in LUT 1 , the first and second training documents are passed to template matching program 156. Program 156 works in accordance with the methods described by Loce et al. in US-A-5,387,985. Generally, the filter design process accomplished by the template matching program will allow for design of optimized template-matching filters that are then stored in a programmable memory as LUT 1 . As is apparent, many aspects of the present invention or the associated template design process may be accomplished or simulated using a programmable data processing system. In the application cited above, the LUT design process produces a filter that results in a minimum number of pixels in error when applied to an input image. In the present case, it may be more important to not make errors in the halftone portion of the image, as opposed to the text portion of the image. Therefore in an alternative embodiment it may be preferable to apply a weighting factor, greater than 1, to halftone pixels in the training document, so the statistical design procedure attempts to minimize halftone pixel classification error more than text pixel classification error. In general, it may be preferable to weight pixels of one structure in the training documents. The weighting could be straightforward such as each halftone pixel is figured into the statistics as N pixels would normally be treated. Once the first stage or iteration of the segmentation filter is generated and stored as LUT 1 , the second stage may be produced. As illustrated by the data flow diagram of Figure 6, the original image 150 is passed to the first-stage segmentation filter represented as LUT 1 . The output, filtered image 154 is then stored so that it can be passed to XOR logic circuit 158, where it is XORed with the text-only training document 152. The output of the XOR operation 158, error image 160, is then passed to template matching program 156 along with the filtered image 154. In this second occurrence of the template matching program, the output will be a series of templates depicted as LUT 2 . It should be noted that additional iterations of the second segmentation filter design process, Figure 6, would be necessary to generate additional templates (LUT n ) to accomplish further segmentation filtering. In recapitulation, the present invention is a method and apparatus for automatic image segmentation using template matching filters. The invention generally segments differing binary textures or structures within an input image by passing one or more structures while removing other structures. More particularly, the method and apparatus segment a stored binary image using a template matching filter that is designed to pass therethrough, for example, text regions while removing halftone regions.

Claims: A method for automatic image segmentation using template matching filters, for processing a document image to determine image structures present therein, the steps comprising: receiving (100), from an image source, a document image having a plurality of pixels therein, each pixel represented by a binary image signal, and storing at least a portion thereof representing a region of the document image in a data buffer; retrieving (102), from the data buffer, the binary image signals for the document image; determining (104-122), using template matching filters, image structures present in the region of the document image, characterised in that the step of determining image structures comprises the steps of: applying a first stage template matching filter (30) to the binary image signals for the document image to produce a first filtered binary output image; applying a second stage template matching filter (40) to the first filtered binary output image to produce a binary error image (44); and, XORing (46) the binary error image (44) and the first filtered binary output image (34) to mask from the first filtered binary output image any binary image signals from segments of the document image not comprised of a first image structure to produce a first binary output image (48), wherein the first binary output image contains only first image structure segments therein. The method of claim 1, wherein the step of determining image structures present in the region of the image retrieved further comprises the step of XORing (50) the first binary output image (48) and the document image (10) to mask from the document image any binary image signals from segments of the document image not comprised of a second image structure to produce a second binary output image (52), wherein the second binary output image contains only second image structure segments therein. The method of claim 1 or claim 2, wherein the step of applying a first stage template matching filter to the binary image signals for the document image comprises the steps of: identifying a window (132) within the document image so as to select a subset of the document binary image signals; and comparing the binary image signals to the window (132) to a pattern (136a-136f) within the template based filter (30), the pattern representing a pattern determined to occur in a first image structure segment of an image. The method of claim 3, wherein a plurality of signals representing a pattern (136a-136f) within the template-based filter are stored in each location of a look-up table memory (32,42) and where the step of comparing the binary image signals in the window to a pattern within the template-based filter comprises the steps of: storing the subset (202) of document binary image signals in a register memory (204); logically comparing (206,208) each binary image signal in the register memory with a uniquely associated signal of the template based filter stored in the look-up table memory (32,42) location; and outputting a logic signal (52) indicative of the result obtained in the logical comparing step. The method of any of claims 1 to 4, wherein the step of applying a second stage template matching filter (40) to the first filtered binary output image comprises the steps of: identifying a window (134) within the first filtered binary output image (34) so as to select a subset of the first filtered binary output image signals; and comparing the density signals in the window (134) to a pattern (138) within the template based filter (40), the pattern representing a pattern determined to occur in a first image structure segment of an image. The method of any of the preceding claims, further comprising the step of segmenting the document image based upon the image structures determined for regions of the document image. The method of any of the preceding claims wherein the first image structure is text and/or the second image structure is halftone. An apparatus for processing binary image pixels in an image represented by a plurality of rasters of binary image pixels, to identify regions exhibiting a particular structure therein, comprising: an image source for producing a document image (10) having a plurality of binary image pixels therein, each pixel represented by a binary image signal; memory for storing at least a portion of the binary image signals representing a region of the document image in a data buffer; and processing means (14) employing template-matching filters (30,40) to identify the presence of the particular structure in the region of the image stored in said memory, characterised by the processing means being adapted to apply a first stage template matching filter (30) to the binary image signals for the document image to produce a first filtered binary output image; to apply a second stage template matching filter (40) to the first filtered output image to produce a binary error image (44); and to XOR the binary error image (44) and the first filtered binary output image (34) to mask from the first filtered output image any binary signals from segments of the document image not comprised of a first image structure to produce a first binary output image (48), wherein the first binary output image contains only first image structure segments therein. Apparatus according to claim 8, wherein the processing means comprises a segmentation circuit (14) having a logic filter.

Inventors: ESCHBACH REINER, JODOIN RONALD E, LOCE ROBERT P, ESCHBACH, REINER, JODOIN, RONALD E., LOCE, ROBERT P.
Applicants: XEROX CORP, XEROX CORPORATION
Application year: 1995
Main classification: H04N1
IPC codes: G06T5, H04N1