Abstract:
An image processing apparatus for changing a layout of a character string and/or a drawing contained in image data is disclosed. The apparatus includes a first detection means, a second detection means, a change means, a recognition means, and a replacing means. The first detection means detects a directive word, which is a character string that indicates a drawing position. The second detection means detects a drawing whose position is indicated by the directive word. The change means changes a layout of the character string and/or the drawing position. The recognition means recognizes positional relation between the directive word and the drawing after a layout change. The replacing means replaces the directive word based on the positional relation.

Description:
BACKGROUND OF THE INVENTION  
         [0001]    1. Field of the Invention  
           [0002]    The invention relates to the image processing apparatus, image processing method, and computer readable storage medium for changing character strings and/or drawing layouts included in image data.  
           [0003]    2. Description of Related Art  
           [0004]    Various editing methods have been known for using image data effectively according to the purpose of usage.  
           [0005]    Japanese Patent Unexamined Publication No. 8-255160 (A) discloses an editing method for automatically laying out visually recognizable information such as characters, graphics, photographs and images within a specified area. The method provides a means of automatically adding layout information for displaying electronic image data on a display device.  
           [0006]    Japanese Patent Unexamined Publication No. 10-228473 (A) discloses a method of automatically generating links between drawings such as diagrams and tables included in the image and the text related thereto, and converting them into hypertexts.  
           [0007]    The method includes the steps of detecting captions, detecting specified character strings related to drawings from the captions, and detecting character strings identical to the detected character strings from character areas to generate links between the character strings in the captions and the character strings in the character areas, based on the positional relations between the areas where the diagrams and tables exist and neighboring character areas.  
           [0008]    Japanese Patent Unexamined Publication No. 11-85741 (A) discloses an editing method for automatically laying out drawing numbers in optimum positions. The method allocates drawing numbers to drawings automatically according to specified drawing number parameters.  
           [0009]    In general, drawings contained in a document are referenced by drawing numbers that contain unique numbers such as “FIG. 1” and “FIG. 2,” or character strings that direct the positions of drawings such as “drawing on the right” and “drawing above.” 
           [0010]    However, contrary to drawing numbers that contain unique numbers, character strings that direct the positions of drawings can cause problems if layout changes are applied. For example, a drawing that used to be referenced as “drawing on the right” can move to the left of the character string that used to constitute the “drawing on the right,” or a drawing that used to be referenced as “drawing below” can move to a position above the character string that used to constitute the “drawing below.” 
           [0011]    Thus, the character string that indicates the position of a drawing may not match with its positional relation of the drawing after an editing process that accompanies a layout change and may develop a contradiction. This causes a problem of reducing its value as a reference material.  
           [0012]    on the other hand, the method disclosed by Publication No. 8-255160 is intended for the layout of documents such as newspapers and magazines where drawing numbers and character strings indicating drawing positions are not indicated. The method disclosed by Publication No. 10-228473 is simply using the existing drawing numbers and character strings indicating drawing positions. The method disclosed by Publication No. 11-85741 is to re-allocate drawing numbers. Therefore, these methods disclosed on the Publications cannot deal with the present problem.  
         SUMMARY OF THE INVENTION  
         [0013]    A general object of the invention is to provide an image processing apparatus, an image process method and a computer readable storage medium capable of maintaining the consistency between character strings that indicate drawing positions and actual drawing positions before and after layout changes.  
           [0014]    It is still more specific object of the invention to provide an image processing apparatus for changing a layout of a character string and/or a drawing contained in image data. The apparatus includes a first detection means, a second detection means, a change means, a recognition means, and a replacing means. The first detection means detects a directive word, which is a character string that indicates a drawing position. The second detection means detects a drawing whose position is indicated by the directive word. The change means changes a layout of the character string and/or the drawing position. The recognition means recognizes positional relation between the directive word and the drawing after a layout change. The replacing means replaces the directive word based on the positional relation.  
           [0015]    A further object of the invention is to provide an image processing method for changing a layout of a character string and/or a drawing contained in image data. The method includes the steps of: (a) detecting a directive word, which is a character string that indicates a drawing position; (b) detecting a drawing whose position is indicated by the directive word; (c) changing a layout of the character string and/or the drawing position; (d) recognizing positional relation between the directive word and the drawing after a layout change; and (e) replacing the directive word based on the positional relation.  
           [0016]    Still a further object of the invention is to provide a computer readable storage medium for storing a program for executing the aforesaid image processing method.  
           [0017]    The objects, characteristics, and advantages of this invention other than those set forth above will become apparent from the following detailed description of the preferred embodiments, which refers to the annexed drawings. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0018]    [0018]FIG. 1 is a block diagram of an image processing system according to an embodiment of this invention;  
         [0019]    [0019]FIG. 2 is an example allocation table used for consistency process on an image processing apparatus of the image processing system;  
         [0020]    [0020]FIG. 3 is a flow chart of the consistency process;  
         [0021]    [0021]FIG. 4 is a flowchart of a first correlating process in the consistency process;  
         [0022]    [0022]FIG. 5 is an example input image;  
         [0023]    [0023]FIG. 6 is an example allocation table after the first correlating process;  
         [0024]    [0024]FIG. 7 is a flow chart of an updating process to an allocation table in the consistency process;  
         [0025]    [0025]FIG. 8 is an example of the allocation table after the updating process;  
         [0026]    [0026]FIG. 9 is a flowchart of a second correlating process in the consistency process;  
         [0027]    [0027]FIG. 10 is a flow chart of a process for detecting positional relation between a first directive word and a drawing in the second correlating process;  
         [0028]    [0028]FIG. 11 is a schematic representation of assistance in explaining the positional relation between the first directive word and the drawing;  
         [0029]    [0029]FIG. 12 is an example allocation table after the second correlating process; and  
         [0030]    [0030]FIG. 13 is an example output image. 
     
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS  
       [0031]    The embodiments of this invention will be described below with reference to the accompanying drawings.  
         [0032]    The image processing system shown in FIG. 1 has an image processing apparatus  10 , a controller  20 , an operating panel  30 , an image input apparatus  40 , a first output apparatus  50 , and a second output apparatus  60 .  
         [0033]    The image processing apparatus  10  has a character recognition unit  11 , an area separation unit  12 , a bitmap processing unit  13 , a vector conversion unit  14 , a binarization unit  15 , a synthesizing unit  16 , a memory  17 , and a format conversion unit  18 .  
         [0034]    The controller  20  has an interface  22  for the operating panel  30 , an interface  23  for the image input apparatus  40 , an interface  22  for the first output apparatus  50  and the second output apparatus  60 , and a central processing unit (CPU)  21  for controlling the interfaces  22  through  24 .  
         [0035]    The operating panel  30  is used by the operator for inputting instructions. The image input apparatus  40  is an image reading apparatus such as a color scanner. The first output apparatus  50  is an image forming apparatus such as a color printer, and the second output apparatus  60  is a apparatus for displaying and data processing the image data to be outputted, for example, a computer equipped with a display device.  
         [0036]    Functions of each unit will be described in detail along the operation flow.  
         [0037]    The user inputs the instruction information using the operating panel  30 . The instruction information can be, for example, an operation start instruction or an instruction for a manual setting item.  
         [0038]    Manual setting items include a scaling factor setting, an instruction of the N-in-1 process, a layout change setting, a consistency process instruction, a post-processing selection, a readout mode, and an output format selection.  
         [0039]    The N-in-1 process is the process of reducing the size of and synthesizing a plurality of sheets of document images and laying them out as a single page image.  
         [0040]    In the layout change setting, one of the three modes, i.e., no priority, character priority, or graphics priority, can be selected. In the character priority mode, it is guaranteed that the character size after the reduction of character areas will not be smaller than the predetermined value. In the graphics priority mode, the size of the character area will be maintained constant when the images are enlarged.  
         [0041]    The consistency process is to maintain the consistency between the directive words as character strings for directing the positions of drawings and the actual positions of the drawings. In other words, it is the process of preventing the characters contained in a directive word that represents a position and the actual position of the related drawing after the layout change from corresponding and contradicting.  
         [0042]    The post-processing selection is a mode for selecting the post-processing that is applied to the three types of areas separated in the character recognition unit  11  and the area separation unit  12 , i.e., character areas, graphics areas,and photographic areas. The post-processing includes character coding at the character recognition unit  11 , bitmap processing at the bitmap processing unit  13 , vector conversion at the vector conversion unit  14 , and binarization at the binarization unit  15 .  
         [0043]    The readout mode consists of the color mode for treating a document image as a color image and a monochromatic mode for treating a document image as a monochromatic image at the image input apparatus  40 .  
         [0044]    The output format selection is a mode for selecting the format of the output file to be prepared at the format conversion unit  18 . The output formats are general-purpose file formats, e.g., the document file format, the page description language format, the file format for document display, and the file format for storing images.  
         [0045]    For example, the document file format is the Rich Text Format, the page description language format is the PostScript (R), the file format for document display is the PDF (Portable Document Format), and the file format for storing images is either the JPEG (Joint Photographic Experts Group) or the TIFF (Tagged Image File Format).  
         [0046]    The instruction information from the operating panel  30  is transmitted to the controller  20  via the interface  22 .  
         [0047]    As it receives instruction information for the manual setting items, the controller  20  inputs the manual setting items to the image processing apparatus  10 . Furthermore, as it receives the operation start instruction, the controller  20  instructs the image input apparatus  40  to start reading images either in the color mode or in the monochromatic mode according to the readout mode setting.  
         [0048]    The image input apparatus  40  reads the document image according to the operation start instruction from the controller  20 . The generated image data is transmitted to the character recognition unit  11  of the image processing apparatus  10  via the interface  23  of the controller  20 .  
         [0049]    The character recognition unit  11  separates character areas from the image data and extract character images existing in the character areas. The image data left after removing the character images are inputted into the area separation unit  12 . The character recognition unit  11  extracts character information including character code data and positional information, and color information from the character images.  
         [0050]    The positional information includes X-Y coordinates, widths, lengths, number of characters, etc. The character information are inputted into the synthesizing unit  16 . When binarization is specified as the post-processing of the output area by the user, the character area is inputted into the binarization unit  15 .  
         [0051]    The area separation unit  12  separates graphics areas and photographic areas from the image data. The photographic area data will be added with positional information such as X-Y coordinates, widths and lengths,.and will be inputted into the bitmap processing unit  13 .  
         [0052]    On the other hand, the data in the graphics area will be added with positional information and will be inputted into the vector conversion unit  14 . If the post-processing is specified, the image data after area separation will be inputted into the bitmap processing unit  13  or the vector conversion unit  14  or the binarization unit  15  according to the details of specified matter.  
         [0053]    The bitmap processing unit  13  applies the bitmap processing to the data in the photographic area. In the bitmap processing, the data of the photographic area is applied with various image processes such as the edge correction, the smoothing and the MTF correction. The bitmap information including the bitmap data and the positional information will be inputted into the synthesizing unit  16 . The bitmap processing unit  13  will execute the similar process to the image data, to which the bitmap processing is specified as the post-processing.  
         [0054]    The vector conversion unit  14  applies vector-conversion to the data in the graphics area to generate vector data. The vector data is inputted into the synthesizing unit  16  together with the attribute data. The vector conversion means converting graphics consisting of dots into vector data such as straight lines, arcs, Bezier curves, etc.  
         [0055]    The attribute data are data obtained by extracting line widths, line types, line colors, end point styles, and colors of enclosed areas surrounded by vector data. The vector conversion unit  14  executes the similar process to image data, to which vector conversion is designated as the post-processing.  
         [0056]    The binarization unit  15  binarizes the image data from the character recognition unit  11  and/or the area separation unit  12 , when the binarization process is specified as the post-processing. The binarization data is inputted into the synthesizing unit  16  with the positional information.  
         [0057]    The synthesizing unit  16  synthesizes the input data from the character recognition unit  11 , the bitmap processing unit  13 , the vector conversion unit  14 , and the binarization unit  15 . The synthesized data is converted into intermediate format data and inputted into the format conversion unit  18 .  
         [0058]    The intermediate format data are intermediate data between the synthesized data and the output format data, and are generated in order to facilitate the processing at the format conversion unit  18 . The synthesizing unit  16  executes the consistency process using the allocation table according to the manual setting items.  
         [0059]    In the consistency process, the corresponding relation between the drawing and the first directive word, which is the character string that directs the position of the drawing before the layout change is detected, the positional information of the first directive word and the drawing are updated according to the layout change, the second directive word, which is character string that directs the position of the drawing after the layout change, is generated based on the corresponding relation and the updated positional information between the first directive word and the drawing, and the first directive word is replaced by the second directive word. The first directive word and the second directive word are, for example, “drawing on the right” or “drawing above.” 
         [0060]    The allocation table has, as shown in FIG. 2, the directive section, the drawing section, the insertion section, and the text section. In the directive section, the first directive word and the second directive word are set up as detected character string and the replacing character string respectively.  
         [0061]    In the drawing section, memory address and the positional information of the drawing that corresponds to the first directive word are setup. In the insertion section, memory address and the positional information of the first directive word are setup. In the text section,memory address and the positional information of the character code data that belongs to the character area are set up.  
         [0062]    The memory  17  is used for storing the allocation table and the input data for and the synthesizing unit  16 .  
         [0063]    The format conversion unit  18  converts the intermediate format data into the data of the specified output format. The output format data is inputted into the first output apparatus  50  and/or the second output apparatus  60  via the interface  24 .  
         [0064]    As an example, the first output apparatus  50  prints the data on paper, and the second output apparatus  60  stores the data and displays it on the monitor.  
         [0065]    Next, the consistency process will be described referring to the flow chart shown in FIG. 3.  
         [0066]    First, character areas are separated from the image data (step Si), and the character information is extracted from character images in the character areas (step S 2 ). The image data, from which the character images are removed, are interpolated using the peripheral pixels of the character images (step S 3 ). Then, photographic areas and graphics areas are separated from the image data (step S 4 ). The photographic area data are treated by the bitmap process, and the graphics area data are treated by the vector conversion process (step S 5 ).  
         [0067]    After that, the first correlating process concerning the allocation table will be executed based on the character information, the bitmap information, and the vector data (step S 6 ). The first correlating process is a process of detecting the corresponding relation between a drawing and the first directive word, which is the character string that directs the position of the drawing before the layout change.  
         [0068]    Next, a judgment will be made whether there is a next page image data (step S 7 ). If it is judged that there is a next page, the process returns to the step S 1 . If the next page does not exist, or if the process of the last page is completed, the allocation table updating process will be executed (step S 8 ). The updating process of the allocation table is a process of updating the positional information of the first directive word and the drawing.  
         [0069]    Then, the second correlating process is executed to obtain the final allocation table that will be used for data synthesizing (step S 9 ). The second correlating process is a process of generating the second directive word, which is the character string that directs the drawing position after the layout change, based on the corresponding relation and the updated positional information of the first directive word and the drawing, and replacing the first directive word with the second directive word.  
         [0070]    Next, the image data will be synthesized based on the allocation table (step S 10 ), and the image data will be converted into the intermediate format data (step Si 1 ). The intermediate format data will be converted into the specified output format data (step S 12 ) and outputted (step S 13 ).  
         [0071]    Lastly, a judgment is made whether there is a next page image data based on the allocation table (step S 14 ). If it is judged that there is a next page, the process returns to the step S 10 . If the next page does not exist, or if the process of the last page is completed, the process will be terminated.  
         [0072]    Next, the first correlating process will be described referring to the flow chart shown in FIG. 4.  721  First, the character code data that belongs to the character area will be stored in the memory  17  (step S 61 ). The memory address and the positional information of the character code data will be set up into the text section of the allocation table (step S 62 ).  
         [0073]    Next, the first directive word contained in the character code data will be detected (step S 63 ), and the first directive word&#39;s data will be stored in the memory  17  (step S 64 ). Then, the first directive word will be set up in the directive section of the allocation table as the detected character string (step S 65 ), and the memory address and the positional information of the first directive word will be set up in the insertion section of the allocation table (step S 66 ).  
         [0074]    After that, the drawing that corresponds to the first directive word will be detected based on the direction the first directive word is directing, the coordinate position of the first directive word, and the bitmap data of the photographic area or the vector data of the graphics area located in the vicinity of the first directive word (step S 67 ).  
         [0075]    The detected drawing&#39;s data will be stored in the memory  17  (step S 68 ). The memory address and the positional information of the detected drawing will be set up in the drawing section of the allocation table being correlated with the first directive word (step S 69 ).  
         [0076]    Then, the data of the remaining drawings that consist of the bitmap data and/or vector data that are not correlated to the first directive word will be stored in the memory  17  (step S 70 ). The memory address and the positional information of the remaining drawings&#39; data will be set up in the drawing column of the allocation table without being correlated with the directive (step S 71 ).  
         [0077]    Next, the first correlating process will be described more specifically using an image shown in FIG. 5.  78   
         [0078]    First, the character code data existing in the character areas  81 ,  82  and  83  is stored in the first storage area of the memory  17 . The memory address and positional information of the character code data are set up in the text section of the allocation table. The starting addresses of the data of the character areas  81 ,  82  and  83  are shown as Cadr 1 , Cadr 3  and Cadr 5 .  
         [0079]    Then, “drawing below,” “drawing on the right” and “drawing on the left” contained in the character code data of character areas  81 ,  82  and  83  will be detected as the first directive words  91 ,  92  and  93 . The data of the first directive words  91 ,  92  and  93  will be stored in the first storage area of the memory  17 . Furthermore, the first directive words  91 ,  92  and  93  will be set up in the directive section of the allocation table as the detected character strings. The memory addresses and positional information of the first directive words  91 ,  92  and  93  will be set up in the insertion section of the allocation table. The starting addresses of the data of the first directive words  91 ,  92  and  93  are shown as Cadr 2 , Cadr 4  and Cadr 6 .  
         [0080]    Next, a drawing  71  that consists of vector data of a graphics area located in the direction indicated by “drawing below,” which is the first directive word  91 , will be detected as the drawing corresponding to the first directive word  91 . Similarly, drawings  72  and  73  consisting of the bitmap data of the photographic areas will be detected as the drawings corresponding to the first directive words  92  and  93 .  
         [0081]    The data of the drawings  71 ,  72  and  73  will be stored in the second storage area of the memory  17 . The memory addresses and positional information of the drawings  71 ,  72  and  73  will be set up into the drawing section of the allocation table in correlation with the first directive words  91 ,  92  and  93 . The starting addresses of the drawings  71 ,  72  and  73  will be shown as Fadr 1 , Fadr 2  and Fadr 3 .  
         [0082]    Thus, the data such as shown in FIG. 6 are set up in the allocation table.  
         [0083]    Next, the updating process of the allocation table will be described referring to the flow chart shown in FIG. 7. Let us take a layout case as an example where the character areas  81  through  83  in the image shown in FIG. 5 are to be enlarged and arranged at the top of the sheet, while the FIGS.  71  through  73  are to be reduced and arranged at the bottom of the sheet.  
         [0084]    First, the data in the character areas  81  through  83  are integrated (step S 81 ), and the enlargement factor of an area  80  to which the character areas  81  through  83  are to be plugged in is calculated (step S 82 ). Then, the data of the character areas  81  through  83  stored in the second storage area will be changed according to the enlargement factor and the layout setting, the memory addresses Cadr 3  and Cadr 5  as well as the positional information stored in the text section will be removed, and the memory address Cadr 1  and the positional information will be changed (step S 83 ).  
         [0085]    Simultaneously, the data of the first directive word  91  through  93  will be changed and the memory addresses Cadr 2 , Cadr 4  and Cadr 6  as well as the positional information will be changed. The memory addresses after the change will be shown as Cadr 1 ′, Cadr 2 ′, Cadr 4 ′ and Cadr 6 ′.  
         [0086]    After that, the reduction factor for drawing area, i.e., the area where the drawings  71  through  73  are to be placed is calculated (step S 84 ). Then, the data of the drawings  71  through  73  stored in the second storage area are changed based on the reduction factor and the layout setting, and the memory addresses and the positional information of the drawing section will be changed (step S 85 ). The memory address after the change will be shown as Fadr 1 ′ through Fadr 3 ′.  
         [0087]    As a result of the above, the data of the allocation table shown in FIG. 6 will be updated and the allocation table shown in FIG. 8 will be obtained.  
         [0088]    Next, the second correlating process will be described referring to the flow chart shown in FIG. 9.  
         [0089]    First, a first directive word that does not have a correlated second directive word will be selected (step S 91 ), and the positional information of the first directive word will be read (step S 92 ). Then, the positional information of the character area and the drawing correlated to the first directive word will be read (step S 93 ).  
         [0090]    Next, the process of detecting positional relation between the first directive word and the drawing will be executed based on the positional information of the character area and the drawing (step S 94 ). After that, a second directive word is generated based on the positional relation (step S 95 ), and the second directive word will be se up in the directive section of the allocation table as the replacing character string (step S 96 ).  
         [0091]    Lastly, a judgment will be made whether any other first directive word that does not have a correlated second directive word exist (step S 97 ). If it is judged that the next first directive word exists, the process returns to the step S 94  and the process will be repeated. On the other hand, if it is judged that the next first directive word does not exist, the process will be terminated.  
         [0092]    Next, the process of detecting the positional relation between the first directive word and the corresponding drawing will be described referring to FIG. 10 and FIG. 11 taking a case of using the starting coordinates and lengths of the character areas and drawings as an example.  
         [0093]    First, the Y-axis value Y C  of a character area  84  where a first directive word  94  exists, correlated to the first directive word  94  and the Y-axis value Y F  of a drawing  74  correlated to the first directive word  94  are compared (step S 941 ). If the value Y C  is judged to be greater than Y F , the drawing  74  is considered to be located about the character area  84  (step S 942 ). Therefore, the second directive word generated in the step S 95  will be “drawing above.” 
         [0094]    If the value Y C  is judged to be equal to or be smaller than YF, a further comparison between the value Y F  and the sum T of the value Y C  and the length L C  of the character area  84  will be made (step S 943 ). If the sum T is judged to be smaller than the value Y F , the drawing  74  is considered to be located below the character area  84  (step S 944 ). Therefore, the second directive word generated in the step S 95  will be “drawing below.” 
         [0095]    If the sum T is judged to be equal to or greater than the value Y F , the X-axis value X C  of the character area.  84  will be further compared with the X-axis value X F  of the drawing  74  (step S 945 ). If X C  is judged to be smaller than X F , the drawing  74  is considered to be located on the right side of the character area  84  (step S 946 ). Therefore, the second directive word generated in the step S 95  will be “drawing on the right.” 
         [0096]    If X C  is judged to be equal to or greater than X F , the drawing  74  is considered to be located on the left side of the character area  84  (step S 947 ). Therefore, the second directive word generated in the step S 95  will be “drawing on the left.” 
         [0097]    Thus, the allocation table shown in FIG. 12 is obtained. The image shown in FIG. 13 is outputted as a result of reading the data from the memory  17  based on the allocation table and synthesizing the data. Therefore, the second directive words  101 ,  102  and  103  are “drawing below,” “drawing below” and “drawing below,” and correspond with the positions of the drawings  71 ,  72  and  73 .  
         [0098]    Consequently, it is possible to maintain the consistency between drawing positions and directive words, which are the character strings that indicate the drawings positions, before and after a layout change. Hence, it eliminates the problem of causing mismatches between the directive words and drawing positions after a layout change and contradictions, and reducing values of the document.  
         [0099]    It is obvious that this invention is not limited to the particular embodiments shown and described above but may be variously changed and modified without departing from the technical concept of this invention.  
         [0100]    The invention is applicable to such layout change processes as the N-in-1 process, the page orientation changes (portrait vs. landscape), etc. In those cases, the only difference is in the allocation table updating process influenced by the layout change process, and the first and second correlating processes remain the same.  
         [0101]    The invention is applicable not only to a system including a plurality of apparatuses, but also to standalone equipment such as digital copying machines.  
         [0102]    It is also possible to make a computer function as an image processing apparatus by providing a program product containing the code data of the programmed image processing method. Such a program product includes a program itself and a storage medium that contains the program.