IMAGE ANALYSIS DEVICE, IMAGE ANALYSIS METHOD, AND STORAGE MEDIUM STORING IMAGE ANALYSIS PROGRAM

An image analysis device includes: an acquisition unit to acquire an input image; a detection unit to analyse the input image acquired by the acquisition unit and to detect an area corresponding to a component image constituting the input image and a type of the component image; an expansion unit to expand the area detected by the detection unit; an evaluation unit to evaluate an image included in the area expanded by the expansion unit on the basis of a rule according to the type; and an output unit to output the area corresponding to the component image from the area expanded by the expansion unit on the basis of an evaluation result given by the evaluation unit.

TECHNICAL FIELD

The present disclosure relates to an image analysis device, an image analysis method, and an image analysis program.

BACKGROUND ART

Technologies for analysing an input image captured by a scanner, etc. to detect areas corresponding to component images that constitute the input image have been conventionally known. According to Patent Document 1, it is possible to detect an area corresponding to a component image of each of elements, such as a figure, a table, or a text, which constitute an input image.

Prior Art Documents

Patent Document

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

Meanwhile, however, a misalignment may occur between the area corresponding to a component image constituting an input image and the detected area. In such cases, the conventional technologies unfortunately cannot output the area corresponding to the component image accurately.

The aim of the present disclosure is to solve the aforementioned problem and thus to control the degradation of output accuracy of an area corresponding to a component image even when a misalignment occurs between the area corresponding to the component image constituting an input image and the detected area.

Means for Solving Problem

An image analysis device includes: an acquisition unit to acquire an input image; a detection unit to analyse the input image acquired by the acquisition unit and to detect an area corresponding to a component image constituting the input image and a type of the component image; an expansion unit to expand the area detected by the detection unit; an evaluation unit to evaluate an image included in the area expanded by the expansion unit on the basis of a rule according to the type; and an output unit to output an area corresponding to the component image from the area expanded by the expansion unit on the basis of an evaluation result given by the evaluation unit.

Effects of the Invention

According to the present disclosure, the degradation of output accuracy of an area corresponding to a component image can be controlled even when a misalignment occurs between the area corresponding to the component image constituting an input image and the detected area.

EMBODIMENTS FOR CARRYING OUT THE INVENTION

FIG.1is a block diagram showing a configuration of an image analysis device1according to Embodiment 1. The image analysis device1is a device that analyses an acquired input image and outputs areas each corresponding to a component image constituting the input image. The input image includes component images, which are represented by types such as figure, table, and text. The image analysis device1includes an acquisition unit101, a detection unit102, a position correction unit108, and an output unit106.

The acquisition unit101acquires an input image read by a device such as a scanner or a camera, which is not shown in the figure. The detection unit102analyses the input image acquired by the acquisition unit101and detects areas and their types each corresponding to a component image constituting the input image. In the following description, an area that is detected by the detection unit102and corresponds to a component image constituting the input image is referred to as a detection area. It is assumed here that some of the detection areas detected by the detection unit102may not exactly match the component images constituting the input image and may include misalignment and that an image included in the detection area is stored in a memory or a storage unit.

An expansion unit103expands the detection area detected by the detection unit102at least twice and until an exit condition of the expansion is satisfied. The expansion unit103expands the detection area, for example, by a predetermined width, height, area, or magnification rate. Hereafter, the area expanded by the expansion unit103will be referred to as an expansion area. An image included in the expansion area shall be stored in a memory or a storage unit. Note that the detection area detected by the detection unit102may be considered as the 0th expansion area expanded by the expansion unit103.

Next, exit conditions of the expansion by the expansion unit103are described. One of the exit conditions of the extension is that the expansion area includes an area of a different type from the type of the detection area. Another one of the exit conditions of the expansion is that the number of the expansions of the detection area reaches a maximum limit. Yet another one of the exit conditions of the expansion is that the expansion area encloses the outline of the component image. Here, the outline of the component image may be a solid line of an outer frame surrounding each element such as a figure, a table, or a text, or may be a virtual line of a rectangular bounding each element. The case where the expansion area encloses the outline of the component image includes, by definition, the case where the detection area is expanded such that the outer frame of the expansion area is of an equal distance from the outline of the component image. To detect the outline, for example, a method described in Reference 1 can be applied. Alternatively, more than one of the above exit conditions may be combined into one exit condition. The exit conditions of the expansion are stored, for example, in a storage unit13described below.Reference 1: Takashi Hirano, Yasuhiro Okada, Fumio Yoda, “Line Extraction Method from Document Images”, General Conference, March 1998, The Institute of Electronics, Information and Communication Engineers

An evaluation unit104evaluates an image included in the area expanded by the expansion unit103on the basis of a rule according to the type of the area concerned. More specifically, the evaluation unit104evaluates an image included in the area obtained as a result of the Nth expansion by the expansion unit103and an image included in the area obtained as a result of the N+1th expansion by the expansion unit103on the basis of a rule according to the type of the area concerned. A rule DB105stores a rule for evaluating the image included in an area expanded by the expansion unit103. The rule DB105may be included in the image analysis device100or provided externally.

The output unit106outputs an image included in the area expanded by the expansion unit103as an image corresponding to the component image on the basis of the evaluation result (evaluation value) given by the evaluation unit104. More specifically, the output unit106outputs an image, included in the expanded area, whose evaluation value given by the evaluation unit104is the largest as an image corresponding to the component image.

FIG.2illustrates a component image that constitutes an input image, a detection area corresponding to the component image, and an expansion area that is an expansion of the detection area. An input image NG1 includes a component image KG1 containing a table, for example. The detection area KR11 indicated by the alternate long and short dash line shows an area detected by the detection unit102.

It is assumed here that detection area KR11 may not exactly match the component image KG1 and may include misalignment. An expansion area KR21 indicated by an alternate long and two short dashes line shows an area obtained as a result of the expansion of the detection area KR11 by the expansion unit103. An expansion area KR22 indicated by the other alternate long and two short dashes line shows an area obtained as a result of the expansion of the expansion area KR21 by the expansion unit103.

Table 1 is an example of the rule for evaluating the image included in the area expanded by the expansion unit103, the rule being stored in the rule DB105.

TABLE 1TypeRuleTableThe smaller the difference between thenumber of cells in the area beforethe expansion and the number of cellsin the area after the expansion is,the higher evaluation value is given tothe area before the expansion.TextThe higher the degree of coincidencebetween the character recognitionresult for the image included in thearea before the expansion and thecharacter recognition result for theimage included in the area after theexpansion is, the higher evaluationvalue is given to the area before theexpansion.FigureThe larger the blank area containedin the image included in the expansionarea is, the higher evaluation value is given.

FIG.3illustrates the rule for evaluating a table included in the detection area. It is assumed here that the detection unit102has detected “a table” as the type corresponding to the component image KG1. An expansion area KRH21 is an area obtained as a result of the N−1th expansion of the detection area detected by the detection unit102and includes one cell therein. An expansion area KRH22 is an area obtained as a result of the Nth expansion of the detection area detected by the detection unit102and includes nine cells therein. An expansion area KRH23 is an area obtained as a result of the N+1th expansion of the detection area detected by the detection unit102and includes nine cells therein. The evaluation unit104gives a higher evaluation value to the area before the expansion as the difference between the number of cells in the area before the expansion and the number of cells in the area after the expansion is smaller. More specifically, the type detected here by the detection unit102is a table; and the difference (zero) between the number (nine) of cells included in the area obtained as a result of the Nth expansion by the expansion unit103and the number (nine) of cells included in the area obtained as a result of the N+1th expansion by the expansion unit103is smaller than the difference (eight) between the number (one) of cells included in the area obtained as a result of the N−1th expansion by the expansion unit103and the number (nine) of cells included in the area obtained as a result of the Nth expansion by the expansion unit103. Therefore, the evaluation unit104gives a higher evaluation value to the area obtained as a result of the Nth expansion. To extract the cells, for example, a method described in Reference 2 can be applied.Reference 2: Y. Itani et. al, “Text Line Extraction Method Using Domain-based Active Contour Model”, ICDAR 2013, August 2013

FIG.4illustrates a rule for evaluating a text included in the detection area. It is assumed here that the detection unit102has detected “a text” as the type corresponding to a component image KG2. An expansion area KRT21 is an area obtained as a result of the N−1th expansion of the detection area detected by the detection unit102and includes a string of “AGE ANALYSIS DEV” as the character recognition result for the image included in the area. An expansion area KRT22 is an area obtained as a result of the Nth expansion of the detection area detected by the detection unit102and includes a string of “IMAGE ANALYSIS DEVICE” as the character recognition result for the image included in the area. An expansion area KRT23 is an area obtained as a result of the N+1th expansion of the detection area detected by the detection unit102and includes a string of “IMAGE ANALYSIS DEVICE” as the character recognition result for the image included in the area. The evaluation unit104gives a higher evaluation value to the area before the expansion as the degree of coincidence between the character recognition result for the image included in the area before the expansion and the character recognition result for the image included in the area after the expansion is higher. More specifically, the type detected here by the detection unit102is a text; and the degree of coincidence between the character recognition result “IMAGE ANALYSIS DEVICE” for the image included in the area obtained as a result of the Nth expansion by the expansion unit103and the character recognition result “IMAGE ANALYSIS DEVICE” for the image included in the area obtained as a result of the N+1th expansion by the expansion unit103is higher than the degree of coincidence between the character recognition result “AGE ANALYSIS DEV” for the image included in the area obtained as a result of the N−1th expansion by the expansion unit103and the character recognition result “IMAGE ANALYSIS DEVICE” for the image included in the area obtained as a result of the Nth expansion by the expansion unit103. Therefore, the evaluation unit104gives a higher evaluation value to the area obtained as a result of the Nth expansion. The degree of coincidence between the character recognition results can be calculated on the basis of, for example, the number of coincident characters and/or the percentage of coincident characters between the character recognition result for the image included in the area obtained as a result of the Nth expansion by the expansion unit103and the character recognition result for the image included in the area obtained as a result of the N+1th expansion by the expansion unit103. For the character recognition of an image included in an area, known technologies can be used.

FIG.5illustrates a rule for evaluating a figure included in the detection area. It is assumed here that the detection unit102has detected “a figure” as the type corresponding to a component image KG3. An expansion area KRZ21 is an area obtained as a result of the N−1th expansion of the detection area detected by the detection unit102and includes part of the figure. An expansion area KRZ22 is an area obtained as a result of the Nth expansion of the detection area detected by the detection unit102and includes the figure and a blank area KR1. An expansion area KRZ23 is an area obtained as a result of the N+1th expansion of the detection area detected by the detection unit102and includes the figure and a blank area KR2. Here, the evaluation unit104gives a higher evaluation value as the blank area contained in the image included in the expansion area is larger. More specifically, the type detected here by the detection unit102is a figure; and the blank area KR2 included in the area obtained as a result of the N+1th expansion by the expansion unit103is larger than the blank area KR1 included in the area obtained as a result of the Nth expansion by the expansion unit103. Therefore, the evaluation unit104gives a higher evaluation value to the area obtained as a result of the N+1th expansion.

That is, the evaluation unit104gives a higher evaluation value to the expansion area KRZ23 than to the expansion area KRZ21 and the expansion area KRZ22. Alternatively, as with the case where the type detected by the detection unit102indicates a text, the evaluation unit104may give a higher evaluation value to the area before the expansion as the degree of coincidence between the image recognition result for the image included in the area before the expansion and the image recognition result for the image included in the area after the expansion is higher. For the image recognition of an image included in an area, known technologies that enable, at least, extraction of image features can be used. The degree of coincidence between the image recognition results can be calculated on the basis of, for example, the number of coincident features and/or the percentage of coincident features between the image recognition result for the image included in the area obtained as a result of the Nth expansion by the expansion unit103and the image recognition result for the image included in the area obtained as a result of the N+1th expansion by the expansion unit103.

FIG.6illustrates an example of a hardware configuration of the image analysis device1. The image analysis device1includes a CPU11, a memory12, a storage unit13, an image acquisition unit14, and a display unit15. The CPU11executes a program, etc. stored in the memory12to realize the functions of the acquisition unit101, the detection unit102, the position correction unit108, and the output unit106shown inFIG.1. The memory12and the storage unit13store various programs to be executed by the CPU11and various data to be used for the processing to be executed by the CPU11. The image acquisition unit14is, for example, an interface to acquire an input image inputted to a device such as a scanner or a camera, which is not shown in the figure. The display unit15is, for example, an LCD display to display an image outputted from the CPU11. The display unit15may be included in the image analysis device100or provided externally. The CPU11and the memory12may be realized, for example, as hardware such as processing circuitry16shown inFIG.7.

FIG.8is a flowchart showing a process of the image analysis device1. The acquisition unit101acquires an input image read by a device such as a scanner or a camera, which is not shown in the figure (SA1). Next, the detection unit102analyses the input image acquired by the acquisition unit101(SA2) and detects an area and a type corresponding to a component image constituting the input image (SA3). If no area and type corresponding to a component image constituting the input image is detected (SA3: NO), the image analysis device1terminates the image analysis process. If an area and a type corresponding to a component image constituting the input image are detected (SA3: YES), the expansion unit103expands the area detected by the detection unit102by a predetermined range (expanse, magnification rate, width, height) (SA4). The evaluation unit104evaluates the image included in the area expanded by the expansion unit103on the basis of the rule according to the type (SA5). If an exit condition of the expansion is satisfied (SA6: YES), the image included in the expansion area with the highest evaluation value given by the evaluation unit104is outputted by the output unit106as an image corresponding to the component image (SA7). If no exit condition of the expansion is satisfied (SA6: NO), the expansion unit103expands the area that was expanded in SA4 again (SA4).

As described above, the image analysis device1includes: the acquisition unit101to acquire an input image; the detection unit102to analyse the input image acquired by the acquisition unit101and to detect an area and a type corresponding to a component image constituting the input image; the expansion unit103to expand the area detected by the detection unit102; the evaluation unit104to evaluate an image included in the area expanded by the expansion unit103on the basis of a rule according to the type; and the output unit106to output the image included in the area expanded by the expansion unit103as an image corresponding to the component image on the basis of an evaluation result given by the evaluation unit104. More specifically, the expansion unit103expands the detection area detected by the detection unit102at least twice and until the exit condition of the expansion is satisfied. The evaluation unit104evaluates an image included in the area obtained as a result of the Nth expansion by the expansion unit103and an image included in the area obtained as a result of the N+1th expansion by the expansion unit103on the basis of a rule according to the type of the area concerned. The output unit106outputs an image, included in the expanded area, whose evaluation value given by the evaluation unit104is the largest as an image corresponding to the component image. This makes it possible to output, as the image corresponding to the component image, an image included in the area expanded by the expansion unit103on the basis of the evaluation results given by the evaluation unit104in accordance with the rule according to the type even if a misalignment occurs between the area corresponding to the component image constituting the input image and the detection area detected by the detection unit102. Thus, the degradation of output accuracy in the area corresponding to the component image can be controlled in accordance with the type of the component image.

In Embodiment 1, the expansion unit103expanded the detection area uniformly, regardless of the type of the detection area. However, the size of the component image varies depending on the type such as a table, a text, and a figure. Therefore, in the case of uniform expansion of the detection area, the expanded area may sometimes be too large or too small for the component image. Therefore, in Embodiment 2, the expansion unit103expands the detection area in a range (expanse, magnification rate, width, height) corresponding to the type of the detection area.

Also, in Embodiment 1, the detection unit102analysed the input image and detected the type corresponding to the component image constituting the input image. However, it is difficult to identify one type corresponding to a component image if, for example, a table includes a text and a figure. Therefore, in Embodiment 2, the detection unit102first detects a plurality of candidate types corresponding to a component image constituting the input image, and then selects an appropriate candidate type as the type of the detection area corresponding to the component image.

FIG.9is a block diagram showing a configuration of an image analysis device1according to Embodiment 2. The image analysis device1includes the acquisition unit101, the detection unit102, a type selection unit107, the position correction unit108, and the output unit106. Hereinafter, configurations different from those in Embodiment 1 will be described.

The detection unit102detects candidates of the type (hereafter, referred to as candidate types) of the detection area corresponding to the component image. The detection unit102also calculates scores that indicate the likelihood of the detected candidate types. The score can be calculated on the basis of a result estimated by using a known machine learning algorithm such as neural networks or a cosine similarity to a pre-determined feature vector by using Expression (1). In Expression (1), V shows a feature vector for each predetermined item, and F shows a feature vector obtained from the image during the analysis.

The type selection unit107selects an appropriate candidate type as the type of the detection area corresponding to the component image from among the candidate types detected by the detection unit102. A selection method for selecting the appropriate candidate type is, for example, a method using scores. In this case, the type selection unit107may select the candidate type for which the score is greater than a threshold value TH1, as in Expression (2). The type selection unit107may select, in addition to the candidate type for which the score shows the highest value, a candidate type for which the difference between the score of the candidate type concerned and the score of the highest value is smaller than a threshold value TH2, as in Expression (3). Furthermore, Expression (2) and Expression (3) may be combined; and the type selection unit107may select a candidate type that satisfies both of these Expressions simultaneously.

FIG.10shows scores calculated for each of the candidate types.FIG.10shows the calculated scores for the detection area as follows: the score with an assumption of the candidate type being a table is 0.8; the score with an assumption of the candidate type being a figure is 0.7; and the score with an assumption of the candidate type being a text is 0.2. When it is assumed that the threshold value TH1 in Expression (2) is 0.6 and the threshold value TH2 in Expression (3) is 0.2, the candidate types satisfying Expressions (2) and (3) are the two types, namely a table and a figure. Therefore, the type selection unit107selects two of the candidate types, namely a table and a figure, as the candidate types of the detection area corresponding to the component image. The threshold value TH1 and the threshold value TH2 shall be stored in advance in the rule DB105or the like.

The expansion unit103determines a range (expanse, magnification rate, width, height) to be applied to the expansion of the area detected by the detection unit102on the basis of the candidate types selected by the type selection unit107. The range to be applied to the expansion of the area detected by the detection unit102shall be stored in the rule DB105or the like in association with the candidate types. The range to be applied to the expansion of the detection area whose candidate type is a table may be set larger than the range to be applied to the expansion of the detection area whose candidate type is a text. Also, the range to be applied to the expansion of the detection area whose candidate type is a figure may be set larger than the range to be applied to the expansion of the detection area whose candidate type is a table. In addition, when a plurality of candidate types is selected, the expansion unit103may determine the range (expanse, magnification rate, width, height) to be applied to the expansion of the detection area on the basis of the candidate type with the largest range of the expansion from among the plurality of candidate types selected.

FIG.11is a flowchart showing a process of the image analysis device1. The acquisition unit101acquires an input image read by a device such as a scanner or a camera, which is not shown in the figure (SA1). Next, the detection unit102analyses the input image obtained by the acquisition unit101(SA2) and detects an area and a candidate type corresponding to a component image constituting the input image (SB3). If no area and candidate type corresponding to a component image constituting the input image is detected (SA3: NO), the detection unit102terminates the image analysis process. If an area and a candidate type(corresponding to a component image constituting the input image are detected (SB3: YES), the type selection unit107selects, if there are more than one, an appropriate candidate type as the type of the detection area corresponding to the component image from among the candidate types detected by the detection unit102(SB41). The expansion unit103expands the area detected by the detection unit102in the range (expanse, magnification rate, width, height) in accordance with the candidate type selected by the type selection unit107(SB42). The evaluation unit104evaluates the image included in the area expanded by the expansion unit103on the basis of the rule according to the type (SA5). If an exit condition of the expansion is satisfied (SA6: YES), the image included in the expansion area with the highest evaluation value given by the evaluation unit104is outputted by the output unit106as an area corresponding to the component image (SA7). If no exit condition of the expansion is satisfied (SA6: NO), the expansion unit103expands the area that was expanded in SA4 again (SA4).

As described above, the expansion unit103can control the degradation of output accuracy in the area corresponding to the component image by changing the range to be applied to the expansion of the detection area for each candidate type of the detection area. Specifically, the expanded range is prevented from being too large or too small with respect to the component image.

In the present invention, the configuration elements of the embodiments may be arbitrarily modified or omitted within the scope of the invention.

DESCRIPTION OF SYMBOLS