METHOD FOR COMPARING DOCUMENTS AND SYSTEM THEREFOR

A method for comparing documents and a system therefor are provided. The method according to some embodiments may include acquiring a first document image and a second document image, extracting a first feature set from the first document image and a second feature set from the second document image through an encoder, generating a correlation feature set by analyzing a correlation between at least part of the first feature set and at least part of the second feature set, and outputting a result of comparison between the first document image and the second document image based on a result of decoding the correlation feature set.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority from Korean Patent Application No. 10-2023-0030368, filed on Mar. 8, 2023, and Korean Patent Application No. 10-2023-0147322, filed on Oct. 31, 2023, in the Korean Intellectual Property Office, and all the benefits accruing therefrom under 35 U.S.C. 119, the contents of which in its entirety are herein incorporated by reference(s).

BACKGROUND

The present disclosure relates to a method for comparing documents and a system therefor, and more particularly, to a method for comparing two documents and detecting differences through deep learning technology and a system therefor.

2. Description of the Related Art

Document comparison technology may be widely utilized in various fields. For example, the document comparison technology may be used to detect major changes in contracts. As another example, the document comparison technology may be employed to check for updates between different versions of documents.

A conventional document comparison technique primarily compares the contents of documents based on text. For example, when scan images of documents are provided, the conventional document comparison technique extracts texts from the scan images using optical character recognition (OCR) and compares the extracted texts.

However, the conventional document comparison technique has various limitations. The conventional document comparison technique includes problems related to OCR accuracy due to the quality of scan images (e.g., frequent misrecognition of characters in low-quality scan images with noise or skew), and difficulties in applying OCR to multi-lingual document images. For example, to apply OCR to multi-lingual document images, suitable OCR models for each language included in the document images are required. However, building OCR models with high accuracy for each language demands considerable time, financial resources, and human resources (e.g., costs for labeling, etc.).

SUMMARY

Aspects of the present disclosure provide a method and system capable of accurately detecting differences (e.g., changes) by comparing the contents of given documents.

Aspects of the present disclosure also provide a method and system that may accurately compare the contents of documents, regardless of the languages included in the documents (or language-independently).

Aspects of the present disclosure also provide a document comparison method that is robust against differences in font, quality, etc. in document images.

Aspects of the present disclosure also provide the structure and learning (training) method of a deep learning model that may accurately compare the contents of documents.

According to an aspect of the present disclosure, there is provided a method for comparing documents performed by at least one processor. The method may include acquiring a first document image and a second document image, extracting a first feature set from the first document image and a second feature set from the second document image through an encoder, generating a correlation feature set by analyzing a correlation between at least part of the first feature set and at least part of the second feature set, and outputting a result of comparison between the first document image and the second document image based on a result of decoding the correlation feature set.

In some embodiments, the first feature set may include a first feature and a second feature of a different scale from the first feature, and the second feature set may include a third feature of a same scale as the first feature and a fourth feature of a same scale as the second feature, and the generating the correlation feature set may include generating a first correlation feature, which belongs to the correlation feature set, by analyzing a correlation between the first feature and the third feature, and generating a second correlation feature, which belongs to the correlation feature set, by analyzing a correlation between the second feature and the fourth feature.

In some embodiments, the generating the correlation feature set may include generating a first attention feature and a second attention feature by performing an attention operation on a first feature, which belong to the first feature set, and a second feature, which belong to the second feature set and generating one or more correlation features that belong to the correlation feature set by performing a correlation operation on the first attention feature and the second attention feature.

In some embodiments, the generating the first attention feature and the second attention feature may include performing a first attention operation on the first feature and the second feature, and generating the first attention feature by performing a second attention operation on the first feature and a result of the first attention operation.

In some embodiments, the first feature corresponds to a query, and the second feature corresponds to a key for the first attention operation, and the generating the first attention feature and the second attention feature may further include performing a third attention operation on the first feature and the second feature, wherein the second feature corresponds to a query for the third attention operation and the first feature corresponds to a key for the third attention operation, and generating the second attention feature by performing a fourth attention operation on the second feature and a result of the third attention operation.

In some embodiments, the first attention feature and the second attention feature are feature maps including a plurality of pixels, respectively, and the generating the one or more correlation features may include determining a first pixel region in the second attention feature that corresponds to a first pixel in the first attention feature, wherein the first pixel region includes a second pixel in the second attention feature that exists at a location corresponding to the first pixel and a neighboring pixel of the second pixel, and generating a first correlation feature by performing a correlation operation on the first pixel and pixels included in the first pixel region.

In some embodiments, the first correlation feature is a feature map including a plurality of pixels, and the generating the first correlation feature may include calculating vector similarities between a channel vector for the first pixel and channel vectors for the pixels included in the first pixel region, and the calculated vector similarities form a channel vector for a third pixel in the first correlation feature that corresponds to the first pixel.

In some embodiments, the generating the one or more correlation features may further include determining a second pixel region in the first attention feature that corresponds to a third pixel in the second attention feature, wherein the second pixel region includes a fourth pixel in the first attention feature that exists at a location corresponding to the third pixel and a neighboring pixel of the fourth pixel, and generating a second correlation feature by performing a correlation operation on the third pixel and pixels included in the second pixel region.

In some embodiments, the first feature set may include multi-scale features, a first feature of a largest scale among the multi-scale features is excluded from the analyzing the correlation, and the outputting the result of comparison may include outputting the result of comparison based on a result of decoding of the correlation feature set and the first feature.

In some embodiments, the outputting the result of comparison may further include performing a first attention operation on the first feature and a first correlation feature that belongs to the correlation feature set, performing a second attention operation on the first correlation feature and a result of the first attention operation, and performing decoding based on a result of the second attention operation.

In some embodiments, the outputting the result of comparison may include generating a segmentation map for the first document image by decoding at least part of the correlation feature set through a first decoder, wherein the segmentation map indicates information on areas in the first document image that are identical to and different from the second document image.

In some embodiments, the method may further include calculating a loss between the generated segmentation map and a ground truth segmentation map, and updating parameters of the encoder and the first decoder based on the calculated loss, wherein the loss is calculated using a dice loss function and a cross-entropy loss function.

In some embodiments, the method may further include acquiring a result of prediction of whether the first document image is identical to the second document image by inputting the generated segmentation map to a classifier, and updating parameters of the encoder and the first decoder based on a loss between the result of prediction and a ground truth.

In some embodiments, the correlation feature set may include a first correlation feature, which is generated based on a feature from the first feature set, and a second correlation feature, which is generated based on a feature from the second feature set, and the outputting the result of comparison may include generating a first segmentation map for the first document image, which indicates information on areas in the first document image that are identical to and different from the second document image, by decoding the first correlation feature through a first decoder, and generating a second segmentation map for the second document image, which indicates information on areas in the second document image that are identical to and different from the first document image, by decoding the second correlation feature through a second decoder.

In some embodiments, the method may further include calculating a first loss between the first segmentation map and a first ground truth segmentation map for the first document image, calculating a second loss between the second segmentation map and a second ground truth segmentation map for the second document image, and updating parameters at least one of the encoder, the first decoder, and the second decoder based on the first loss and the second loss.

According to another aspect of the present disclosure, there is provided a system for comparing documents. The system may include at least one processor, and a memory configured to store a computer program that is executed by the at least one processor, wherein the computer program includes instructions to perform: acquiring a first document image and a second document image, extracting a first feature set from the first document image and a second feature set from the second document image through an encoder, generating a correlation feature set by analyzing a correlation between at least part of the first feature set and at least part of the second feature set, and outputting a result of comparison between the first document image and the second document image based on a result of decoding the correlation feature set.

According to yet another aspect of the present disclosure, there is provided a non-transitory computer-readable recording medium storing a computer program, which, when executed by at least one processor, causes the at least one processor to perform acquiring a first document image and a second document image, extracting a first feature set from the first document image and a second feature set from the second document image through an encoder, generating a correlation feature set by analyzing a correlation between at least part of the first feature set and at least part of the second feature set, and outputting a result of comparison between the first document image and the second document image based on a result of decoding the correlation feature set.

DETAILED DESCRIPTION

In adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are assigned to the same components as much as possible even though they are shown in different drawings. In addition, in describing the present disclosure, when it is determined that the detailed description of the related well-known configuration or function may obscure the gist of the present disclosure, the detailed description thereof will be omitted.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that may be commonly understood by those skilled in the art. In addition, the terms defined in the commonly used dictionaries are not ideally or excessively interpreted unless they are specifically defined clearly. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase.

In addition, in describing the component of this disclosure, terms, such as first, second, A, B, (a), (b), may be used. These terms are only for distinguishing the components from other components, and the nature or order of the components is not limited by the terms. If a component is described as being “connected,” “coupled” or “contacted” to another component, that component may be directly connected to or contacted with that other component, but it should be understood that another component also may be “connected,” “coupled” or “contacted” between each component.

Various embodiments of the present disclosure will hereinafter be described in detail with reference to the accompanying drawings.

FIG.1is a schematic drawing for explaining the operation of a document comparison system10according to some embodiments of the present disclosure.

Referring toFIG.1, the document comparison system10is a computing device/system that compares first and second documents and output the results of the comparison (e.g., whether the first and second documents are identical, parts where the first and second documents are different, the rate of similarity (or match rate) between the first and second documents, etc.). For example, the document comparison system10may compare the first and second documents at an image level, i.e., compare first and second document images12and13, through a deep learning model11. In this manner, the time and computing cost required for training in various languages may be reduced because image-level document comparison, unlike optical character recognition (OCR)-based document comparison, does not require language-specific training, and document comparison may be performed regardless of the languages included in given documents (or language-independently). Accordingly, multilingual documents may also be easily compared. For convenience, the document comparison system10will hereinafter be referred to simply as the comparison system10.

The first document (or the second document) may also be referred to as a source document or a comparison reference document, and the second document (or the first document) may also be referred to as a target document, a comparison target document, etc.

The deep learning model11is a model used for document comparison and is a model that performs document comparison at the image level. That is, the deep learning model11may be configured and trained to receive the first and second document images12and13as inputs and output comparison results for the two document images.

For example, as illustrated inFIG.2, the deep learning model11may receive a first document image21and a second document image22as inputs and output a segmentation map23for the first document image21and/or the second document image22. The segmentation map23represents a map that indicates information on areas where the first and second document images21and22are identical or match and areas where they are different (or do not match) (e.g., a map where the matching areas and the non-matching areas are defined as separate classes, and class prediction values are assigned to each pixel). A semantic segmentation task and a segmentation map that results from the semantic segmentation task are already well known in the art to which the present disclosure pertains, and thus, detailed descriptions thereof will be omitted. The segmentation map23may also be referred to as a segmentation mask, a segmentation label, etc.

FIG.2illustrates an example where the deep learning model11outputs a single segmentation map23for either the first document image21or the second document image22, but the present disclosure is not limited thereto.

The structure, training method, etc., of the deep learning model11will be described later with reference toFIG.3and the subsequent drawings.

The comparison system10may be implemented with at least one computing device. For example, all functions of the comparison system10may be implemented on a single computing device. Alternatively, first and second functionalities of the comparison system10may be implemented on first and second computing devices, respectively. Yet alternatively, a particular functionality of the comparison system10may be implemented on multiple computing devices.

Here, the term “computing device” may encompass nearly any device equipped with computing capabilities, and an example computing device will be described later with reference toFIG.21. As a computing device is an assembly where various components (e.g., memories, processors, etc.) interact, it may also be referred to as a computing system, which obviously, may also include the concept of an assembly where multiple computing devices interact.

The operation of the comparison system10has been briefly described so far with reference toFIGS.1and2. The structure and operation of an example deep learning model11will hereinafter be described with reference toFIG.3and the subsequent drawings.

FIG.3is a schematic drawing illustrating the structure and operation of the deep learning model11. Referring toFIG.3and the subsequent drawings, each feature symbol, for example, “F1_1”, consists of a combination of an alphabet letter (e.g., “F”, “A”, and “C”) indicating the type of feature, a number indicating which document image the corresponding feature is associated with (e.g., “1” for association with a first document image, “2” for association with a second document image, and “12” for association with both the document images, but indicating that document comparison has been performed based more on the first document image than on the second document image), an underscore “_”, and a number indicating the scale of the corresponding feature (e.g., “1” for a largest scale and “3” for a smallest scale).

As illustrated inFIG.3, the deep learning model11may include an encoder31, a correlation analyzer32, and one or more decoders, i.e., first and second decoders33and34.FIG.3illustrates an example where the deep learning model11has two independent decoders to two output comparison results (38-1and38-2) respectively corresponding to two document images (35-1and35-2), but the present disclosure is not limited thereto. Alternatively, the deep learning model11may be configured to have only one decoder. For convenience, however, the deep learning model11will hereinafter be described as including two decoders, i.e., the first and second decoders33and34, as illustrated inFIG.3.

The encoder31is a common module that extracts features (or feature sets) from input document images, for example, the first document image35-1. For example, the encoder31may be used to extract a feature36-1from the first document image35-1and may also be used to extract a feature36-2from the second document image35-2.

The encoder31may be implemented based on, for example, a convolutional neural network (CNN). For example, the encoder31may be configured to include multiple convolutional layers to extract multi-scale features from the input document images (seeFIG.4).

In some embodiments, multi-scale features may be extracted by the encoder31. For example, as illustrated inFIG.4, a largest scale feature (or a feature at the lowest level of abstraction), for example, a feature43-3may be extracted in a first layer (e.g., a convolutional layer) of the encoder31, an intermediate scale feature (or a feature at an intermediate level of abstraction), for example, a multi-scale feature41-1, may be extracted in a second layer of the encoder31at the rear of the first layer, and a smallest scale feature (or a feature at a highest level of abstraction), for example, a multi-scale feature36-1, may be extracted in a third layer (or the last or rearmost layer) of the encoder31. However, the present disclosure is not limited to this example, and the number of features extracted may vary. The deep learning model11may extract features36-1,41-1, and42-1for the first document image35-1and features36-2,41-2, and42-2for the second document image35-2. In this case, the features36-1,41-1, and42-1may have the same scale as the features36-2,41-2, and42-2, respectively. By performing document comparison using multi-scale features, the accuracy of comparison between document images that include characters of various sizes may be enhanced.

In some embodiments, as illustrated inFIG.4, a skip-connection may be formed so that a largest scale feature containing a largest amount of information, among the extracted multi-scale features, is input to each decoder. For example, the feature42-1, among the features36-1,41-1, and42-1, may be input to the first decoder33, and the feature42-2, among the features36-2,41-2, and42-2, may be input to the second decoder34. In this case, the first decoder33may easily generate a sophisticated segmentation map38-1by performing decoding using the richer information contained in the feature42-1. This decoding process will be described later in further detail with reference toFIGS.10and11.

The multi-scale features (or feature sets) will hereinafter be described as being extracted by the encoder31of the deep learning model11.

Referring back toFIG.3or4, the correlation analyzer32is a module that analyzes the correlation between input features, i.e., the features36-1and36-2. In other words, the correlation analyzer32may generate one or more correlation features (or sets of correlation features), i.e., correlation features37-1and43-1, by analyzing feature-level correlations. Here, the correlation features may represent features containing correlation information, and correlation analysis may be understood as a process of comparing document images at a feature level. Comparing document images at the feature level may minimize the influence of font and quality differences on the accuracy of document comparison, as font differences, quality differences (e.g., differences in document tilt), etc., are less pronounced at the feature level.

Correlation may also be referred to as association, relatedness, similarity, etc.

If multi-scale features are extracted by the encoder31, the correlation analyzer32may generate correlation features for each scale. For example, the correlation analyzer32may analyze the correlation between features of a first scale, i.e., the features36-1and36-2, to generate a first-scale correlation feature, i.e., the correlation feature37-1, and analyze the correlation between features of a second scale, i.e., the features41-1and41-2, to generate a second-scale correlation feature, i.e., the correlation feature43-1.

In some embodiments, as illustrated inFIG.5, the correlation analyzer32may be configured to include a first attention operator51and a correlation operator52, and this will hereinafter be described with reference toFIGS.5through9.

FIG.5is a schematic drawing for explaining the structure and operation of the correlation analyzer32.FIG.5illustrates an example where the correlation feature37-1is generated from the smallest scale features36-1and36-2, and correlation features may also be generated for other scales (e.g., the scale of the features41-1and41-2) in a manner that will be described later. For convenience, the feature36-1and an attention feature53associated with the first document image35-1will hereinafter be referred to as the first feature36-1and the first attention feature53, respectively, and the feature36-2and an attention feature54associated with the second document image35-2will hereinafter be referred to as the second feature36-2and the second attention feature54, respectively.

As illustrated inFIG.5, the correlation analyzer32may be configured to include the first attention operator51and the correlation operator52.

The first attention operator51is a module that performs an attention operation (i.e., cross-attention operation) on the first and second features36-1and36-2. Here, the attention operation may be an operation based on a query, a key, and a value, as exemplified by Equation 1 below. In Equation 1, Q, K, and V represent a query, a key, and a value, respectively, and de represents the dimensionality of a key vector. The attention operation according to Equation 1 and how to derive the query Q, the key K, and the value V for the attention operation (e.g., by applying corresponding weight matrices to input data) are already well known in the art to which the present disclosure pertains, and thus, detailed descriptions thereof will be omitted.

The first attention operator51may perform an attention operation on the first and second features36-1and36-2, thereby generating the first and second attention features53and54. Specifically, the first attention operator51may correspond the first feature36-1with a query (i.e., generate query vectors from the first feature36-1) and the second feature36-2with a key and a value, and then perform an attention operation, thereby generating the first attention feature53. Here, the term “attention feature” refers to a feature that reflects the result of an attention operation. Additionally, the first attention operator51may correspond the second feature36-2with the query Q and the first feature36-1with the key K and the value V, and then perform the same attention operation again, but in the opposite direction, thereby generating the second attention feature54. In this case, the first attention feature53may be considered more associated with the first document image35-1because it is based on the first feature36-1. Similarly, the second attention feature54may be considered more associated with the second document image35-2.

In some embodiments, as illustrated inFIG.6, the first attention operator51may be configured to include a first attention module61and a second attention module62. That is, the first attention operator51may be configured to consecutively perform two attention operations. However, the present disclosure is not limited to this. Alternatively, the first attention operator51may be configured to consecutively perform three or more consecutive attention operations. Specifically, the first attention module61of the first attention operator51may perform the first attention operation (or 1-th/primary attention operation) on the first feature36-1, thereby obtaining the first attention operation result63. In this first attention operation for obtaining the first attention operation result63, the first feature36-1corresponds to a query Q, and the second feature36-2corresponds to a key K and a value V. Then, the second attention module62of the first attention operator51may perform the second attention operation (or 2-th/secondary attention operation) on the first feature36-1and the first attention operation result63, as indicated by reference numeral62, thereby generating the first attention feature53. Additionally, the first attention operator51may consecutively perform attention operations in the opposite direction, thereby generating the second attention feature54. Specifically, the first attention module61of the first attention operator51may perform the first attention operation on the second feature36-2and the first feature36-1, thereby obtaining the first attention operation result64. In this first attention operation for obtaining the first attention operation result64, the second feature36-2corresponds to the query Q, and the first feature36-1corresponds to the key K and the value V for the first attention operation. Then, the second attention module62of the first attention operator51may perform the second attention operation on the second feature36-2and the first attention operation result64, thereby generating the second attention feature54.

Referring back toFIG.5, the correlation operator52is a module that performs a correlation operation on input attention features (53and54). As illustrated, the input attention features (53and54) may include the first attention feature53, associated with the first document image35-1, and the second attention feature54, associated with the second document image35-2.

A correlation operation may be performed, for example, by calculating the similarities (i.e., vector similarities) between the channel vectors that form the first attention feature53and the channel vectors that form the second attention feature54. An example correlation operation (i.e., the operation of the correlation operator52) will hereinafter be described with reference toFIGS.7and8.

FIGS.7and8are schematic drawings for explaining the operation of the correlation operator52.

Referring toFIG.7, it is assumed that the first and second attention features53and54are configured as three-dimensional (3D) feature maps. In other words, each pixel (e.g., a pixel71) in a two-dimensional (2D) feature map (i.e., an H-W feature map) represents a channel vector.

In this case, the correlation operator52may calculate a vector similarity (e.g., cosine similarity) between the pixel71of the first attention feature53and a pixel region72of the second attention feature54. The pixel region72may include a pixel73of the second attention feature54that corresponds to the pixel71and neighboring pixels74around the pixel73.

A padding technique may be applied to the second attention feature54, so that all the pixels in the first attention feature53correspond to respective pixel regions of the second attention feature54, as indicated by dashed lines. Nearly any type of padding technique may be used.

The correlation operator52may generate channel vectors (e.g., a channel vector76) that form the correlation feature37-1based on the vector similarities between an individual pixel (e.g., the pixel71) of the first attention feature53and the corresponding pixel region (e.g., the pixel region73) of the second attention feature54. For example, the value of the channel vector76may be the similarity between the channel vector for the pixel71and the channel vector for a pixel74. Furthermore, the correlation operator52may repeat this process for other pixels of the first attention feature53, thereby generating the correlation feature37-1.

For a better understanding, a detailed explanation of the operation of the correlation operator52will be provided with reference toFIG.8.FIG.8illustrates an example where the padding technique is applied to a second feature map83. Referring toFIG.8, a first feature map81and the second feature map83may be understood as corresponding to the first attention feature53and the second attention feature54, respectively.

Referring toFIG.8, it is assumed that a correlation operation is performed between a center pixel82(see “A”) of the first feature map81and a pixel region (or pixel area)84of the second feature map83. As previously mentioned, the pixel region84may include a pixel85-1of the second feature map83that corresponds to the center pixel82and neighboring pixels (e.g., pixels85-2and85-3) around the pixel85-1. Here, the size of the pixel region84(e.g., the number of neighboring pixels) may be a hyperparameter set in advance, but the present disclosure is not limited thereto.

In this case, the correlation operator52may determine the value of a pixel87-1of a feature map86-1of a first channel of a correlation feature based on the vector similarity between the center pixel82and a first neighboring pixel85-2of the pixel region84. For example, pixel “A1” may be a pixel to which the result of a correlation operation performed on pixel “A” and pixel “1” (see the first neighboring pixel85-2) is assigned. As mentioned earlier, the vector similarity between the center pixel82and the first neighboring pixel85-2implies the similarity between the channel vectors for the center pixel82and the first neighboring pixel85-2.

Moreover, the correlation operator52may determine the value of a pixel87-2of a feature map86-2of a second channel of the correlation feature based on the vector similarity between the center pixel82and a second neighboring pixel85-3of the pixel region84. By repeating this process for other pixels of the pixel region84, channel vectors containing correlation information between the center pixel82and the pixel region84(i.e., channel vectors with the values of the pixels87-1and87-2as their elements) may be generated.

Meanwhile, in some embodiments, the correlation operator52may perform correlation operations bidirectionally. For example, as illustrated inFIG.9, the correlation operator52may perform a correlation operation between the first and second attention features53and54based on the first attention feature53, thereby generating the correlation feature37-1, which may also be referred to as the first correlation feature37-1. Here, the correlation operation performed based on the first attention feature53means a correlation operation performed between an individual pixel of the first attention feature53and the corresponding pixel region of the second attention feature54. Then, the correlation operator52may perform a correlation operation between the first and second attention features53and54based on the second attention feature54, thereby generating the correlation feature37-2, which may also be referred to as the second correlation feature37-2. That is, the correlation operator52may generate the second correlation feature37-2through a correlation operation between an individual pixel of the second attention feature54and the corresponding pixel region of the first attention feature53. The second correlation feature37-2may be considered as a feature more associated with the second document image35-2, which is input to the second decoder34.

Referring back toFIG.3or4, the first decoder33is a module that decodes, for example, the first correlation feature37-1, to output or predict a segmentation map38-1(hereinafter, the first segmentation map38-1) for the first document image35-1. Here, the first segmentation map38-1may be understood as being a map representing information on areas in the first document image35-1that are identical to or different from the second document image35-2.

For example, as illustrated inFIG.4, the first decoder33may generate the first segmentation map38-1by decoding the correlation features (37-1and43-1) generated based on the features (36-1and41-1) from the first document image35-1.

The first decoder33may be implemented or configured based on, for example, deconvolution layers, upsampling layers, fully-connected layers, etc., but the present disclosure is not limited thereto. The first decoder33may be implemented in any manner as long as it may properly produce and output the first segmentation map38-1.

In some embodiments, as illustrated inFIG.10, the first decoder33may be configured to include a second attention operator101and a segmentation predictor102, and this will hereinafter be described with reference toFIGS.10and11.

FIG.10is a schematic drawing for explaining the structure and operation of the first decoder33.FIG.10illustrates an example where the largest scale feature42-1among the multi-scale features from the first document image35-1is input to the first decoder33.

Referring toFIG.10, the first decoder33may be configured to include the second attention operator101and the segmentation predictor102.

The second attention operator101is a module that performs an attention operation (i.e., a cross-attention operation) between input correlation features. The second attention operator101may generate an attention feature103through an attention operation between the correlation feature37-1and the feature42-1, and may also generate an attention feature104through an attention operation between the correlation feature43-1and the feature of another scale. The second attention operator101may perform attention operations bidirectionally (for more information, refer to the description of the first attention operator101).

In some embodiments, referring toFIG.11, the second attention operator101may be configured to consecutively perform two attention operations. That is, the second attention operator101may be configured to include a third attention module111and a fourth attention module112. Specifically, the third attention module111of the second attention operator101may perform the first attention operation (or 1-th/primary attention operation) on the correlation feature37-1and the feature42-1, thereby obtaining the first attention operation result113. Here, the correlation feature37-1corresponds to the query Q, and the feature42-1corresponds to the key K and the value V. Then, the fourth attention module112of the second attention operator101may perform the second attention operation (or 2-th/secondary attention operation) on the correlation feature37-1and the first attention operation result113, thereby generating the attention feature103.

For more information on the second attention operator102, refer back to the descriptions of the first attention operator51inFIGS.5through7.

Referring back toFIG.10, the segmentation predictor102is a module that predicts and outputs the first segmentation map38-1based on the attention feature103(or set of attention features). For example, the segmentation predictor102may aggregate the attention features (103,104), analyze them, and predict the class (i.e., class label) for each pixel of the first document image35-1, resulting in the creation of the first segmentation map38-1.

The segmentation predictor102may be implemented or configured based on, for instance, deconvolution layers, upsampling layers, fully-connected layers, etc., but the present disclosure is not limited thereto. The segmentation predictor102may be implemented in any manner as long as it may properly produce and output the first segmentation map38-1.

Referring back toFIG.3or4, the second decoder34is a module that decodes, for example, the second correlation feature37-2to output or predict a segmentation map38-2, which may also be referred to as the second segmentation map38-2, for the second document image35-2. Here, the second segmentation map38-2may be understood as being a map indicating information on areas in the second document image35-2that are identical to or different from the first document image35-1.

For example, as illustrated inFIG.4, the second decoder34may decode the correlation features (37-2and41-1) generated based on the features (36-2and41-2) from the second document image35-2to create the second segmentation map38-2. For more information on the second decoder34, refer to the description of the first decoder33.

FIG.12illustrates an actual implementation of the deep learning model11according to some embodiments of the present disclosure.

Referring toFIG.12, the deep learning model11may be configured to include a common encoder31and first and second decoders33and34, which correspond to their respective document images. The second decoder34may be configured to have the same structure as the first decoder33. That is, the second decoder34, like the first decoder33, may be configured to include an attention operator121(e.g., a third attention operator) and a segmentation predictor122.

A positional encoding module as shown inFIG.12refers to a module that adds position information to input features so that the first attention operator51may distinguish between the input features. The concept and operating principles of the positional encoding module are already well known in the art to which the present disclosure pertains, and thus, detailed descriptions thereof will be omitted.

“Correlation” and “Marginalization” ofFIG.12refer to the correlation operator52.

FIG.13illustrates another actual implementation of the first attention operator51or the second attention operator101. For example, as illustrated inFIG.13, an attention module131on the left and an attention module132on the right may be understood as corresponding to the first and second attention modules61and62, respectively.FIG.13illustrates an example where the result of an attention operation is configured to be reflected back into each input feature (e.g., input features are reflected again to “Conv_Block3” and a subsequent addition operation).

The structure and internal operations of the deep learning model11have been described so far with reference toFIGS.3through13. Various methods that may be performed in the comparison system10will hereinafter be described.

For a better understanding, it is assumed that all steps/actions of methods that will hereinafter be described are performed by the comparison system10. Thus, even if the subject of a particular step/action is not specifically mentioned, it may be understood as being performed by the comparison system10. However, in reality, some steps of the methods that will hereinafter be described may be performed on other computing devices. For example, the training of the deep learning model11may be performed on a different computing device.

FIG.14is an example flowchart illustrating a document comparison method according to some embodiments of the present disclosure. The embodiment ofFIG.14is merely example for achieving the purposes of the present disclosure, and some steps may be added to or omitted from the embodiment ofFIG.14, as necessary.FIG.14depicts steps commonly performed in both the training and inference procedures of the deep learning model11.

Referring toFIG.14, the document comparison method may begin with step S141, which involves acquiring images of first and second documents. Here, the first and second documents refer to two documents on which document comparison is to be performed.

A method to acquire the two document images in step S141may vary.

For example, during the inference procedure of the deep learning model11, the comparison system10may convert first and second documents, which are in the format of text, into image format to acquire first and second document images. Alternatively, the comparison system10may initially receive the first and second document images in the image format.

As another example, in the training procedure of the deep learning model11, the comparison system10may generate various positive pairs (i.e., pairs of identical first and second document images) and/or negative pairs (i.e., pairs of different first and second document images) through a data augmentation technique. Specifically, the comparison system10may create a negative pair by making some changes to the content of the first document in the text format to create the second document and then converting the first and second documents into the image format. Alternatively, the comparison system10may create a positive pair by slightly changing the font of the first document in the text format to create the second document and then converting the first and second documents into the image format. Alternatively, the comparison system10may change the quality of the first document image (e.g., changing the document tilt or adding noise) to create the second document image, in which case, the first and second document images form a positive pair.

In some embodiments, the comparison system10may also automatically generate ground truth labels (e.g., a ground truth segmentation maps) based on the changed parts of the first document when generating negative pairs. Obviously, the comparison system10may also automatically generate ground truth labels (or correct answer labels) for positive pairs.

Additionally, the document images used in the training procedure may also be referred to as document image samples, etc.

In step S142, through the encoder31of the deep learning model11, a first feature set may be extracted from the first document image, and a second feature set may be extracted from the second document image. Here, each of the first and second feature sets may include at least one feature.

For example, the comparison system10may extract the first and second feature sets, each consisting of multi-scale features, from the first and second document images, respectively, through the encoder31(for more information, refer to the descriptions inFIG.4).

In step S143, a correlation feature set may be generated by analyzing the correlation between at least parts of the first and second feature sets. Here, the correlation feature set may include at least one correlation feature.

For example, as illustrated inFIG.15, the comparison system10may perform attention operations bidirectionally on a first feature of the first feature set and a second feature of the second feature set that has the same scale as the first feature, thereby generating first and second attention features (S151) (for more information, refer to the descriptions inFIG.6). Thereafter, the comparison system10may perform correlation operations on the first and second attention features, thereby generating one or more correlation features (S152). For example, the comparison system10may perform correlation operations bidirectionally on the first and second attention features, thereby generating first and second correlation features (for more information, refer to the descriptions inFIGS.7through9).

Steps S151and S152may be repeatedly performed for features of other scales. However, in some embodiments, largest scale features among sets of multi-scale features may be excluded from correlation analysis. Then, the excluded features may be used in a decoding process to generate more sophisticated segmentation maps. Specifically, as depicted inFIG.4, the excluded features may be input to the first and second decoders33and34of the deep learning model11through skip-connections.

In some embodiments, the comparison system10may extract attention features through multiple consecutive attention operations (i.e., cross-attention operations). For example, as illustrated inFIG.16, the comparison system10may perform the first attention operation (or 1-th/primary attention operation) on the first and second features (S161) and then generates the first attention feature by performing the second attention operation (or 2-th/secondary attention operation) on the first feature and the result of the first attention operation (S162). The comparison system10may generate the second attention feature in the same manner (for more information on steps S161and S162, refer to the descriptions inFIG.6).

Referring back toFIG.14, in step S144, comparison results for the first and second documents may be output based on decoding results for the correlation feature set. Here, the comparison results may be, for example, segmentation maps output by the first decoder33of the deep learning model11or analysis results for the segmentation maps.

For example, the comparison system10may input the first correlation feature from the correlation feature set, which is associated with the first document image, into the first decoder33to generate the first segmentation map38-1. Additionally, the comparison system10may input the second correlation feature from the correlation feature set, which is associated with the second document image, into the second decoder34to generate the second segmentation map38-2(for more information, refer to the descriptions inFIGS.3and4).

FIG.17is a detailed flowchart illustrating step S144, particularly, the step of generating the first segmentation map.

Referring toFIG.17, the comparison system10may generate the first segmentation map based on the results of the decoding of the correlation feature set and the largest scale feature.

Specifically, the comparison system10may perform the first attention operation (or 1-th/primary attention operation) on the first correlation feature from the correlation feature set and the largest scale feature (S171) and then the second attention operation (or 2-th/primary attention operation) on the first correlation feature and the result of the first attention operation (S172). Then, the comparison system10may generate the first segmentation map for the first document image by decoding the result of the second attention operation (for more information, refer to the descriptions inFIGS.10and11).

During the training procedure of the deep learning model11, the comparison system10may further perform the steps of calculating losses using the comparison results for the first and second documents, and updating the parameters of the deep learning model11based on the calculated losses.

For example, as illustrated inFIG.18, the comparison system10may calculate a loss187between a first segmentation map183and a first ground truth segmentation map185(i.e., the ground truth label for a first document image181). Similarly, the comparison system10may calculate a loss188between the second segmentation map184and a second ground truth segmentation map186(i.e., the ground truth label for a second document image182). Then, the comparison system10may update the parameters of the deep learning model11(e.g., the encoder31, the first and second decoders33and34, the first attention operator51, etc.) based on the losses187and188. For example, the loss187may be calculated based on a dice loss function (e.g., “1-Dice Score”) and a cross-entropy loss function, but the present disclosure is not limited thereto. The dice loss function and the cross-entropy loss function are already well known to the field to which the present disclosure pertains, and thus, detailed descriptions thereof will be omitted.

As another example, as illustrated inFIGS.18and19, the comparison system10may update the parameters of the deep learning model11based on a loss (or cross-entropy loss)194, which is calculated through a classifier191. Specifically, the comparison system10may input the first and second segmentation maps183and184into the classifier191to obtain a prediction result192on whether the first and second document images181and182are identical. Then, the comparison system10may update the parameters of the classifier191and the deep learning model11based on the loss194between the prediction result192and a ground truth193. The classifier191may be implemented in any manner. In some embodiments, the classifier191may be configured to further receive a max segmentation map in addition to the first and second segmentation maps183and184. Here, the max segmentation map refer to a segmentation map created through a max operation (i.e., a maximum value operation) between the first and second segmentation maps183and184.

As another example, the parameters of the deep learning model11may be updated based on various combinations of the aforementioned embodiments. For example, the comparison system10may calculate a total loss based on a weighted sum of the losses187,188, and194and update the parameters of the deep learning model11based on the calculated total loss.

The aforementioned step of updating the parameters of the deep learning model11may be performed repeatedly for various document images. As a result, the deep learning model11may be equipped with accurate document comparison capabilities.

The document comparison method according to some embodiments of the present disclosure has been described so far with reference toFIGS.14through19. As described, the deep learning model11trained to compare documents at the image level may be used to compare the first and second documents. Accordingly, time and computing costs required for training in various languages may be reduced because image-level document comparison, unlike OCR-based document comparison, does not require language-specific training, and document comparison may be performed independently of the languages included in the documents (e.g., multilingual documents may be easily compared).

Moreover, the deep learning model11may be configured and trained to output comparison results (e.g., segmentation maps) for input document images using the results of feature-level correlation analysis. Thus, the influence of differences in fonts, quality (e.g., differences in document tilt), etc., on the accuracy of document comparison may be minimized (e.g., cases where minor font differences lead to the documents being considered different may be minimized), thereby enhancing the performance of the deep learning model (refer to the experimental results inFIG.20). Furthermore, by performing a correlation operation between the channel vector of each individual pixel and the channel vector of the corresponding pixel region (i.e., the pixel region including neighboring pixels around the corresponding individual pixel), the performance of the deep learning model11may be further improved.

Experimental results for the document comparison method, referred to as the proposed method, according to some embodiments of the present disclosure will hereinafter be described.

The inventors of the present disclosure conducted experiments to evaluate the performance of the proposed method using the deep learning model with the structure illustrated inFIGS.12and13. Specifically, the inventors trained the deep learning model using the losses illustrated inFIGS.18and19, i.e., the losses186,187, and194, and conducted experiments comparing segmentation maps of document images output by the trained deep learning model with the ground truth segmentation maps. Additionally, for performance comparison, the inventors also conducted experiments to output segmentation maps of document images using UNet, which is a representative model for semantic segmentation tasks. The structure and operating principles of UNet are already well known in the field to which the present disclosure pertains, and thus, detailed descriptions thereof will be omitted.

The experimental results are as shown inFIG.20. InFIG.20, “source” and “target” refer to pairs of document images being compared, “Ground Truth” refers to ground truth segmentation maps, and “Ours” refers to the proposed method.

Referring toFIG.20, it may be observed that the segmentation maps according to the proposed method are almost identical to the ground truth segmentation maps, regardless of the language type. This demonstrates that accurate document comparison may be performed using the proposed method, even if the language included in each pair of document images changes.

Moreover, while the segmentation maps from the proposed method are almost identical to the ground truth segmentation maps, the segmentation maps produced by UNet display considerable differences from the ground truth segmentation maps. This suggests that performing correlation operations at the feature level may significantly enhance the accuracy of document comparison.

The experimental results for the document comparison method according to some embodiments of the present disclosure have been described so far with reference toFIG.20. An example computing device210that may implement the comparison system10will hereinafter be described with reference toFIG.21.

FIG.21is a hardware configuration view illustrating the computing device210.

Referring toFIG.21, the computing device210may include at least one processor211, a bus213, a communication interface214, a memory212, which loads a computer program216executed by the processor211, and a storage215that stores the computer program216. Even thoughFIG.21depicts only components related to the embodiments of the present disclosure, it is obvious to one of ordinary skill in the art to which the present disclosure pertains that the computing device210may further include other generic components, in addition to the components depicted inFIG.21. Moreover, in some embodiments, the computing device210may be configured with some of the components depicted inFIG.21omitted. The components of the computing device210will hereinafter be described.

The processor211may control the overall operation of each of the components of the computing device210. The processor211may be configured to include at least one of a central processing unit (CPU), a micro-processor unit (MPU), a micro-controller unit (MCU), a graphics processing unit (GPU), or any form of processor well-known in the field of the present disclosure. Additionally, the processor211may perform computations for at least one application or program to execute operations/methods according to some embodiments of the present disclosure. The computing device210may be equipped with one or more processors.

The memory212may store various data, commands, and/or information. The memory212may load the computer program216from the storage215to execute the operations/methods according to some embodiments of the present disclosure. The memory212may be implemented as a volatile memory such as a random-access memory (RAM), but the present disclosure is not limited thereto.

The bus213may provide communication functionality between the components of the computing device210. The bus213may be implemented in various forms such as an address bus, a data bus, and a control bus.

The communication interface214may support wired or wireless Internet communication of the computing device210. Additionally, the communication interface214may also support various other communication methods. To this end, the communication interface214may be configured to include a communication module well-known in the technical field of the present disclosure.

The storage215may non-transitorily store at least one computer program216. The storage215may be configured to include a non-volatile memory such as a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, as well as a computer-readable recording medium in any form well-known in the technical field of the present disclosure, such as a hard disk or a removable disk.

The computer program216, when loaded into the memory212, may include one or more instructions that enable the processor211to perform the operations/methods according to some embodiments of the present disclosure. That is, by executing the loaded one or more instructions, the processor211may perform the operations/methods according to some embodiments of the present disclosure.

For example, the computer program216may include instructions for performing the operations of: acquiring first and second document images; extracting first and second feature sets from the first and second document images, respectively, through the encoder31; generating a correlation feature set by analyzing the correlations between at least parts of the first and second feature sets; and outputting comparison results for the first and second document images based on decoding results for the correlation feature set.

As another example, the computer program216may include instructions to perform at least some of the steps/operations described above with reference toFIGS.1through20.

In this example, the computing device210may implement the comparison system10according to some embodiments of the present disclosure.

Meanwhile, in some embodiments, the computing device210ofFIG.21may refer to a virtual machine implemented based on cloud technology. For example, the computing device210may be a virtual machine operating on one or more physical servers included in a server farm. In this example, at least some of the processor211, the memory212, and the storage215may be virtual hardware, and the communication interface214may also be implemented as a virtualized networking element such as a virtual switch.

The computing device210that may implement the comparison system10according to some embodiments of the present disclosure has been described so far with reference toFIG.21.

Various embodiments of the present disclosure and their effects have been mentioned thus far with reference toFIGS.1through21.

According to some embodiments of the present disclosure, documents may be compared at an image level using a deep learning model trained for document comparison. In this case, the time and computing cost associated with training in various languages because document comparison at the image level, unlike OCR-based document comparison, does not require language-specific training. Also, language-independent (or non-dependent) document comparison may be conducted, and thus, multilingual documents may be easily compared.

Moreover, a deep learning model may be configured and trained to use the results of feature-level correlation analysis to output comparison results (e.g., segmentation maps) for input document images. In this case, the influence of font differences, quality differences (e.g., variations in the tilt of documents), etc., on the accuracy of document comparison may be minimized (e.g., cases where minor font differences lead to the documents being considered different may be minimized), thereby enhancing the performance of the deep learning model (refer to the experimental results inFIG.20). Furthermore, when performing a correlation operation between a first feature (e.g., attention feature) extracted from a first document image and a second feature (e.g., attention feature) extracted from a second document image, the correlation operation may be performed between the first feature and the pixel region including the second feature and the neighboring pixels around the second feature). As a result, the performance of the deep learning model may be further improved.

Additionally, by configuring the deep learning model's correlation analyzer to perform correlation analysis using multi-scale features, accurate comparison may be achieved even between document images containing characters of various sizes.

Furthermore, by configuring the deep learning model's attention operator to perform consecutive attention operations, the performance of the deep learning model may be further enhanced.

Also, by configuring the deep learning model's decoder to further receive features of a largest scale (i.e., the lowest level of abstraction) among the multi-scale features, sophisticated segmentation maps for the input document images may be easily generated.

In addition, by configuring the deep learning model's correlation analyzer to perform operations in both directions, the performance of the deep learning model may be further improved. For example, through bidirectional operations, sophisticated segmentation maps for both the first and the second document images may be generated.

It should be noted that the effects of the present disclosure are not limited to those described above, and other effects of the present disclosure will be apparent from the following description.

The effects according to the technical idea of the present disclosure are not limited to those mentioned above, and other effects not mentioned may be clearly understood by one of ordinary skill in the related art from the description below.

The technical features of the present disclosure described so far may be embodied as computer readable codes on a computer readable medium. The computer readable medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disc, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer equipped hard disk). The computer program recorded on the computer readable medium may be transmitted to other computing device via a network such as internet and installed in the other computing device, thereby being used in the other computing device.

Although operations are shown in a specific order in the drawings, it should not be understood that desired results may be obtained when the operations must be performed in the specific order or sequential order or when all of the operations must be performed. In certain situations, multitasking and parallel processing may be advantageous. According to the above-described embodiments, it should not be understood that the separation of various configurations is necessarily required, and it should be understood that the described program components and systems may generally be integrated together into a single software product or be packaged into multiple software products.