Patent Description:
The disclosure herein generally relates to deep learning networks, and, more particularly, to deep learning based table detection and associated data extraction from scanned image documents.

With the widespread use of mobile phones and scanners to photograph documents, the need for extracting the information trapped in unstructured document images such as retail receipts, insurance claim forms and financial invoices is becoming more acute. A major hurdle to this objective is that these images often contain information in the form of tables and extracting data from tabular sub-images presents a unique set of challenges. This includes accurate detection of the tabular region within an image, and subsequently detecting and extracting information from the rows and columns of the detected table. While some progress has been made in table detection, extracting the table contents is still a challenge since this involves table structure (rows and columns) recognition. Document <NPL>", discloses saliency-based fully-convolutional neural network performing multi-scale reasoning on visual cues followed by a fully-connected conditional random field (CRF) for localizing tables and charts in digital/digitized documents. Performance analysis carried out on an extended version of ICDAR <NUM>.

Embodiments of the present disclosure present technological improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems. , In one aspect, there is provided a processor implemented method for table detection and associated data extraction from scanned image documents using a deep learning network as defined in independent claim <NUM>.

In another aspect, there is provided a system for deep learning based table detection and associated data extraction from scanned image documents using a deep learning network as defined in independent claim <NUM>.

In yet another aspect, there are provided one or more non-transitory machine readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause deep learning based table detection and associated data extraction from scanned image documents and is defined in independent claim <NUM>.

It is intended that the following detailed description be considered as exemplary only, with the true scope being indicated by the following claims.

With the proliferation of mobile devices equipped with cameras, increasingly more customers are uploading documents via these devices, making the need for information extraction from these images more pressing. Currently, these document images are often manually processed resulting in high labour costs and inefficient data processing times. In addition, these documents often contain data stored in tables with multiple variations in layout and visual appearance. A key component of information extraction from these documents therefore involves digitizing the data present in these tabular sub-images. The variation in the table structure, and in the graphical elements used to visually separate the tabular components make extraction from these images a very challenging problem. Most existing approaches to tabular information extraction divide the problem into the two separate sub-problems of <NUM>) table detection and <NUM>) table structure recognition, and attempt to solve each sub-problem independently. While table detection involves detection of the image pixel coordinates containing the tabular sub-image, tabular structure recognition involves segmentation of the individual rows and columns in the detected table.

In the present disclosure, systems and methods and embodiments associated thereof implement an end-to-end deep learning model/network that exploits the inherent interdependence between the twin tasks of table detection and table structure identification. The model utilizes a base network that is initialized with pre-trained VGG-<NUM> features. This is followed by two decoder branches for <NUM>) Segmentation of the table region and <NUM>) Segmentation of the columns within a table region. Subsequently, rule based row extraction is employed to extract data in individual table cells.

A multi-task approach is used for the training of the deep model. The model takes a single input image and produces two different semantically labelled output images for tables and columns. The model shares the encoding layer of VGG-<NUM> for both the table and column detectors, while the decoders for the two tasks are different. The shared common layers are repeatedly trained from the gradients received from both the table and column detectors while the decoders are trained independently. Semantic information about elementary data types is then utilized to further boost model performance. The utilization of the VGG-<NUM> as a base network, which is pre-trained on the ImageNet dataset allows for exploitation of prior knowledge in the form of low-level features learnt via training over ImageNet.

The present disclosure has also evaluated TableNet's (or model of the present disclosure) performance on the ICDAR-<NUM> dataset, demonstrating that method of the present disclosure outperforms other deep models as well as other state-of-the art methods in detecting and extracting tabular information in image documents. The present disclosure further demonstrates that the model can generalize to other datasets with minimal fine-tuning, thereby enabling transfer learning. Furthermore, the Marmot dataset which has previously been annotated for table detection was also manually annotated for column detection, and these new annotations can be further utilized.

There is significant prior work on identifying and extracting the tabular data inside a document. Before the advent of deep learning, most of the work on table detection was based on heuristics or metadata. For instance, one of the prior research work exploited the structural information to identify tables and their component fields, while other research work used hierarchical representations based on the MXY tree for table detection and was the first attempt at using Machine Learning techniques for this problem. Yet another research work identified intersecting horizontal, vertical lines and low-level features and used an SVM classifier to classify an image region as a table region or not. Probabilistic graphical models were also used to detect tables; wherein joint probability distribution was modeled over sequential observations of visual page elements and the hidden state of a line (HMM) to join potential table lines into tables resulted in a high degree of completeness. Yet further research work used table header as the starting point to detect the table region and decompose its elements. Yet another research work made an attempt to detect borderless tables. They utilized whitespaces as a heuristic rather than content for detection.

In all prior deep learning based works, table detection and column detection are considered as two different problems, which can be solved independently. However, intuitively if all the columns present in a document are known apriori, the table region can be determined easily. But by definition, columns are vertically aligned word/numerical blocks. Thus, independently searching for columns can produce a lot of false positives. Thus, knowledge of the tabular region can greatly improve results for column detection. Since both tables and columns have common regions. Therefore, if convolutional filters utilized to detect table, can be reinforced by column detecting filters, this should significantly improve the performance of the model.

The model as implemented by the present disclosure is based encoder-decoder model for semantic segmentation. The encoder of the model is common across both tasks, but the decoder emerges as two different branches for tables and columns. Concretely, encoding layers have been enforced by the present disclosure and its systems and methods to use the ground truth of both tables and columns of document for training. However, the decoding layers are separated for table and column branches. Thus, there are two computational graphs to train.

The input image for the model of the present disclosure, is first transformed into RGB image and then, resized to for example say, <NUM> * <NUM> resolution. This modified image is processed using tesseract OCR (optical character recognition) technique as described and known in the art. Since a single model produces both the output masks for the table and column regions, these two independent outputs have binary target pixel values, depending on whether the pixel region belongs to the table/column region or background respectively.

The problem of detecting tables in documents is similar to the problem of detecting objects in real world images. Similar to the generic object detection problem, visual features of the tables can be used to detect tables and columns. The difference is that the tolerance for noise in table/column detection is much smaller than in object detection. Therefore, instead of regressing for the boundaries of tables and columns, the method of the present disclosure is implemented to predict table and column regions pixel-wise. Recent research work on semantic segmentation has been based on pixel wise prediction. In the recent research work, fully convolution network (FCN) architecture has demonstrated the accuracy of encoder-decoder network architectures for semantic segmentation. The FCN architecture uses the skip-pooling technique to combine the low-resolution feature maps of the decoder network with the high-resolution features of encoder networks. VGG-<NUM> was used as the base layers in their model and a fractionally-strided convolution layers is used to upscale the found low-resolution semantic map which is then combined with high resolution encoding layers. The model as implemented by the present disclosure uses the same intuition for the encoder/decoder network as the FCN architecture. The model/network of the present disclosure, uses a pre-trained VGG-<NUM> layer as the base network. The fully connected layers (layers after pool5) of VGG-<NUM> are replaced with two (1x1) convolution layers. Each of these convolution layers (conv6) uses the ReLU activation followed by a dropout layer having probability of <NUM> (conv6+dropout as shown in <FIG>). Following this layer, two different branches of the decoder network are appended. This is according to the intuition that the column region is a subset of the table region. Thus, the single encoding network can filter out the active regions with better accuracy using features of both table and column regions. The output of the (conv6 + dropout) layer is distributed to both decoder branches. In each branch, additional layers are appended to filter out the respective active regions. In the table branch of the decoder network, an additional (1x1) convolution layer, conv7 table is used, before using a series of fractionally strided convolution layers for upscaling the image. The output of the conv7 table layer is also up-scaled using fractionally strided convolutions, and is appended with the pool4 pooling layer of the same dimension. Similarly, the combined feature map is again up-scaled and the pool3 pooling is appended to it. Finally, the final feature map is upscaled to meet the original image dimension. In the other branch for detecting columns, there is an additional convolution layer (conv7 column) with a ReLU activation function and a dropout layer with the same dropout probability. The feature maps are up-sampled using fractionally strided convolutions after a (1x1) convolution (conv8 column) layer. The up-sampled feature maps are combined with the pool4 pooling layer and the combined feature map is up-sampled and combined with the pool3 pooling layer of the same dimension. After this layer, the feature map is upscaled to the original image. The outputs of the two computational graphs yields the mask for the table and column regions. The overview of the deep learning TableNet architecture is shown in <FIG>.

<FIG> an exemplary block diagram of a system <NUM> for deep learning based table detection and associated data extraction from scanned image documents, in accordance with an embodiment of the present disclosure. The system <NUM> may also be referred as 'table detection and data extraction system' and may be interchangeably used hereinafter. In an embodiment, the system <NUM> includes one or more hardware processors <NUM>, communication interface device(s) or input/output (I/O) interface(s) <NUM> (also referred as interface(s)), and one or more data storage devices or memory <NUM> operatively coupled to the one or more hardware processors <NUM>. The one or more processors <NUM> may be one or more software processing components and/or hardware processors. In an embodiment, the hardware processors can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor(s) is configured to fetch and execute computer-readable instructions stored in the memory. In an embodiment, the system <NUM> can be implemented in a variety of computing systems, such as laptop computers, notebooks, hand-held devices, workstations, mainframe computers, servers, a network cloud and the like.

The I/O interface device(s) <NUM> can include a variety of software and hardware interfaces, for example, a web interface, a graphical user interface, and the like and can facilitate multiple communications within a wide variety of networks N/W and protocol types, including wired networks, for example, LAN, cable, etc., and wireless networks, such as WLAN, cellular, or satellite. In an embodiment, the I/O interface device(s) can include one or more ports for connecting a number of devices to one another or to another server.

The memory <NUM> may include any computer-readable medium known in the art including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. In an embodiment, a database <NUM> is comprised in the memory <NUM>, wherein the database <NUM> comprises scanned image documents, for example, scanned image document containing text, table(s) and associated text comprised in the table(s), and the like. In an embodiment, the memory <NUM> may store (or stores) one of more techniques (e.g., OCR technique(s) such as masking technique(s), and the like). The memory <NUM> further comprises one or more domain rules applied on masked table region to obtain one or more row. The memory <NUM> further comprises (or may further comprise) information pertaining to input(s)/output(s) of each step performed by the systems and methods of the present disclosure. In other words, input(s) fed at each step and output(s) generated at each step are comprised in the memory <NUM>, and can be utilized in further processing and analysis.

<FIG>, with reference to <FIG>, depicts an exemplary deep learning network comprised in the system <NUM> of <FIG> for table detection and associated data extraction from scanned image documents along with a modified dataset, in accordance with an embodiment of the present disclosure. More specifically, on the left side of <FIG>, sample training image from Marmot dataset is depicted along with highlighted texts. On right side of the <FIG>, deep learning network architecture is depicted wherein pre-trained layers of VGG-<NUM> are shown. Layers starting from conv1 to pool5 are used as common encoder layers for both table and column graph. Two decoder branches, conv7 column and conv7 table emerging after encoder layers, generate separate table predictions and column predictions.

<FIG>, with reference to <FIG>, is an exemplary flow diagram illustrating a method for deep learning based table detection and associated data extraction from scanned image documents using the system <NUM> of <FIG>, in accordance with an embodiment of the present disclosure. In an embodiment, the system(s) <NUM> comprises one or more data storage devices or the memory <NUM> operatively coupled to the one or more hardware processors <NUM> and is configured to store instructions for execution of steps of the method by the one or more processors <NUM>. The steps of the method of the present disclosure will now be explained with reference to components of the system <NUM> of <FIG>, the deep learning network architecture of <FIG> and the flow diagram of <FIG>. At step <NUM> of the present disclosure, the one or more hardware processors <NUM> receive, via the deep learning network architecture of <FIG>, a scanned image document comprising one or more tables and text. <FIG>, with reference to <FIG>, depicts a sample input scanned image document, in accordance with an embodiment of the present disclosure. At step <NUM> of the present disclosure, the one or more hardware processors <NUM> extract and highlight, via the deep learning network architecture, at least one of one or more non-numerical values and one or more numerical values containing image from the one or more tables and associated text, using a text detection and recognition technique (e.g., Tesseract OCR tool as known in the art). In an embodiment, the step of extracting and highlighting at least one of one or more non-numerical values and one or more numerical values from the one or more tables and associated text, using a text detection and recognition technique, is based on identified semantic information using the text from the scanned image document. <FIG>, with reference to <FIG>, depicts detection of words regions using the text recognition technique, wherein text are extracted and highlighted and region is created on basis of text content.

At step <NUM> of the present disclosure, the one or more hardware processors <NUM> input the extracted and highlighted non-numerical and numerical values to the deep learning network of <FIG> to obtain a set of learnt features. <FIG> depicts masked content and/or text in the scanned image document, in accordance with an embodiment of the present disclosure. <FIG> depicts a modified dataset, with masked content appended to the modified dataset, in accordance with an embodiment of the present disclosure. More specifically, the modified dataset with masked content appended to it as shown in <FIG> is fed as an input to the deep learning network of <FIG>. In other words, the dataset as shown in <FIG> is overlaid with the masked content. Alternatively, the masked content is overlaid with the dataset of <FIG> wherein the overlaid representation is depicted in <FIG>. At step <NUM> of the present disclosure, the one or more hardware processors <NUM> generate a masked table region and one or more masked column regions using the set of learnt features. <FIG>, with reference to <FIG>, depicts mask for the column regions in the scanned image document, in accordance with an embodiment of the present disclosure. <FIG>, with reference to <FIG>, depicts masked table region comprised in the scanned image document, in accordance with an embodiment of the present disclosure. More specifically, the trained deep learning network architecture or trained deep learning model is executed on the above modified image, to obtain two output: (i) mask for the column regions in an image (also referred as scanned image document and interchangeably used hereinafter) and (ii) mask for the table region in image as depicted in <FIG>. In other words, the masked table region and the one or more masked column regions are generated to determine boundary of the one or more columns and the one or more tables comprised in the scanned image document. Extracting the proper boundary using the word boundaries and above masks gives (i) <NUM> boxes representing columns, and (ii) <NUM> large box representing table. <FIG>, with reference to <FIG>, depicts boundary of columns and table(s) comprised in the input scanned image document in accordance with an embodiment of the present disclosure.

At step <NUM> of the present disclosure, the one or more hardware processors <NUM> apply one or more domain-based rules on the masked table region to obtain one or more rows. In a nutshell, steps <NUM> till <NUM> are described as below for better understanding of the embodiments of the present disclosure. After processing the scanned image document from the deep learning network of <FIG>, masks for table and column regions are generated as mentioned in step <NUM>. These masks are used to filter out the table and its column regions from the image. Since, all word positions of the document are already known (using Tesseract OCR), only the word patches lying inside table and column regions are filtered out. Now, using these filtered words, a row can be defined as the collection of words from multiple columns, which are at the similar horizontal level. However, a row is not necessarily confined to a single line, and depending upon the content of a column or line demarcations, a row can span multiple lines. Therefore, to cover the different possibilities, the systems and methods of the present disclosure apply various domain rules for row segmentation as depicted below, and these rules shall not be construed as limiting the scope of the present disclosure:.

At step <NUM> of the present disclosure, the one or more hardware processors <NUM> identify tuples in the one or more tables based on one or more columns from the masked column regions and the one or more obtained rows from the masked table region. Below table depicts tuples identified in the one or more tables based on one or more columns from the masked column regions and the one or more obtained rows from the masked table region, in accordance with an embodiment of the present disclosure. More specifically, words are arranged on basis of spatial positions to different columns and inside table. It is to be understood by a person having ordinary skill in the art that texts are pre-filtered with table bounds and then it is filtered in respective columns. While traversing each rows, in each column, text regions are traversed in top-bottom and left to right approach. wherein data is converted into table comprising rows and columns. Below is an example of the tuples identified in the table <NUM> wherein data is converted into table comprising rows and columns.

Below description provides explanation on dataset preparation, preparing semantic information (as depicted in step <NUM>), and training Data Preparation for the deep learning network of <FIG>.

Deep-learning based approaches are data-intensive and require large volumes of training data for learning effective representations. There are very few datasets for example, but not limited to, Marmot (e.g., refer '<NPL>. '), UW3 (e.g., refer '<NPL>. '), etc. for table detection and even these contain only a few hundred images. There are even fewer datasets for table structure identification such as the (International Conference on Document Analysis and Recognition) ICDAR <NUM> table competition dataset for both table detection and its structural analysis (e.g., refer '<NPL>. This creates a constraint for deep learning models to solve both table detection & table structural analysis.

For training the deep learning network of <FIG>, the present disclosure has used the Marmot table recognition dataset. This is the largest publicly available dataset for table detection but did not have annotations for table columns or rows. Dataset has been (manually) annotated for table structure recognition since the dataset had ground truth only for table detection. The dataset was (manually) annotated by labeling the bounding boxes around each of the columns inside the tabular region. The manually annotated modified dataset is publicly released with the name Marmot Extended for table structure recognition.

Intuitively, any table has common data types in the same row/column depending on whether the table is in row major or column major form. For example, a name column contains strings, while, a quantity header contains numbers. To provide this semantic information to the deep model (or deep learning network of <FIG>), text regions with similar data types are coded (e.g., color coded). This modified image is then fed to network resulting in improved performance.

System and methods of the present included spatial semantic features by highlighting the words with patches as shown in <FIG> this dataset is also made publicly available. The document images are first processed with histogram equalization. After pre-processing, the word blocks are extracted using tesseract OCR. These word patches are colored (not shown in FIGS. ) depending upon their basic datatype. The resulting modified images are used as input to the network. The deep learning network of <FIG> takes the input image and generates the binary mask image of both table and columns separately. The achieved result is filtered using rules outlined on the basis of the detected table and column masks. An example of the generated output are shown in <FIG> as indicated in step <NUM>.

To provide the basic semantic type information to the model/the deep learning network, the word patches are (color) coded. The image is first processed with tesseract OCR, to get all word patches in the image document. Then the words are processed via regular expressions to determine their data-type. The intuition is to color the word bounding boxes to impart both the semantic and spatial information to the network. Each datatype is given a unique color (not shown in FIGS. ) and, bounding-boxes of words with similar datatypes are shaded in the same color (not shown in FIGS. Word bounding boxes are filtered out to remove the spurious detections. However, since word detection and extraction from the present OCR cannot be accurate in detecting all words, the model/the deep learning network needs to learn these cases. Therefore to simulate the case of incomplete word detection in the training image, few randomly selected word patches are dropped deliberately. The formed color coded image (not shown in FIGS. ) can be used for training, but a lot of relevant information are dropped in the process. Many visual features for example, line demarcations, corners, color highlights, etc. are lost, while using only the word annotated document image. Therefore, to retain those important visual features in the training data, the word highlight image is pixel-wise added to the original image. This modified document images are used for training.

The experiments and results section describe different experiments performed on the ICDAR <NUM> table competition dataset (e.g., refer '<NPL>. ') and the model performance is evaluated based on the Recall, Precision & F1-score. These measures are computed for each document and their average is taken across all the documents.

The deep learning network of <FIG> requires both table and structure annotated data for training. The present disclosure and its systems and methods used the Marmot table detection data and manually annotated the structure information. There were a total of <NUM> documents containing tables including both Chinese and English documents, out of which <NUM> English documents were annotated and used for training. In Experiment <NUM>, the deep learning network of <FIG> was trained on all positive samples of Marmot and tested on ICDAR <NUM> table dataset for both table and structure detection. There are two computation graphs which require training. Each training sample is a tuple of a document image, table mask and column mask. With each training tuple, the two graphs are computed at-least twice. In the initial phase of training, the table branch and column branch were computed in the ratio of <NUM>:<NUM>. With each training tuple, the table branch of the computational graph was computed twice, and then the column branch of the deep learning network was computed. It is worth noting that, although the table branch and column branch are different, the encoder is the same for both. During initial iterations of training, the learning is more focused on training the model to generate big active tabular regions that on subsequent training specializes to column regions. After around <NUM> iterations with a batch size of <NUM>, when training loss of both table and column are comparable and close, this training scheme is modified. However, it should be noted by a person having ordinary skill in the art, that the table classifier at this stage must exhibit a converging trend, otherwise, the training is extended with the same <NUM>:<NUM> scheme. The deep learning network of <FIG> is then trained in the ratio of <NUM>:<NUM> for both branches until convergence. Using the same training pattern, the model (also referred as deep learning network of <FIG> or TableNet and interchangeably used hereinafter) was trained for <NUM> iterations with a batch size of <NUM> and learning rate of <NUM>. Adam optimizer was used for improving and optimizing training with parameters beta1=<NUM>, beta2=<NUM> and epsilon=1e-<NUM>. The convergence and over-fitting behavior was monitored by observing the performance over the validation set (taken from ICDAR <NUM> dataset). During testing, <NUM> was taken as the threshold probability for pixel-wise prediction. The results have been compiled in Table <NUM> and table <NUM> as depicted below. More specifically, Table <NUM> depicts results on Table Detection and shall not be construed as limiting the scope of the present disclosure.

Table <NUM> depicts results on Table Structure Recognition and Data Extraction as shown below:.

Similarly, in another experiment, the present disclosure and its systems and methods used the modified Marmot dataset where, the words in each documents were highlighted to provide semantic context as explained in earlier description sections. All the parameters were identical to earlier experiment. There was slight improvement in the results, when these spatial, semantic information are appended to the image in visual forms (see table for comparison). Output of the model is shown in <FIG>. Additionally, yet another experiment was carried out to generate the result so that it can be compared with the closest deep-learning based solution, DeepDSert (e.g., refer '<NPL>. In DeepDSert, separate models were made for each task, which was trained on different datasets such as MarmotC for table detection, and ICDAR <NUM> table dataset for table structure recognition. To generate the comparable result, the present disclosure and its systems and methdos fine-tuned the Marmot trained model, on ICDAR train and test data split. As done in DeepDSert, <NUM> images were randomly chosen for testing and rest of data images for fine-tuning the model of the present disclosure. The model/deep learning network of <FIG> was fine-tuned, with same parameters as in original, in the ratio of <NUM>:<NUM> for both branches for <NUM> iterations with batch size <NUM>. The performance of the model of the present disclosure further improved after the fine-tuning, possibly due to the introduction to the domain of ICDAR documents. The results of this experiment are also compiled in Tables <NUM> and <NUM> as depicted above.

Embodiments of the present disclosure implement an end-to-end model for jointly performing both table detection and structure recognition in an end-to-end fashion. Existing approaches for information extraction treat them as two separate tasks and approach the problem with different models. The present disclosure and its systems and methods jointly address both tasks simultaneously, by exploiting the inherent interdependence between table detection and table structure identification. The model of the present disclosure utilizes the knowledge from previously learned tasks wherein this information or knowledge can be transferred to newer, related ones demonstrating transfer learning. This is particularly useful when the test data is sparse. Through experiments and results, the present disclosure also shows that by highlighting the text based to provide data-type information improve the performance of the model.

Such other modifications are intended to be within the scope of the claims if they have similar elements that do not differ from the literal language of the claims.

The hardware device can be any kind of device which can be programmed including e.g. any kind of computer like a server or a personal computer, or the like, or any combination thereof. The device may also include means which could be e.g. hardware means like e.g. an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of hardware and software means, e.g. an ASIC and an FPGA, or at least one microprocessor and at least one memory with software processing components located therein. Alternatively, the embodiments may be implemented on different hardware devices, e.g. using a plurality of CPUs.

Alternatives (including extensions, variations, deviations, etc., of those described herein) will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein.

Claim 1:
A processor implemented method, comprising:
receiving, via one or more hardware processors, a scanned image document comprising one or more tables and text (<NUM>);
extracting and highlighting, via the one or more hardware processors, at least one of one or more non-numerical values and one or more numerical values from the one or more tables and associated text, using a text recognition technique based on identified semantic information using the text from the scanned image document (<NUM>);
inputting, via the one or more hardware processors, the extracted and highlighted non-numerical and numerical values being appended to the scanned image document, to a deep learning network to obtain a set of learnt features (<NUM>), wherein training the deep learning network comprises:
providing basic semantic type information of the scanned image to the deep learning network by:
processing the scanned image by using optical character recognition technique to get word patches in the image document;
processing the word patches via regular expressions to determine their data-type;
giving each datatype a unique color and shading bounding-boxes of words with similar datatypes in the same color;
filtering the word bounding boxes to remove spurious detections;
dropping randomly selected word patches to simulate the case of incomplete word detection;
pixel-wise adding the word shaded image to the scanned image to obtain modified image used for training;
generating, via the one or more hardware processors, a masked table region and one or more masked column regions using the extracted and highlighted non-numerical and numerical values based on the set of learnt features (<NUM>) to determine boundary of the one or more columns and the one or more tables comprised in the scanned image document to obtain: (i) <NUM> boxes representing columns, and (ii) <NUM> large box representing table;
applying, via the one or more hardware processors, one or more domain-based rules on the masked table region to obtain one or more rows (<NUM>); and
identifying, via the one or more hardware processors, tuples in the one or more tables based on one or more columns from the masked column regions and the one or more obtained rows from the masked table region (<NUM>).