Patent Description:
Official documents are often used to verify the details of people. This is typically done manually. For example, a representative of a car rental company may check a customer's driving license before authorizing the rental of a car to the customer. Such checking implicitly requires manual verification of the presented documents and extraction of the data contained therein. Manual verification and extraction is a skilled and labour intensive process that leads to uncertainty and processing bottlenecks. For this reason, various techniques have been developed to automate aspects of this process.

One example is the use of computers to extract document data using optical character recognition (OCR) from scanned images of official documents. However, the efficiency of such methods is limited by a variety of factors. For example, the acquired images may be poor quality or the official documents may be damaged or tampered with in some way. Additionally, there may be a large number of official documents, with different structures of data, which may be presented for consideration. Automated data extraction from official documents is therefore a formidable computational challenge. Consequentially, typically, it is difficult to have confidence in automated data extraction results compared to the equivalent manually obtained results.

The computational challenge in extracting the data can be greatly reduced if the type of the official document is known. This is because a known type of document will generally be formatted and structured according to an official template, which will provide the expected type, location, and structure of data. Knowing the type of a document therefore provides the data extraction process with prior knowledge of what is expected to be present in an image of the document. As data extraction comprises matching of characters to the processed image, such prior knowledge limits the range of possibilities and ensures better performance of the data extraction process.

The ideal data extraction program would be able to extract data from and verify any type of official document. However, the number of official documents that may be presented for consideration is vast. This is clear by considering that most countries issue unique passports, birth certificates, driving licences, etc. which will all likely have changed in form over time.

Known data extraction programs which cover a large number of official document types have two main issues. Firstly, such data extraction programs are slow, since the program has to check which type of standard template fits the considered document, testing one standard template at a time. Secondly, when there are a limited number of the considered documents or access to a standard template is restricted, it may be impossible to have good confidence that a presented document is of the considered type as opposed to a similar document of a different type, or even a forged document.

<CIT> relates to processing of images of documents captured using a mobile device, and to real-time processing and feature extraction of images of payment documents for classifying the payment document therein.

An aspect of an invention is defined by independent claim <NUM>. Further, optional features of embodiments are defined by the dependent claims.

According to a first aspect, there is a computer-implemented method for extracting information from an image of an document comprising: acquiring an image comprising image data relating to at least a part of a document; classifying the image data as comprising a determined document type; using knowledge of the determined document type to extract image data from the document image; segmenting the extracted image data to obtain segmentation data; performing OCR on the segmentation data to obtain recognized data; post-processing the recognized data to obtain classification data; and reporting the classification data.

Preferably, classifying the image data as comprising a determined document type comprises using a deep CNN to assign the document image as comprising a determined document type. The use of a single classification method ensures a consistent performance, which makes the system more predictable. Additionally, all documents are thoroughly inspected regardless of type.

Using knowledge of the determined document type to extract image data from the document image may be performed by a process that comprises: finding a transform to define a bounding box of a template of the determined document type in the document image. The transform may comprise a rotation or scaling of the determined document image relative to the document image axes. This ensures that the method can accept image data that has not been acquired under ideal conditions. For example, it may use image data acquired using a camera that is skewed due the angle at which the camera was operated.

Segmenting the extracted image data to obtain segmentation data is performed by a process that comprises: searching the extracted image data to find at least one text field; associating a label with each of the at least one text field; obtaining segmentation data from the extracted image data, the segmentation data comprising the position of each of the at least one text field and the associated label; returning the segmentation data. Generally, segmenting the extracted image data to obtain segmentation data comprises using per pixel based segmentation. Segmenting the extracted image data to obtain segmentation data comprises using a deep CNN. Semantic labelling thereby quickly identifies the relevant sections of the document and automatically identifies the reason for their importance.

Performing OCR on the segmented data comprises: cropping the image data using the segmentation data; recognizing text in the cropped image data; obtaining recognized data, the recognized data comprising the position of the cropped image data and the recognized text. Recognizing text in the cropped image data comprises using prior knowledge about the determined document type. The performance of the OCR is assisted by the prior knowledge to help return the most likely text for a given field, thereby improving the performance and accuracy of the OCR.

Recognizing text in the cropped image data comprises using a recognizing network comprising a deep CNN with long short-term memory (LSTM) network layers on top.

Optionally, the recognizing network has been trained with connectionist-temporal-classification as a cost function. The recognizing network inference is done via beam-search.

Post-processing the recognized data comprises removal of special characters, and/or standardizing the format of the elements of the recognized data. This prevents data contamination of any system using the extracted data results.

In some embodiments, there may be a first set of steps; and a second set of steps, wherein documents are processing according to the first set of steps, or start being processed according to the first set of steps but, before the step of manipulating the image data to obtain a document image, are switched and processed according to the second set of steps. The decision to switch processing methods can be made according to difficulty of classification, in response to processing load, or as required by a user.

Preferably, acquiring an image comprising image data comprises at least one of: loading a file; taking a photograph; scanning an image; and receiving an image. Thus, all common methods of acquiring image data are supported increasing the versatility of the method.

If classification data has been obtained and a document type determined, the classification data may be validated by a process comprising comparing the classification data with standard data of a document of the determined document type. Validating the classification data by comparing the classification data with standard data of a document of the determined document type may comprise: summing the number of elements of the classification data; comparing the number of elements of the classification data to the number of elements of the standard data; and validating if the number of elements is the same.

Validating the classification data by comparing the classification data with standard data of a document of the determined document type may comprise: comparing the structure of each of the elements of the classification data to the corresponding element of the standard data using regular expressions; validating if the structures are the same.

The present disclosure is made by way of example only with reference to the accompanying drawings in which:.

<FIG> shows a system <NUM> in which according to one embodiment the invention is implemented. The system comprises electronic devices <NUM>, <NUM>, including mobile electronic devices <NUM>, fixed location electronic devices <NUM> and servers <NUM>. The electronic devices are in communication with at least one communication network <NUM> (which may include, but not necessarily include wireless network <NUM>). Data may be communicated between the electronic devices. The at least one communication network may comprise the internet, The wireless network may for example be a cellular or WiFi communication network, or any other conventionally known
wireless communication network. The described network architecture is only exemplary and modifications to it, including removing or adding of network components, are possible.

<FIG> shows selected aspects of the network system <NUM> shown in <FIG>. Specifically, it shows a mobile electronic device <NUM> in communication, over the wireless network <NUM>, with a server <NUM>. The server <NUM> is an electronic device that can be accessed across the network <NUM> by device <NUM>, <NUM> to perform computational tasks. The mobile electronic device <NUM> comprises a communication subsystem <NUM> to enable communication across the wireless network <NUM>. The mobile electronic device <NUM> further comprises at least one application <NUM> that can be executed on a processor <NUM> and a camera <NUM> that can be used to acquire image data. The image data and applications <NUM> are stored in memory <NUM> on the mobile electronic device.

<FIG> also shows a server <NUM> which is connected to the wireless network <NUM> by a wireless network interface <NUM> and a network interface <NUM>. The server <NUM> further comprises applications <NUM> that can be executed on a processor <NUM>. The server further comprises memory <NUM> on which the applications <NUM> and any data that is received from the wireless network <NUM>, and any electronic device connected thereto, can be stored. The server <NUM> may be distributed and comprise multiple servers, several processors and/or several memory storage locations. Such a distributed server <NUM> may operate by distributing computational tasks and data across its constitute parts and may communicate with other servers to perform computational operations.

<FIG> provide further details of the mobile electronic device <NUM> through which a user may work the invention as described herein. The mobile electronic device <NUM> comprises a display <NUM>, the camera <NUM>, and an electromagnetic (EM) radiation source <NUM> for illuminating the area to be imaged with the camera <NUM>. The mobile electronic device <NUM> is an example of a user electronic device by which a camera <NUM> may be used to capture image data of an official document <NUM>. This image data may be communicated over the wireless network <NUM> to the server <NUM> and stored in the server memory <NUM>.

In the server <NUM>, application software <NUM> of the stored applications <NUM> executes on the processor <NUM> to extract data from the official document <NUM> to which the image data relates. The result of this extraction may be communicated back across the wireless network <NUM> to the mobile electronic device <NUM> and displayed on the display <NUM> to a user. It will be understood that the system <NUM> described above is merely an exemplary system <NUM> for implementing the invention defined herein.

Referring to <FIG>, an official document <NUM> may take one of many forms such as a driving license, a passport, a utility or other bill, a birth certificate, a benefits book, an state identify card, or a residency permit. The term official document <NUM> is therefore intended to cover any document that contains structured information that may be used to verify a person's identity or an aspect relating a person, for example their address.

The official document <NUM> will comprise at least one feature such as a photo <NUM>, a MRC, e.g. barcode <NUM>, one or more lines of text <NUM>, or a symbol <NUM> such as a national flag. The text <NUM> may be positioned in set locations within the official document <NUM>. The text <NUM> may also be in a particular format and type, possibly including a particular font. The text in each location may also be limited to one or a subset of possible options. As an example, in an exemplary official document, <NUM> in from the left edge and <NUM> down from the top edge may have the term "DRIVING LICENSE" printed in <NUM> point size of a special font. To reduce the likelihood of counterfeiting, the structuring and format of the official document <NUM> may be restricted or difficult to obtain.

The official document <NUM> may be a machine-readable travel document (MRTD), such as a machine-readable passport (MRP), which comprises a machine-readable zone (MRZ) <NUM>. The MRZ <NUM> comprises information encoded into a series of special characters which may be in the structure and format according to the standards described by International Civil Aviation Organization document <NUM>. The MRZ <NUM> is designed to be read by a computer using OCR and enables faster or easier processing of documents than manually assessed passports.

The official document <NUM> may comprise an MRC, e.g. barcode <NUM>. The MRC, e.g. barcode <NUM>, comprises an image that encodes information. The image is machine-readable by optical scanning. An MRC may be a barcode comprising parallel lines of varying widths and spacing in-between, or it may be a QR code comprising a two dimensional pattern which may use rectangles, dots, and other geometric shapes. An example of a two dimensional barcode is a QR code.

Referring to <FIG>, the data extraction process for a document will now be described. Firstly, an image of an official document is acquired in an input step <NUM>. This acquisition could be by loading a file, taking and/or transferring a photograph, scanning an official document, or receiving and/or loading an image on to a computer. The described processing is thus versatile and not limited to a particular method of acquiring the image data. As an example, image data may be acquired by using the camera <NUM> of the mobile electronic device <NUM> to take an image of an official document <NUM>, this data is then communicated to the server <NUM> for processing using the below described method.

The acquired image may include multiple documents or a document in a background setting. In which case, the subset of the image data relating to the document is selected from the full image data. In other words the image is cropped. The image data or the cropped image data may not display the full side of the official document. For example, a photograph may only display <NUM>% of the front of an official document due to a thumb of a hand holding the document obscuring the remainder of the official document. This is not a problem provided there is enough of the official document visible in the image for the process to identify the official document. What aspects of the official document are necessary for it to be identified as a particular type will depend on the distinctiveness and form of a particular type of official document. For example, identifying the official document may only require one text field to be identified or it may require two, three, or more text fields to be identified.

Before or after cropping, the image data of the image may be rotated, preferably to align the document with the image axis using known methods. The image data may be processed with standard image processing tools to improve the quality of the image and help to distinguish the important features. The image data proceeds to a classification sub-process <NUM>, comprising at least one classification method.

The image data is then also processed by OCR, in a blind OCR step <NUM>. The OCR is performed blind, which means that all text in the image is identified and read without knowing anything about the structure of the text in the image data. The result of the blind OCR process is recognized data.

As part of the identification of text process, one or more bounding boxes containing textual characters are identified. The recognized data therefore comprises the position and size of the bounding boxes as well as any textual characters that have been identified. Textual characters relate to characters, or groups of characters, that are designed to be read by a human. Textual characters may include characters from at least one alphabet, punctuation marks, and may also include symbols that serve as textual character replacements, such as Japanese kanji. Non-textual characters are characters, or a groups of characters, that are not designed to be read by a human. Non-textual characters include groups of characters that are designed to be recognized by machines. Examples of non-textual characters include a MRZ and a barcode.

The image data is then assessed to see if the corresponding document is a MRTD. This is assessed by ascertaining if the document contains MRZ. One method to do this is to check if a pattern of one or more chevrons are present, in one of more lines, as this may indicate that the document is an MRTD.

If the image data is found to relate to a MRTD the system reads the MRZ with the OCR system. Since the data encoded in the MRZ is encoded in a known method, the result of the OCR that is the data that is recognized by the OCR can be decoded to form classification data, which will also reveal the type of the official document. The decoding may include contacting an external database and using the result of the OCR to obtain classification data. Once classification data is obtained, the official document that the image data relates to is thereby classified as a particular type of document. The advantage of doing the MRTD classification <NUM> first is that it is, by design, a fast and efficient way to read official document and therefore produces a result faster than with other classification and/or data extraction methods.

A validation <NUM> is then performed to ensure the data from the MRZ has been extracted correctly and to ensure that the official document has been correctly classified. To validate the classification data, the classification data is compared to standard data, or template data, of a document of the particular type. One method by which this is performed is to sum the number of elements in the classification data and compare with the number of elements in the standard data. If these are the same the classification data is said to be valid, and validation <NUM> is therefore deemed to have completed successfully.

Alternatively or additionally, validation <NUM> may comprise comparing the structure of each of the element of the classification data to the corresponding element in the standard data. If the structures are the same the classification data is said to be valid, and validation <NUM> is therefore deemed to have completed successfully.

Due to this separate independent test of the classification, the confidence in the classification data is increased. In this case, the processing of image data can stop. Since MRTD classification <NUM> is fast by stopping after MRTD classification <NUM> additional processing can be avoided and thereby the turnover of the classification system can be improved.

If validation <NUM> fails or if MTRD classification was not possible as no MRZ <NUM> was identified in the image data, the system will continue processing the image data and will move to Barcode classification.

For barcode classification <NUM>, the image data is assessed to see if the document contains a barcode <NUM>. If at least one barcode is found, the barcode is analysed. Since the data encoded in a barcode <NUM>, is encoded in a known method, the result of the barcode reading is data that can be decoded to form classification data, which will also reveal the type of the official document. The decoding may include contacting an external database and using the coded data to obtain classification data. Once classification data is obtained, the official document that the image data relates to is thereby classified as a particular type of document. Barcode classification <NUM> is a fast and efficient way to read official document and therefore produces a result faster than with other classification and/or data extraction methods.

A validation <NUM> is then performed to ensure the data from the barcode has been extracted correctly and to ensure that the official document has been correctly classified. To validate the classification data, the classification data is compared to standard data of a document of the particular type as previously described.

If validation <NUM> of the barcode classification <NUM> data fails or if barcode classification <NUM> was not possible as no barcode <NUM> was identified in the image data, the system will continue processing the image data.

The MTRD classification <NUM> and barcode classification <NUM> may run sequentially in either order or they may run consecutively. The resultant classification data may be compared with other or it may be combined and validated as previously described. Both classifications are relatively fast compared to other classification processes. Therefore, if either or both classifications are successful further processing can be avoided which reduces the computational load to classify an official document thereby improving the rate at which official documents can be classified and classification data extracted.

If barcode <NUM> and MRTD classification <NUM> have failed, keyword classification <NUM> is attempted. MRTD <NUM> and barcode classification <NUM> are generally started first because they don't rely on the recognized data from the blind OCR process <NUM>. This means that the blind OCR process <NUM> can run while these classifiers are processing.

One method of keyword classification <NUM> comprises analysing example images to form a list of template data. An example image may be an exemplary image of the official document of a particular type or it may be a document specific template, which is a template relating to particular type of official document. The term "word" should be considered as merely as one of, or a group of, textual characters. A keyword is one or more words that are part of the standard format of official document of a particular type, for example, "Passport Number". A keyword may also be a word that is commonly found on official document of the particular type, for example "Expired".

An exemplary process by which the list of template may been formed is as follows. Acquiring at least one document template image, a document template image being an example image of an official document of a first type. The document template image is then processed, preferably using OCR, to extract data associated with the document template. The extracted data may comprise an extracted keyword and a position of the extracted keyword. Preferably the extracted data comprises a plurality of extracted keywords and their positions. Preferably, a plurality of example images of official documents of the first type are processed and the extracted data combined to produce extracted data, as then the extracted data better characterizes official documents of the first type. The extracted data for each document template image are then combined into a list of template data. The list represents a two dimensional array of data. An example of such a list for a driving license, passport and ID card from a particular country might be:.

The produced list of template data is then used to obtain a weighting matrix, which is a term frequency-inverse document weighting matrix that serves to characterize the difference in term frequency of each extracted keyword relative to a corpus.

The weight matrix is then used as a linear classifier to classify the official document to which the image data relates. Using a linear classification means that the confidence in the classification can be easily calculated. If this confidence is above a threshold, or pre-set level, the official document that the image data relates to is classified as a particular type of document. If this threshold level of confidence is not reached the keyword classification <NUM> will be considered to have failed and a different classification method will be attempted.

It is easy to add to the list of template data and therefore keyword classification <NUM> can easily support a lot of different documents. Typically, keyword classification <NUM> supports classification of more official document types than all the other classification methods and it can operate even if there is only one example of the official document. This is why it generally occurs after MRTD <NUM> and barcode classification <NUM>. The keyword classifier will also be the first classification attempted documents for those official document on which a barcode <NUM> and MRZ <NUM> was not located. The accuracy of the keyword classifier is dependent on the number of keywords in the specific document and how distinct the set of keywords for a specific official document is different from all other documents.

Once a document has been classified by the keyword classification <NUM> a semantic labelling <NUM> process is performed to extract classification data. The following is one exemplary description of a process comprising the semantic labelling <NUM> process. Firstly standard data, from the list of template data, of the official document that the image data relates to is retrieved. The recognized data, from the blind OCR, is then matched with the standard data by searching the recognized data for each keyword from the standard data. An example image of an official document is then retrieved, and a homography calculated between the example image of the official document and the image data. The homography defines the perspective transform, or mapping, between the keywords read by the blind OCR system and the keywords from the example image.

Once the homography has been evaluated semantic labelling <NUM> can occur. In other words, the homography is used to match other textual data in the recognized data from the blind OCR to the location in the example image, thereby additional classification data is acquired. Pattern based matching, preferably using regular expression, is used to validate the text being matched to the positions in the official document.

A validation <NUM> is then performed to ensure the data from the keyword classification <NUM> has been extracted correctly and to ensure that the official document has been correctly classified. To validate the classification data, the classification data is compared to standard data of a document of the particular type as previously described.

If validation <NUM> of the keyword classification <NUM> data fails, the system will continue processing the image data. Generally, the next classification method to be attempted is a CNN based classification <NUM>. Preferably, the CNN is a deep CNN. Typically, for each official document of a particular type, the CNN been trained on approximately <NUM> images of the official document of the particular type.

Given the large number of documents used in the training of the CNN, when the CNN is classifying an official document on which it has been trained, the CNN classification <NUM> will provide high accuracy. Whilst the CNN classification <NUM> only supports the document types on which it has been trained, it may also be trained to recognize documents of a type on which it has not been trained as unknown. Therefore, it can report that it does not know about a certain document with a high accuracy. The CNN may therefore report that a document is not of one of a particular type or not one of a particular group of types, such as known types. The ability to accurately deal with unknown documents is why the CNN classification <NUM> occurs after the keyword classification <NUM>, which generally cannot assign a document as an unknown type.

The CNN classification <NUM> follows the same steps as a keyword classification <NUM>. Therefore, the CNN is used to classify the official document that the image data relates to as being a particular type. If a threshold of confidence in the classification has been achieved, semantic labelling <NUM> of the image data is then performed to obtain classification data.

A validation <NUM> is then performed to ensure the data from the CNN classification <NUM> has been extracted correctly and to ensure that the official document has been correctly classified. To validate the classification data, the classification data is compared to standard data of a document of the particular type as previously described.

If validation <NUM> of the CNN classification <NUM> data fails, the system will continue processing the image data. Generally, the next classification method to be attempted is a visual classifier based classification <NUM>. Preferably, the visual classifier is a bag-of-visual-words model using Kaze features and a support vector machine classifier. Typically, the visual classifier is trained on approximately <NUM> or <NUM> images of the official document of the particular type. Given the lower numbers of images of official documents needed for training, the visual classifier generally supports more types of official documents. As this classifier doesn't know which documents it doesn't know about it always tries to make a classification. For this reason, the visual classifier is generally last.

The visual classification <NUM> follows the same steps as a keyword classification <NUM>. Therefore, the visual classifier <NUM> is used to classify official documents that the image data relates to as being a particular type. If a threshold of confidence in the classification has been achieved, semantic labelling <NUM> of the image data is performed to obtain classification data.

A validation <NUM> is then performed to ensure the data from the visual classification <NUM> has been extracted correctly and to ensure that the official document has been correctly classified. To validate the classification data, the classification data is compared to standard data of a document of the particular type as previously described.

The above classification methods, or a subset thereof, can be completed in the described order or in a different order if necessary. The order of MRTD classification <NUM>, barcode classification <NUM>, keyword classification <NUM>, CNN classification <NUM> and visual classification <NUM> is described herein as the first processing pipeline <NUM>. The first processing pipeline <NUM> runs the fastest classification methods (MRTD <NUM> and barcode classification <NUM>) first, falling back to keyword classification <NUM> if the faster classification methods fail, then failing back to CNN classification <NUM> if the keyword classification <NUM> fails, and finally failing back to the visual classification <NUM>, which never fails to make a classification of the type of official document to which the image data relates.

In addition to the above mentioned advantages, one further advantage of the first processing pipeline <NUM> is that that experienced users can adapt the system to deal with new document types. To adjust the method to account for a new official document type merely requires a manual process of creating a template of the keyword for the new document, for the keyword classification <NUM>, and retraining the CNN and visual classifiers. This may be part of a processes triggered by the failure of a processed document.

One classification data has been made acquired, a consistency check <NUM> is performed on the classification data. The consistency check <NUM> takes the recognized data from the blind OCR <NUM> and compares it with the classification data. If a threshold of similarity is satisfied the consistency check <NUM> is said to be passed. Optionally, the threshold of similarity is that the classification data comprises a set number of elements of the recognized data or a decoded form of the recognized data. Optionally, the set number of elements corresponds to all classification data that would be expected to be visible in an official document of the classified type. As an example, consider a passport that is processed by MRTD classification <NUM> to result in classification data including: the name "Alice", the nationality "British", and the expiry date "<NUM>/<NUM>/<NUM>". This exemplary classification data would pass a consistency check <NUM> if the recognized data from the blind OCR comprises the words "Alice", "British", and "<NUM>/<NUM>/<NUM>".

The consistency check <NUM> confirms that the official document that the image data relates to was classified and processed correctly. There is always the risk that a document gets misclassified and the processing proceeds to extract erroneous classification data progressing any further. Therefore, the consistency is a simply a check that at least most of the relevant fields have been populated with plausible information.

Generally, if the validation <NUM> of all classification fails another is attempted until all possible classification methods have been attempted. If all classification methods have been tried, and none have produced classification data that has passed validation <NUM>, the system will typically report data extraction to have failed. If validation <NUM> has been passed, but consistency checking of the validated classification data fail, the system will typically report data extraction to have failed. Failure may flag the considered document or image for further processing or consideration. The further processing may be manual assessment or further automatic processing. Additionally, the further processing may involve assessing whether the failed document is a new official document type. In some cases, the failure may trigger automatic requests for further documents or images.

The second processing pipeline <NUM> is shown in <FIG>.

In one mode of operation, all processed official documents enter the first processing pipeline <NUM>, but selected documents may be switched based on user selection or preconfigured settings to the second processing pipeline <NUM>. In addition, the processing path by which a particular documents is processed, be that following the first processing pipeline <NUM> to the end, or switching to the second processing pipeline <NUM> can be made decided according to an assessment made after the image data from the document has been acquired and processed. Alternatively, in one mode of operation, all documents to be processed proceed only via the first processing pipeline <NUM>. In another mode of operation, all documents to be processed proceed only via the second processing pipeline <NUM>. In another alternative embodiment, the selection of documents to switch to the second processing pipeline <NUM> can be made: (i) by a user, (ii) according to an algorithm based on one or more document characteristics, e.g. the type of document, the size of the document, or document image resolution; (iii) according to operational parameters of the system corresponding to the classification methods, (iv) or randomly.

The second processing pipeline <NUM> has only one classification method, which is similar to the CNN classifier mentioned previously.

The second processing pipeline <NUM> comprises the following steps: classifying <NUM> the document image as comprising a determined document type; pre-processing <NUM>, possibly using knowledge of the determined document type to extract image data from the document image; segmenting <NUM> the extracted image data to obtain segmentation data; performing OCR <NUM> on the segmentation data to obtain recognized data; conditioning, or post-processing <NUM>, the recognized data to produce extracted data; and reporting the extracted data.

The step of classifying <NUM> the document image comprises determining the document type using a CNN, to assign a flag to the document image as being one comprising a document type determined by the CNN.

In a preprocessing step <NUM>, the official document, or a portion of an official document, is located within the image data and the rotation of the official document with the image axis is calculated. The relevant image data is then identified or extracted.

The segmenting step <NUM> processes the extracted image data to obtain segmentation data to find labels for each keyword in the image data. In other words, it identifies the relevant classification data and its position. Segmenting <NUM> the extracted image data to obtain segmentation data comprises: searching the extracted image data to find at least one text field; associating a label with each of the at least one text field; obtaining segmentation data from the extracted image data, the segmentation data comprising the position of each of the at least one text field and the associated label; returning the segmentation data. Preferably segmenting the extracted image data to obtain segmentation data comprises uses pixel based segmentation and is thus done in a purely visual way. The segmentation is performed using a deep CNN. This semantic labelling thereby identifies the relevant sections of the document and the reason for their importance.

The relevant sections are then read with a line-OCR step <NUM> that takes in a crop of a line of text and outputs the text in that image crop. The line-OCR system which performs this step uses further data inputs on document constraints, for example how dates are formatted, specific formatting of document numbers, etc., to provide further certainty on the likely text being returned for a given field. The line-OCR system is a deep convolutional network with LSTM network layers. It has been trained using a connectionist-temporal-classification cost function which is used to perform OCR. Inference is done via a beam-search process.

The step of post-processing <NUM> involves the classification data being cleaned-up from special pre-defined characters that sometimes occur (dash, apostrophe, etc.) and to format the data in a preferred format (i.e. data to yyyy-mm-dd format).

The CNN classifier of the first processing pipeline <NUM> and the second processing pipeline <NUM> are quite similar and the same advantages therefore also apply to the second processing pipeline <NUM>, as in the first processing pipeline <NUM>, which are discussed previously. However, since the other classification methods are not used in the first processing pipeline <NUM>, pipelines two is more accurate and faster due to more performant CNNs. The line-OCR system is trained on a plurality of documents types and can thus generate an associated model for each document type which is used within the pipeline and utilised with the system. This makes the line-OCR system utilised with the invention described herein perform better than a general blind-OCR system.

Further advantages of the second processing pipeline <NUM> are that every official document passing through it is processed in the same way. This improves predictability of processing times and means that the predictability, performance and stability of a computer using the methods is improved.

Additionally, the second processing pipeline <NUM> enables data to be more easily labelled by non-experts and the models can be trained automatically for each new document type. In contrast, the first processing pipeline <NUM> benefits from more involvement from experts to adapt it to account for new documents.

As explained, the pipelines described above, and shown in <FIG> and <FIG>, may in one embodiment be executed by the server <NUM>. The acquired image data may be image data of an official document <NUM> captured by camera <NUM> of mobile electronic device <NUM> that is communicated to the server <NUM>. The mobile electronic device <NUM> can include an application executable on the device <NUM> which coordinates the use of the camera <NUM>, the communication of the image data to the server <NUM>, and the reporting on the display <NUM> of the assessment result reported by the server <NUM>. A user may therefore work the invention via mobile electronic device <NUM> or, alternatively, via any other user electronic device that is connected to the wireless network <NUM>.

Such user electronic devices <NUM>, <NUM> are generally termed communication devices and may be a mobile or handheld device, such as a mobile or handheld communication device. It may also have the capability to communicate with other computer systems; for example, via a data link or network, such as a short-range radio frequency link, e.g. Bluetooth, or via a data network, which may be wireless and/or may be connected to the Internet. In certain embodiments, the user electronic device is a multiple-mode communication device configured for both data and voice communication, a mobile telephone, such as a smartphone, a wearable computer such as a watch, a tablet computer, a personal digital assistant, or a computer system such as a notebook, laptop, or desktop system. The user electronic device may take other forms apart from those specifically listed above, for example a fixed location server or a remotely accessed computer system. The user electronic device may also be referred to as a mobile, handheld or portable communications device, a communication device, or a mobile device. In the context of this disclosure, the term "mobile" means the device is of a size or weight which makes it readily portable by a single individual.

The electronic devices <NUM>, <NUM> may include a controller including a processor <NUM> (such as a microprocessor) which controls the operation of the electronic device <NUM>, <NUM>. In certain electronic devices <NUM>, <NUM>, more than one processor is provided, typically, with each processor in communication with each other and configured to perform operations in parallel, so that they together control the overall operation of the electronic device. The processor <NUM> interacts with device subsystems, such as a wireless communication subsystem <NUM> for exchanging radio frequency, or microwave frequency, signals with a wireless network <NUM> to perform communication functions. The processor <NUM> is communicably coupled with additional device subsystems, some of which are shown on <FIG>, including:.

Some of the subsystems perform communication-related functions, whereas other subsystems may provide "resident" or on-device functions.

The electronic device <NUM>, <NUM> stores data <NUM> in an erasable persistent memory, which in one embodiment is the memory <NUM>. In various embodiments, the data <NUM> includes service data including information used by the electronic device <NUM> to establish and maintain communication with the wireless network <NUM>. The data <NUM> may also include user application data such as email messages, address book and contact information, calendar and schedule information, notepad documents, presentation documents and information, word processor documents and information, spread sheet documents and information; desktop publishing documents and information, database files and information; image files, video files, audio files, internet web pages, services, applications, games and other commonly stored user information stored on the electronic device <NUM> by its user. The data <NUM> may also include program application data such as functions, controls and interfaces from an application such as an email application, an address book application, a calendar application, a notepad application, a presentation application, a word processor application, a spread sheet application, a desktop publishing application, a database application, a media application such as a picture viewer, a video player or an audio player, and a web browser. The data <NUM> stored in the persistent memory (e.g. flash memory) of the electronic device <NUM> may be organized, at least partially, into one or more databases or data stores.

In at least some embodiments, the electronic device <NUM>, <NUM> includes a touchscreen which acts as both an input interface <NUM> (e.g. touch-sensitive overlay) and an output interface <NUM> (i.e. display). The touchscreen may be constructed using a touch-sensitive input surface which is connected to an electronic controller and which overlays the display <NUM>.

As noted above, in some embodiments, the electronic device <NUM>, <NUM> includes a communication subsystem <NUM> which allows the electronic device <NUM> to communicate over a wireless network <NUM>. The communication subsystem <NUM> includes a receiver, a transmitter, and associated components, such as one or more antenna elements <NUM>, local oscillators (LOs) <NUM>, and a processing module such as a digital signal processor (DSP) <NUM> which is in communication with the processor <NUM>. The antenna elements <NUM> and <NUM> may be embedded or internal to the electronic device <NUM>, <NUM> and a single antenna may be shared by both receiver and transmitter. The particular design of the wireless communication subsystem <NUM> depends on the wireless network <NUM> in which electronic device <NUM>, <NUM> is intended to operate.

In at least some embodiments, the electronic device <NUM>, <NUM> also includes a device orientation subsystem <NUM> including at least one orientation sensor which is connected to the processor <NUM> and which is controlled by one or a combination of a monitoring circuit and operating software. The orientation sensor detects the orientation of the device electronic <NUM>, <NUM> or information from which the orientation of the electronic device <NUM>, <NUM> can be determined, such as acceleration. An orientation sensor may generate orientation data which specifies the orientation of the electronic device <NUM>, <NUM>. In various embodiments, the orientation subsystem <NUM> may include a gravity sensor, a gyroscope, a tilt sensor, an electronic compass or other suitable sensor, or combinations thereof. The device orientation subsystem <NUM> may include two or more orientation sensors such as an accelerometer and an electronic compass.

The electronic device <NUM>, <NUM> includes a microphone or one or more speakers. In at least some embodiments, the electronic device <NUM>, <NUM> includes a plurality of speakers <NUM>. Each speaker <NUM> may be is associated with a separate audio channel. The multiple speakers may, for example, be used to provide stereophonic sound (which may also be referred to as stereo).

The electronic device <NUM>, <NUM> may also include one or more cameras <NUM>. The one or more cameras <NUM> may be capable of capturing images in the form of still photographs or motion video. In at least some embodiments, the electronic device <NUM>, <NUM> includes a front facing camera <NUM>. A front facing camera is a camera which is generally located on a front face of the electronic device <NUM>. The front face is typically the face on which a display <NUM> is mounted. That is, the display <NUM> is configured to display content which may be viewed from a side of the electronic device <NUM>, <NUM> where the camera <NUM> is directed. The front facing camera <NUM> may be located anywhere on the front surface of the electronic device; for example, the camera <NUM> may be located above or below the display <NUM>. The camera <NUM> may be a fixed position camera which is not movable relative to the display <NUM> of the electronic device <NUM>, <NUM> or the housing of the electronic device <NUM>, <NUM>. In such embodiments, the direction of capture of the camera is always predictable relative to the display <NUM> or the housing. In at least some embodiments, the camera may be provided in a central location relative to the display <NUM> to facilitate image acquisition of a face. A back facing camera may be used alternatively to, or in addition to, in some embodiments.

In at least some embodiments, the electronic device <NUM>, <NUM> includes an electromagnetic (EM) radiation source <NUM>. In at least some embodiments, the EM radiation source <NUM> is configured to emit electromagnetic radiation from the side of the electronic device which is associated with a camera <NUM> of that electronic device <NUM>, <NUM>. For example, where the camera is a front facing camera <NUM>, the electronic device <NUM>, <NUM> may be configured to emit electromagnetic radiation from the front face of the electronic device <NUM>, <NUM>. That is, in at least some embodiments, the electromagnetic radiation source <NUM> is configured to emit radiation in a direction which may visible by the camera. That is, the camera <NUM> and the electromagnetic radiation source <NUM> may be disposed on the electronic device <NUM>, <NUM> so that electromagnetic radiation emitted by the electromagnetic radiation source <NUM> is visible in images detected by the camera.

In some embodiments, the electromagnetic radiation source <NUM> is an infrared (IR) radiation source which is configured to emit infrared radiation. In at least some embodiments, the electromagnetic radiation source <NUM> may be configured to emit radiation which is not part of the visible spectrum. The camera <NUM> may be a camera which is configured to capture radiation of the type emitted by the electromagnetic radiation source <NUM>. Accordingly, in at least some embodiments, the camera <NUM> is configured to capture at least some electromagnetic radiation which is not in the visible spectrum.

The electronic device <NUM>, <NUM> also includes a battery <NUM> as a power source, which is typically one or more rechargeable batteries that may be charged. The processor <NUM> operates under stored program control and executes software modules <NUM> stored in memory such as persistent memory; for example, in the memory <NUM>. The software modules <NUM> include operating system software <NUM> and other software applications <NUM>.

The electronic device <NUM>, <NUM> processor <NUM> is configured to execute executable code stored in memory, wherein the executable code comprises instructions for performing the method of the present invention. The code can be stored in any suitable memory.

The electronic device <NUM>, <NUM> can be supplied with the code preinstalled. Alternatively, the code can be loaded by the user or others on to the phone in the ways that are known to the skilled person, such as by data transfer through a USB cable or by downloading the code via a wireless communication Preinstalling or loading the code is equivalent to installing the code. Preferably, the code is in the form of an application. The application can be provided by a third party application providing service, as is common on modern electronic devices. Code updates may be loaded on to the electronic device <NUM>, <NUM> in a similar manner.

The code may operate by contacting one or more external systems, such as a server <NUM>, and exchanging data with the external systems. This prevents all the processing, or calculations, having to occur on the electronic device <NUM>, <NUM> which is useful to spare processing load and thus battery power. The electronic device <NUM>, <NUM> may use one preferred communication method to exchange data or it may select the optimal communication method in light of those that are available, The selection of communication methods can be adaptive or responsive. By way of nonlimiting example, if a wireless network communication signal using the IEEE <NUM> standard (WiFi) is initially available but lost, as the electronic device moves out of WiFi range, the electronic device may switch to a wireless network communication signal using the CDMA200 standard (<NUM>) to continue the data exchange with the server <NUM>. The data may be seamlessly transferred without interruption or the data transfer may pause during the switch over and be restarted thereafter either automatically or by the user.

In some embodiments, all the processing can occur on a user electronic device to prevent the need to contact external systems. This is especially useful if the user electronic device is a portable electronic device that may move into area in that is outside of all useful communications networks, since the functionality of the method is then not dependent of the availability of a communication network. In some cases, the execution of the code may cause the user electronic device to ascertain whether or not a communications network is available and select the operation mode accordingly, the assessment may be ongoing, periodic, or occur a limited number of times.

The code may provide flags, signals, or indications to other applications or services that the user electronic device is equipped with the extra functionality afforded by the present invention. Additionally, the code may be accessible by other applications or services to provide its functionality within the other application and services. For example, once installed the code may flag a financial application that extra security features are installed. The financial application may thus unlock, or enable, more sensitive functions and execute the code, to increase security, when these features are used. An exemplary use of code, which executes in accordance with the present invention, is described below.

Consider a user who wishes to register for a secure service, which requires registered users to be authenticated, this can be achieved via an application (or webpage) accessed via electronic device <NUM>, <NUM>. When the application is first accessed it checks the features and applications loaded onto the electronic device <NUM>, <NUM> and proceeds to advise the user to install an identification authentication application. It may also direct the user to a location to download the identification authentication application. The user proceeds to download the identification authentication application and load it on to the electronic device <NUM>, <NUM>. When the user returns to the service, the service detects that the identification authentication application is loaded and executes, or calls, the identification authentication application. The identification authentication application then prompts the user, via display <NUM> or speaker <NUM>, to use the camera <NUM> to take a photo of an official identification document, possibly using a separate camera application. Once a photo of an official document has been acquired, the identification authentication application sends the image data to a server <NUM> to extract data. The data extraction uses the process described above, either or both of the first and second processing pipelines <NUM>, <NUM>. The extracted data is then communicated from the server <NUM> back to the identification authentication application. The identification authentication application then communicates this extracted data to the service. The service knows the identification authentication application provides authentic extracted data. Therefore, the service can use the extracted data to register the new user.

If the server <NUM> could not extract the data or had low confidence in the extracted data, it may alert the service. Alternatively, it may alert the user and request further images of the official document or request images of alternative official documents.

Claim 1:
A computer-implemented method for extracting information from an image of a document comprising:
acquiring an image comprising image data relating to at least a part of a document (<NUM>);
classifying (<NUM>) the image as comprising a determined document type;
using knowledge of the determined document type to extract image data from the document image;
segmenting (<NUM>) the extracted image data to obtain segmentation data,
wherein segmenting the extracted image data to obtain segmentation data comprises using a deep convolutional neural network and comprises:
searching the extracted image data to find at least one text field (<NUM>);
associating a label with each of the at least one text field;
obtaining segmentation data from the extracted image data, the segmentation data comprising the position of each of the at least one text field and the associated label; and
returning the segmentation data;
performing optical character recognition (<NUM>) on the segmentation data to obtain recognized data, wherein performing optical character recognition on the segmented data comprises:
cropping the image data using the segmentation data;
recognizing text in the cropped image data, wherein recognizing text in the cropped image data comprises using prior knowledge about the determined document type, and using an associated recognizing network for each document type, each recognizing network comprising a deep convolutional neural network with long short-term memory network layers on top, and wherein the recognizing networks are trained on a plurality of document types to generate the associated network for each document type; and
obtaining recognized data, the recognized data comprising the position of the cropped image data and the recognized text;
post-processing (<NUM>) the recognized data to remove special characters and/or standardize the format of the recognized data to obtain classification data; and
reporting the classification data.