Patent Description:
Official documents are often used to check and test the identity of people. This is typically done manually. For example, a representative of a car rental company may check a customer's driving license before authorising the rental of a car to the customer. Such checking implicitly requires manual verification of the presented documents to check their authenticity.

One problem that is encountered is that people may use forged or counterfeit documents to present false details. In the case of manual verification, the authenticity of a document is assessed using an assessor's experience and many factors are taken into account. Often, an assessor will suspect a presented document is not authentic, but will not be able to identify on what factor they base their assessment. An assessor's ability to assess documents will also be limited by their experience with particular types of documents.

Manual verification is therefore a highly skilled and labour intensive process that may lead to uncertainty and processing bottlenecks. For this reason, various systems have been developed to assist and automate aspects of identity verification, in particular for official document authentication. One such system uses a scanner to identify and read machine-readable zones (MRZ) on a document and check that the information contained in the MRZ is authentic. Another such system includes a device to image the document and perform optical character recognition (OCR) on the image in order to extract textual information from the document, and similarly check whether the extracted information is authentic. Further systems include extracting a facial photograph from a document and comparing the extracted facial photograph with a photograph of the user via face matching.

Such systems have meant that there is no longer a need for a person having their identity checked to physically present their document to an assessor for authentication. Instead, the person may capture an image of their document using a user electronic device, such as a mobile electronic device, and send the image on for automated official document authentication. This has proved popular for those having their identity checked due to its convenience, and for those which require identity verification, such as the car rental company mentioned above, due to the increased speed of verification.

However, one drawback of capturing an image of an official document in such a way is that the image of the official document may not be aligned within the captured image. For example, image of the official document may appear rotated within the captured image. This rotation may not only be perpendicular to the image plane, but may be a three-dimensional rotation such that one or more edges of the document may appear larger in the captured image than opposing edges. This misalignment is problematic because the information contained in the image of the official document does not have the appearance that it would have done if the image would have been aligned, making it difficult to perform automated official document authentication for the official document. Accordingly, it is usually necessary to align the image of an official document before performing automated official document authentication.

The present invention therefore relates to a method for aligning a set of images which have shared structural characteristics, such as official documents.

<NPL>), according to its abstract, states that modern Convolutional Neural Networks (CNN) are extremely powerful on a range of computer vision tasks. However, their performance may degrade when the data is characterised by large intra-class variability caused by spatial transformations. The Spatial Transformer Network (STN) is currently the method of choice for providing CNNs the ability to remove those transformations and improve performance in an end-to-end learning framework. In this paper, they propose Densely Fused Spatial Transformer Network (DeSTNet), which is the first dense fusion pattern for combining multiple STNs. Specifically, they show how changing the connectivity pattern of multiple STNs from sequential to dense leads to more powerful alignment modules.

<NPL>, describes a deep network model that combines a spatial transformer for image deformation and a convolutional autoencoder for unsupervised feature learning for robust ssEM image alignment.

The present invention is defined by the independent claims, with further optional features being defined by the dependent claims.

There is provided a computer-implemented method for aligning a set of images according to claim <NUM>. The image data is preferably large-scale image data (e.g. may contain up to <NUM> images, each image at a resolution of up to 256x256 pixels).

To calculate the alignment error, the aligned image data may be normalised to have zero mean and unit standard deviation, and the alignment error calculated using the L1-norm using the normalised aligned image data. Optionally, the alignment error may be calculated using each of the pixels of the normalised aligned image data and each of the pixels of the template image.

The reconstruction error may be calculated using the L1-norm, and/or may be calculated using the L2-norm.

Optionally, prior to outputting the set of aligned images, the steps of applying, using a first deep neural network, at least one image transform to the image data to form aligned image data in which each image of the set of images is substantially aligned with a template image, compressing the aligned image data, and reconstructing the image data from the compressed image data may be iterated to improve the alignment of the set of images. Outputting the set of aligned images from the reconstructed image data may comprise using the reconstructed image data of the iterated steps.

The set of images have shared structural characteristics. For example, each image of the set of images may comprise an image of an official document. The official document may be the same type of official document for each image of the set of images. Similarly, the template image may comprise an image of an official document. In such examples, the official document of the template image is preferably the same type of official document as each image of the set of images. The set of images may comprise between <NUM> and <NUM> images. Additionally or alternatively, images in the set of images may have a working resolution of up to 256x256 pixels.

The at least one image transform applied using the first deep neural network may be a linear transformation. For example, the at least one image transform may comprise one or more of: rotation, translation, scaling, and shearing.

The reconstructed image data may have a reduced data size compared to the aligned image data. To this end, a low-rank penalty may be applied to the output of the encoder module (i.e. to the latent-space representation). The low-rank penalty may be approximated by, for example, a monotonically increasing functional penalty.

There is also provided a computer-implemented method for annotating a set of aligned images obtained by the above-describe method for aligning a set of images according to claim <NUM>.

There is also provided a computer-readable medium comprising executable instructions for performing the above-described computer-implemented methods.

There is further provided a computer comprising a processor configured to execute executable code stored in memory, wherein the executable code comprises instructions for performing the above-described computer-implemented methods.

The present disclosure is made by way of example and with reference to the accompanying drawings in which:.

<FIG> shows a system <NUM> in which, according to one embodiment, the disclosed method is implemented. The system comprises user electronic devices <NUM>, <NUM>, including mobile electronic device <NUM>, fixed location electronic device <NUM>, and server <NUM>. The user electronic devices are in communication with at least one communication network <NUM> (which may, but not necessarily, include wireless network <NUM>). Data may be communicated between the user electronic devices <NUM>, <NUM>. The at least one communication network <NUM> may include the internet, an internet of things (IoT) network, a cellular network, or the like. The wireless network <NUM> may, for example, be a <NUM> LTE network or WiFi communication network, or any other conventionally known wireless communication network. The described network architecture is only exemplary and modifications to it, including removing or adding of network components, are possible.

<FIG> shows selected aspects of the network system <NUM> shown in <FIG>. Specifically, <FIG> shows a mobile electronic device <NUM> in communication, over the wireless network <NUM>, with a server <NUM>. The server <NUM> is an electronic device that can be accessed across the network <NUM> by user electronic devices <NUM>, <NUM> to perform computational tasks. The mobile electronic device <NUM> comprises a communication subsystem <NUM> to enable communication across the wireless network <NUM>. The mobile electronic device <NUM> further comprises at least one application <NUM> that can be executed on a processor <NUM> and a camera <NUM> that can be used to acquire image data. The image data and applications <NUM> are stored in memory <NUM> on the mobile electronic device <NUM>.

<FIG> also shows a server <NUM> which is connected to the wireless network <NUM> by a wireless network interface <NUM> and a network interface <NUM>. The server <NUM> further comprises applications <NUM> that can be executed on a processor <NUM>. The server further comprises memory <NUM> on which the applications <NUM> and any data that is received from the wireless network <NUM>, and any user electronic device <NUM>, <NUM> connected thereto, can be stored. The server <NUM> may be distributed and comprise multiple servers, several processors and/or several memory storage locations. Such a distributed server <NUM> may operate by distributing computational tasks and data across its constituent parts and may communicate with other servers to perform computational operations.

<FIG> provide further details of mobile electronic device <NUM>. The mobile electronic device <NUM> comprises a display <NUM>, the camera <NUM>, and an electromagnetic (EM) radiation source <NUM> for illuminating the area to be imaged with the camera <NUM>. The mobile electronic device <NUM> is an example of a user electronic device <NUM>, <NUM> by which a camera <NUM> may be used to capture images. Due to differences between cameras <NUM> of known user electronic devices <NUM>, <NUM>, the resolution of any images captured may vary, depending on the user electronic device <NUM>, <NUM> used to capture the image. Captured images from currently known user electronic devices <NUM>, <NUM> are typically found to have a resolution of between <NUM>×<NUM> pixels and <NUM>×<NUM> pixels. These captured images may be communicated over the wireless network <NUM> to the server <NUM> and stored in the server memory <NUM>. Other components of the mobile electronic device <NUM> may include an input device, such as a touch screen or a keyboard, a microphone, and an orientation system, such as a gyroscope or GPS positioning system.

The memory <NUM> of mobile electronic device <NUM> includes an operating system <NUM> which stores the computer-readable code for operating the mobile electronic device <NUM>. As mentioned, the memory <NUM> also includes applications <NUM>, such as identity authentication application <NUM>, which are downloadable to memory <NUM> from server <NUM> via the at least one communication network <NUM>, or are pre-stored on the memory <NUM>. Other data <NUM> may also be present in memory <NUM> such as current and historical metadata of the mobile electronic device <NUM>.

The fixed location device <NUM> may have similar components to the mobile electronic device <NUM>. The components may be integrated into the fixed location device <NUM>, or may be in communication with the fixed location device via a port in the fixed location device <NUM>. For example, camera <NUM> may be connected to the fixed location device <NUM> via a USB port or similar in the fixed location device <NUM>.

In the server <NUM>, application software of the stored applications <NUM> executes on the processor <NUM> to perform one or more of the methods disclosed herein. In other words, the stored applications <NUM> include executable code which are stored in server memory <NUM>, where the executable code comprises instructions for performing one or more of the methods disclosed herein. Any such method may use captured images previously acquired from user electronic device <NUM>, <NUM>, which has been stored in the server memory <NUM>.

It will be understood that the system <NUM> described above is merely an exemplary system <NUM> for implementing the disclosed method defined herein.

The images acquired by user electronic devices <NUM>, <NUM> have shared structural characteristics, such as images of an official document <NUM>, as shown in <FIG>. An official document <NUM> may take one of many forms such as a driving license, a passport, a utility or other bill, a birth certificate, a benefits book, a state identity card, or a residency permit. The term "official document" is therefore intended to cover any document that contains structured information that may be used to verify a person's identity or an aspect relating a person, for example their address. The different forms of official document <NUM> may be referred to herein as being a particular "type" of official document <NUM>, which may be further restricted by the territory of issue. For example, an official document type might be a 'UK passport', a 'Finnish driving licence', etc..

Images of official document <NUM> have shared structural characteristics in that the official document <NUM> includes one or more features which are common to that type of official document <NUM>. Referring to <FIG>, there is an exemplary official document <NUM> which comprises the features of a facial photograph <NUM>, one or more lines of text <NUM>, and a machine-readable zone (MRZ) <NUM>. The at least one feature may be positioned in set locations for the type of official document <NUM>. The at least one feature may also have set size and format. For example, in an exemplary official document <NUM>, there may be positioned at <NUM> in from the left edge and <NUM> down from the top edge a facial photograph in a particular size and format. For example, the facial photograph may be <NUM> high and <NUM> wide. To reduce the likelihood of counterfeiting, the structure and format of the official document <NUM>, and its constituent features, may be restricted or difficult to obtain and therefore reproduce. The skilled person is aware that particular types of official document <NUM> have particular structures and formats.

As mentioned above, a known drawback of capturing an image of an official document <NUM> via a user electronic device <NUM>, <NUM> is that the image of the official document <NUM> may not be aligned within the captured image. For example, the image of the official document <NUM> may appear rotated within the captured image. This rotation may not only be perpendicular to the image plane, but may be a three-dimensional rotation such that one or more edges of the official document <NUM> may appear larger in the captured image than opposing edges. This misalignment is problematic because the information contained in the official document <NUM> does not have the appearance that it would have done has the image of official document <NUM> been aligned, making it difficult to perform automated official document authentication on the official document <NUM>. Accordingly, it is usually necessary to align the image of the official document <NUM> before performing automated official document authentication.

In certain circumstances, it may be necessary to align more than one image of an official document <NUM>, referred to herein as a set of images. For example, there may be a dedicated server, such as server <NUM>, which processes the alignment of images via its processor <NUM> using stored applications <NUM> in server memory <NUM>. In such circumstances, it is possible to queue the set of images in server memory <NUM>, and then align each image one-by-one using processor <NUM>. However, such a method is computationally intensive and may take a significant length of time if the number of images is large. Accordingly, jointly aligning the set of images is preferred. Such joint alignment is commonly referred to as image congealing.

Previously proposed approaches for jointly aligning a set of images having shared structural characteristics, such as the matrix rank minimisation method adopted in <NPL>, have unsatisfactory alignment results for images captured by a user electronic device <NUM>, <NUM> due to the extent of misalignment in the set of images.

Moreover, due to the way in which the joint alignment is computed in these previously proposed approaches, these approaches are limited in the amount of images that can be jointly aligned, and also in the resolution of the images that can be jointly aligned. Thus, such approaches cannot adequately align images captured with high resolution, as is the case for images captured by certain user electronic devices <NUM>, <NUM>, or images which have been acquired from different user electronic devices <NUM>, <NUM> having different resolutions.

There are further challenges for images captured by user electronic devices <NUM>, <NUM> which make these images difficult to jointly align, such as non-uniform illumination, shadows, and compression noise. Moreover, there may be present in the captured image a highly variable background which may include clutter and non-target objects. Further, there may be occlusion in the image of the official document <NUM>, for example the presence of fingers covering part of the official document <NUM> when held for image capture. Thus, there is a need for a method of aligning a set of images which addresses one or more of these issues.

Referring to <FIG>, an overview of the disclosed method for aligning a set of images is provided. The method is preferably performed by processor <NUM> using the stored applications <NUM> in server memory <NUM>.

As noted from the steps above, the method is not restricted to a set of images which contain an image of an official document <NUM>, but is used for any set of images having shared structural characteristics. Other image which have shared structural characteristics include, for example, images of a particular font, images of a particular person, and images of a particular product, amongst others.

The steps introduced above are described below in detail with respect images of an official document <NUM>. However, the skilled person would appreciate that the steps are applicable to any images having shared structural characteristics.

With reference to step <NUM> of <FIG>, the first step of the method is to acquire image data <NUM> comprising a set of images. Example image data <NUM> is depicted in <FIG>.

The set of images, as used herein, refers to images of an official document <NUM>. In other words, each image of the set of images comprises an image of an official document <NUM>. The set of images, as used herein, is not intended to refer to the images captured by user electronic devices <NUM>, <NUM> since an aim of the disclosed methods is to jointly align a set of images of an official document <NUM> within captured images, rather than aligning the captured images themselves. However, it will be appreciated that an image of an official document <NUM> may be present in a captured image.

The set of images have shared structural characteristics. For example, the image of official document <NUM> may be the same type of official document <NUM> for each image of the set of images. For instance, the set of images may comprise five images corresponding to images of UK driving licences of five different people.

At a minimum, the set of images comprises at least two images. The upper limit for the number of images in the set of images depends on the available computational power of server processor <NUM>. It is recommended based on currently commercially available server processors <NUM> that the set of images should include no more than about <NUM> images. This upper limit is significantly higher than previously proposed approaches for jointly aligning a set of images which can typically only process a maximum of <NUM> images at once.

The set of images may comprise images having a higher resolution than has been demonstrated in previously proposed approaches for jointly aligning a set of images. In particular, the set of images may comprise images having a resolution of up to and including 256x256 pixels for a large set of images (i.e. up to <NUM> images). Conversely, previously proposed approaches for jointly aligning a set of images have been demonstrated resolutions of 28x28 pixels for a large set of images, up to 60x80 pixels for a few tens of images (e.g. <NUM> images) and 100x100 pixels for a small set of images (e.g. <NUM>-<NUM> images). This higher resolution results in a more efficient optimisation of joint alignment of the set of images.

The resolution for each image of the set of images is determined by resizing the image captured by user electronic device <NUM>, <NUM>, which are typically in the range of 422x215 pixels to 6016x3910 pixels, to a lower resolution. This lower resolution is the resolution of up to and including 256x256 pixels mentioned above. Accordingly, the disclosed method is capable of jointly aligning images which are originally captured with a range of resolutions, as is generally the case for images captured by different user electronic devices <NUM>, <NUM>.

Preferably the image data <NUM> is acquired from server memory <NUM>, where the image data <NUM> is stored. This allows each of the set of images of the image data <NUM> to be individually received at the server memory <NUM> at various times, and for the set of images to therefore subsequently be batch processed by the server processor <NUM>. To this end, server memory <NUM> may include permanent data storage, such as an optical hard drive, a solid state hard drive, a removable optical disk, etc. Alternatively or additionally, server memory <NUM> may include non-permanent data storage, such as a cache or random access memory (RAM).

Prior to storing in the server memory <NUM>, each of the set of images may have been captured via a camera <NUM> of a user electronic device <NUM>, <NUM>. For example, a captured image may be acquired at the server memory <NUM> via an identification authentication application <NUM> (or webpage) accessed via user electronic device <NUM>, <NUM>. When the application <NUM> is first accessed it loads and executes the applications on to the user electronic device <NUM>, <NUM>. The identification authentication application <NUM> may then prompt the user, via display <NUM> or a speaker, to use the camera <NUM> to capture an image of an official document <NUM>, possibly using a separate camera application. This image is then communicated to server memory <NUM> via any of the previously described communication networks <NUM>, <NUM>. In another example, each of the set of images may not be communicated immediately to the server memory <NUM> after being captured, but may instead be stored in memory <NUM> of the user electronic device <NUM>, <NUM>. In such case, the identification authentication application <NUM> may allow the user to upload an image stored in memory <NUM> on the user electronic device <NUM>, <NUM> to the server memory <NUM>.

With reference to step <NUM> of <FIG>, the second step of the method is to apply, using a first deep neural network <NUM>, at least one image transform to the image data <NUM> to form aligned image data <NUM> in which each image of the set of images is substantially aligned with a template image <NUM>. A particular example of the second step is depicted in <FIG>.

The first deep neural network attempts to align each image of the set of images of official document <NUM> with template image <NUM> such that each image and the template image <NUM> are at least prima facie aligned (i.e. appears to be aligned at a glance). However, as the skilled person would appreciate, it is difficult, if not impossible, to achieve full alignment (i.e. pixel-by-pixel alignment) in this way, especially when each image is misaligned in three-dimensions, as is the case for images captured by user electronic device <NUM>, <NUM>. Hence, where it is mentioned herein that each image is substantially aligned with template image <NUM>, it is intended to refer to a degree of alignment between "full alignment" and "prima facie" alignment. This degree of alignment (or, more accurately, misalignment) is characterised by an alignment error, as is discussed below.

As depicted in <FIG>, the template image <NUM> is an image of an official document <NUM>. In particular, the official document <NUM> of the template image <NUM> is the same type of official document <NUM> as each image of the set of images. For example, when the set of images comprises five images corresponding to images of UK driving licences of five different people, the template image <NUM> is an image of a UK driving licence. To improve the alignment result of the first deep neural network <NUM>, the template image <NUM> is preferably a high resolution image which is fully aligned. The template image <NUM> may be retrieved from server memory <NUM>. Accordingly, the server memory <NUM> may contain a repository which includes at least one template image of various types of official document <NUM>. The template image <NUM> may be obtained by the server <NUM> by, for example, a high-resolution image scanner.

As mentioned above, the first deep neural network <NUM> applies at least one image transform to the image data <NUM> to form aligned image data <NUM>. The at least one image transform may be a linear transformation. For example, the at least one image transform may be one or more of: rotation, translation, scaling, and shearing. In rotation, an image is rotated about a particular angle from its origin. In translation, an image is moved laterally from its origin. In scaling, an image is increased or decreased in size. In shearing, also referred to as skewing, the image is slanted in a particular direction.

The first deep neural network <NUM> applies at least one image transform to the image data <NUM> so that each image of the set of images is substantially aligned with the template image <NUM>. Thus, the first deep neural network <NUM> may apply at least one image transform to each image of the set of images. The least one image transform applied to each image may be different from the at least one image transform applied to other images due to the difference in alignment of the image of official document <NUM> in the images captured by user electronic devices <NUM>, <NUM>.

Deep neural networks such as first deep neural network <NUM> are known for being able to solve a range of computer vision tasks, such as image alignment. However, the performance of first deep neural network <NUM> may degrade when the image data <NUM> includes large variations in alignment in the images, such as is the case for images captured by user electronic devices <NUM>, <NUM>. Spatial transformer networks, such as that described in <NPL>, may be used to provide first deep neural network <NUM> with the ability to remove the misalignment and thus improve performance in alignment. Spatial transformer networks may also be sequentially connected to further improve alignment performance, see for example <NPL>.

First deep neural network <NUM> may have any number l of layers, including l-<NUM> hidden layers, and <NUM> output layer. The parameters of each layer l includes a weight parameter and a bias parameter. The weight parameter is said to indicate the influence of a change in the input of the layer to the output of the layer. For instance, for layers with weights parameters of zero, changing the input of the layer will not change the output of the layer. The weight parameters of first deep neural network <NUM> may be updated by the disclosed methods, as discussed below. The bias parameter is said to indicate the strength of the assumptions that should be made about the template image <NUM>.

As shown in <FIG>, the first deep neural network <NUM> is based on a densely fused spatial transformer network (DeSTNet). As described in <NPL>, DeSTNet includes multiple spatial transformer networks which are connected in a dense fusion pattern. This type of connection of spatial transformer networks causes improved image alignment over sequential connection described above.

It has been shown that DeSTNet can be trained in a supervised manner to detect the corners of an official document <NUM> in an image of an official document <NUM>. The disclosed method defines further ways to train DeSTNet (and other types of first deep neural network <NUM>) but in an unsupervised manner, as is discussed further herein. Unsupervised learning is preferable to supervised learning because, unlike supervised learning, unsupervised learning does not require large numbers of images to be pre-aligned for training purposes. This means that the first deep neural network <NUM> can be trained quickly for new types of official document <NUM>, and requires only one image of that type of official document <NUM>, as is discussed further herein.

An advantage of using first deep neural network <NUM> to align images is that first deep neural network <NUM> is robust to occlusion which is typically seen in images captured by user electronic device <NUM>, <NUM>, such as a finger holding official document <NUM>. Similarly, first deep neural network <NUM> is robust to partial crop of the image of official document <NUM>, for example where the captured image does not contain the whole of the official document <NUM>, but rather a corner or edge of the official document is cropped. Previously proposed approaches for aligning images typically fail when the image contains either occlusion or partial crop.

Once the first deep neural network <NUM> has formed aligned image data <NUM>, it may be desirable to remove from the aligned image data <NUM> any image data which is not an image of official document <NUM>. For example, if the image data <NUM> contains, in addition to the set of images, image data representing a background behind official document <NUM>, as would be the case from an image captured by user electronic device <NUM>, <NUM>, then this background image data may be removed by any known method.

With reference to steps <NUM> and <NUM> of <FIG>, the third and fourth steps of the disclosed method are to compress the aligned image data <NUM>, and to reconstruct image data <NUM> from compressed image data <NUM>. The third and fourth steps according to the invention are depicted in <FIG>.

It has been previously observed that a set of images sharing similar appearance and structure characteristics which are accurately aligned require less modelling capacity to be reconstructed well than a set of images which are not accurately aligned, see for example <NPL>. Motivated by this observation, the aligned image data <NUM> is compressed and then reconstructed to form reconstructed image data <NUM> to further improve the image alignment, where the reconstructed image data <NUM> has a reduced data size compared to the aligned image data <NUM>. For example, the aligned image data <NUM> may have a resolution of 256x256 pixels, and the reconstructed image data <NUM> may have a resolution of 64x64 pixels.

The aligned image data <NUM> is compressed and reconstructed using a second deep neural network <NUM> (different from first deep neural network <NUM>). With reference to the example in <FIG>, the second deep neural network <NUM> is an autoencoder. Autoencoders comprise an encoder module <NUM> and a decoder module <NUM>. The aligned image data <NUM> is compressed using the encoder module <NUM> of the autoencoder and the reconstructed image data <NUM> is formed from the compressed image data <NUM> using the decoder module <NUM> of the autoencoder.

The encoder module <NUM> and the decoder module <NUM> are fully-connected, rather than being convolutional. Advantageously, a fully-connected encoder module <NUM> and decoder module <NUM> enforces pixel level alignment which would not be enforced with convolutional layers due to translation equivariance.

The compressed image data <NUM> is a latent-space representation of the aligned image data <NUM>, which means that compressed image data <NUM> is a compressed representation of the aligned image data <NUM>. From the compressed image data <NUM>, i.e. the compressed representation of the aligned image data <NUM>, the decoder attempts to reconstruct the aligned image data <NUM>.

The autoencoder may be a low-capacity autoencoder, that is to say that the compressed image data <NUM>, i.e. the compressed representation of the aligned image data <NUM>, is limited in the number of relationships between pixels in the aligned image data <NUM> which it can model. In contrast, a high-capacity autoencoder is able to model more relationships between more pixels in the aligned image data <NUM> than a low-capacity autoencoder.

The low-capacity type of autoencoder is enforced by applying a low-rank penalty to the output of the encoder module <NUM>. The low-rank penalty may be approximated by a monotonically increasing functional penalty. For example, where the output of the encoder module <NUM> is <MAT>, and <MAT> is a positional weighting, then the functional penalty P may be: <MAT> where each component of w, <MAT> with i ∈ {<NUM>,. ,N}, and <MAT>. This functional penalty explicitly encourages the autoencoder to represent the aligned image data <NUM> using primarily the top components of z, zi, i ∈ {<NUM>,. ,K} and K << N, resulting in a smaller data size for the reconstructed image data <NUM>.

An advantage of using an autoencoder for compression and reconstruction of the aligned image data <NUM> is autoencoders are robust to occlusion which is typically seen in images captured by user electronic device <NUM>, <NUM>, such as a finger holding official document <NUM>. Similarly, autoencoders are robust to partial crop of the image of official document <NUM>, for example where the captured image does not contain the whole of the official document <NUM>, but rather a corner or edge of the official document is cropped. Previously proposed approaches for aligning images typically fail when the image contains either occlusion or partial crop.

With reference to step <NUM> of <FIG>, the fifth step of the disclosed method is to output a set of aligned images <NUM> (not shown) from the reconstructed image data <NUM>.

The set of aligned images <NUM> contains only images of official document <NUM> which, due to the alignment caused by the disclosed method, are in a suitable form for automated official document authentication. Accordingly, the set of aligned images <NUM> may be output and stored to server memory <NUM> or elsewhere for automated official document authentication. Particular aspects of automated official document authentication which benefit from a set of aligned images which are aligned using the disclosed method are discussed in detail below.

However, the set of aligned images <NUM> may not be output immediately after completion of the fourth step <NUM>. Instead, the first deep neural network <NUM> and the second deep neural network <NUM> may be trained based on the aligned image data <NUM> and/or the reconstructed image data <NUM>. Then, image data <NUM> may be passed through the first deep neural network <NUM> (i.e. the third step <NUM>) and the second deep neural network <NUM> (i.e. the fourth step <NUM> and fifth step <NUM>) again. After this, the reconstructed image data <NUM> from the repeated fifth step <NUM> may be used in place of the reconstructed image data <NUM> of the previous fifth step <NUM> to output a set of aligned images <NUM>.

An advantage of repeating, or iterating, the passing of image data <NUM> through the first deep neural network <NUM> (i.e. the third step <NUM>) and the second deep neural network <NUM> (i.e. the fourth step <NUM> and fifth step <NUM>), is that the alignment of the image achieved by the first deep neural network <NUM> in combination with the second deep neural network <NUM> is improved, as is further discussed below. To this end, step <NUM>, step <NUM>, and step <NUM> may be iterated any number of times to optimise the alignment of the set of aligned images. For example, the step <NUM>, step <NUM>, and step <NUM> may be iterated only once, or up to thousands of times, depending on the alignment quality desired. Typically only a few iterations are required to produce satisfactory alignment quality for automated official document authentication.

With reference to <FIG>, improved alignment may be achieved in the iterations by calculating an alignment error <NUM> (step <NUM>), which is the error between the template image <NUM> and the aligned image data <NUM>, and propagating the alignment error <NUM> to the first deep neural network <NUM>. Alternatively or additionally, a reconstruction error <NUM> may be calculated (step <NUM>) which is the error between the aligned image data <NUM> and the reconstructed image data <NUM>. The reconstruction error <NUM> may then be propagated to the second deep neural network <NUM> and/or the first deep neural network <NUM>. These steps and errors are described in more detail below. The remaining steps of <FIG> are the same as those shown in and discussed in relation to <FIG>.

An advantage of improving the image alignment using the disclosed method is that the training of first deep neural network <NUM> and second deep neural network <NUM> is unsupervised. This means that only one image of an official document <NUM> is required, i.e. the template image <NUM>, in order to jointly align a large set of images of an official document <NUM>. In practice, the disclosed method for aligning a set of images with unsupervised training is significantly faster than known methods (i.e. within a few hours).

The first deep neural network <NUM> and second deep neural network <NUM> are trained for a specific type of official document <NUM>. Thus, a different first deep neural network <NUM> and neural network <NUM> are required to be trained each type of official document <NUM>. For instance, one first deep neural network <NUM> and second deep neural network <NUM> may be trained for aligning images of UK driving licences, and another deep neural network <NUM> and second deep neural network <NUM> may be trained for aligning images of a Finnish passports. As unsupervised training of first deep neural network <NUM> and second deep neural network <NUM> is fast, and the first deep neural network <NUM> and the second deep neural network <NUM> have a minimal memory footprint, the disclosed method when aligning a set of images of a new type of official document <NUM> may be quickly deployed.

As mentioned above, each image of the image data <NUM> is substantially aligned with template image <NUM> by a degree of alignment between "full alignment" and "prima facie" alignment. This degree of alignment (or, more accurately, misalignment) is characterised by the alignment error <NUM>. The alignment error <NUM> thus provides a measure of how accurately first deep neural network <NUM> has aligned the set of images in the aligned image data <NUM>. For instance, a higher alignment error <NUM> implies a higher amount of misalignment for the set of images of the aligned image data <NUM>. The alignment error is calculated using the aligned image data <NUM> and the template image <NUM>.

One method for calculating the alignment error <NUM> is to normalise the aligned image data <NUM>, such that the aligned image data <NUM> has zero mean and unit standard deviation. Once the aligned image data <NUM> is normalised, the alignment error <NUM> may be calculated by using the L1-norm with the normalised aligned image data <NUM> and the template image <NUM>. The L1-norm minimises the sum of the absolute differences between the template image <NUM> and the normalised aligned image data <NUM>. The L1-norm is also known as least absolute deviations, and least absolute errors. The L1-norm is advantageous because its use always provides one solution which is stable. Alternative differentiable methods for calculating the alignment error <NUM> are also possible.

For the alignment error <NUM>, the L1-norm may be calculated using each of the pixels of the normalised aligned image data <NUM> and each of the pixels of the template image <NUM>. This means that the alignment error <NUM> may be based on all pixels locations (dense) rather than using a few key points (sparse). It is noted that the calculated alignment error <NUM> is estimated based on computer vision measure of image alignment (e.g. normalised cross-correlation), rather than physical ones (e.g. mm), as the latter depends on the image resolution.

Once calculated, the alignment error <NUM> is used to train the first deep neural network <NUM>. To train the first deep neural network using the calculated alignment error, the calculated alignment error may be back-propagated to the first deep neural network. Additionally or alternatively, training the first deep neural network using the calculated alignment error may comprise updating the weight parameters of the first deep neural network using the calculated alignment error. In particular, the alignment error <NUM> may be back-propagated to the first deep neural network <NUM>, and used to update the weight parameters of the first deep neural network <NUM>.

By training the first deep neural network <NUM> using the alignment error <NUM>, the first deep neural network <NUM> achieves improved (i.e. more accurate) image alignment during an iteration of the step of applying the at least one image transform to the image data <NUM> to form aligned image data <NUM> (step <NUM>). Subsequently, the first deep neural network <NUM> may be further trained, in the same manner as set out above, using the alignment error <NUM> from the iterated step. This may be repeated any number of times to achieve a desired image alignment quality.

The reconstruction error <NUM> is the difference between the aligned image data <NUM> which is input into second deep neural network <NUM> and the reconstructed image data <NUM> which is output from second deep neural network <NUM>.

The reconstruction error <NUM> may be calculated using the L1-norm. Similar to the alignment error <NUM>, the reconstruction error <NUM> may be calculated using each of the pixels of the reconstructed image data <NUM> and each of the pixels of the aligned image data <NUM>. The L1-norm is advantageous because its use always provides one solution which is stable.

Alternatively, the reconstruction error <NUM> may be calculated using the L2-norm. The L2-norm is also known as least squares. The L2-norm minimises the sum of the square of the differences between the aligned image data and the reconstructed image data. Similar to L1-norm, the reconstruction error <NUM> may be calculated using the L2-norm for each of the pixels of the reconstructed image data <NUM> and each of the pixels of the aligned image data <NUM>.

The reconstruction error <NUM> provides a measure of how accurate the alignment of the first deep neural network <NUM> is between each image in the set of images in the aligned image data <NUM>. Thus, for example, a higher reconstruction error <NUM> implies a higher amount of misalignment between the set of images of the aligned image data <NUM>. This is because, as mentioned above, a set of images sharing similar appearance and structure characteristics which are accurately aligned requires less modelling capacity to be reconstructed well than a set of images which are not accurately aligned. As a consequence, the second deep neural network <NUM>, which has limited modelling capacity, cannot accurately reconstruct a set of images which are not accurately aligned. In contrast, a set of images which are accurately aligned is reconstructed by the second deep neural network <NUM> with a low reconstruction error <NUM>. Thus, similar to the alignment error <NUM>, the reconstruction error <NUM> provides a measure of the alignment achieved by the first deep neural network <NUM>.

Accordingly, the reconstruction error <NUM> is used to train the first deep neural network <NUM>. Additionally, the reconstruction error <NUM> is used to train the second deep neural network <NUM>.

Training the first deep neural network <NUM> using the calculated reconstruction error <NUM> may be performed by back-propagating the calculated reconstruction error <NUM> to the first deep neural network <NUM>. The weight parameters of the first deep neural network <NUM> may then be updated using the calculated reconstruction error <NUM>.

Training the second deep neural network <NUM> using the calculated reconstruction error <NUM> may be performed by back-propagating the calculated reconstruction error <NUM> to the second deep neural network <NUM>.

By training the first deep neural network <NUM> and the second deep neural network <NUM> using the reconstruction error <NUM>, the first deep neural network <NUM> and second deep neural network <NUM> achieves improved (i.e. more accurate) image alignment during an iteration of the step of applying the at least one image transform to the image data <NUM> to form aligned image data <NUM> (step <NUM>). Subsequently, the first deep neural network <NUM> and the second deep neural network <NUM> may be further trained, in the same manner as set out above, using the alignment error <NUM> from the iterated step. This may be repeated any number of times to achieve a desired image alignment quality.

Outputting a set of aligned images <NUM> is useful for various automated official document authentication processes, including annotation. With reference to <FIG>, annotation of an image of an official document <NUM> refers to the marking of salient portions of the official document <NUM>, including image fields <NUM>, text fields <NUM> and/or machine readable zones <NUM>. These portions include information which is extractable by optical character recognition, image processing, and the like, and may be used to determine if the official document <NUM> is authentic. For example, a name may be extracted from text field <NUM> and compared against a name existing in a server memory <NUM>, or input into user electronic device <NUM>, <NUM>. If the names appear to be the same, then the official document <NUM> may be authentic.

Accordingly, the invention provides a computer-implemented method for annotating the set of aligned images <NUM>. The method may be performed by server processor <NUM>. The method comprises the steps of:.

With reference to <FIG>, the portion in the template image <NUM> which is annotated may comprise one or more of: a text field <NUM>, an image field <NUM>, and a machine readable zone <NUM>. A text field <NUM> may contain, for example, a name, date of birth, nationality of a person. An image field <NUM> may contain, for example, a facial image or another biometric image of a person. The machine readable zone <NUM> may contain, for example, a barcode or a QR code relevant to a person. The annotations may be determined manually and input into server processor <NUM>, or may be determined by the server processor <NUM>, and checked manually.

The first coordinates of the portion are relative to an edge or a corner of the official document image <NUM> in the template image <NUM>. This edge or corner may also be annotated. In addition to the portion with first coordinates, a second portion with second coordinates may be annotated, similarly a third portion with third coordinates may be annotated. In fact, there may be up to an nth portion with nth coordinates, subject to the number n of portions chosen for automated document authentication. The coordinates may be stored in server memory <NUM>.

The set of aligned images <NUM> may be acquired from server memory <NUM>, where the set of aligned images <NUM> is stored after the disclosed method of aligning.

Once the template image <NUM> is annotated, and the set of aligned images <NUM> acquired, corresponding portions can be obtained from the set of aligned images <NUM> using the coordinates of the portions. This is not possible using known method for aligning a set of images as the inferior quality of alignment with these known methods means that the coordinates would not accurate for all images of an official document <NUM> in the set of images. Accordingly, when using known methods for aligning a set of images, it is generally required that each image of an official document <NUM> be passed through a data extraction pipeline to be individually annotated. One such data extraction pipeline is described in <CIT>.

An advantage of this approach to annotation is that, for a new type of official document <NUM>, only one image of that type of official document <NUM> (i.e. the template image <NUM>) is required for alignment and annotation. Further, only the template image <NUM> needs to be annotated, then corresponding annotations in a large set of images of the official document <NUM> may be extracted in a fully unsupervised manner.

Claim 1:
A computer-implemented method for aligning a set of images, the method comprising:
a. acquiring (<NUM>) image data comprising a set of images, the set of images having shared structural characteristics;
b. applying (<NUM>), using a densely fused spatial transformer neural network, at least one image transform to the image data to form aligned image data in which each image of the set of images is substantially aligned with a template image, wherein each image of the set of images is substantially aligned with the template image in that there is an alignment error;
c. compressing (<NUM>) the aligned image data using an autoencoder neural network having an encoder module and a decoder module, wherein the compressed image data is a latent-space representation of the aligned image data and is output by the encoder module, and wherein the encoder module and the decoder module are fully connected;
d. reconstructing (<NUM>) the image data from the compressed image data using the decoder module, calculating (<NUM>) a reconstruction error using the aligned image data and the reconstructed image data, calculating (<NUM>) the alignment error using the aligned image data and the template image; training the autoencoder neural network using the calculated reconstruction error, and training the densely fused spatial transformer neural network using the calculated reconstruction error and the calculated alignment error; and
e. outputting (<NUM>), from the reconstructed image data, a set of aligned images.