Patent Description:
Today's increasing demand for online and mobile verification of identity documents has created a strong need for authentication solutions with various fraud detection capabilities.

Existing systems can generally be divided between active and passive solutions.

Active solutions rely on embedding anti-counterfeiting security features such as watermarks, linecodes, etc. in an identity document to increase the difficulty of tampering with the document. Such solutions however may not be effective at detecting tampering that does not affect the embedded security features.

Passive solutions are based on checking whether an identity document contains traces of forgery or manipulation in any portion of the document. Typically, these solutions use pixel-level analysis to identify predetermined tampering operations, and generally focus on checking manipulations around personally identifiable information (PII) areas only.

Document by <NPL> discloses a method of detecting forgery in an identity document by analyzing texture patches on the document. The document by <NPL> discloses a method of detecting forgery in identity documents by taking binary patches in the text area to detect the presence of copy pasted areas.

Systems and methods for texture-based authentication of digital identity documents are defined by the independent method claim <NUM>, system claim <NUM> and medium claim <NUM>.

Examples not covered by the claims include authentication based on global texture information extracted from a digital image representative of an identity document.

Global texture-based authentication may include generating a global texture profile for an identity document image and comparing the global texture profile with a stored profile associated with a jurisdiction class of the identity document.

In examples global texture-based authentication may be configured to be insensitive to PII and to tolerate wide variations in the ambient illumination captured in.

In the invention as defined by the claims embodiments include authentication based on local texture information extracted from the digital image.

Local texture-based authentication includes generating one or more local texture patches representative of texture information of one or more select local blocks of the identity document. The one or more local texture patches are provided as input to one or more local detectors each trained to detect the presence of a forgery based on a target manipulation space.

The target manipulation space is physical portrait photo substitution.

In embodiments, the local detectors may each include a texture-based convolutional neural network (CNN) classifier; a machine learning-based classifier; or an Error Level Analysis (ELA)-based classifier.

Classifiers may be trained using augmented training data designed to simulate the target manipulation space of the classifier. In an embodiment, a style-transfer based network may be used to synthesize images with a particular global document style.

The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the present disclosure and, together with the description, further serve to explain the principles of the disclosure and to enable a person skilled in the pertinent art to make and use the disclosure.

<FIG> illustrates an example authentication system <NUM> according to an embodiment of the present disclosure. As would be understood by a person of skill in the art based on the teachings herein, system <NUM> is provided for the purpose of illustration only and is not limiting of embodiments of the present disclosure.

As shown in <FIG>, system <NUM> includes a normalization module <NUM>, a texture information generator <NUM>, a jurisdiction class detector <NUM>, a gallery indexing module <NUM>, a database <NUM>, a fraud detector <NUM>, and a decision module <NUM>.

In an embodiment, operation of system <NUM> begins by receiving a digital image <NUM>. The digital image <NUM> represents an identity document (ID), such as a national identity card, a passport, or a driver's license, for example.

Normalization module <NUM> may perform various transformations on the image <NUM>. For example, if the image is in color, the image <NUM> may be transformed to a grayscale image. Additionally, normalization module <NUM> may flat render the image <NUM> to remove any content not related to the ID represented by the image. For example, content outside of a detected boundary of the ID may be removed.

The normalized image <NUM> is then provided to texture information generator <NUM>. In another embodiment, the normalized image <NUM> may also be provided to jurisdiction class detector <NUM>.

Texture information generator <NUM> is configured to generate, based on the digital image <NUM>, a global texture profile <NUM> representative of global background texture information of the ID and/or one or more local texture patches <NUM> representative of texture information at one or more select local blocks of the ID.

In the invention texture information generator <NUM> includes a texture block extractor <NUM>.

Texture block extractor <NUM> is configured to extract local texture information <NUM> from the image <NUM>. As would be understood by a person of skill in the art based on the teachings herein, texture information refers to information that quantifies the perceived texture of an image. The perceived texture is a function of the spatial variation of brightness intensity in the image.

Global texture information <NUM> may correspond to texture information extracted at a plurality of blocks of the image distributed over the entire area of the digital image. Local texture information <NUM> may correspond to texture information extracted at specific local areas of the image, for example at or around specific features of the image. In an embodiment, the local texture information <NUM> may include one or more sets of texture information (<NUM>-<NUM>,. , <NUM>-n) each tailored to enable the detection of a particular forgery manipulation (manipulation space) and/or tailored for a specific type of local detector. Each set of texture information <NUM>-<NUM>,. , <NUM>-n may include one or more local texture patches as further described below.

<FIG> illustrates an example approach for extracting global texture information from a digital image according to an embodiment. As shown in <FIG>, the digital image may be partitioned into a plurality of blocks <NUM>, according to a defined number of rows and columns. Texture information may then be extracted at each of the blocks <NUM>. The extracted texture information may accordingly capture texture with a wider correlation, as illustrated, for example, by texture blocks <NUM><NUM>,<NUM> and <NUM><NUM>,<NUM> corresponding respectively to blocks <NUM><NUM>,<NUM> and <NUM><NUM>,<NUM> of the digital image.

In another approach, as shown in <FIG>, the global texture information may be extracted from smaller, regional blocks <NUM> distributed over the entire area of the digital image. The extracted texture information may accordingly capture regional texture as illustrated, for example, by texture blocks <NUM><NUM>,<NUM> and <NUM><NUM>,<NUM> corresponding respectively to blocks <NUM><NUM>,<NUM> and <NUM><NUM>,<NUM> of the digital image.

<FIG> illustrates an example approach for extracting local texture information from a digital image according to the invention. According to this approach, the local texture information includes one or more local texture patches representative of texture information of one or more select local blocks of the ID.

Generating the one or more local texture patches comprises detecting, based on a target manipulation space, one or more features of interest of the ID.

According to the invention, the target manipulation space is physical portrait photo substitution.

As would be understood by a person of skill in the art based on the teachings herein, physical portrait photo substitution refers to the replacement of the portrait photo on a physical ID; digital splicing refers to the digital manipulation of an image to add external content (e.g., to change the PII) into the image; inpainting refers to the removal of content from an image using pixels coming from the same image; resampling refers to the use of interpolation to geometrically transform a digital image or a portion of an image; photocopy recapture refers to the generation of a digital image by photographing a physical copy of a document; and LCD screen recapture refers to the generation of a digital image by photographing an LCD screen display of a document.

For example, in <FIG>, the target manipulation space is physical portrait photo substitution, and the one or more features of interest include a portrait photo of the ID. Various feature detection techniques may be used to detect features of interest. For example, to detect a portrait photo in an ID, the Hough transform may be used to detect the lines corresponding to the frame of the portrait photo.

After detecting the one or more features of interest in the ID, the one or more select local blocks of the ID are defined as a function of the detected features of interest. Subsequently, texture information is extracted at each of the one or more select local blocks to generate the one or more local texture patches. For example, in <FIG>, after detecting the portrait photo <NUM>, one or more local select blocks <NUM> are defined as rectangular or square blocks straddling the top, bottom, left, and right boundaries of the portrait photo. Texture information is then extracted from one or more of the select local blocks <NUM> to obtain the one or more local texture patches.

In an embodiment, texture block extractor <NUM> may be configured to automatically set to black pixel blocks of the digital image that include personally identifiable information (PII), prior to extracting the respective texture information at each of the plurality of blocks.

Returning to <FIG>, in an example not covered by the claims, texture block extractor <NUM> may be configured to provide global texture information <NUM> to global texture profile generator <NUM>. Texture block extractor <NUM> may also be configured to provide local texture information <NUM> to one or more local detectors <NUM> of fraud detector <NUM>.

Global texture profile generator <NUM> may be configured to generate a global texture profile, based on global texture information <NUM>, representative of global background texture information of the ID.

<FIG> is a flow chart that illustrates an example process <NUM> for generating a global texture profile according to an example. One or more steps of process <NUM> may be performed by texture block extractor <NUM> or by global texture profile generator <NUM>.

As shown in <FIG>, process <NUM> begins in step <NUM>, which includes extracting, based on the digital image, respective texture information at each of a plurality of blocks distributed over the entire area of the digital image. Step <NUM> may be performed by text texture block extractor <NUM>, for example.

Next in step <NUM>, the process includes generating a respective texture descriptor based on the extracted respective texture information for each of the plurality of blocks. Step <NUM> may be performed by texture profile generator <NUM>.

In an embodiment, the texture descriptor generated for each of the plurality of blocks includes one or more of: a histogram-based texture descriptor, a correlation-type texture descriptor, or a local binary pattern (LBP)-based texture descriptor. The histogram-based texture descriptor captures statistics of finer local (<NUM>nd-order) differences among texture pixels. The correlation-type descriptor captures coarser and wider correlation between larger areas (patches) of texture pixels (like moiré patterns). The LBP-based texture descriptor captures statistics of relative variations of local intensity of texture pixels.

In an embodiment, the texture descriptor for a given block includes a combination of descriptors of the different types mentioned above. For example, the texture profile can include a concatenation of multiple segments each obtained using a different type of texture descriptor. The advantage of combining descriptors of different types to form the texture descriptor of a given block is that the resulting descriptor conveys a richer characterization of the texture of the given block.

Process <NUM> terminates in step <NUM>, which includes combining the respective texture descriptors of the plurality of blocks to generate the global texture profile of the ID. In an embodiment, the combination of the respective texture descriptors of the plurality of blocks includes concatenating the respective texture descriptors to obtain the global texture profile of the ID.

<FIG> is an example that illustrates the generation of texture descriptors according to an embodiment. As shown, texture information is extracted from a plurality of regional blocks <NUM> distributed over the entire area of the digital image. The texture information extracted from the blocks <NUM> may be as illustrated by texture blocks <NUM><NUM>,j-<NUM> and <NUM><NUM>,j corresponding respectively to blocks <NUM><NUM>,j-<NUM> and <NUM><NUM>,j.

The texture descriptors may be histogram-based as illustrated by descriptors <NUM><NUM>,j-<NUM> and <NUM><NUM>,j which correspond respectively to texture blocks <NUM><NUM>,j-<NUM> and <NUM><NUM>,j.

The texture descriptors of the plurality of blocks <NUM> are combined to generate a global texture profile of the digital image. In the case of histogram-based texture descriptors, the combination includes adding together the bins of the histograms to obtain a combined histogram.

Returning to <FIG>, jurisdiction class detector <NUM> may be configured to detect a jurisdiction class <NUM> of the identity document based on the digital image <NUM>. The jurisdiction class of an identity document may be defined by an issuing authority (e.g., country, state, etc.) and a category of the identity document (national identity card, passport, driver's license, etc.).

In an embodiment, jurisdiction class detector <NUM> may be configured to scan the image <NUM> and to use character recognition techniques to extract information relating to issuing authority and/or category of the identity document.

Jurisdiction class detector <NUM> provides the detected jurisdiction class <NUM> to gallery indexing module <NUM>. In an embodiment, gallery indexing module <NUM> queries database <NUM>, using the detected jurisdiction class <NUM>, to retrieve a stored texture profile <NUM> associated with the detected jurisdiction class <NUM>. In an embodiment, the stored texture profile <NUM> corresponds to a previously generated global texture profile for the detected jurisdiction class <NUM>.

The previously generated global texture profile for a jurisdiction class may be generated based on a gallery image for the jurisdiction class and is referred to hereinafter as a gallery profile. The gallery image for a jurisdiction class is an image of an identity document that is known to be authentic for the jurisdiction class.

In an embodiment, the gallery profile for a jurisdiction class may be generated in the same manner as global texture profile <NUM>, namely using normalization module <NUM>, texture block extractor <NUM>, and global texture profile generator <NUM>. The gallery profile is then provided to gallery indexing module <NUM>, which indexes and stores the gallery profile into database <NUM> based on its associated jurisdiction class detected by jurisdiction class detector <NUM>.

According to embodiments, gallery profiles for a multitude of jurisdiction classes may be generated and stored in database <NUM> prior to live authentication using system <NUM>.

Fraud detector <NUM> may be configured to detect the presence of a forgery in the digital image based on the global texture profile <NUM> or the one or more local texture patches <NUM>. According to embodiments, the detected forgery may belong to one or more manipulation spaces, such as physical portrait photo substitution, digital image splicing, inpainting, resampling, photocopy recapture, or LCD screen recapture. In an embodiment, fraud detector <NUM> may include a global detector <NUM> and one or more local detectors <NUM>.

Global detector <NUM> may be configured to detect forgeries based on global texture profile <NUM>. In an embodiment, global detector <NUM> compares the global texture profile <NUM> to the stored texture profile <NUM> associated with the detected jurisdiction class of the identity document represented by the digital image <NUM>.

Based on the comparison, the global detector <NUM> and/or a decision module <NUM> may identify the digital image <NUM> as a recaptured image from a photocopy or a computer screen, or as having an incorrect background layout for the jurisdiction class of the identity document, for example.

In an embodiment, two texture profiles are compared by measuring a difference for each type of descriptor inside the profile. The descriptor differences are then merged to obtain a final texture profile score. In an embodiment, the descriptor differences are calculated as distance metrics. For a histogram-based descriptor, the distance metric may be the intersection of two histograms (one from the global texture profile <NUM>, the other from the stored texture profile <NUM>). In an embodiment, a dropout policy to avoid high-noise bins may be employed. For a correlation-type descriptor, the distance metric may be a score mapping with a threshold to ensure that an aliasing signal of the global texture profile <NUM> is not larger than that of the stored texture profile <NUM>.

In an embodiment, the global detector <NUM> may be configured to account, in the comparison, for ambient illumination variations between the digital image <NUM> and the gallery image used to generate the gallery profile <NUM>. In an embodiment, this is made possible by ensuring that the descriptors used in the texture profile consist of only relative information, such as differences (or gradients), correlation coefficients, and statistics of intensity variations and gradients.

The setting to black of PII areas in the generation of the profiles as described above allows the profile comparison to be PII insensitive.

Local detectors <NUM>-<NUM>,. , <NUM>-n may each be configured to detect forgeries based on respective local texture patches <NUM>. In an embodiment, each detector <NUM>-i is configured to receive respective one or more local patches <NUM>-i tailored for the specific manipulation space that the detector <NUM>-i is intended to detect.

According to embodiments, local detectors <NUM>-<NUM>,. , <NUM>-n may be configured to detect, for example, one or more of: physical portrait photo substitution, digital splicing, inpainting, resampling, photocopy recapture, or LCD screen recapture.

In an embodiment, a local detector <NUM>-i configured to detect physical portrait photo substitution may be configured to receive one or more local textures patches as illustrated in the example of <FIG> described above. In a similar manner, a local detector configured <NUM>-j configured to detect digital splicing of PII may be configured to receive one or more local patches extracted at or in the neighbourhood of the PII of interest on the ID. On the other hand, a local detector <NUM>-k configured to detect photocopy recapture or LCD screen recapture may require only randomly extracted local texture patches.

Local detectors <NUM>-<NUM>,. , <NUM>-n may each be implemented in various ways. Without limitation, local detectors <NUM>-<NUM>,. , <NUM>-n may each include a texture-based convolutional neural network (CNN) classifier; a machine learning-based classifier; or an Error Level Analysis (ELA)-based classifier.

For the purpose of illustration and not limitation, a texture-based CNN classifier may be implemented as described in "<NPL>," which is incorporated herein by reference in its entirety. As would be understood by a person of skill in the art, the implementation would be modified as needed to address the particular problem of the present disclosure.

For the purpose of illustration and not limitation, a machine learning-based classifier may be implemented as a Support Vector Machine (SVM) as described in "<NPL>," which is incorporated herein by reference in its entirety. As would be understood by a person of skill in the art, the implementation would be modified as needed to address the particular problem of the present disclosure.

For the purpose of illustration and not limitation, an ELA-based classifier is a classifier that acts on an ELA signal obtained from the image, rather than on the raw image. The ELA signal includes error levels computed at one or more pixels of the image. Typically, the error levels are computed by compressing the image at a known error rate and then by taking the difference pixel-by-pixel between the original image and the compressed image. Pixels with error levels above a defined relative threshold may be identified as having been subject to manipulation.

In an embodiment, rather than using an absolute threshold, the ELA-based classifier may use a relative threshold for different regions of the image. For example, the error levels over the non-portrait photo region of the image may be used to estimate a dynamic and relative threshold for the error levels over the portrait photo region of the image for the classification.

In an embodiment, to improve classification performance, classifiers may be trained using augmented training data designed to simulate the target manipulation space of the classifier. Data augmentation is generally relevant to all types of classifiers, and especially for CNN classifiers.

In an embodiment, the augmented training data may be generated using a style-transfer based network as described in "<NPL>)," which is incorporated herein by reference in its entirety. Such network may be configured to generate images that mix the content of a first image with the style of a second image. As would be understood by a person of skill in the art, the implementation would be modified as needed to address the particular problem of the present disclosure.

For example, <FIG> illustrates the synthesis of example digital images representative of a manipulated identity document with photo recapture style. Specifically, image <NUM> represents the image which content is to be reproduced in the synthesized image (the content image), and image <NUM> represents the image which style (photo recapture) is to be transferred into the synthesized image (the style image).

Images <NUM> and <NUM> represent respectively coarse-detail and fine-detail synthesized images resulting from mixing content image <NUM> and style image <NUM> using a style-transfer based network.

Similarly, <FIG> illustrates the synthesis of a coarse-detail image <NUM> and of a fine-detail image <NUM> by mixing content image <NUM> with a style image <NUM> having an LCD recapture style, using a style-transfer based network. It is noted that style image <NUM> may have an entirely different content than content image <NUM>.

As shown in <FIG> and <FIG>, the mixing of content and style images according to this approach results in the synthesized images having a particular global document style. As such, data augmentation may include not only pixel-level geometric, photometric, and noise-adding operations, but also texture-level data augmentation to generate fraudulent training images with a wider range of pixel correlation or with a periodic texture structure.

Returning to <FIG>, decision module <NUM> may be configured to receive the outputs of each of global detector <NUM> and local detectors <NUM>-<NUM>,. Based on the received outputs, decision module <NUM> may be configured to determine whether the image <NUM> has been manipulated or represents a fraudulent identity document.

<FIG> is a flowchart that illustrates an example process <NUM> for authenticating a digital identity document according to an embodiment. As would be understood by a person of skill in the art based on the teachings herein, process <NUM> is provided for the purpose of illustration only and is not limiting of embodiments of the present disclosure.

As shown in <FIG>, process <NUM> begins in step <NUM>, which includes generating, based on a digital image representative of an ID, a global texture profile representative of global texture information of the ID. The global texture profile may be generated as described above with reference to <FIG>, <FIG>, <FIG>.

Before, after, or concurrently with step <NUM>, in step <NUM>, the process may include detecting a jurisdiction class of the ID based on the digital image. In an embodiment, step <NUM> may be performed by a detector such as jurisdiction class detector <NUM>.

Next, step <NUM> includes retrieving, from a database, a stored texture profile associated with the detected jurisdiction class of the ID. In an embodiment, the stored template corresponds to a previously generated global texture profile for the detected jurisdiction class. The previously generated global texture profile for a jurisdiction class may be generated based on a gallery image for the jurisdiction class.

Subsequently, step <NUM> includes comparing the global texture profile to a stored texture profile associated with the jurisdiction class of the identity document. In an embodiment, the comparison is made PII insensitive by removing the PII from the digital images before generating the global texture profile and/or the stored texture profile. In another embodiment, the comparison is made less sensitive to ambient illumination variations between the digital image and the gallery image by the choice of descriptors that form the texture profile. In an embodiment, this is made possible by ensuring that the descriptors used in the texture profile consist of only relative information, such as differences (or gradients), correlation coefficients, and statistics of intensity variations and gradients.

If, in step <NUM>, the global texture profile does not match the stored texture profile, process <NUM> proceeds to step <NUM>, which includes identifying the presence of a forgery in the digital image. For example, step <NUM> may include identifying the digital image as a recaptured image from a photocopy or a computer screen. Alternatively or additionally, step <NUM> may include identifying the digital image as having an incorrect background layout for the jurisdiction class of the identity document.

Otherwise, if, in step <NUM>, the global texture profile matches the stored texture profile, process <NUM> transitions to step <NUM>, which includes generating the one or more local texture patches based on the digital image. The one or more local texture patches may be generated as described above with reference to <FIG> and <FIG> based on one or more target manipulation spaces. In an embodiment, the target manipulation spaces may include physical portrait photo substitution, digital image splicing, inpainting, resampling, photocopy recapture, or screen recapture.

Process <NUM> then proceeds to step <NUM>, which includes detecting the presence of a forgery based on the one or more local texture patches. In an embodiment, the detection may be performed as described above using local detectors each configured and trained to detect a specific type of forgery.

<FIG> illustrates a computer system <NUM> which may be used to implement embodiments of the present invention. According to an embodiment, the above-described authentication system <NUM> may be implemented using computer system <NUM>.

As shown in <FIG>, computer system <NUM> includes a processor <NUM> and a memory <NUM>. A computer program (PROG) may be stored on memory <NUM>. The computer program may include instructions that, when executed by the processor <NUM>, cause the processor <NUM> to execute a method for authenticating a digital image representative of an identity document according to any of the embodiments described herein.

The foregoing description of the specific embodiments will so fully reveal the general nature of the disclosure that others can, by applying knowledge within the skill of the art, readily modify and/or adapt for various applications such specific embodiments, without undue experimentation, without departing from the general concept of the present disclosure. Therefore, such adaptations and modifications are intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. It is to be understood that the phraseology or terminology herein is for the purpose of description and not of limitation, such that the terminology or phraseology of the present specification is to be interpreted by the skilled artisan in light of the teachings and guidance.

Claim 1:
A method for authenticating a digital image representative of an identity document (ID), comprising:
generating, based on the digital image, at least two local texture patches representative of texture information of at least two select local blocks of the ID; wherein generating the at least two local texture patches comprises:
detecting, for a target manipulation space, at least two features of interest of the ID;
defining the at least two select local blocks of the ID based on the detected at least two features of interest; and
extracting, based on the digital image, texture information at each of the at least two select local blocks to generate the at least two local texture patches and
detecting the presence of a forgery in the digital image based on the at least two local texture patches,
characterized in that the target manipulation space is physical portrait photo substitution, wherein the at least two features of interest comprise a portrait photo of the ID, and wherein the at least two select local blocks comprise at least two blocks straddling a boundary of the portrait photo.