Patent Publication Number: US-11663489-B2

Title: Machine learning systems and methods for improved localization of image forgery

Description:
RELATED APPLICATIONS 
     The present application claims the benefit of U.S. Provisional Application Ser. No. 62/865,414 filed on Jun. 24, 2019, the entire disclosure of which is expressly incorporated herein by reference. 
    
    
     BACKGROUND 
     Technical Field 
     The present disclosure relates generally to the field of machine learning. Specifically, the present disclosure relates to machine learning systems and methods for improved localization of image forgery. 
     RELATED ART 
     Photo-realistically altering the contents of digital images and videos is problematic as society becomes increasingly reliant on digital images and videos as dependable sources of information. Altering image contents is facilitated by the availability of image editing software and aggravated by recent advances in deep generative models such as generative adversarial networks (GAN). Digital image forensics focuses on this issue by addressing critical problems such as establishing the veracity of a digital image (e.g., manipulation detection), localizing a tampered region within the image (e.g., manipulation localization), and identifying an alteration type. Different alteration types require different forensic techniques. One type of alteration includes introducing foreign material into an image. For example, splicing can be utilized to insert a part of one image into another image or inpainting can be utilized to insert an object into an image via a specialized algorithm. Semantic information has had limited success in solving such operations because skilled attackers utilize semantic structures to hide image alterations. Non-semantic pixel-level statistics have proven more successful since these statistics amplify low-level camera model specific distortions and noise patterns (i.e., a camera model&#39;s digital fingerprint). A camera model digital fingerprint can aid in resolving an integrity of an image by determining whether the camera model fingerprint is consistent across an entirety of the image. Several hand-engineered, low-level statistical approaches have been explored. However, given the aforementioned availability of image editing software and the technological improvement of recent deep generative models, there is a need for forensic algorithms that can provide data-driven deep learning solutions for the localization of image forgery. 
     Therefore, there is a need for machine learning systems and methods which can improve the localization of image forgery while improving an ability of computer systems to more efficiently process data. These and other needs are addressed by the machine learning systems and methods of the present disclosure. 
     SUMMARY 
     The present disclosure relates to machine learning systems and methods for improved localization of image forgery. The system generates a variational information bottleneck objective function and works with input image patches to implement an encoder-decoder architecture. The encoder-decoder architecture controls information flow between the input image patches and a representation layer. The system utilizes information bottleneck to learn useful noise-residual patterns and discard semantic content present in each input image patch. In particular, the system extracts noise-residual patterns by considering learned local noise models and learns a suitable representation (e.g., a statistical fingerprint of a source camera model of each input image patch) from the extracted noise-residual patterns. The system trains a neural network to learn the representation indicative of the statistical fingerprint of the source camera model of each input image patch while excluding the semantic content thereof. The system determines a splicing manipulation localization by the trained neural network. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The foregoing features of the invention will be apparent from the following Detailed Description of the Invention, taken in connection with the accompanying drawings, in which: 
         FIG.  1    is a diagram illustrating the system of the present disclosure; 
         FIG.  2    is a flowchart illustrating overall processing steps carried out by the system of the present disclosure; 
         FIG.  3    is a flowchart illustrating step  54  of  FIG.  2    in greater detail; 
         FIG.  4    is a flowchart illustrating step  72  of  FIG.  3    in greater detail; 
         FIG.  5    is a table illustrating a neural network architecture of the system of the present disclosure; 
         FIG.  6    is a flowchart illustrating step  56  of  FIG.  2    in greater detail; 
         FIGS.  7 A-B  are graphs illustrating a range of regularization parameter values of an RD curve and a performance of selected regularization parameter values; 
         FIGS.  8 A-B  are tables illustrating quantitative processing results of the system of the present disclosure on different datasets and according to different scoring metrics; 
         FIG.  9    is a compilation of images illustrating qualitative processing results of the system of the present disclosure in comparison to a variety of models based on several input images and ground truth masks thereof; 
         FIG.  10    is a compilation of images illustrating qualitative processing results of the system of the present disclosure based on a detection of a variety of known generative adversarial network signatures present in input images; 
         FIG.  11    is a diagram illustrating hardware and software components capable of being utilized to implement the system of the present disclosure; and 
         FIG.  12    is a diagram illustrating the neural network  16  of the present disclosure, in greater detail. 
     
    
    
     DETAILED DESCRIPTION 
     The present disclosure relates to systems and methods for improved localization of image forgery, as described in detail below in connection with  FIGS.  1 - 12   . 
     By way of background, the image formation process broadly consists of three stages: (1) sensor measurements; (2) in-camera processing; and (3) storage which may include compression. The image formation process is unique for every camera model and yields subtle distortions and noise patterns (i.e., a digital camera model fingerprint) in the image which are invisible to the eye. These subtle distortions and noise patterns are useful in forensic applications because they are specific to each camera model. Sensor pattern noise originates from imperfections in the sensor itself and has shown to be sensitive to several manipulation types. Accordingly, sensor pattern noise has been utilized for the detection and localization of forgeries. However, sensor pattern noise is difficult to detect in image regions with high texture and is absent or suppressed in saturated and dark regions of an image. Color filter array (CFA) demosaicking is an in-camera processing step that produces pixel colors. Different detection and localization strategies based on CFA signature inconsistencies are known. However, the scope of such specialized CFA models is often limited. Joint Photographic Experts Group (JPEG) is a common storage form and carries camera model signatures such as dimples or can contain clues regarding post-processing steps such as traces of multiple compressions. Although, JPEG statistics have been utilized for detection and localization tasks, these statistics are format specific and do not generalize to other common or new formats. 
     More generic approaches include modeling noise-residuals. A noise-residual is a statistical pattern that is not attached to a specific source but is instead a result of the combined processes of an imaging pipeline. Noise-residuals can be discerned by suppressing the semantic contents of an image. For example, one known approach utilizes a wavelet transform as a high-pass filter to estimate noise-residuals and then determine its inconsistencies. Other known approaches utilize spatial rich filters (RFs). Spatial RFs are a set of alternate high-pass filters to model local noise-residuals. For example, one known approach explores co-occurrences of one RF while another known approach utilizes three residual filters along with color information in a convolutional neural network (CNN) to localize manipulations. Learned RFs utilizing constrained convolutions have also been employed for localizing manipulations. Noiseprint, another known approach, utilizes a denoising CNN to estimate properties of noise-residuals and changes therein to discover manipulations. Additionally, another known approach, utilizes a CNN trained for camera model identification to discover manipulations but does not exploit noise-residuals and relies on a CNN architecture to learn semantic contents. 
     The system of the present disclosure utilizes a constrained convolution layer to mimic RFs and information bottleneck to learn noise-residual patterns and localize manipulations. The system of the present disclosure demonstrates that information bottleneck provides for a formal framework to interpret mutual information-based regularization. This interpretation provides for a more efficient solution utilizing variational approximation and provides for tuning the regularization in a principled manner to enhance forensic performance. 
     Information theory is a framework that provides for improving various aspects of deep machine learning including, but not limited to, representation learning, generalizability and regularization, and an interpretation of deep neural network functionality. Mutual information plays an important role in many of these methods. For example, InfoGAN has shown that maximizing mutual information between latent code and a generator&#39;s output improves the representations learned by a GAN thereby providing for the representations to be more disentangled and interpretable. Mutual information is challenging to compute and therefore InfoGAN maximizes a variational lower bound. A similar maximization approach has been explored to improve unsupervised representation learning utilizing a numerical estimator. 
     Information bottleneck curbs information flow between an input and a representation layer. This curbing of information flow encourages a model to learn task related features and helps improve its generalization ability. It should be understood that information bottleneck Lagrangian is challenging to solve in practice. Accordingly, variational approximations suitable for deep learning have been proposed and demonstrate that information bottleneck is closely related to variational autoencoders (VAEs). Information bottleneck can be utilized to learn disentangled representations. For example, information bottleneck approaches to improve the disentanglement of representations learned by VAEs have been investigated empirically. Additionally, a known insightful rate-distortion interpretation utilizing information bottleneck has been applied to VAEs. Information bottleneck has also been proposed as an effective regularization and shown to improve imitation learning, reinforcement learning, and the training of GANs. The system of the present disclosure leverages a variational information bottleneck formulation developed for deep neural networks utilizing a reparameterization process. 
     The information bottleneck framework and its variational approximation will now be described. Learning a predictive model p(y|x) is hindered when a model overfits nuisance detractors that exist in the input data X instead of focusing on relevant information for a task Y. This is of importance in deep learning when an input is high-dimensional (e.g., an image), a task is a simple low-dimensional class label, and a model is a flexible neural network. An objective of information bottleneck is to overcome this problem by learning a compressed representation Z, of X, which is optimal for the task Y in terms of mutual information. It is applied by maximizing the information bottleneck Lagrangian based on mutual information values I(Z, X) and I(Z, Y) as follows in Equation 1:
 
 = I ( Z,Y )−β I ( Z,X )   Equation 1
 
     By penalizing an information flow between X and Z while maximizing the mutual information required for the task, information bottleneck extracts the relevant information that X contains regarding Y and discards non-informative signals. This provides for learning a representation Z with an improved generalization ability. 
     As mentioned above, it should be understood that mutual information is challenging to compute in a general setting and even more so with high dimensional variables. As such, a known approach applies a variational approximation to a neural network. In particular, let Z be a stochastic encoding layer and based on the definition of mutual information, Equation 2 yields: 
     
       
         
           
             
               
                 
                   
                     I 
                     ⁡ 
                     
                       ( 
                       
                         Z 
                         , 
                         Y 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       ∫ 
                       
                         
                           p 
                           ⁡ 
                           
                             ( 
                             
                               y 
                               , 
                               z 
                             
                             ) 
                           
                         
                         ⁢ 
                         log 
                         ⁢ 
                         
                           
                             p 
                             ⁡ 
                             
                               ( 
                               
                                 y 
                                 , 
                                 z 
                               
                               ) 
                             
                           
                           
                             
                               p 
                               ⁡ 
                               
                                 ( 
                                 y 
                                 ) 
                               
                             
                             ⁢ 
                             
                               p 
                               ⁡ 
                               
                                 ( 
                                 z 
                                 ) 
                               
                             
                           
                         
                         ⁢ 
                         dydz 
                       
                     
                     = 
                     
                       
                         ∫ 
                         
                           
                             p 
                             ⁡ 
                             
                               ( 
                               
                                 y 
                                 , 
                                 z 
                               
                               ) 
                             
                           
                           ⁢ 
                           log 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             p 
                             ⁡ 
                             
                               ( 
                               
                                 y 
                                 | 
                                 z 
                               
                               ) 
                             
                           
                           ⁢ 
                           dydz 
                         
                       
                       - 
                       
                         ∫ 
                         
                           
                             p 
                             ⁡ 
                             
                               ( 
                               y 
                               ) 
                             
                           
                           ⁢ 
                           log 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             p 
                             ⁡ 
                             
                               ( 
                               y 
                               ) 
                             
                           
                           ⁢ 
                           dy 
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   2 
                 
               
             
           
         
       
     
     In Equation 2, the last term is ignored as it is the entropy of y and is constant. The first term p(y|z) is intractable and is approximated utilizing a variational distribution q(y|z), the decoder network. Then, a lower bound of I(Z, Y) is determined because the KL divergence KL[p(y|z)∥q(y|z)]≥0⇒∫p(y|z)log p(y|z)dy≥∫p(y|z)log q(y|z)dy and by assuming a Markov chain relation Y→X→Z, yields Equation 3:
 
 I ( Z,Y )≥   x,y˜p(x,y)     z˜p(z|x) [log  q ( y|z )]]   Equation 3
 
     where p(z|x) is an encoder network and p(x, y) can be approximated utilizing the training data distribution. Therefore, the right hand side of Equation 3 becomes the average cross-entropy (with stochastic sampling over z). Proceeding similarly, Equation 4 yields: 
     
       
         
           
             
               
                 
                   
                     I 
                     ⁡ 
                     
                       ( 
                       
                         Z 
                         , 
                         X 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       ∫ 
                       
                         
                           p 
                           ⁡ 
                           
                             ( 
                             
                               x 
                               , 
                               z 
                             
                             ) 
                           
                         
                         ⁢ 
                         log 
                         ⁢ 
                         
                           
                             p 
                             ⁡ 
                             
                               ( 
                               
                                 x 
                                 , 
                                 z 
                               
                               ) 
                             
                           
                           
                             
                               p 
                               ⁡ 
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                             ⁢ 
                             
                               p 
                               ⁡ 
                               
                                 ( 
                                 z 
                                 ) 
                               
                             
                           
                         
                         ⁢ 
                         dxdz 
                       
                     
                     = 
                     
                       
                         ∫ 
                         
                           
                             p 
                             ⁡ 
                             
                               ( 
                               
                                 x 
                                 , 
                                 z 
                               
                               ) 
                             
                           
                           ⁢ 
                           log 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             p 
                             ⁡ 
                             
                               ( 
                               
                                 z 
                                 | 
                                 x 
                               
                               ) 
                             
                           
                           ⁢ 
                           dxdz 
                         
                       
                       - 
                       
                         ∫ 
                         
                           
                             p 
                             ⁡ 
                             
                               ( 
                               z 
                               ) 
                             
                           
                           ⁢ 
                           log 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             p 
                             ⁡ 
                             
                               ( 
                               z 
                               ) 
                             
                           
                           ⁢ 
                           dz 
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   4 
                 
               
             
           
         
       
     
     In this case, p(z) is intractable and is approximated by a prior marginal distribution r(z). An upper bound for I(Z, X) is determined because KL[p(z)∥r(z)]≥0⇒∫p(z) log p(z)dz≥∫p(z)log r(z)dz, therefore Equation 5 yields: 
     
       
         
           
             
               
                 
                   
                     
                       I 
                       ⁡ 
                       
                         ( 
                         
                           Z 
                           , 
                           X 
                         
                         ) 
                       
                     
                     ≤ 
                     
                       ∫ 
                       
                         
                           p 
                           ⁡ 
                           
                             ( 
                             x 
                             ) 
                           
                         
                         ⁢ 
                         
                           p 
                           ⁡ 
                           
                             ( 
                             
                               z 
                               | 
                               x 
                             
                             ) 
                           
                         
                         ⁢ 
                         log 
                         ⁢ 
                         
                           
                             p 
                             ⁡ 
                             
                               ( 
                               
                                 z 
                                 | 
                                 x 
                               
                               ) 
                             
                           
                           
                             r 
                             ⁡ 
                             
                               ( 
                               z 
                               ) 
                             
                           
                         
                         ⁢ 
                         dxdz 
                       
                     
                   
                   = 
                   
                     
                       𝔼 
                       
                         x 
                         ∼ 
                         
                           p 
                           ⁡ 
                           
                             ( 
                             x 
                             ) 
                           
                         
                       
                     
                     ⁡ 
                     
                       [ 
                       
                         KL 
                         ⁡ 
                         
                           [ 
                           
                             
                               p 
                               ⁡ 
                               
                                 ( 
                                 
                                   z 
                                   | 
                                   x 
                                 
                                 ) 
                               
                             
                             ⁢ 
                             
                                
                                
                             
                             ⁢ 
                             
                               r 
                               ⁡ 
                               
                                 ( 
                                 z 
                                 ) 
                               
                             
                           
                           ] 
                         
                       
                       ] 
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   5 
                 
               
             
           
         
       
     
     p(x) can be approximated utilizing the data distribution. Replacing Equations 3 and 5 in Equation 1 yields the variational information bottleneck function as shown in Equation 6: 
     
       
         
           
             
               
                 
                   
                     
                       J 
                       IB 
                     
                     ⁡ 
                     
                       ( 
                       
                         p 
                         , 
                         q 
                       
                       ) 
                     
                   
                   = 
                   
                     
                       
                         
                           1 
                           N 
                         
                         ⁢ 
                         
                           
                             ∑ 
                             
                               i 
                               = 
                               1 
                             
                             N 
                           
                           ⁢ 
                           
                             
                               𝔼 
                               
                                 z 
                                 ∼ 
                                 
                                   p 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       z 
                                       | 
                                       
                                         x 
                                         i 
                                       
                                     
                                     ) 
                                   
                                 
                               
                             
                             ⁡ 
                             
                               [ 
                               
                                 
                                   - 
                                   log 
                                 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 
                                   q 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       
                                         y 
                                         i 
                                       
                                       | 
                                       z 
                                     
                                     ) 
                                   
                                 
                               
                               ] 
                             
                           
                         
                       
                       + 
                       
                         β 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           KL 
                           ⁡ 
                           
                             [ 
                             
                               
                                 p 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     z 
                                     | 
                                     
                                       x 
                                       i 
                                     
                                   
                                   ) 
                                 
                               
                               ⁢ 
                               
                                  
                                  
                               
                               ⁢ 
                               
                                 r 
                                 ⁡ 
                                 
                                   ( 
                                   z 
                                   ) 
                                 
                               
                             
                             ] 
                           
                         
                       
                     
                     ≥ 
                     
                       - 
                       ℒ 
                     
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   6 
                 
               
             
           
         
       
     
     The variational information bottleneck function can be minimized utilizing a known reparameterization process. According to a rate-distortion interpretation of information bottleneck, the loss term is denoted as distortion D and approximates the non-constant part of −I(Z, Y) while the unweighted regularization term is denoted as rate R and approximates I(Z, X). R measures an excess number of bits required to encode representations. The RD-plane provides for visualizing a family of solutions to the information bottleneck Lagrangian for different values of β and provides insight into properties of the encoder-decoder network. 
     The system of the present disclosure allows for localization of a digital manipulation that inserts foreign material into a host image to alter its contents (e.g., a splicing operation). Since a splicing operation is often camouflaged by semantic structures, such manipulations can be localized by inspecting low-level pixel statistics. Generally, a splicing operation will contain a different statistical profile (e.g., fingerprint) than the host image because the splicing operation likely originates from a different camera model or a different image formation process (e.g., inpainting). The system of the present disclosure allows for localization of an image manipulation by utilizing an information bottleneck-based loss to learn to ignore semantic content of an image. In particular, the system trains a deep neural network to learn a representation that captures a statistical fingerprint of a source camera model from an input image patch while ignoring the semantic content of the input image patch. Then, the system computes the fingerprint representation for different parts of a test image. Lastly, the system searches for inconsistencies among the computed fingerprint representations to localize splicing manipulations. It should be understood that the system trains the neural network with a large number of camera models to improve the ability of the system to distinguish even unseen camera models. Accordingly, the network can be effective in a blind test context when applied on images acquired from unknown camera models. 
     Turning to the drawings,  FIG.  1    is a diagram illustrating the system  10  of the present disclosure. The system  10  includes a neural network  16  having an information bottleneck function generation module  14  that receives input data  12 . The neural network  16  can receive training input data  20  and validation input data  24 . The neural network  16  further includes a model training system  18  and a trained model system  22 , and outputs output data  26 . The neural network  16  can be any type of neural network or machine learning system, or combination thereof. For example, the neural network  16  can be a deep neural network capable of, for example, image forgery localization and can use one or more frameworks (e.g., interfaces, libraries, tools, etc.). 
       FIG.  2    is a flowchart  50  illustrating overall processing steps carried out by the system  10  of the present disclosure. Beginning in step  52 , the information bottleneck function generation module  14  generates a variational information bottleneck objective function. The information bottleneck function generation module  14  receives input data  12 . Utilizing information bottleneck provides for framing the issue of localizing manipulations in an image as a deep representation learning problem and provides a framework for interpreting a deep neural network. In particular, utilizing information bottleneck casts the problem of modelling and distinguishing low-level camera model statistics as a data-driven representation learning problem. The system  10  works with input image patches to implement an encoder-decoder architecture. The encoder-decoder architecture controls an information flow between the input image patches and a representation layer. This constriction of mutual information allows the neural network  16  to ignore irrelevant semantic content contained in any image patch and focus its capacity on learning useful features required to classify a source camera model. As such, the system  10  utilizes information bottleneck to learn useful residual noise patterns and ignore semantic content. As mentioned above, a learned noise pattern representation can be considered as a camera model signature (i.e., fingerprint). 
     In step  54 , the model training system  18  trains the neural network  16  utilizing the information bottleneck function on training input data  20 . In particular, the model training system  18  trains the neural network  16  to learn a representation indicative of a statistical fingerprint of a source camera model from an input image patch while excluding semantic content thereof. The training input data  20  can include, but is not limited to, a predetermined number of images of the Dresden Image Database that contains more than 17,000 JPEG images from 27 source camera models. It should be understood that the neural network  16  can be any type of neural network or machine learning system, or combination thereof, utilizing the information bottleneck function. Additionally, it should be understood that the system  10  may utilize a different neural network  16  based on the training input data  20 . For example, the system  10  may utilize a model with mutual information regularization and a model without mutual information regularization when training on the Dresden Image Database. Then, in step  56 , the trained model system  22  processes validation input data  24  to determine whether the system  10  can localize an image manipulation. The validation input data  24  can include, but is not limited to, a predetermined number of images of the DSO-1, Nimble Challenge 2016 (NC16) and the Nimble Challenge 2017 (NC17-dev1) datasets. 
       FIG.  3    is a flowchart illustrating step  54  of  FIG.  2    in greater detail. The system  10  learns a low level representation by extracting noise-residuals and learning suitable representations from the extracted noise-residuals. Beginning in step  70 , the system  10  extracts noise-residuals by considering learned local noise models. In particular, the system  10  considers a constrained convolution layer according to Equation 7 as follows: 
     
       
         
           
             
               
                 
                   
                     ℛ 
                     
                       ( 
                       k 
                       ) 
                     
                   
                   = 
                   
                     
                       
                         
                           w 
                           k 
                         
                         ⁡ 
                         
                           ( 
                           
                             0 
                             , 
                             0 
                           
                           ) 
                         
                       
                       + 
                       
                         
                           ∑ 
                           
                             i 
                             , 
                             
                               j 
                               ≠ 
                               0 
                             
                             , 
                             0 
                           
                         
                         ⁢ 
                         
                           
                             w 
                             k 
                           
                           ⁡ 
                           
                             ( 
                             
                               i 
                               , 
                               j 
                             
                             ) 
                           
                         
                       
                     
                     = 
                     0 
                   
                 
               
               
                 
                   Equation 
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   7 
                 
               
             
           
         
       
     
     The constrained convolution layer binds the weights of the k th  filter to determine a mismatch or noise-residual, between a pixel&#39;s value at position (0, 0) and its value as interpolated from its S×S neighbors. These are high-pass filters similar to RFs that model noise-residuals locally by suppressing semantic content and can be trained end-to-end by including a penalty  =(Σ k (   (k) ) 2 ) 1/2  in the optimization. 
     It should be understood that since these noise models are high-pass filters, the models also capture high-frequency semantic content such as edges and textures which carry scene related information the system  10  seeks to suppress. Ideal noise-residuals are considered to be high-frequency content uncorrelated to semantic information. It is possible to learn these noise-residuals by regularizing mutual information between an input and a feature layer in a neural network. Intuitively, this would discourage a correlation between learned features and semantic content in the input. However, and as mentioned above, mutual information is challenging to compute. Accordingly, the system  10  re-interprets the mutual information regularization through the information bottleneck framework. As such, the system  10  can employ an efficient variational solution and explore longer training processes and provide an RD-plane that can be evaluated to select the best regularization parameter β of Equation 6. In step  72 , the system  10  learns suitable representations from the extracted noise-residuals. Step  72  will be described in further detail below in connection with  FIG.  4   . 
       FIG.  4    is a flowchart illustrating step  72  of  FIG.  3    in greater detail. Beginning in step  90 , the system  10  implements a stochastic encoder-decoder architecture to learn suitable representations utilizing information bottleneck. As the input, the system  10  considers X to be an image patch, Y to be a class label for a task of camera model identification and Z to be a stochastic encoding layer. The system  10  selects a classification task to learn non-semantic features from the input image patch since the semantic content of an image is not relevant to the classification of a camera model. Additionally, the system  10  can exploit large camera tagged untampered image databases for training which provides for avoiding specialized manipulated datasets and averting the chances of overfitting to the specialized manipulated datasets. As implemented, the system  10  can train the neural network  16  by minimizing the variational IB objective function of Equation 6. 
     In step  92 , the system  10  configures the encoder p(z|x). The system  10  can utilize an architecture inspired by residual network  18  (ResNet-18) version-1 including an initial constrained convolution layer (as shown in Equation 7) to model noise-residuals and discard operations that quickly shrink the input and encourage learning high level (i.e., semantic) features. Namely, the system  10  discards an initial max-pooling layer, convolutions with a stride greater than one, and a final global average pooling layer. The system  10  also inserts additional 7×7 and 5×5 convolutions to end the network with a single “feature-pixel” with a large bank of filters to avoid fully connected layers. 
       FIG.  5    is a table  100  illustrating the architecture of the neural network  16  of the system  10  of the present disclosure. In particular, the neural network  16  is a CNN having 27 layers where every convolution is followed by batch normalization and ReLU activation. The convolutions have a stride of 1. The input patch size is 49×49×3 and the output encoding size is 1×1×72. To yield a stochastic encoding Z, the system  10  splits the CNN&#39;s output vector of 72 filters into μ x  and σ x  and model p(z|x)=N(μ x , diag(σ x )). The neural network  16  also includes a decoder, discussed in greater detail below in connection with  FIG.  12   . 
     Returning to  FIG.  4   , in step  94  the system  10  configures the decoder q(y|z). The decoder deters the CNN from degenerating to the autodecoder limit, an issue faced by VAEs with powerful decoders. As such, the system  10  configures the decoder with a simple logistic regression model having a dense (log it generating) layer that is connected to the stochastic code layer Z and is activated by the softmax function. 
     Then, in step  96 , the system  10  determines the regularization parameter β. The system  10  utilizes the RD-plane to determine the characteristics of the encoder-decoder. It should be understood that an RD curve divides the plane into practical feasible and infeasible regions. Evaluating the RD curve provides for selecting a regularization parameter β to balance a trade-off between the distortion which affects task accuracy and the rate which affects compression and hence the generalization capacity. In addition to a primary task of localizing splicing manipulations, the system  10  also provides for training the neural network  16  on a secondary task of camera model identification. As such, the system  10  employs the RD curve of the training task to identify a potential range for the regularization parameter β and then selects optimal value(s) of the regularization parameter β from this range through empirical testing. 
       FIG.  6    is a flowchart illustrating step  56  of  FIG.  2    in greater detail. The system  10  simplifies localizing a splicing manipulation into a two-class feature segmentation problem based on the assumption that the untampered region of a test image is the largest part of the test image. Beginning in step  120 , the system  10  determines the neural network&#39;s  16  representation (μ, σ) as a predictive signature of a camera model for juxtaposed patches in the test image. Then, in step  122 , the system  10  segments the predictive signature via a Gaussian mixture model with two components utilizing expectation maximization (EM). This segmentation step  122  is described in greater detail below in connection with  FIG.  12   , and it is noted that the Gaussian mixture model is not a part of the neural network  16  described herein. It should be understood that the Gaussian distributions are approximate statistics of the features of the two classes and aid in separating the classes probabilistically. It should also be understood that the system  10  does not perform forgery detection since the system  10  identifies two classes. 
     Training and testing of the neural network  16  of the system  10  will now be described. The system  10  evaluates input patches of size 49×49×3 and k=64 constrained convolutions with support S=3. The encoder has a fixed number of 64 filters in every layer. Additionally, for the variational prior distribution, the system  10  utilizes a factorized standard Gaussian r(z)=Π i N i (0, 1) and trains the neural network  16  utilizing the loss of Equation 8 as follows:
 
 J=J   IB +λ +ω 1   ∥W∥   1 +ω 2   ∥W∥   2    Equation 8
 
     In Equation 8, W denotes all weights of the neural network  16  and the system  10  empirically selects λ=1 and ω 1 =ω 2 =1e−4. As mentioned above, the system  10  can utilize the Dresden Image Database as the training input data  20 . The Dresden Image Database consists of more than 17,000 JPEG images corresponding to 27 source camera models. For each camera model, the system  10  randomly selects 70% of the images for training, 20% for validation and 10% for testing. The system  10  trains with a mini-batch of 200 patches for 700 epochs with 100,000 randomly selected patches in every epoch. The system  10  maintains a constant learning rate of 1e-4 for 100 epochs which then linearly decays to 5e-6 in the next 530 epochs and finally exponentially decays by a factor 0.9 over the last 70 epochs. As such, the system  10  provides for a camera model prediction accuracy of ˜80% on the test and validation sets for various values of the regularization parameter β. 
     The system  10  was implemented with TensorFlow and trained on a NVIDIA Tesla V100-SXM2 (16 GB) GPU with 32 CPUs and 240 GB RAM. It should be understood that the system  10  can be implemented utilizing other software and hardware configurations. For comparison, a deep network having 18 layers with 64 filters (instead of 19) was also trained. This network trained with 72×72×3 input patches for the same number of epochs but with a decreased batch size of 100. Additionally, the system  10  trains two neural network models. In particular, the system  10  trains a neural network model with mutual information regularization (MI) and a neural network model without mutual information regularization (NoMI). Training the variational model of the system  10  required 14 hours whereas training the MI model required eight days thereby highlighting the efficiency of the variational solution in contrast to the numerically expensive binning method (i.e., the MI model). 
     The processing results of the system  10  will now be described. The system  10  can be tuned by evaluating the RD curve and selecting an optimal regularization parameter β. It is noted that an ablation study is carried out to gauge a relevance of information bottleneck. The system  10  is tested on three standard manipulated datasets and scores are generated via three distinct metrics. The manipulation datasets include, but are not limited to, a predetermined number of images of the DSO-1, NC16, and the NC17-dev1 datasets. The DSO-1 dataset consists of 100 spliced images in Portable Network Graphics (PNG) format such that the tampered regions are relatively large but well camouflaged by the semantic content of each image. The NC16 dataset consists of 564 spliced images mostly in JPEG format. The NC17-dev-1 dataset consists of 1,191 images having different types of manipulations. Of these images, only spliced images are selected thereby yielding 237 images. The NC16 and NC17-dev-1 images contain a series of manipulations, some of which are complex operations that attempt to erase traces of manipulations. Furthermore, the tampered regions are often small. Each of the DSO-1, NC16 and NC17-dev1 datasets contain difficult to detect manipulations and are accompanied by ground truth manipulation masks. Additionally, manipulations created by three well known inpainting GANs are generated. 
     Performance of the system  10  was evaluated via three metrics including the F1 score, the Matthews Correlation Coefficient (MCC) and an area under the receiver operating characteristic curve (ROC-AUC). These metrics are known for evaluating a splicing manipulation localization. F1 and MCC require a binarized forgery prediction mask while the system  10  predicts probabilities from the EM segmentation. It is customary to generate and report scores for optimal thresholds computed from the ground truth masks. As such, scores from automatic thresholding utilizing a known method (e.g., Otsu&#39;s method) are generated and reported. 
     For comparison with the system  10 , two ablated models are considered including the neural network model with mutual information regularization (MI) and the neural network model without mutual information regularization (NoMI) as described earlier. An optimal regularization parameter β is selected in addition to a variational model with no information bottleneck regularization (β=0). These models and regularization values aid in gauging the importance of information regularization and provide a comparison of the efficient variational approach of the system  10  and the expensive numerical binning approach (i.e., the MI model). Additionally, other models are considered including the SpliceBuster (SB) which is a state-of-the-art splice localization algorithm and top performer of the NC17 challenge and the EX-SC which is a deep learning based algorithm that predicts meta data self-consistency to localize tampered regions. 
       FIGS.  7 A-B  are graphs illustrating a range of regularization parameter values of an RD curve and a performance of selected regularization parameter values. In particular,  FIG.  7 A  is a graph  140  illustrating a plot of an RD curve  142  from which regularization parameter values β can be selected. As shown in  FIG.  7 A , the system  10  yields low distortion values for β≤5e-3 for the training task.  FIG.  7 B  is a graph  150  illustrating F1 scores on the DSO-1 dataset. In selecting a regularization parameter β for the forensic task, F1 scores are determined on the DSO-1 dataset for all values of β until 0. As shown in  FIG.  7 B , a peak in the F1 scores is evident from 2e-3 to 1e-4 (1e−3 is an anomaly that can be attributed to stochastic training). As such, testing is carried out for the central regularization parameter values of β=1e−3 and 5e−4. 
       FIGS.  8 A- 8 B  are tables illustrating quantitative processing results of the system  10  on different datasets and according to different scoring metrics. In particular,  FIG.  8 A  is a table  160  illustrating quantitative results for splicing manipulation localization of the system  10  and other models on the DSO-1, NC16, and NC17-dev1 datasets according to the F1 and MCC scoring metrics. Scores listed in each of the left columns are indicative of optimal thresholds and scores listed in each of the right columns are indicative of automatic thresholding utilizing a known approach (e.g., Otsu&#39;s method). As shown in  FIG.  8 A , the F1 scores indicate up to a 6% point improvement over SB and a 15% point improvement over EX-SC on the DSO-1 dataset and the best scores on the NC16 and NC17-dev1 datasets. The system  10  IP1e-3 and IP5e-4 model MCC scores are also high in comparison to the other models with a margin of up to 8% points on the DSO-1 dataset in comparison to SB.  FIG.  8 B  is a table  170  illustrating quantitative results for splicing manipulation localization of the system  10  and other models on the DSO-1, NC16, and NC17-dev1 datasets according to the AUC scoring metric. As shown in  FIG.  8 B , a comparison of the ablated models NoMI and MI evidences that information regularization improves forensic performance. Additionally,  FIG.  8 B  illustrates that the variational approach of the system  10  (e.g., the IP1e-3 and IP5e-4 models) outperforms the numerically expensive MI model. 
       FIG.  9    is a diagram  180  of a compilation of images illustrating qualitative processing results of the system  10  in comparison to the MI, EX-SC and SB models based on several input images  182   a - f  and ground truth masks  184   a - f  thereof. The input images  182   a - f  are sourced from the DSO-1, NC16 and NC17-dev1 datasets. As shown in  FIG.  9   , the system  10  images  186   a - f  closely resemble the ground truth images  184   a - f  of the input images  182   a - f . Further, the system  10  images  186   a - f  more closely resemble the ground truth images  184   a - f  than the MI model images  188   a - f , the EX-SC model images  190   a - f  and the SB model images a-f. 
       FIG.  10    is a compilation of images  200  illustrating qualitative processing results of the system  10  based on a detection of a variety of known GAN signatures present in input images  210   a - f . As shown in  FIG.  10   , the system  10  detects the signatures of the inpainting GANs in the input images  210   a - f . It should be understood that most of the input images  210   a - f  are processed (e.g., resized or compressed) which destroys camera model traces but nevertheless the system  10  is able to detect the splicing manipulations present in the input images  210   a - f . Accordingly, this indicates that synthetically generated pixels carry a different low level signature that the system  10  can detect and exploit. 
     The system  10  utilizes an information bottleneck formulation that converts a classical feature modelling problem for identifying camera models into a deep representation learning problem. This is a unique application of information bottleneck to a growing real-world problem with serious consequences. The application of information bottleneck via the system  10  is also unique in that it encourages learning low level noise patterns rather than semantic information which is contrary to the conventional application of information bottleneck. A comparison of the system  10  with the expensive number estimation method (i.e., the MI model) evidences that the computationally efficient approximated solution based on variational information bottleneck of the system  10  outperforms the MI model. As such, the representation learning problem can be solved numerically or approximated via the variational inference where the latter outperforms the former in regards to the task of splicing manipulation localization. Additionally, the system  10  outperforms several state of the art models on a suite of standard test datasets and can detect the signatures of deep generative models (e.g., inpainting GANs). 
       FIG.  11    is a diagram  300  showing hardware and software components of a computer system  302  (i.e. a server) on which the system  10  of the present disclosure can be implemented. The computer system  302  can include a storage device  304 , computer software code  306 , a network interface  308 , a communications bus  310 , a central processing unit (CPU) (microprocessor)  312 , a random access memory (RAM)  314 , and one or more input devices  316 , such as a keyboard, mouse, etc. It is noted that the CPU  312  could be one or more graphics processing units (GPUs), if desired. The server  302  could also include a display (e.g., liquid crystal display (LCD), cathode ray tube (CRT), etc.). The storage device  304  could comprise any suitable, computer-readable storage medium such as disk, non-volatile memory (e.g., read-only memory (ROM), erasable programmable ROM (EPROM), electrically-erasable programmable ROM (EEPROM), flash memory, field-programmable gate array (FPGA), etc.). The computer system  302  could be a networked computer system, a personal computer, a server, a smart phone, tablet computer etc. It is noted that the server  302  need not be a networked server, and indeed, could be a stand-alone computer system. 
     The functionality provided by the present disclosure could be provided by computer software code  306 , which could be embodied as computer-readable program code stored on the storage device  304  and executed by the CPU  212  using any suitable, high or low level computing language, such as Python, Java, C, C++, C#, .NET, MATLAB, etc. The network interface  308  could include an Ethernet network interface device, a wireless network interface device, or any other suitable device which permits the server  302  to communicate via the network. The CPU  312  could include any suitable single-core or multiple-core microprocessor of any suitable architecture that is capable of implementing and running the computer software code  306  (e.g., Intel processor). The random access memory  314  could include any suitable, high-speed, random access memory typical of most modern computers, such as dynamic RAM (DRAM), etc. 
       FIG.  12    is a diagram illustrating the neural network  16  of  FIG.  1    in greater detail. As can be seen, the network takes an input patch  350  from an input image  360 , during training of the neural network. The input patch  350  is processed by an encoder  354  having one or more rich filters, which generate semantic edges as can be seen in the image  364 . A feature extractor  356  extracts a noise fingerprint using the processes disclosed herein, to produce a camera noise fingerprint shown in image  366 . The noise fingerprint is processed by a decoder  358  and associated classifier to perform a proxy task of camera classification. Also, the noise fingerprint is processed by a Gaussian mixture model  360  to perform a main task of splice localization, as illustrated in image  368 . The Gaussian mixture model  360  is not a part of the network  16 , and is instead a separate model utilized for splice localization. The splice localization shown in image  368  identifies the genuine patches of the image from the forged ones, thereby indicating the forged portions of the image. As can be seen in  FIG.  12   , the network  16  utilizes the information bottleneck function described herein during the filtration, encoding, feature extraction, sampling, decoding, and classification steps. The information bottleneck function improves over rich filters by suppressing the semantic contents and amplifying the true camera noise fingerprints in the image, thereby enhancing the ability of the system to determined forged portions of an image. 
     Having thus described the system and method in detail, it is to be understood that the foregoing description is not intended to limit the spirit or scope thereof. It will be understood that the embodiments of the present disclosure described herein are merely exemplary and that a person skilled in the art can make any variations and modification without departing from the spirit and scope of the disclosure. All such variations and modifications, including those discussed above, are intended to be included within the scope of the disclosure.