Towards Order of Processing Operations Detection in JPEG-compressed Images with Convolutional Neural Networks

Determining which processing operations were used to edit an image and the order in which they were applied is an important task in image forensics. Existing approaches to detecting single manipulations have proven effective, however, their performance may significantly deteriorate if the processing occurs in a chain of editing operations. Thus, it is very challenging to detect the processing used in an ordered chain of operations using traditional forensic approaches. First attempts to perform order of operations detection were exclusively limited to a certain number of editing operations where feature extraction and order detection are disjoint. In this paper, we propose a new data-driven approach to jointly extract editing detection features, detect multiple editing operations, and determine the order in which they were applied. We design a constrained CNN-based classifier that is able to jointly extract low-level conditional fingerprint features related to a sequence of operations as well as identify an operation’s order. Through a set of experiments, we evaluated the performance of our CNN-based approach with different types of residual features commonly used in forensics. Experimental results show that our method outperforms the existing approaches.  Identifying the processing history of an image has been investigated over the last decade given its importance in wide variety of settings . Digital images are used as evidence in legal proceedings and criminal investigations. Therefore, it is important to determine the types of editing operation that an image has undergone and the order in which they have been applied. This can provide a forensic analyst a complete information about a forged image, such as the multiple types of processing operations frequently used by a forger, and may enlighten directions to determine the party who created the forgery. Image editing operations typically leave behind unique artifacts, known as fingerprints, that can be used to determine its type. Early approaches proceeded by extracting representative features of these fingerprints then developed associated algorithms to determine the type of a particular image tampering. This approach has proven effective in detecting many types of image tampering such as median filtering [19, 17], contrast enhancement [24], resizing and resampling [21, 18], etc. While exiting forensic approaches can determine if and how an image has undergone a single processing operation, their performance typically deteriorates if the processing occurs as a sequence of image editing operations [23, 8]. This is because fingerprints produced by latter processing operations can destroy processing fingerprints left by a former processing operation [23, 8]. Furthermore, detecting the order of operations can give a forensic investigator a complete picture of the global processing operations in order to understand to which extend a subject image has been altered from its raw version captured by a camera. Little work has been done to determine the order of a processing operation within a sequence of image editing. Early approaches proposed the notion of conditional fingerprints to determine the order of an operation occurring in an ordered chain of image editing. More specifically, Stamm et. al [23] proposed to differentiate between the ordered chains using a sequence of intermediate grouped hypothesis tests. Then, each intermediate stage searches for the presence of a specific fingerprint or conditional fingerprint. However, human discovery of conditional fingerprints and complex hierarchical hypothesis test must be designed for each chain, which is difficult and time consuming. This problem has been also studied from the information-theoretical perspective in . An alternative steganalytic approach has been proposed by Boroumand and Fridrich  where a rich model features based approach with quantization step 1 (SRMQ1)  was used to determine an operations order. This method operates by building local models of pixel dependencies by analyzing the joint distribution of pixel value prediction-errors. While this approach has proven effective, it still requires a forgery analyst to separately extract features then perform order of operations detection. Thus, several questions remain open: How should low-level predictive feature extractors be designed? Can order of operation detection features be learned directly from data? Can we devise a generic approach that can jointly extract features and perform order of operations detection? Furthermore, in a realistic scenario images are typically JPEG compressed to be saved after being processed. JPEG compression makes order of processing operation detection problem harder because a substantial component of the processing artifacts for forensic purposes is significantly suppressed or distorted. Our ultimate goal is to devise a robust forensic approach to JPEG compression that can jointly (1) learn pixel-value relationship traces left by editing processing chains and (2) determine the order in which these editing operations have been applied. In this paper, we propose a more practical data-driven approach to perform order of operations detection that is able to distinguish between conditional fingerprints left by different ordered chains of editing operations. To accomplish this, we employed a constrained convolutional neural network (CNN) based approach which can adaptively learn low-level prediction error features directly from data. In fact, CNNs tend to learn contentdependent features from images.Therefore, researchers in forensics employed several prediction error feature extractors associated with the CNNs. In our experiments, we evaluated our CNNbased approach with different types of prediction error feature extractors commonly used in forensics, i.e., the adaptive constrained convolutional layer used in forensics , the median filter residual (MFR) features employed in median filtering detection [9], and the high-pass filter (HPF) first adopted in steganalysis [20]. In what follows, we provide an overview of our proposed data-driven approach including details on the employed contsrained CNN architecture. Next, we evaluated our approach to perform order of processing operations detection with three different types of image editing operations. Additionally, we studied the impact of the training data size on CNN’s performance. Results of these experiments showed that our approach outperforms the state-of-the-art method with JPEG re-compressed images and can achieve 96.38% with three processing operations using a large scale training dataset. Proposed method The goal of this paper is to devise a more practical datadriven approach to performing order of processing operations detection in JEPG re-compressed images. As noted above, early approaches rely on theoretical analysis and parametric models of image’s data  which may not be accurate enough particularly in challenging scenarios. More specifically, in realistic scenarios images are in general JPEG compressed such as in social media and photo-sharing websites. This makes forensic traces very difficult to detect, hence, the performance of traditional forensic approaches significantly deteriorates . Typically, a forger uses a sequence of several editing operations to create a forgery, also called a chain of processing operations. It is very challenging to detect chains of editing operations using traditional approaches. This is because operations that occur later in the processing sequence can potentially destroy or alter fingerprints left by operations that occur earlier in the sequence. Moreover, as mentioned above traditional approaches rely on theoretical model of image’s data  which may not be accurate enough to detect manipulation fingerprints left by a chain of processing operation. Chains of processing operations are associated with an order in which a forger employs to perform a sequence of manipulations within an image. These chains leave behind manipulation traces, known as conditional fingerprints [23] , which are dependent on the employed order of operations in the chain. It is very challenging to detect traces induced by every single editing operation and determine its corresponding order in a processing chain. Therefore, forensic investigators must first measure and analyze traces left by the cumulative effect induced by an ordered processing operations chain . Instead of relying on theoretical analysis of parametric models, we propose a data-driven approach to directly learn from data the cumulative effect induced by an ordered chain of processing operations. To accomplish this, we cast the order of processing operations detection as a classification problem where every processing chain corresponds to a class. To better understand this, Fig. 1 illustrates two chains of processing operations that consist of the same set of N editing operations. If we invert the order of operations i and j in chain 1, this will correspond to a different class of ordered chain 2 that leaves behind different and unique cumulative effect in an image. Our approach works for general scenarios such that the total number of operations N that a chain consists of is arbitrary and can be as low as 1. Furthermore, our method requires a forensic investigator to assign a label to each Figure 1: Two chains of the same N processing operations using different order; Chains 1 and 2 leave behind different traces. different ordered processing chain as a separate class in the training data. To formulate this problem, let us consider the detection problem where a forgery analyst would like to build a forensic detector algorithm g(·) that identifies the type of editing operation which an image X has undergone. Typically, the forensic detector g is designed as the composition of two functions f (·) and d(·) such that g(X) = d ◦ f (X) = d( f (X)), (1