Histogram Layer, Moving Convolutional Neural Networks Towards Feature-Based Steganalysis

Feature-based steganalysis has been an integral tool for detecting the presence of steganography in communication channels for a long time. In this paper, we explore the possibility to utilize powerful optimization algorithms available in convolutional neural network packages to optimize the design of rich features. To this end, we implemented a new layer that simulates the formation of histograms from truncated and quantized noise residuals computed by convolution. Our goal is to show the potential to compactify and further optimize existing features, such as the projection spatial rich model (PSRM). Motivation In steganography, the main goal is to communicate secretly through an overt channel. Steganalysis, on the other hand tries to detect the presence of steganography. For a long time, detection relied on classifiers trained in a supervised fashion on examples of cover and stego images [3]. To detect content-adaptive schemes, researchers hand crafted various high-dimensional feature representations (rich models). Such features are typically formed from noise residuals extracted from the input image by convolutions with high-pass linear filters (kernels). The array of residuals would then be represented either with co-occurrence matrices (empirical joint densities) [19, 12, 2] or via histograms of projections on random directions [6]. Due to the inherent complexity of digital images, the design of suitable kernels has been based entirely on heuristics. While there were attempts to optimize the kernels by parametrizing them and determining the parameters by minimizing the classifier detection error using, e.g., the Nelder–Mead algorithm [5], the complexity of evaluating the objective function (training a classifier on thousands of images and evaluating its performance using, e.g., the minimal total error probability PE) makes such approaches non-scalable and unable to optimize a sufficient number of kernels to build a rich model. The main flaw is the need for a constant feedback from the classifier, which is very time consuming due to the complexity of training with highdimensional features on thousands of images. The main advantage of Convolutional Neural Networks (CNNs) is their ability to optimize the feature extraction and classification steps simultaneously and thus close the loop between feature extraction and classification. Such networks are not only capable of learning the best decision boundary between different classes but also the best representation of each data class that would improve their separability. Originally developed for numerous computer vision problems, these networks were recently adapted for steganalysis [16, 14, 13, 17, 18]. Since in a typical computer vision problem, CNNs are used to learn patterns or objects, in steganalysis the signal of interest is hidden within the noise component of the image. In order to deal with this issue, in one of the early works in this direction Qian et al. [14] proposed to adjust the network design by using one of the successfully hand-designed high-pass kernels in the Spatial Rich Model (SRM) [3] as the first (fixed) convolutional layer in the network. The main reason behind this step is to suppress most of each image content and thus force the network to “pay attention” to high-frequency details. A closer look at the CNN structure reveals a similarity with feature-based steganalysis. The convolutional layers play the role of residual extraction while activation and pooling layers mimick truncation and quantization. Various researches tried to improve the performance of CNNs for steganalysis using insights acquired from feature-based steganalysis. In one of the most recent works in this direction, Xu et al. [17] proposed a novel CNN architecture capable of approaching the performance of feature-based steganalysis. In their five-layer CNN, after initial high-pass filtering and the first convolutional layer, they proposed to take the absolute value (ABS) of the feature maps (residuals) as the activation function followed by a hyperbolic tangent (TanH) in order to preserve the sign symmetry of the residuals in a process similar to computing SRM residuals. Additionally, by using 1×1 convolutional kernels in the last three layers, the authors forced the network to collect the local statistics from the feature maps in a pixel by pixel fashion. Inspired by similarities between conventional featurebased and CNN detectors, in this paper we investigate the possibility to use existing infrastructure behind CNNs to optimize the design of kernels (linear pixel predictors) in feature-based steganalysis. The replacement of the objective function in the form of some scalar classifier performance criterion by the loss function in a CNN, powerful gradient descend algorithms could be used for the optimization task provided there was a way to form histograms within a CNN. Thus, as our first step, we implemented a histogram layer for the Caffe CNN package. We use meanshifted Gaussian functions as the building blocks of this layer to obtain a proper back-flow of gradients through the layer and to facilitate learning of the parameters behind this layer. As a proof of concept, in this paper we use this layer to model individual submodels of the PSRM [6]. Instead of forming co-occurrences of neighboring quantized residual samples, PSRM projects unquantized residual values on random directions, which are subsequently quantized and represented using histograms as steganalysis features. In PSRM, each SRM kernel is projected on 55 random two-dimensional kernels and their rotated and mirrored versions. The higher detection rate of this feature set is the result of a large number of projections, which comes at a high computational cost. This problem can render this powerful feature set unusable for applications with limited time and computational power [10]. Modeling these submodels within the CNN framework with the histogram layer enables us to reduce the high dimensionality of this feature set by replacing random kernels with fewer optimized kernels. Our study also hints at the possibility to extract more information in the final layers of CNNs to pave the way for better network designs. In the next section, we introduce the histogram layer and discuss its design and internal building blocks. In Section “Experimental setup,” we simulate PSRM models within the CNN framework using the histogram layer and outline the training procedure. The proposed histogram layer is tested in Section “Analyzing results,” where we compare the detection results of our CNN model with PSRM submodels and discuss the results. The paper is summarized in the last section. Histogram layer To capture the local statistics of feature maps using histograms without any information loss we need to use step functions centered on each histogram bin. While shifted step functions are the best choice to compute independent histogram bins, they are unusable within a CNN framework because their derivative is zero everywhere except for the edges. This prevents the back flow of gradients through the layer and the back-propagation algorithm stops working. In contrast, a Gaussian kernel seems to be a good candidate to form histogram bins. Unlike the sharp edges of a step function, its smooth slopes will create a fine path for gradients to flow backwards through each histogram bin towards previous layers. A Gaussian activation function has the form g(x) = e− (x−μ)2 σ2 (1) where μ is the center of the histogram bin and σ controls the tradeoff between the accuracy of each binning operation (and the overlap between adjacent bins) and the flow of gradients through the layer. In our experiments, we fixed σ = 0.6 to obtain a close match between the exact histogram and a histogram computed using a Gaussian activation function. Fig. 1 shows the structure of an 8-bin histogram function using Gaussian activations. The value of μ for each kernel is chosen from the set μ∈ {−3.5,−2.5, . ,2.5,3.5}. The tails of the Gaussian kernels on the sides are replaced with a constant value of 1 in order to simulate the residual truncation done in featurebased steganalysis. To compute the value of the histogram bin B(k) centered at μ= μk for an M ×N activation map −3.5 −2.5 −1.5 −0.5 0.5 1.5 2.5 3.5 0 0.2 0.4 0.6 0.8 1 Figure 1. Structure of an 8-bin histogram using mean-shifted Gaussian