# Neural Reversible Steganography with Long Short-Term Memory

Authors: Ching-Chun Chang
Journal: Security and Communication Networks
 (2021)

---

## Abstract

Deep learning has brought about a phenomenal paradigm shift in digital steganography. However, there is as yet no consensus on the use of deep neural networks in reversible steganography, a class of steganographic methods that permits the distortion caused by message embedding to be removed. The underdevelopment of the field of reversible steganography with deep learning can be attributed to the perception that perfect reversal of steganographic distortion seems scarcely achievable, due to the lack of transparency and interpretability of neural networks. Rather than employing neural networks in the coding module of a reversible steganographic scheme, we instead apply them to an analytics module that exploits data redundancy to maximise steganographic capacity. State-of-the-art reversible steganographic schemes for digital images are based primarily on a histogram-shifting method in which the analytics module is often modelled as a pixel intensity predictor. In this paper, we propose to refine the prior estimation from a conventional linear predictor through a neural network model. The refinement can be to some extent viewed as a low-level vision task (e.g., noise reduction and super-resolution imaging). In this way, we explore a leading-edge neuroscience-inspired low-level vision model based on long short-term memory with a brief discussion of its biological plausibility. Experimental results demonstrated a significant boost contributed by the neural network model in terms of prediction accuracy and steganographic rate-distortion performance.


---

## Body

## 1. Introduction

Steganography is the art and science of concealing a message within a cover object (e.g., image, audio, video, and text) in an imperceptible manner [1]. Applications of modern steganography include copyright protection [2–4], tamper detection [5–7], covert communication [8–10], etc. The distortion caused by message embedding, albeit usually minimal and invisible, may to some extent contaminate the cover object. In this era of data-driven artificial intelligence, steganographic distortion might entail uncontrollable risks to the reliability of some autonomous machines since the robustness against steganographic distortion is probably not taken into consideration when building those machines. Accurate and consistent data lay a sound foundation of modern analytics platforms [11], and accordingly, the ability to reverse steganographic distortion and restore data integrity is of paramount importance.Reversible steganographic methods have undergone rapid development over the past decades [12–22]. Although there are various principles and practices, a reversible steganographic method can be broadly compartmentalised into coding and analytics modules. In general, the coding module is devised to encode a message in an imperceptible and reversible way, whereas the analytics module exploits data redundancy with the aim of maximising steganographic capacity.Deep learning has revolutionised both academia and industry [23]. The phenomenal advances in deep learning have also introduced a paradigm shift in digital steganography [24–29]. However, research on reversible steganography with deep neural networks remains largely undeveloped. A possible explanation might be that perfect reversal of steganographic distortion seems to be hardly achievable at first glance. A coding module often involves sophisticated designs and procedures in order to regulate imperceptibility and guarantee reversibility. Any faulty operation may result in malfunctioning or failure of steganographic systems. A lack of transparency and interpretability in present neural networks could deter one from employing neural networks to realise or even upgrade these delicate reversible mechanisms. From our perspective, it is advisable to seek an alternative use of neural networks in reversible steganographic schemes. In contrast to the coding module, the analytics module has no demand for complete perfection, thereby allowing deep learning to serve its purpose. Recently, an exploratory study on adversarial learning for reversible image steganography was presented [30]. The author investigated a neural analytics module compatible with the regular singular (RS) coding module [31]. The neural analytics module was configured as a bit-plane predictor and implemented by a conditional generative adversarial network (GAN) called the pix2pix [32]. It has been suggested that transforming the analytics module into a neural network (neuralisation) could deliver a significant improvement to the original RS method.Contemporary reversible steganographic schemes for digital images are based primarily on the histogram-shifting (HS) method on account of its sterling rate-distortion performance [33–40]. In general, this type of scheme consists of two procedures: histogram generation and histogram modification, linked to the analytics module and the coding module, respectively. The objective of histogram generation is to compute from an image a frequency distribution of which the data values are as concentrated as possible or, alternatively, the entropy is as small as possible. A more sharply distributed histogram would normally result in a finer steganographic rate-distortion performance. A simple example is the frequency distribution of pixel intensities. However, the distribution of pixel intensities is apparently diverse and not necessarily concentrated, and the entropy of such distribution might not be minimal. A better option is to consider the histogram of prediction errors. Providing a well-behaved predictor, the frequencies of prediction errors typically have a peak around zero and fall off exponentially from the peak on both sides (following a zero-mean Laplace distribution). The more accurate the predictor is, the more sharply distributed the histogram will become. To this end, scientists have proposed various approaches for pixel intensity prediction [41–46].Given a fixed HS coding module, we can reasonably confine our attention to the design of an accurate pixel intensity predictor. Through experimental analysis, we found that although conventional (non-neural) predictors could estimate smooth image patches with a high degree of precision and are arguably less computationally demanding, their ability to predict textural patches is far from satisfactory. In view of this problem, we propose to employ a deep neural network model to refineprior estimation from a conventional predictor. While many deep neural network models may be employed to carry out the refinement, this task seems closest to low-level vision task (e.g., noise removal and super-resolution imaging) [47–53]. Therefore, we explore a seminal low-level vision model, the MemNet [54], of which the foundation is long short-term memory (LSTM) [55]. LSTM models were designed to mitigate the vanishing gradient problem encountered when training deep neural networks. The problem was overcome with the use of an internal mechanism called the gate unit which regulates the flow of information and learns to maintain important hidden states over extended time intervals. Although LSTM models are typically used for sequential data (e.g., time series, natural languages, and audio signals), the MemNet is a computer vision model that deals with low-level image features (e.g., edges, contours, and textures). Due to its state-of-the-art performance in image denoising and image super-resolution, we may reasonably expect to see an improvement delivered by the MemNet in the visual quality of pre-estimated images.In this paper, we study a neural analytics module compatible with the HS coding module. While there are wide variations across HS methods (e.g., multiple histograms, multidimensional shifting, and optimal bin selection), we eliminate intricate mechanisms and focus on a prototype coding module in order to underline the performance gain contributed by the neural network model. The proposed neural analytics module comprises a pre-processing stage that generates a pre-estimated image via a linear predictor and a post-processing stage that refines the prior estimation via an LSTM-based vision model. Experimental results from large-scale assessments validated the effectiveness of the neural network model and demonstrated a significant improvement in steganographic rate-distortion performance.The remainder of this paper is organised as follows. Section2 reviews a prototype HS coding module and formulates some principal concepts. Section 3 presents the proposed neural analytics model which utilises an LSTM-based vision model for refining the prior estimation from a linear predictor. Section 4 validates the effectiveness of the neural network model and evaluates steganographic performance through simulation experiments. The paper draws conclusions in Section 5.


## 2. Coding Module

In this section, we revisit the coding module of a prototype HS method. We start with a workflow of the encoding and decoding processes, as illustrated in Figure1. Suppose that a sender, Alice, wants to communicate a message to a receiver, Bob, through a reversible steganographic scheme. For a cover image X, Alice defines a set of context pixels preserved for predicting the other set of query pixels. The prediction can be fulfilled by either a conventional predictor or a neural network, resulting in a reference image X˜. By subtracting X˜ from X, cover residuals (prediction errors) are obtained. The HS coding module is applied to embed a message into cover residuals, yielding stego residuals along with an overflow map for later use in the reverse process. The stego image X′ is finally generated by adding the stego residuals to X˜. Addition may cause the problem of pixel intensity overflow; pixel intensities that are unexpectedly small or large wrap around the minimum and maximum after addition. In order to handle this exception, an overflow map is pre-calculated to flag pixels of which intensity would be off-boundary after message embedding. The overall collection of sent data includes a stego image and a compressed overflow map. At the receiving end, Bob computes X˜ from X′ via a shared prediction mechanism. The reference image will be the same because only the query pixels have been modified and the context pixels in X and X′ are unchanged. The remaining decoding procedures for message extraction and image recovery are virtually a reverse process of the encoding procedures. Next, we explain the details of the coding module under the assumption that the reference image X˜ has already been obtained.Figure 1
A workflow for a reversible steganographic scheme based on histogram shifting. (a) Encoding process. (b) Decoding process.
(a)(b)


### 2.1. Histogram of Prediction Errors

Let us denote byXi,j a pixel at position i,j and X˜i,j its predicted counterpart, where Xi,j,X˜i,j∈0,255. For each query pixel, a prediction error is calculated by(1)Ei,j=Xi,j−X˜i,j,where Ei,j∈−255,255. Then, we count the occurrence of each error value and construct a histogram of prediction errors. We select one or more bins on the histogram as the steganographic channel. A bin is a container into which errors of the same value are grouped together. Selecting bins as the steganographic channel indicates defining which values of the prediction errors can be used to carry the message. In general, an increase in the number of selected bins will help to enhance steganographic capacity while simultaneously aggravating steganographic distortion. Let us denote by hε a bin for error value ε. According to the law of error [56], the frequency of an error could be expressed as an exponential function of its magnitude, disregarding its sign. In other words, small deviations would be observed more frequently than large deviations in normal circumstances. Hence, we may reasonably assume that the frequency of errors follows a zero-mean Laplacian distribution (i.e., double exponential distribution), in which the peak bin occurs around zero and the height of bins decays exponentially with the absolute magnitude of errors. Accordingly, we may explicitly define a channel selection rule that selects from h0 and moves outwards in both positive and negative directions.


### 2.2. Encoding and Decoding

A summary for the HS coding mechanism is presented visually in Table1. While the code chart allows us to develop a simpler understanding of the coding mechanism, we provide mathematical details to avoid confusion.Table 1
Code charts with different thresholds for the steganographic channel (grey cells).
θ=0E−254⋯−3−2−10+1+2+3⋯+254E⇑−255⋯−4−3−20+2+3+4⋯+255E′−255⋯−4−3−2−10+1+2+3+4⋯+255θ=1E−253⋯−3−2−10+1+2+3⋯+253E⇑−255⋯−5−4−20+2+4+5⋯+255E′−255⋯−5−4−3−2−10+1+2+3+4+5⋯+255θ=2E−252⋯−3−2−10+1+2+3⋯+252E⇑−255⋯−6−4−20+2+4+6⋯+255E′−255⋯−6−5−4−3−2−10+1+2+3+4+5+6⋯+255Letθ denote a threshold for the steganographic channel such that(2)θ=0,ifh0is selected,1,ifh0andh±1are selected,2,ifh0,h±1,andh±2are selected,⋯According to the threshold, we derive the following three intervals:(3)U1=0,U2=±1,±2,±3,…,±θ,U3=±θ+1,±θ+2,…,±255.The encoding process begins by shifting the bins selected as the steganographic channel (inner bins) and the remaining unselected bins (outer bins) outwards in order to empty out bins for carrying message digits. Shifting the inner and outer bins is equivalent to modifying prediction errors that fall into different intervals. We shift the value of each error by(4)Ei,j⇑=2Ei,j,ifEi,j∈U1∪U2,Ei,j+sgnEi,j·θ+1,ifEi,j∈U3.For an intended message, we divide it into two segments and convert them into the binary and ternary numeral systems, respectively. Then, we embed them depending on the error value that is currently observed. A pre-scanning is required in order to determine the length of each segment. Let us denote bymtrit a ternary message digit and by mbit a binary message digit, where mtrit∈−1,0,1 and mbit∈0,1. We embed a ternary digit (log23 bits) if the error value is 0, embed a binary digit (1 bit) if the error value other than 0 originally falls into the steganographic channel, and skip the current error otherwise, as given by(5)Ei,j′=mtrit,ifEi,j⇑∈U1,Ei,j⇑+sgnEi,j⇑·mbit,ifEi,j⇑∈2·U2,Ei,j⇑,otherwise,where(6)2·U2=±2,±4,…,±2θ.Finally, we add each modified prediction error to the estimated pixel at the corresponding position to obtain a stego image:(7)Xi,j′=X˜i,j+Ei,j′.It is worth noting that pixel intensities after addition are not guaranteed within range of possible values from 0 to 255. Therefore, an overflow map is pre-calculated to flag pixels whose intensity might be out-of-bound. For pixels that may incur overflow, we skip the process of message embedding and record the positions by marking with flags on the map as(8)Ωi,j=⊥false,ifXi,j′∈0,255,⊤true,otherwise.The overflow map can be compressed and sent along or else embedded into the image as a part of the payload. For simplicity, we opt for the first approach in our implementation. Nevertheless, for fair evaluations, we deduct from the overall payload the size of the compressed overflow map when assessing steganographic capacity.Decoding is simply the reverse process of encoding. It begins by generating the reference imageX˜ using the same set of context pixels as in the encoding process. For pixels where Ωi,j=⊥, we calculate the prediction errors by(9)Ei,j′=Xi,j′−X˜i,j.Following the threshold and the coding mechanism, we divide pixels into the intervals:(10)U1′=0,±1,U2′=±2,±3,…,±2θ+1,U3′=±2θ+2,±2θ+3,…,±255.A ternary or binary digit is extracted based on different interval conditions such that(11)m^trit=Ei,j′∀Ei,j′∈U1,m^bit≡Ei,j′mod2∀Ei,j′∈U2,and the cover image can be recovered by(12)E^i,j=0,ifEi,j′∈U1′,Ei,j′2,ifEi,j′∈U2′,Ei,j′−sgnEi,j′·θ+1,ifEi,j′∈U3′.


## 2.1. Histogram of Prediction Errors

Let us denote byXi,j a pixel at position i,j and X˜i,j its predicted counterpart, where Xi,j,X˜i,j∈0,255. For each query pixel, a prediction error is calculated by(1)Ei,j=Xi,j−X˜i,j,where Ei,j∈−255,255. Then, we count the occurrence of each error value and construct a histogram of prediction errors. We select one or more bins on the histogram as the steganographic channel. A bin is a container into which errors of the same value are grouped together. Selecting bins as the steganographic channel indicates defining which values of the prediction errors can be used to carry the message. In general, an increase in the number of selected bins will help to enhance steganographic capacity while simultaneously aggravating steganographic distortion. Let us denote by hε a bin for error value ε. According to the law of error [56], the frequency of an error could be expressed as an exponential function of its magnitude, disregarding its sign. In other words, small deviations would be observed more frequently than large deviations in normal circumstances. Hence, we may reasonably assume that the frequency of errors follows a zero-mean Laplacian distribution (i.e., double exponential distribution), in which the peak bin occurs around zero and the height of bins decays exponentially with the absolute magnitude of errors. Accordingly, we may explicitly define a channel selection rule that selects from h0 and moves outwards in both positive and negative directions.


## 2.2. Encoding and Decoding

A summary for the HS coding mechanism is presented visually in Table1. While the code chart allows us to develop a simpler understanding of the coding mechanism, we provide mathematical details to avoid confusion.Table 1
Code charts with different thresholds for the steganographic channel (grey cells).
θ=0E−254⋯−3−2−10+1+2+3⋯+254E⇑−255⋯−4−3−20+2+3+4⋯+255E′−255⋯−4−3−2−10+1+2+3+4⋯+255θ=1E−253⋯−3−2−10+1+2+3⋯+253E⇑−255⋯−5−4−20+2+4+5⋯+255E′−255⋯−5−4−3−2−10+1+2+3+4+5⋯+255θ=2E−252⋯−3−2−10+1+2+3⋯+252E⇑−255⋯−6−4−20+2+4+6⋯+255E′−255⋯−6−5−4−3−2−10+1+2+3+4+5+6⋯+255Letθ denote a threshold for the steganographic channel such that(2)θ=0,ifh0is selected,1,ifh0andh±1are selected,2,ifh0,h±1,andh±2are selected,⋯According to the threshold, we derive the following three intervals:(3)U1=0,U2=±1,±2,±3,…,±θ,U3=±θ+1,±θ+2,…,±255.The encoding process begins by shifting the bins selected as the steganographic channel (inner bins) and the remaining unselected bins (outer bins) outwards in order to empty out bins for carrying message digits. Shifting the inner and outer bins is equivalent to modifying prediction errors that fall into different intervals. We shift the value of each error by(4)Ei,j⇑=2Ei,j,ifEi,j∈U1∪U2,Ei,j+sgnEi,j·θ+1,ifEi,j∈U3.For an intended message, we divide it into two segments and convert them into the binary and ternary numeral systems, respectively. Then, we embed them depending on the error value that is currently observed. A pre-scanning is required in order to determine the length of each segment. Let us denote bymtrit a ternary message digit and by mbit a binary message digit, where mtrit∈−1,0,1 and mbit∈0,1. We embed a ternary digit (log23 bits) if the error value is 0, embed a binary digit (1 bit) if the error value other than 0 originally falls into the steganographic channel, and skip the current error otherwise, as given by(5)Ei,j′=mtrit,ifEi,j⇑∈U1,Ei,j⇑+sgnEi,j⇑·mbit,ifEi,j⇑∈2·U2,Ei,j⇑,otherwise,where(6)2·U2=±2,±4,…,±2θ.Finally, we add each modified prediction error to the estimated pixel at the corresponding position to obtain a stego image:(7)Xi,j′=X˜i,j+Ei,j′.It is worth noting that pixel intensities after addition are not guaranteed within range of possible values from 0 to 255. Therefore, an overflow map is pre-calculated to flag pixels whose intensity might be out-of-bound. For pixels that may incur overflow, we skip the process of message embedding and record the positions by marking with flags on the map as(8)Ωi,j=⊥false,ifXi,j′∈0,255,⊤true,otherwise.The overflow map can be compressed and sent along or else embedded into the image as a part of the payload. For simplicity, we opt for the first approach in our implementation. Nevertheless, for fair evaluations, we deduct from the overall payload the size of the compressed overflow map when assessing steganographic capacity.Decoding is simply the reverse process of encoding. It begins by generating the reference imageX˜ using the same set of context pixels as in the encoding process. For pixels where Ωi,j=⊥, we calculate the prediction errors by(9)Ei,j′=Xi,j′−X˜i,j.Following the threshold and the coding mechanism, we divide pixels into the intervals:(10)U1′=0,±1,U2′=±2,±3,…,±2θ+1,U3′=±2θ+2,±2θ+3,…,±255.A ternary or binary digit is extracted based on different interval conditions such that(11)m^trit=Ei,j′∀Ei,j′∈U1,m^bit≡Ei,j′mod2∀Ei,j′∈U2,and the cover image can be recovered by(12)E^i,j=0,ifEi,j′∈U1′,Ei,j′2,ifEi,j′∈U2′,Ei,j′−sgnEi,j′·θ+1,ifEi,j′∈U3′.


## 3. Analytics Module

The previous coding module works under the assumption that a prediction mechanism has been developed and it is time to unveil and deliver the analytics module for estimating a reference image from the preserved context pixels. We begin by dividing pixels into the context and the query according to a pre-determined pattern. Next, we introduce a pre-processing stage for generating aprior reference image. Then, we explore a neural network model based on the long short-term memory for refining the pre-processed image into a posterior reference image.


### 3.1. Prior Estimation

The initial step of pixel prediction is typically to define the set of preserved pixels for estimating a query pixel, namely, the context. Amongst various ways to define the context and the query, thechequerboard pattern can be regarded as the most common one. Consider a chequerboard pattern that divides pixels into a black set and a white set, as illustrated in Figure 2. We may appoint the black set as the query and the white set as the context, or the other way round, which can be written mathematically as(13)Xi,j∈blackquery,ifi+jis even,whitecontext,ifi+jis odd.Figure 2
An illustration of the chequerboard pattern which divides pixels into the query and the context.There are a variety of strategies for predicting the query pixels given the context pixels, but the most naïve strategy is to estimate by the mean of four immediate context pixels, formulated as(14)X˜i,j=Xi−1,j+Xi+1,j+Xi,j−1+Xi,j+14.This approach is, however, far from optimal due to a relatively restricted receptive field and limit of linearity. In other words, estimation is based solely on a linear combination of immediate local neighbours and any information outside the local field is completely ruled out.In order to manage this issue, we may refine this pre-processed output by a nonlinear neural network model. We refer to the pre-processed image as the prior imageX˜pre and the refined image as the posterior image X˜post. Also, the pre-processor (linear non-neural model) is termed the prior predictor and the post-processor (nonlinear neural model) is termed the posterior predictor. We model this refinement process as a special type of low-level vision task and employ a vision model, the MemNet, to improve the visual quality of a pre-estimated reference image en route from input to output through hidden layers:(15)X˜post=MemNetX˜pre.Our implementation of the MemNet involves minor modifications. Consequently, the following description details the network architecture in order to ensure understanding, reproducibility, and replicability.It is worth noting that the chequerboard-based prediction mechanism can be operated in two rounds, resulting in a dual-layer embedding scheme [57]. Suppose that the black set is assigned as the query and the white set as the context in the first round. After the first-layer embedding, the black set will be modified to carry a message segment. For the second-layer embedding, the white set will be assigned as the query and the modified black set as the context. Decoding is carried out in a first-in last-out manner; that is, pixels in the white set are recovered first and then used to recover pixels in the black set. We would like to emphasise that the dual-layer embedding scheme is not considered in our simulation experiments since our primary aim is to analyse the performance gain from neuralisation and an extended dual-layer embedding scheme would have few implications for the findings of this study.


### 3.2. Long Short-Term Memory

A fundamental component of the MemNet is thememory cell, which consists of neurons connected in a recurrent form and a gating mechanism that regulates persistent memories (i.e., important hidden states). From a practical and engineering standpoint, a slavish adherence to biological plausibility is not necessary for building neural network models; nonetheless, a neurobiological perspective may afford some interesting insights and provide guidance at a high level of abstraction [58]. Anatomical evidence has shown that recurrent synapses typically outnumber feedback and feedforward synapses, and it is believed that recurrent circuitry might play a crucial role in shaping the responses of neurons in the visual cortex [59]. Neuroscience studies also suggest that the mammalian brain has an evolved mechanism to avoid catastrophic forgetting called synaptic consolidation, whereby previously acquired knowledge, or memory, is durably encoded by rendering a proportion of synapses less plastic and thus stable over a long period of time [60].Recurrent connections could be modelled as a recurrent neural network (RNN) [61]. For processing image data, it would be more convenient to construct a residual neural network (ResNet) [62] in such a way that the same weights are shared amongst layers. In fact, there is an intriguing equivalence between an RNN and a ResNet with weight sharing [63]. It can be seen from Figure 3 that a ResNet with weight sharing approximates an RNN when being unfolded into a feedforward network. Apart from a biological interpretation, recurrent connections can reduce the number of trainable parameters (i.e., weights and biases) substantially and thereby result in a comparatively lightweight model for storage. A gating mechanism mimicking synaptic consolidation could be represented by a convolutional layer that learns weights for preserving or erasing memories. After passing through the convolutional gate unit, a series of ephemeral recollections (short-term memories) become a recollection that persists (long-term memory).Figure 3
An equivalence relation between an RNN and a ResNet with weight sharing.Architectural details of the MemNet are described as follows. The MemNet is composed of a pre-processing layerfpre, an LSTM module, and a post-processing layer fpost, as illustrated in Figure 4(a) and expressed symbolically by(16)MemNetA=fpostLSTMfpreA+A,where fpre and fpost are both convolutional layers with kernel size 3, stride 1, and padding 1. The post-processing layer takes not only the output of the LSTM module but also the original input. Shortcuts or skip connections are essential to deep neural networks. It has been shown that when the model gets deeper, skip connections allow the information from shallow layers to propagate more effectively to deep layers [64]. From our viewpoint, bypassing the intermediate layers and connecting the prior image directly to the last layer could guide the neural network to learn delicate textural information in images, namely, minute differences between the prior estimation and the ground truth (i.e., the pristine image). The distance between the refined output and the ground truth is measured by the ℓ1 norm. The model is trained to minimise this loss function with the backpropagation algorithm [65].Figure 4
Architectural details of the MemNet. (a) MemNet. (b) Long short-term memory. (c) Memory cell. (d) Residual unit.
(a)(b)(c)(d)The LSTM module comprises interconnected memory cells. Each current cell takes long-term memories produced from all previous cells as the input, as illustrated in Figure4(b). Let l denote the number of memory cells and ℒt the output from the t-th memory cell. The LSTM module inputs the 0-th memory and outputs the l-th memory:(17)ℒl=LSTMℒ0,where(18)ℒ0=fpreA.A memory cell has several residual units connected to each other in a recurrent manner (with weight sharing) and a gate unit placed at the end of the cell, as illustrated in Figure4(c). The outputs from all residual units (i.e., short-term memories) along with the outputs from previous cells (i.e., long-term memories) go through a gate unit to produce a persistent memory for subsequent cells, as expressed by(19)ℒ1=fgate1S11S12…S1sℒ0,ℒ2=fgate2S21S22…S2sℒ0ℒ1,⋮ℒl=fgatelSl1Sl2…Slsℒ0ℒ1…ℒl−1,where(20)St1=Restℒt−1,St2=RestRestℒt−1,⋮Sts=RestRest…Restℒt−1….Residual unit is illustrated in Figure4(d) and laid out as follows:(21)ResA=fresfresA+A.The structure of bothfres and fgate follows the basic building block, composed of a convolutional layer [66–68], a batch normalisation [69], a ReLU activation function [70], and a dropout regularisation [71], written as(22)fA=DropoutReLUBNConvA.In implementation, the convolutional layer offres was configured to kernel size 3, stride 1, and padding 1, whereas the convolutional layer of fgate was set to kernel size 1, stride 1, and padding 1. We applied a dropout rate of 0.1 to fres and fgate.


## 3.1. Prior Estimation

The initial step of pixel prediction is typically to define the set of preserved pixels for estimating a query pixel, namely, the context. Amongst various ways to define the context and the query, thechequerboard pattern can be regarded as the most common one. Consider a chequerboard pattern that divides pixels into a black set and a white set, as illustrated in Figure 2. We may appoint the black set as the query and the white set as the context, or the other way round, which can be written mathematically as(13)Xi,j∈blackquery,ifi+jis even,whitecontext,ifi+jis odd.Figure 2
An illustration of the chequerboard pattern which divides pixels into the query and the context.There are a variety of strategies for predicting the query pixels given the context pixels, but the most naïve strategy is to estimate by the mean of four immediate context pixels, formulated as(14)X˜i,j=Xi−1,j+Xi+1,j+Xi,j−1+Xi,j+14.This approach is, however, far from optimal due to a relatively restricted receptive field and limit of linearity. In other words, estimation is based solely on a linear combination of immediate local neighbours and any information outside the local field is completely ruled out.In order to manage this issue, we may refine this pre-processed output by a nonlinear neural network model. We refer to the pre-processed image as the prior imageX˜pre and the refined image as the posterior image X˜post. Also, the pre-processor (linear non-neural model) is termed the prior predictor and the post-processor (nonlinear neural model) is termed the posterior predictor. We model this refinement process as a special type of low-level vision task and employ a vision model, the MemNet, to improve the visual quality of a pre-estimated reference image en route from input to output through hidden layers:(15)X˜post=MemNetX˜pre.Our implementation of the MemNet involves minor modifications. Consequently, the following description details the network architecture in order to ensure understanding, reproducibility, and replicability.It is worth noting that the chequerboard-based prediction mechanism can be operated in two rounds, resulting in a dual-layer embedding scheme [57]. Suppose that the black set is assigned as the query and the white set as the context in the first round. After the first-layer embedding, the black set will be modified to carry a message segment. For the second-layer embedding, the white set will be assigned as the query and the modified black set as the context. Decoding is carried out in a first-in last-out manner; that is, pixels in the white set are recovered first and then used to recover pixels in the black set. We would like to emphasise that the dual-layer embedding scheme is not considered in our simulation experiments since our primary aim is to analyse the performance gain from neuralisation and an extended dual-layer embedding scheme would have few implications for the findings of this study.


## 3.2. Long Short-Term Memory

A fundamental component of the MemNet is thememory cell, which consists of neurons connected in a recurrent form and a gating mechanism that regulates persistent memories (i.e., important hidden states). From a practical and engineering standpoint, a slavish adherence to biological plausibility is not necessary for building neural network models; nonetheless, a neurobiological perspective may afford some interesting insights and provide guidance at a high level of abstraction [58]. Anatomical evidence has shown that recurrent synapses typically outnumber feedback and feedforward synapses, and it is believed that recurrent circuitry might play a crucial role in shaping the responses of neurons in the visual cortex [59]. Neuroscience studies also suggest that the mammalian brain has an evolved mechanism to avoid catastrophic forgetting called synaptic consolidation, whereby previously acquired knowledge, or memory, is durably encoded by rendering a proportion of synapses less plastic and thus stable over a long period of time [60].Recurrent connections could be modelled as a recurrent neural network (RNN) [61]. For processing image data, it would be more convenient to construct a residual neural network (ResNet) [62] in such a way that the same weights are shared amongst layers. In fact, there is an intriguing equivalence between an RNN and a ResNet with weight sharing [63]. It can be seen from Figure 3 that a ResNet with weight sharing approximates an RNN when being unfolded into a feedforward network. Apart from a biological interpretation, recurrent connections can reduce the number of trainable parameters (i.e., weights and biases) substantially and thereby result in a comparatively lightweight model for storage. A gating mechanism mimicking synaptic consolidation could be represented by a convolutional layer that learns weights for preserving or erasing memories. After passing through the convolutional gate unit, a series of ephemeral recollections (short-term memories) become a recollection that persists (long-term memory).Figure 3
An equivalence relation between an RNN and a ResNet with weight sharing.Architectural details of the MemNet are described as follows. The MemNet is composed of a pre-processing layerfpre, an LSTM module, and a post-processing layer fpost, as illustrated in Figure 4(a) and expressed symbolically by(16)MemNetA=fpostLSTMfpreA+A,where fpre and fpost are both convolutional layers with kernel size 3, stride 1, and padding 1. The post-processing layer takes not only the output of the LSTM module but also the original input. Shortcuts or skip connections are essential to deep neural networks. It has been shown that when the model gets deeper, skip connections allow the information from shallow layers to propagate more effectively to deep layers [64]. From our viewpoint, bypassing the intermediate layers and connecting the prior image directly to the last layer could guide the neural network to learn delicate textural information in images, namely, minute differences between the prior estimation and the ground truth (i.e., the pristine image). The distance between the refined output and the ground truth is measured by the ℓ1 norm. The model is trained to minimise this loss function with the backpropagation algorithm [65].Figure 4
Architectural details of the MemNet. (a) MemNet. (b) Long short-term memory. (c) Memory cell. (d) Residual unit.
(a)(b)(c)(d)The LSTM module comprises interconnected memory cells. Each current cell takes long-term memories produced from all previous cells as the input, as illustrated in Figure4(b). Let l denote the number of memory cells and ℒt the output from the t-th memory cell. The LSTM module inputs the 0-th memory and outputs the l-th memory:(17)ℒl=LSTMℒ0,where(18)ℒ0=fpreA.A memory cell has several residual units connected to each other in a recurrent manner (with weight sharing) and a gate unit placed at the end of the cell, as illustrated in Figure4(c). The outputs from all residual units (i.e., short-term memories) along with the outputs from previous cells (i.e., long-term memories) go through a gate unit to produce a persistent memory for subsequent cells, as expressed by(19)ℒ1=fgate1S11S12…S1sℒ0,ℒ2=fgate2S21S22…S2sℒ0ℒ1,⋮ℒl=fgatelSl1Sl2…Slsℒ0ℒ1…ℒl−1,where(20)St1=Restℒt−1,St2=RestRestℒt−1,⋮Sts=RestRest…Restℒt−1….Residual unit is illustrated in Figure4(d) and laid out as follows:(21)ResA=fresfresA+A.The structure of bothfres and fgate follows the basic building block, composed of a convolutional layer [66–68], a batch normalisation [69], a ReLU activation function [70], and a dropout regularisation [71], written as(22)fA=DropoutReLUBNConvA.In implementation, the convolutional layer offres was configured to kernel size 3, stride 1, and padding 1, whereas the convolutional layer of fgate was set to kernel size 1, stride 1, and padding 1. We applied a dropout rate of 0.1 to fres and fgate.


## 4. Experimental Results

In this section, we present experimental results based on large-scale statistical evaluations. Our primary aim is to demonstrate the performance difference between the prior (linear non-neural) and posterior (nonlinear neural) predictors. We begin by validating the effectiveness of the neural network model for refining the visual quality of pre-estimated images. Then, we examine the error distribution with regard to entropy and cumulative frequency. In order to understand how the visual quality of reference images and the entropy of error distribution would influence steganographic capacity, we carried out regression analysis. This section ends with an evaluation of steganographic rate-distortion performance.


### 4.1. Experimental Setup

The image samples for training and testing the MemNet were from the BOSSbase [72], which contains a collection of 10,000 greyscale photographs covering a wide variety of subjects and scenes. All the images were resampled to a resolution of 256×256 pixels through the Lanczos algorithm [73]. The number of convolution kernels per layer was configured to 64, the number of total memory cells was configured to 3, and the number of residual units per cell was configured to 3. The kernel weights were initialised by the Xavier initialisation [74]. The model was trained on 8,000 images over 100 epochs by the Adam optimiser [75] with an initial learning rate set to 10−3 and scheduled to decay exponentially after every epoch. Large-scale assessments were conducted on 2,000 test images. The inference process was simulated on selected standard test images from the USC-SIPI database [76].


### 4.2. Visual Quality Analysis

Starting from Figures5 and 6, we can catch a glimpse of the extent to which the model can refine the pre-processed images. It can be observed that the visual quality of posterior images is better than that of prior images, especially at the edges and in textural areas. The same outcome is reflected in the peak signal-to-noise ratio (PSNR) of images, measured in decibel (dB). Results suggest that the neural network model indeed has a stronger ability to model nonlinearity and complex pattern.Figure 5
Visual comparisons between the prior and posterior predictors based on images from the USC-SIPI database. Numerical data express the PSNRs. (a) Ground truth/aeroplane. (b) Ground truth/Lena. (c) Ground truth/mandrill. (d) Ground truth/peppers. (e) Prior/32.9055 dB. (f) Prior/34.5666 dB. (g) Prior/28.5752 dB. (h) Prior/35.3336 dB. (i) Posterior/36.3609 dB. (j) Posterior/37.6200 dB. (k) Posterior/29.3495 dB. (l) Posterior/39.9355 dB.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 6
Visual comparisons between the prior and posterior predictors based on images from the BOSSbase. Numerical data express the PSNRs. (a) Ground truth/0021. (b) Ground truth/0916. (c) Ground truth/1366. (d) Ground truth/1953. (e) Prior/31.5765 dB. (f) Prior/32.4795 dB. (g) Prior/34.9142 dB. (h) Prior/28.4180 dB. (i) Posterior/33.6192 dB. (j) Posterior/34.6770 dB. (k) Posterior/37.5122 dB. (l) Posterior/31.7238 dB.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)


### 4.3. Entropy Analysis

Figure7 shows that the posterior error distribution is more concentrated and its entropy smaller, whereas the prior error distribution is comparatively more diffuse. However, it is striking that the height of the peak bin (usually h0) on the posterior histogram is not always higher than the height of the same bin on the prior histogram. A possible explanation would be that some image samples contain a relatively large number of smooth patches on which a naïve linear predictor may perform sufficiently well.Figure 7
Prediction error distributions. Numerical data express the entropy of distribution. (a) Aeroplane: prior/2.9529; posterior/2.6493. (b) Lena: prior/2.8683; posterior/2.5383. (c) Mandrill: prior/3.9392; posterior/3.8131. (d) Peppers: prior/2.7781; posterior/2.4074
(a)(b)(c)(d)


### 4.4. Cumulative Frequency Analysis

In order to better understand how the prior and posterior prediction errors distribute, we analyse their cumulative frequencies. Figure8 presents cumulative distribution function (CDF) plots, where the 95th percentile gives the maximum error value below which 95% of errors fall. It is evident that the rate of convergence of the posterior error distribution is faster than that of the prior error distribution, confirming again that posterior errors are more concentrated and the magnitude of these is smaller on average.Figure 8
Cumulative distribution function of prediction error distributions. Numerical data express the 95th percentile. (a) Aeroplane: prior/20; posterior/12. (b) Lena: prior/16; posterior/10. (c) Mandrill: prior/30; posterior/28. (d) Peppers: prior/15; posterior/7.
(a)(b)(c)(d)


### 4.5. Large-Scale Assessment

In addition to evaluating the performance on individual selected images from the USC-SIP database, we provide a large-scale assessment based on a large number of test samples from the BOSSbase. Figure9(a) depicts the probability density of PSNRs of prior and posterior images. Figure 9(b) shows the probability density of entropies of prior and posterior error distributions. Figure 9(c) reveals the average rates of convergence of prior and posterior errors. On average, the visual quality of the posterior errors is higher, the distribution of them is more peaked, and the convergence rate is faster.Figure 9
Statistical comparisons between prior and posterior predictors with respect to PSNRs of reference images, entropies of prediction error distributions, and cumulative distribution functions. (a) Prior/34.29 dB; posterior/37.27 dB. (b) Prior/2.863; posterior/2.546. (c) Prior/17; posterior/13.
(a)(b)(c)


### 4.6. Regression Analysis

While we have shown that our neural network model offers better visual quality and smaller entropy, it is still unclear how these factors may benefit steganographic capacity. As a consequence, we carried out regression analysis amongst the PSNR of reference images, entropy of prediction errors, and maximum embedding rate, measured in bits per pixel (bpp). Figure10 plots the results using the test samples from the BOSSbase with different threshold values θ which regulate the steganographic channel. As expected, the general trends suggest that the embedding rate is directly proportional to the PSNR of reference images and inversely proportional to the entropy of prediction errors.Figure 10
Regression analysis amongst PSNR of reference images, entropy of prediction errors, and maximum embedding rate at different threshold values that regulate the steganographic channel. (a)θ = 0. (b) θ = 1. (c) θ = 2. (d) θ = 3.
(a)(b)(c)(d)


### 4.7. Rate-Distortion Evaluation

We evaluate capacity and distortion by rate-distortion curves, as plotted in Figure11. It can be observed that the maximum embedding rate increases with the increase of the threshold (the width of steganographic channel). The reason is straightforward: an increase in the threshold implies an increase in the number of bins for carrying the message. In addition to this, the observations suggest that the maximum embedding rate tends to be smaller for images containing more complex textures. It is because the prediction errors of such images are less concentrated and thus fewer bins are covered within the steganographic channel. There is a gradual and steady decline in the visual quality of stego images as embedding rate increases. The difference between the rate-distortion performances of the prior and posterior predictors is subtle for a small threshold value, but it becomes significant as the threshold value grows, with the posterior outperforming the prior. The underlying explanation for the trend may be that the naïve predictor and the neural network model have similar abilities to estimate smooth patches, for which both methods can often estimate perfectly. Nonetheless, the latter excels over the former when estimating textural patches, for which neither methods can offer accurate prediction but the neural network gives smaller error magnitude in general. Figure 12 lists stego images generated by embedding a simulated message into the cover images. The intended message is often assumed to have been compressed and encrypted and thus can be reasonably simulated by a random bit stream and a random trit stream.Figure 11
Rate distortion curves of applying prior and posterior predictors at different threshold values that regulate the steganographic channel. (a) Aeroplane. (b) Lena. (c) Mandrill. (d) Peppers.
(a)(b)(c)(d)Figure 12
Stego images generated by embedding a random bitstream with different threshold values that regulate the steganographic channel. Numerical data express the PSNRs and maximum embedding rates. (a)θ = 0, 51.52 dB, 0.1652 bpp. (b) θ = 0, 51.51 dB, 0.1607 bpp. (c) θ = 0, 51.29 dB, 0.0470 bpp. (d) θ = 0, 51.50 dB, 0.1554 bpp. (e) θ = 1, 46.72 dB, 0.3240 bpp. (f) θ = 1, 46.71 dB, 0.3248 bpp. (g) θ = 1, 45.62 dB, 0.1057 bpp. (h) θ = 1, 46.68 dB, 0.3191 bpp. (i) θ = 2, 44.46 dB, 0.4039 bpp. (j) θ = 2, 44.53 dB, 0.4150 bpp. (k) θ = 2, 42.48 dB, 0.1581 bpp. (l) θ = 2, 44.53 dB, 0.4210 bpp.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)


## 4.1. Experimental Setup

The image samples for training and testing the MemNet were from the BOSSbase [72], which contains a collection of 10,000 greyscale photographs covering a wide variety of subjects and scenes. All the images were resampled to a resolution of 256×256 pixels through the Lanczos algorithm [73]. The number of convolution kernels per layer was configured to 64, the number of total memory cells was configured to 3, and the number of residual units per cell was configured to 3. The kernel weights were initialised by the Xavier initialisation [74]. The model was trained on 8,000 images over 100 epochs by the Adam optimiser [75] with an initial learning rate set to 10−3 and scheduled to decay exponentially after every epoch. Large-scale assessments were conducted on 2,000 test images. The inference process was simulated on selected standard test images from the USC-SIPI database [76].


## 4.2. Visual Quality Analysis

Starting from Figures5 and 6, we can catch a glimpse of the extent to which the model can refine the pre-processed images. It can be observed that the visual quality of posterior images is better than that of prior images, especially at the edges and in textural areas. The same outcome is reflected in the peak signal-to-noise ratio (PSNR) of images, measured in decibel (dB). Results suggest that the neural network model indeed has a stronger ability to model nonlinearity and complex pattern.Figure 5
Visual comparisons between the prior and posterior predictors based on images from the USC-SIPI database. Numerical data express the PSNRs. (a) Ground truth/aeroplane. (b) Ground truth/Lena. (c) Ground truth/mandrill. (d) Ground truth/peppers. (e) Prior/32.9055 dB. (f) Prior/34.5666 dB. (g) Prior/28.5752 dB. (h) Prior/35.3336 dB. (i) Posterior/36.3609 dB. (j) Posterior/37.6200 dB. (k) Posterior/29.3495 dB. (l) Posterior/39.9355 dB.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)Figure 6
Visual comparisons between the prior and posterior predictors based on images from the BOSSbase. Numerical data express the PSNRs. (a) Ground truth/0021. (b) Ground truth/0916. (c) Ground truth/1366. (d) Ground truth/1953. (e) Prior/31.5765 dB. (f) Prior/32.4795 dB. (g) Prior/34.9142 dB. (h) Prior/28.4180 dB. (i) Posterior/33.6192 dB. (j) Posterior/34.6770 dB. (k) Posterior/37.5122 dB. (l) Posterior/31.7238 dB.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)


## 4.3. Entropy Analysis

Figure7 shows that the posterior error distribution is more concentrated and its entropy smaller, whereas the prior error distribution is comparatively more diffuse. However, it is striking that the height of the peak bin (usually h0) on the posterior histogram is not always higher than the height of the same bin on the prior histogram. A possible explanation would be that some image samples contain a relatively large number of smooth patches on which a naïve linear predictor may perform sufficiently well.Figure 7
Prediction error distributions. Numerical data express the entropy of distribution. (a) Aeroplane: prior/2.9529; posterior/2.6493. (b) Lena: prior/2.8683; posterior/2.5383. (c) Mandrill: prior/3.9392; posterior/3.8131. (d) Peppers: prior/2.7781; posterior/2.4074
(a)(b)(c)(d)


## 4.4. Cumulative Frequency Analysis

In order to better understand how the prior and posterior prediction errors distribute, we analyse their cumulative frequencies. Figure8 presents cumulative distribution function (CDF) plots, where the 95th percentile gives the maximum error value below which 95% of errors fall. It is evident that the rate of convergence of the posterior error distribution is faster than that of the prior error distribution, confirming again that posterior errors are more concentrated and the magnitude of these is smaller on average.Figure 8
Cumulative distribution function of prediction error distributions. Numerical data express the 95th percentile. (a) Aeroplane: prior/20; posterior/12. (b) Lena: prior/16; posterior/10. (c) Mandrill: prior/30; posterior/28. (d) Peppers: prior/15; posterior/7.
(a)(b)(c)(d)


## 4.5. Large-Scale Assessment

In addition to evaluating the performance on individual selected images from the USC-SIP database, we provide a large-scale assessment based on a large number of test samples from the BOSSbase. Figure9(a) depicts the probability density of PSNRs of prior and posterior images. Figure 9(b) shows the probability density of entropies of prior and posterior error distributions. Figure 9(c) reveals the average rates of convergence of prior and posterior errors. On average, the visual quality of the posterior errors is higher, the distribution of them is more peaked, and the convergence rate is faster.Figure 9
Statistical comparisons between prior and posterior predictors with respect to PSNRs of reference images, entropies of prediction error distributions, and cumulative distribution functions. (a) Prior/34.29 dB; posterior/37.27 dB. (b) Prior/2.863; posterior/2.546. (c) Prior/17; posterior/13.
(a)(b)(c)


## 4.6. Regression Analysis

While we have shown that our neural network model offers better visual quality and smaller entropy, it is still unclear how these factors may benefit steganographic capacity. As a consequence, we carried out regression analysis amongst the PSNR of reference images, entropy of prediction errors, and maximum embedding rate, measured in bits per pixel (bpp). Figure10 plots the results using the test samples from the BOSSbase with different threshold values θ which regulate the steganographic channel. As expected, the general trends suggest that the embedding rate is directly proportional to the PSNR of reference images and inversely proportional to the entropy of prediction errors.Figure 10
Regression analysis amongst PSNR of reference images, entropy of prediction errors, and maximum embedding rate at different threshold values that regulate the steganographic channel. (a)θ = 0. (b) θ = 1. (c) θ = 2. (d) θ = 3.
(a)(b)(c)(d)


## 4.7. Rate-Distortion Evaluation

We evaluate capacity and distortion by rate-distortion curves, as plotted in Figure11. It can be observed that the maximum embedding rate increases with the increase of the threshold (the width of steganographic channel). The reason is straightforward: an increase in the threshold implies an increase in the number of bins for carrying the message. In addition to this, the observations suggest that the maximum embedding rate tends to be smaller for images containing more complex textures. It is because the prediction errors of such images are less concentrated and thus fewer bins are covered within the steganographic channel. There is a gradual and steady decline in the visual quality of stego images as embedding rate increases. The difference between the rate-distortion performances of the prior and posterior predictors is subtle for a small threshold value, but it becomes significant as the threshold value grows, with the posterior outperforming the prior. The underlying explanation for the trend may be that the naïve predictor and the neural network model have similar abilities to estimate smooth patches, for which both methods can often estimate perfectly. Nonetheless, the latter excels over the former when estimating textural patches, for which neither methods can offer accurate prediction but the neural network gives smaller error magnitude in general. Figure 12 lists stego images generated by embedding a simulated message into the cover images. The intended message is often assumed to have been compressed and encrypted and thus can be reasonably simulated by a random bit stream and a random trit stream.Figure 11
Rate distortion curves of applying prior and posterior predictors at different threshold values that regulate the steganographic channel. (a) Aeroplane. (b) Lena. (c) Mandrill. (d) Peppers.
(a)(b)(c)(d)Figure 12
Stego images generated by embedding a random bitstream with different threshold values that regulate the steganographic channel. Numerical data express the PSNRs and maximum embedding rates. (a)θ = 0, 51.52 dB, 0.1652 bpp. (b) θ = 0, 51.51 dB, 0.1607 bpp. (c) θ = 0, 51.29 dB, 0.0470 bpp. (d) θ = 0, 51.50 dB, 0.1554 bpp. (e) θ = 1, 46.72 dB, 0.3240 bpp. (f) θ = 1, 46.71 dB, 0.3248 bpp. (g) θ = 1, 45.62 dB, 0.1057 bpp. (h) θ = 1, 46.68 dB, 0.3191 bpp. (i) θ = 2, 44.46 dB, 0.4039 bpp. (j) θ = 2, 44.53 dB, 0.4150 bpp. (k) θ = 2, 42.48 dB, 0.1581 bpp. (l) θ = 2, 44.53 dB, 0.4210 bpp.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)(k)(l)


## 5. Conclusions

This paper studies a neural analytics module compatible with the HS coding module. We propose a novel prediction mechanism which follows a two-step pipeline: first a pre-estimated image is generated by a conventional linear predictor and then the prior estimation is refined by an LSTM-based vision model called the MemNet. It is believed that this neural network model is to some extent biologically plausible and we have validated the effectiveness of the model for refining the prior estimation in terms of the visual quality and the entropy of error distribution. Furthermore, the impact of refinement to steganographic capacity has been analysed and a better rate-distortion performance was achieved. We envision that by joining this neural analytics module with a state-of-the-art HS coding module, the steganographic performance can be further improved. It is also interesting to investigate the possibility of combining different pre-processing predictors and post-processing neural network models to achieve a higher prediction accuracy. We hope this paper can prove instructive for future research on reversible steganography with deep learning.



---

*Source: 5580272-2021-04-05.xml*