Learning Iterative Neural Optimizers for Image Steganography
Authors: Xiangyu Chen
Varsha Kishore
Kilian Q Weinberger
Venue: ICLR 2023 poster (2023)

Abstract: Image steganography is the process of concealing secret information in images through imperceptible changes. 
Recent work has formulated this task as a classic constrained optimization problem. In this paper, we argue that image steganography is inherently performed on the (elusive) manifold of natural images, and propose an iterative neural network trained to perform the optimization steps. In contrast to classical optimization methods like L-BFGS or projected gradient descent, we train the neural network to also stay close to the manifold of natural images throughout the optimization. We show that our learned neural optimization is faster and more reliable than classical optimization approaches. In comparison to previous state-of-the-art encoder-decoder based steganography methods, it reduces the recovery error rate by multiple orders of magnitude and achieves zero error up to 3 bits per pixel (bpp) without the need for error-correcting codes. 

Published as a conference paper at ICLR 2023
LEARNING ITERATIVE NEURAL OPTIMIZERS FOR IM- AGE STEGANOGRAPHY
Xiangyu Chen, Varsha Kishore & Kilian Q Weinberger Department of Computer Science
Conell University Ithaca, NY 14850, USA {xc429, vk352, k ) corne
ABSTRACT
Image steganography is the process of concealing secret information in images through imperceptible changes. Recent work has formulated this task as a classic constrained optimization problem. In this paper, argue that image steganog- raphy is inherently performed on the (elusive) manifold of natural images, and propose an iterative neural network trained to perform the optimization steps. In contrast to classical optimization methods like L-BFGS or projected gradient de- scent, we train the neural network also stay close to the manifold natural im- ages throughout the optimization. We show that our learned neural optimization is faster and more reliable than classical optimization approaches. comparison to previous state-of-the-arr encoder-decoder based steganography methods, it re- duces the recovery error rate by multiple orders of magnitude and achieves zero error up 3 bits per pixel (bpp) without the need for error-crreccing codes.
INTRODUCTION
Image steganography aims alter image, to imperceptibly hide (secret) bit string, such that previously defined decoding method can then extract the message from the altered im- age. Steganography has been used in many applications such as digital watermarking to establish ownership (Cox et al., 2007), copyright certification (Bilal et al., 2014), anonymized image shar- ing (Kishore et al., 2021). and for hiding information coupled with images (e.g. patient name and ID in medical CT scans (Srinivasan et al., 2004)). Following the success of neural networks on var- ious tasks, prior work has used end-to-end encoder-decoder networks for steganography (Zhu al., 2018; Zhang et al., 2019; Hayes & Danezis, 2017a; Baluja, 2017). The encoder takes input an image X and message M, and produces steganographic image X that is visually imperceptible from the original image X. The decoder recovers the message M from the steganographic image X. Such methods can hide large amounts of data in images with great image quality, due to con- volutional networks' ability generate realistic outputs that close to the input image along the manifold of natural images (Zhu et al., 2016; Zhang et al., 2019). However, these approaches can only reliably encode bits per pixel. At higher bit rates, they suffer from poor recovery for the message M (around 5-25% at 4 bpp). Consequently, they cannot be used for some steganog- raphy applications that require the message to be recovered with 100.0% accuracy, for example, the message is encrypted or hashed. Although error- eeror-correcting codes (Crandall, 1998; Munuera, 2007) can be used recover spurious mistakes, their reliance on additional parity bits reduces the payload, negating much of the advantages of neural approaches.
rates
Recent work has ahandoned learned encoders altogether and reformulated image steganography as constrained optimization problem (Kishore et al., 2021), 1), where the steganographic image is opti- mized with respect to the outputs of a fixed (random or pre-trained) decoder - by employing tech- nique based adversarial image perturbations (Szegedy et al., 2013). The optimization problem is solved with off-the-shelve gradient-bart based optimizers, such as projected gradient descent (Carlini & Wagner, 2017) L-BFGS (Fletcher, 2013). Such approaches achieve low error rates with high payloads (2-3% error at 4 bpp), but are slow and prone to getting stuck local minima. Further,
"Equal contribution.

---

Published as a conference paper at ICLR 2023
each pixel is optimized in isolation, and pixel-level constraints only ensure that the steganographic image stays close to the input image according to an algebraic norm, rather than along the natural image manifold. Although similar manifold-unawarn approaches versarial attacks, steganography aims to precisely control millions of binary decoder outputs, instead of single class prediction; the resulting optimization problem is thus harder and prone producing unnatural-looking images.
successfully deployed for ad-
In this paper we introduce Learned Iterative Sleganography Optimization (LISO), a method that combines the ability of neural networks to learn image manifolds with the rigor of con- strained optimization. LISO is highly effi- cient in hiding messages inside images, and achieves very low message recovery error. LISO achieves these feats with an encoder- decoder-critic architecture. The encoder is learned recurrent neural network that itera- tively solves (Teed Deng, 2020; Alaluf et al., 2021) the constraint optimization problem pro- posed in Kishore et al. (2021) and mimics the steps of a gradient-based optimization method. Figure demonstrates how the steganographic image and error rate change over subsequent it- erations. We also use a critic network (Zhang al., 2019) to ensure the changes stay im- perceptible and that the steganographic im- age looks natural. Our resulting architecture can be trained end-to-end. We show that LISO learns a more efficient descent direction than standard (manifold unaware) optimization al- gorithms and produces better steganography results with great consistency. Finally, the error rate (almost) always be driven to flat O if the optimization is finished with few iterations of L-BFGS within the vicinity of LISO's solution (LISO + L-BFGS).
PSKR:
Perturbation
Stegano- graphic Image
Message Decoding Errce
Eser 59%
Eror 0.46% Step
Enor: 0.000%
Erox 0% Stop
Step
Step
Figure LISO iteratively optimizing a sample image with 3 bits encoded in each pixel. The covery error goes down drastically to exactly zero in only 8 steps. The perturbation (difference be- tween the steganographic image and the original image) is large in the beginning, but becomes im- perperceptible with LISO's s optimization, leading natural-looking steganographic image.
We evaluate the efficacy of LISO extensively across multiple datasets. We demonstrate that test- time, with unseen cover images and random bit strings, the optimizer can reliably circumvent bad local minima and find a low-error solution within only few iterative steps that already outperforms all previous sendddr-dcooderbbaeee approaches. If the optimization is followed by few additional updates of L-BFGS optimization, we can reliably reach 100% error-free recovery even with 3 bits per pixel (bpp) hidden information across all (thousands of) images we tested on.
Our contributions are as follows. We introduce novel gradient- based neural optimization algo- rithm, LISO, that learns preferred descent directions and is image manifold aware. 2) We show that its optimization process based on learned descent directions orders of magnitudes faster than classical optimization procedures. 3) As far as we know, our variant LISO + L-BFGS is by far the most accurate steganography algorithm in existence, resulting in perfect recovery on all images tried with 3bpp and the vast majority for 4bpp implying that it avoids bad local minima and can be deployed without error correcting code. The code for LISO is available at https: //gi////iii thub. httmm/cxy199//1111 LISSO.
2 RELATED WORK
Steganography. Classic steganography methods operate directly on the spatial image domain to encode data. Least Significant Bit (LSB) Steganography encodes data by replacing the least signifi- cant bits of the input image pixels with bits from the secret data sequentially (Chan & Cheng, 2004). Pixel-value Differencing (PVD) (Wu & Tsai, 2003) hides data by comparing the differences between the intensity values of two successive pixels. Following this, many methods data while minimizing distortion to the input images and these methods differed how the dis- tortion measured. Highly Undetectable Steganography (HUGO) (Pevný al., 2010) identifies
presented to hide
2

---

Published as a conference paper at ICLR 2023
pixels that can be modified without resulting in a large distortion based on handcrafted steganalytic features. Wavelet Obtained Weights (WOW) (Holub & Fridrich, 2012) uses syndrome- trellis codes to minimize the expected distortion and to identify noisy high-textured regions to modify.
With the advent of deep learning, many deep learning approaches have been proposed for steganog- raphy. These approaches either neural networks in some stages of the pipeline (e.g., as a feature extractor) or learn networks end-to-end for message encoding and decoding. HiDDeN (Zhu et al,, 2018) proposed an end-to-end encoder-decoder architecture to hide up 0.4 bpp of information in images. Hayes & Danezis (2017b) uses adversarial training to approximately hide 0.4 bpp in 32 x 32 images. SteganoGAN Zhang al. (2019) adopts adversarial training to generate steganographic im- ages with better quality, and can hide up to 6 bpp with eπror rates of about 13-33%. Tan et al. (2021) improved upon SteganoGAN and introduce CHAT-GAN which uses an additional channel attention module to identify favorable channels in feature maps to hide bits of information.
HiDDeN, SteganoGAN, and CHAT-GAN hide arbitrary bit strings but other methods hide (struc- tured) image within an image (Baluja, 2017; Dong et al., 2018; Yu, 2020; Rahim et al., 2018) by learning image priors and condensing information; arbitrary bit strings can not be similarly con- densed. Invertible Steganography Network (Lu et al., 2021) hides multiple images in single cover image using an invertible neural network. Deep Multi-Image Steganography with Private Keys (Shang et al., 2020) hides multiple images in cover image while producing many secret keys, each of which is used to extract exactly one of the hidden images. HiNet (Jing et al., 2021) is an invertible network hide information in the wavelet domain. Zhang et al. (2020) explore finding cover-agnostic universal perturbations that can be added to any natural image to hide a secret im- age. There have also been attempts conduct steganography with non-digital images (Tancik al., 2020; Wengrowski & Dana, 2019), but in this paper we focus on digital images.
Steganalysis. Many applications require that steganographic images are indistinguishable from nat- ural images. Steganalysis systems are used to detect whether secret information is hidden in images and they can broadly be divided into two categories statistical and neural. The former makes use of statistical tests on pixels to determine whether a message is encoded in an image (Dumitrescu et al., 2002b; Fridrich et al., 2001; Westfeld & Pfitzmann, 1999; Dumitrescu et al., 2002a), while the latter employs neural networks to detect the existence of hidden messages. Classic steganography methods can be easily detected by statistical tools, but more recent methods are complex and harder to detect with statistical tools and require neural tools. Neural steganalysis methods have shown state-of-the-ar performances for detecting steganography with high accuracy even when low pay- loads of information are hidden. GNCNN (Qian et al., 2016) used handerafted features and CNNs to perform steganalysis. Following that, many methods have used deep networks to automatically extract features and perform steganalysis (You et al., 2020; Qian al., 2015; Chen et al., 2017; Wu et al., 2019; Qian et al., 2018). In response to the advances in steganalysis, neural steganography methods add steganalysis systems into their end-to-end pipeline in order to find robust perturbations that evade detection by steganalysis systems (Shang et al., 2020; Dong et al., 2018).
Learning to Optimize. Recent methods have investigated incorporating optimization problems into neural network architectures (Amos & Kolter, 2017; Agrawal et al., 2019; Wang et al., 2019). These methods design custom neural network layers that mimic certain canonical optimization problems. As a result, parametrized optimization problems can be inserted into neural networks. Not all op- timization algorithms can easily be written as a layer and another strategy is to train a network to learn iterative updates from data (Adler & Öktem, 2018; Flynn et al., 2019; Gregor & LeCun, 2010). Most similar to our approach is arguably RAFT (Teed & Deng, 2020), which poses optical flow estimation descent direction. In a similar vein, we aim to learn the steganography optimization problem with a convolutional neural network without specially designed optimization layers.
optimization problem and uses recurrent neural network to find an optimal
3 NOTATION AND SETUP
Let X E [0, =33 denote an RGB cover image with height H and width W. The hidden message M is a bit string of length m H HxWx x B that is reshaped to match the cover image's size,
In practice, we PNG-24/JPGG with quantized pixel values.
3

---

Published as a conference paper at ICLR 2023
Le. M E {0,1)">xxBB, where payload B is the number of encoded bits per pixel (bpp). Image steganography aims to conceal M in steganographic image X E [0, 1| x that looks visually identical to X, such that M is transmitted undetected through X. A steganographic encoder Enc takes as imput X and M and outputs X. A decoder Dec recovers the message, M' = Dec (X), with minimal recovery error
llo (ideally = 0). Below we describe two recent paradigms for steganography (Appendix F summarizes the similarities and differences between the paradigms).
Learned Image Steganography, Recent deep-learning based image steganography meth- ods (Zhang et al., 2019; Zhu et al., 2018; Tan et al., 2021) use straight-
convolutional neural network architectures for encoders and decoders that maintain the same feature dimensions, H x W, throughout (i.e. no downsampling). SteganoGAN (Zhang et al., 2019) and CHAT-GAN (Tan et al., 2021) use such an encoder network to generate the steganographic image X with a single forward pass and decoder network to predict the message M'. The final layer of the decoder usually consists of H x Wx x B sigmoidal outputs, producing real values within [0, 1] that turn into the bi- nary recovered message M' after rounding. These methods also have a Critic/ Discriminator to predict whether an image looks natural. The encoder, decoder, and critic are all trained end-to-end.
Optimization- based Image Steganography. Given differentiable decoder with random or learned weights (from a method mentioned in the last paragraph). Kishore et al. (2021) express the steganog- raphy encoding step as a per sample optimization problem that is inspired by adversarial perturba- tions (Szegedy et al., 2013), Their method, Fixed Neural Network Steganography (FNNS), finds a steganographic image by solving the following constrained optimization problem while ensuring that the perturbed image lies within the [0, 1] 1| W hypercube:
min Lace (Dec [ xxxxx
(x)
.MM) ),M)+Nou + (x,x)
Luce (M', M) :- MM, log M') + ((1 - MM, - -M'))
Lqua
X
:
(I)
where (-) denotes the dot product operation, À is a weight factor and N xxW x3. Note that only the perturbed image X is being optimized and the decoder Dec is kept fixed throughout. The accu- racy loss Lice (M', M) is binary cross entropy loss that encourages the message recovery error to be low, and the quality loss Lqua( (X, X) uses mean squared error to ensure that the steganographic image looks similar to the cover image. We denote the objective in Equation as 8(X, M). Several solvers can be used to solve Equation 1 as shown in Algorithm 1, but the most popular choices are iterative, gradient- based hill-climbing algorithms. In the algorithm, 17 > denotes the step size, and (-) is an update function defined by the specific optimization method. The perturbation 5 is iteratively optimized to minimize the loss within the image pixel constraints.
Drawbacks of Existing Methods Fxisting learned methods are fast because single forward passes with the trained encoder and decoder are all that are needed for encoding and decoding, but they yield high error rates of up 5-25% for bpp. It is also hard to incorporate additional constraints to these models without retraining. On the other hand, optimization-bann based approaches can result in lower error rates, but they are much slower (requiring thousands of iterations). Furthermore, their resulting recovery error depends heavily on the weights of the decoder and the initialization of 8 (Kishore al., 2021). Especially with B 2 4, the optimization can get stuck in local minima.
4 LEARNED STEGANOGRAPHIC OPTIMIZATION
Ideally, we would like method that is both fast and has low recovery error, SO we combine ideas from learned methods and iterative optimization methods to propose Learned Iterative Steganog- raphy Optimization (LISO). LISO consists of an encoder, decoder, and critic network. but the encoder is itcrative and it approximates the function g(.) in Algorithm 1. The optimization proce- dure can be viewed as a recurrent neural network with hidden state that iteratively optimizes the perturbation 5 with respect to the loss f. At each iteration, it obtains the previous estimate 5,-1 and the gradient of é as input and produces new 8, The hidden state allows LISO learn what
II is not multiple of H W, it padded with zeros.
4

---

Published as a conference paper at ICLR 2023
optimization steps best suited for this task. After multiple iterative steps, the encoder outputs the steganographic image. The decoder is a simple ferd-forward convolutional neural network trained to retrieve the message from the steganographic image. The critic is neural classifier (akin to GAN discriminators) trained to discriminate between cover images and steganographic images (Hayes & Danezis, 2017b). As everything is differentiable, the iterative encoder, the decoder, and the critic can be jointly learned on a dataset of images.
Algorithm Iterative Optimization
LISO architecture. The LISO decoder and critic have convolutional blocks, each con- taining a convolutional layer (with 3 kernels and stride 1), batch-norm, and leaky-ReLU, similar to Zhang et al. (2019). In the decoder, the convolutional blocks are followed by an- W B
ōn for 1=1 to do ô, + δ
199
l l(x 16-1.M), ;,,,x,,-.
81-1
other convolutional layer to obtain dimensional output. In the critic, the convolu- tional blocks are followed by an adaptive mean pooling layer to obtain scalar output that de- notes a prediction of whether an image has hidden code.
+ X dr
Comz
perturbation 6.01
The core of LISO is its encoder, which has a re- current GRU-based update operator that func- tions like Algorithm 1. The function g() in Al- gorithm is approximated using a fully convo- lutional network designed around gated recur- rent unit (GRU) (Cho et al., 2014). The GRU has an internal state and keeps track of infor- mation relevant to the optimization (more de- tails about the GRU cell are in Appendix A). To avoid information loss, the LISO networks have layers, which
Com
WW EL
updtite S+
Nidee stite
Dona
penurbutton
COUEET imanee
Figure 2: Model architecture of LISO's iterative optimization network that functions as optimizer g in Algorithm 1.
no down wwwpssammmmppaaamaaiaaaaa means LISO can process images of any size. At high level, the LISO encoder (illustrated in Figure 2) takes as input the cover image X, the current perturbation 6,-1 and the gradient of the loss with respect to the perturbed image Vs, ,((X + 8 M) and predicts the perturbation d, for the next step. The GRU hidden state ho is initialized with a feature extractor (not shown) that is simple 3-layer fully convolutional network that extracts features from the input image. The input to the GRU in subsequent steps I, is constructed by concatenating the extracted features from current perturbation 8,-1, the gradient V 1,x + 8,-1, M) and the cover image X. The GRU unit up- dates its own hidden state h, and the hidden state is processed by additional convolutional layers to produce a gradient-type step update gt- The final update becomes 8, = 8 + 119t. where 1) is the step size for the new update. Perturbation 8, can also be clipped by truncation factor to limit how much the X changes. The steganographic image, that the final output the LISO encoder. is produced through recurrent applications of the iterative encoder; the final steganographic image becomes X = X + 17 E. 9t-
Training. The entire LISO pipeline (the iterative encoder, decoder, and critic) is trained end-to- end dataset of images. As with GAN training, optimize the critic and the encoder-decode networks in alternating steps. During the training, we compute loss for all intermediate updates with exponentially increasing weights TT- at step 1). With intermediate predictions denoted as X1. i....., Xx the loss is:
T Σx (M.X.) + ALyua (x.X.) + pLorii (x, (x.X.) [=1
Lirain
(2)
where E (0, 1) is decay factor and Lerit denotes the critic loss (with weight μr > 0). Despite the predictions being sub-optimal from earlier iterations, we still make use of the loss from those steps because they do provide useful update directions for later iterations. Note that this loss is similar to Equation 1, but there is additional critic loss term that ensures that the steganographic image looks like a natural image (Zhang et al., 2019). As the qualitative and critic losses are computed at all steps, the optimizer encouraged stay on the manifold of natural images throughout.
5

---

Published as a conference paper at ICLR 2023
Table 1: Image steganography results of LISO on three different datasets with different payloads.
Error Rate (%)
FSNR
SSIM
Method
[bil
bits
3 bits
bbis
1
2 bits
bils
4 bils
bii
2 bits
biils
bils
SteganoCiAN
5.12
8.31
13.74
22.85
21.33
21.06
21.42
21.84
0.76
0.76
0.77
0.78
Div2k
FNNS-D
0.00
0.00
0.10
5.45
29.30
26.25
22.90
25.74
0.82
0.73
0.53
0.65
LISO
4B-05
4E-04
2E-03
3E-02
33.83
33.18
30.24
27.37
0.90
0.90
0.85
0.76
LISO+L-BFGS
0.00
0.00
0.00
IE-05
33.12
32.77
29.58
27.14
0.89
0.89
0.82
0.74
SteganoGAN
3.94
7.36
8.84
10.00
25.98
25.53
25.70
25.08
0.85
0.86
0.85
0.82
MS COCO CelebA
FNNS-D
0.00
0.00
0.00
3,17
36.06
34,43
30.05
33.92
0.87
0.86
0.71
0.84
LISO
4K-04
1E-03
4H-03
1F-01
35.62
36.02
32.25
30.07
0.89
0.90
0.82
0.79
LISO+L-BRGS
0.00
0.00
0.00
71:05
35.40
35.63
31.61
28.36
0.89
0.89
0.79
SteganoGAN
3.40
6.29
11.13
15.70
25.32
24.27
25.01
24.94
0.84
0.82
0.82
0.82
FNNS-D
0.00
0.00
0.00
13.65
37.94
34,51
27.77
34.78
0.95
0.90
0.72
0.88
LISO
2IB-04
5E-04
3E-03
4E-02
33.83
32.70
27.98
25. 46
0.90
0.89
0.75
0.68
LISO+L-BFGS
0.00
0.00
0.00
1B-05
33.42
31.80
25.17
0.90
0.87
0.70
Inference. The learned LISO iterative encoder and decoder are used during inference. The iterative encoder performs a gradient-style optimization on the cover image to produce the steganographic image with the desired hidden message. Mathematical optimization algorithms like L-BFGS have strong theoretical convergence guarantees, which assure fast and highly accurate convergence within the vicinity of global minimum, but for non-convex optimization problems they have a tendency to get stuck in local minima. Although our learned optimizer lacks theoretical convergence guar- antees, LISO is very efficient at finding local vicinity of good minima for natural images it can reliably find solutions with only a few bits decoded incorrectly (on the order of 0.01 %) as shown in Table 1. FNNS (Kishore et al., 2021) 1) works in complementary manner trained encoder-decoder networks and optimizes steganographic images using fixed networks and off-the-shelf optimization algorithms. Consequently, we can use LISO in conjunction with FNNS; we refer to this procedure as LISO+(PGD) or LISO+(LBFGS) dependent on which optimization algorithm is used for FNNS optimization. Concretely, we encode message into cover image by using the LISO encoder, and then use L-BFGS/PGD to further reduce the error rate. The error rate after LISO is already quite low and hence only a few additional L-BFGS iterations are required to reduce the error. Compared to FNNS, LISO can reliably achieve 0% error for up to 3 hpp, with drastically fewer iterations.
5 EXPERIMENTS AND DISCUSSION
We evaluate our method on three public datasets: 1) Div2k (Agustsson & Timofte, 2017) which a scenic images dataset, 2) CelebA (Liu et al., 2018) which consists of facial images of celebrities, and 3) MS COCO (Lin et al., 2014) which contains images of common household objects and scenes. For CelebA and MS COCO we use the first 1000 for validation and the following 1,000 for training. Note that 1,000 training images are sufficient for LISO to achieve SOTA performance, and using full datasets will only make training longer. The messages are random binary bit strings sampled from an independent Bernoulli distribution with = 0.5, which is close to the distribution of compressed/encryypte messages.
During training, we set the number of encoder iterations T = 15, the step size = 1, the decay
y 0.8 and loss weights X = 1u- = During inference. we use smaller step size 1) = 0.1 for larger number of iterations T'; we iterate until the error rate converges. We show the average number of iterations required minimize the recovery error in Table 3. The average number is consistently below the number of iterations used during training T = 15, although some images require up to 25 iterations. These results suggest that in practice LISO just needs a few iterations to achieve low recovery error rates. Figure 3 shows how the recovery error decreases over time during testing. The monotonicity of the graphs indicates that the optimization loss is well aligned with recovery error and that LISO is successful at learning to find suitable descent directions (more details are in the subsequent optimization analysis). The final steganographic images are stored in PNG format.
Image Steganography Performance. Most traditional steganography methods hide < 1 bpp mes- sages in images, and methods that hide large payloads of information usually hide structured infor- mation like images. Very few methods are designed for hiding high payload arbitrary bit string messages in color images. We compare LISO with three other methods that are designed to do so, SteganoGAN (Zhang et al., 2019), FNNS (Kishore et al., 2021) and CHAT-GAN (Tan et al., 2021).

---

Published as a conference paper at ICLR 2023
Since the source code for CHAT-GAN not available, we were only able compare against the numbers provided in the paper on MS-COCO and these results are presented in Appendix E. In Table 1, evaluate LISO's performance image steganography in terms of both message recov- ery accuracy and image quality. Following previous work (Zhang et al., 2019), we measure how much the steganographic image changes by computing the structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) (Wang et al., 2004) between the cover and steganographic im- ages. In Appendix B, we also evaluate with the Learned Perceptual Image Patch Similarity (LPIPS) metric (Zhang et al., 2018) and find trends similar to that of PSNR and SSIM.
we observe that the error rates of LISO are an order of magnitude lower when com- pared to SteganoGAN. Interestingly, the image quality as measured by PSNR scores is also superior. LISO able to achieve an error rate of exactly 0% for payloads Σ hpp for all images when paired with few steps of direct L-BFGS optimization (the numbers in Table 1 are averaged and for many samples 0% error is obtained just with LISO). Using a learned iterative method, we are able to achieve the same performance as that obtained from a direct optimization method (FNNS), but in far fewer optimization steps and time (see Table 4). Essentially LISO's s ability to learn from past optimizations and to stay on the image manifold, allows it to achieve low error rates, high image quality, and initializations (for further L-BFGS optimization) near a good minimum.
From Table
Figure 4 shows the steganographic images produced by LISO under different payloads. Qualitatively we observe that the image quality is quite good even when a large number of bits per pixel are hidden in the cover images. The CelehA example shows rare example where the hue changes slightly while the image remains un-pixelated and natural: in contrast, the low-quality steganographic images produced by FNNS tend to be pixelated. Different from previous image steganography methods that produce unnatural Gaussian noise, LISO's perturbation is smooth and natural because LISO implicitly learns the manifold of natural images and produces steganographic images that stay close it. These color shifts disappear when a larger image quality factor À or more diverse dataset is used (a similar observation was made by Upchurch al. (2017) for image transformations). We can also explicitly control the trade-off between error rate and image quality by changing the image quality factors À and μ on the loss terms if desired. Additional qualitative images obtained by varying these provided in Appendix J.
Optimization with LISO. In Figure 3, we compare optimizing with the LISO encoder against popular numerical optimization al- gorithms including projected gradient de- scent (Madry et al., 2017) and L-BFGS (Liu & Nocedal, 1989) the steganographic encoding task. Note that L-BFGS- L-BFGS-Random is equivalent to FNNS-R and L-BFGSS-reerained equiva- lent to FNNS-D (Kishore et al., 2021). LISO converges faster than any other optimization al- gorithm and this validates our hypothesis that LISO can learn a descent direction that is bet- ter than the one found by common gradient- based optimization methods; LISO converges to 100x lower loss and 80x lower error rate. This is because the LISO encoder can lever- age information learned from optimizing many samples, while off-the-shelf gradient-based based methods perform per-sample optimization. Further- more, we show that recovery error rate from LISO can further be reduced by taking only few additional L-BFGS steps. This suggests that the LISO encoder output is quite close to the optimum and is good initialization for using conventional optimization methods like L-BFGS or PGD.
10a
10-1
1
Error Rate (1g
-
10
PGD PGD-Ran Random
L-8GGS-Random
10->
PGD Pretra ined
L-FFGS- Pretrained
10-c
LISO
LISO+PGD
10-7
USO+LLBFGS
400
1000
Number Iterations
Figure 3: Error-iteration curves for different opti- mization methods (at 4 bpp). The pretrained net- work is SteganoGAN.
Steganography with JPEG Compression. JPEG (Wallace, 1992) is a commonly used lossy im- age compression method, which removes high-frequency components for reduced file size. JPEG- compression-resistan steganography is hard because steganography and JPEG compression have opposing objectives; steganography attempts to encode information through small imperceptible changes and JPEG compression attempts to eliminate texture details. General-purpose steganogra- phy methods have about 50% recovery error rate when decoding JPEG-compressed images. Many methods have been specifically developed to perform
steganography to hide < 0.5
7

---

Published as a conference paper at ICLR 2023
Image
Pixel
Bits Per Pixel
Bits Pixel
Pixel
Div2K
CalebA
MS-COCO
Figure 4: Random images with corresponding steganographic images under different payloads.
bpp messages (Zhang et al., 2015; Fan et al., 2011; Zhang et al., 2016; Holub et al., 2014). Some of these find perturbations specifically in low-frequency space to avoid being removed by compression. HiDDeN (Zhu et al., 2018) and FNNS (Kishore et al., 2021) include differentiable JPEG layers to approximate compression part of their network architecture. Following (Athalye et al., 2018), adopt an approximate JPRG layer where the forward pass performs normal JPEG compression and the backward pass is identity function. We call this adapted method LISO-JPEG. Table 2 com- pares LISO-JPEG with existing methods. LISO is able to find perturbations that won't 't be lost in JPEG compression, while maintaining low error rate (<IE-0I). The PSNR values for LISO-JPEG are low but qualitative 1pp steganographic image examples from LISO-JPEG (see Figure 5) show that there only slight pixelation. seen in Table 2, the PSNR of FNNS-JPEG and LISO-JPEG are similar but LISO-JPEG has significantly lower eπro rate. Higher quality steganographic im- ages can be obtained by training LISO model with a higher weight on the qualitative loss l Luguas but this will cause slight increase in the error rate.
Table 2:
LISO-JPEG results compared with baselines. JPEG quality 80 is used.
Method LISO-PNG
bpp
Error (%)
PSNR t
2E-05
31.99
FNNS-JPEG
32.03
22.65
HiDDeN
0.2
2 10
unknown
LISO-JPEG
6E-02
19.72
Figure 5: LISO-JPEG steganographic images.
Computational Time. One of the advantages of using a learned method as opposed to an optimiza- tion method is that it is much faster. The average inference time required for the diffferent learned and optimization-bant bbasdd methods is presented in Table 4. The fastest method is SteganoGAN be- cause all that required use SteganoGAN is single forward passes. The FNNS variants, on the other hand, are much slower. Our proposed method LISO, is slightly slower than SteganoGAN but not by a significant magnitude. The reported times were the average times on Div2k's validation set and the methods were run on Nvidia Titan RTX GPU.
Steganalysis. To investigate the robustness our method against statistical detection techniques, we use Stegexpose (Boehm, 2014), tool devised for steganography detection using four well-known steganalysis approaches and find that the detection rate is lower than 25% for images produced by LISO. Recently, more powerful neural-based methods are being used to detect steganography (see related work). is hard for any steganography method, including classical methods like WOW and S-UNIWARD, to evade detection from these neural methods even with low payload messages of about 0.4bpp (You et al., 2020). However, we show that in the following two scenarios we can evade detection by Sia (You al., 2020).
8

---

Published as a conference paper at ICLR 2023
Table 4: Average computation time (in seconds). (* indicates the optimization was stopped after zero error was reached.)
Table 3: Number of iterations it takes to reach optimum under payload of 1-4 bits per pixel on Div2k's validation set.
Method
1 bit
2 bits
3 bits
4 bits
Bits per Pixel
Iterations (avg)
Max Iterations
SteganoGiAN
0.09
0.09
0.08
0.09
6.44
18
FNNS-R
42.18
114.44
156.91
159.19
2
9.50
17
FNNS-D
4.95*
10.53*
44.39
44.29
3
11,36
21
LISO
0.16
0.29
0.32
0.33
4
11.15
24
LISO+LBFGS
1.26*
2.81
6.64
28,59
Table 5: Steganalysis results using images produced by different methods, evaluated on MS-COCO
dataset. SG is used to denote SteganoGAAN.
Setting
Method
Error
Rate (9))
FSNR
Detection Accuracy Rate (%)
1 bit
2 bits
bits
bits
bbt
2 bits
bits
bits
I bit
2 bits
3
bbtts
4 bits
6.29
11.13
25.32
24.27
25.01
w/o defense
FNNS-D
0.00
0.00
0.01
13.65
37.94
34.51
27.77
34.78
73
75
99
98
LISO
1E:04
3E.04
5B-03
4F:02
33,83
34.15
30.08
24.58
51
42
98
FNNS-D
0.00
0.00
0.01
22.28
22.35
21.2
21.36
12
14
w/ defense
LISO
4E-03
1E-02
8E-03
5E-02
31.62
28.67
26.63
25.10
4
8
5
In the first scenario, "w/o defense", the steganography model M trained without any explicit precautions avoid steganalysis detection. We assume the attacker (i.e. the party applying ste- ganalysis) knows the model architecture of M but has no access to the actual weights, exact training set, or hyperparameters. However, they can train a surrogate model M' in order create their own training set of steganographic images. To simulate this setting, we used a steganalysis model trained on CelebA detect steganographic MS COCO images. From Table we see that in the "w/o de- fense" scenario. steganographic images from SteganoGAN and FNNS can be detected much more easily when compared to images from LISO.
In the second scenario, "w/ defense", we take advantage of the fact that neural steganalysis methods are fully differentiable, and both LISO and FNNS involve gradient-based optimization. It is there- fore possible to avoid detection by adding additional loss term from the steganalysis system into the LISO optimization or the FNNS optimization. Specifically, during evaluation we add the logit value of the steganographic class to the loss if an image is detected as steganographic. Note that we cannot add an additional loss term to SteganoGAN during inference because SteganoGAN just uses forward passes of a trained neural network for inference. We also tried training SteganoGAN network with an auxiliary steganalysis loss and found that we couldn't 't reliably avoid detection even when we did so. As seen Table 5, we obtain single-digit detection accuracy rates for LISO in the "w/ defense" scenario for all bit rates. Compared to FNNS, LISO is able to avoid detection more effectively. The image quality of steganographie images produced by LISO also better than those produced by FNNS under this scenario. We also tested LISO with two additional steganalysis sys- tems, SRNet (Boroumand et al., 2018) and XuNet (Xu et al., 2016), and the corresponding results are presented in Appendix H.
CONCLUSION
We propose a novel iterative encoder-based method, LISO, for image steganography. The LISO encoder is designed to learn the update rule of general gradient-based optimization algorithm for steganography, and is trained simultaneously with corresponding decoder. learns more efficient dynamic update rule for steganography when compared to PGD or L-BFGS, and the learned decoder is particularly suitable for further L-BFGS optimization. LISO also implicitly learns the manifold of images and therefore produces high-quality steganographic images. LISO achieves state-of-the-ane results for steganography, while being fairly fast. It is also flexible and can incorporate additional constraints, like producing JPEG-resistant steganographic images making steganographic images undetectable by specific steganalysis systems. In the future, we plan to investigate ways to increase image quality while maintaining low error for JPEG-resistant steganography.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This research is supported by grants from DARPA AIE program, Geometries of Learning (HR001 12290078), DARPA Techniques for Machine Vision Disruption grant (HR001 12090091] I), the National Science Foundation NSF (I1--100716, II112600, IIS-1149888, and I1242222,2,, and the Cornell Center for Materials Research with funding from the NSF MRSEC program (DMR- 1719875). We would like thank Oliver Richardson and all the reviewers for their feedback.
REFERENCES
Jonas Adler and Ozan Öktem. Learned primal-dual reconstruction. IEEE transactions on medical imaging. 1323):1322-3332,222 2018.
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and Zico Kolter. Differentiable convex optimization layers. arXiv preprint arXiv:1910.12430,, 2019.
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 126-135, 2017.
Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or. Restyle: A residual-based stylegan encoder via iterative refinement. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 671 1-6700, 2021.
Brandon Amos and Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, pp. 136-145. PMLR, 2017.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense se- curity: Circumventing defenses adversarial examples. In International conference on machine learning, pp. 274-283. PMLR, 2018.
Shumeet Baluja. Hiding images in plain sight: Deep steganography. Advances in Neural Information Processing Systems, (0022099-2079, 2017.
Ifra Bilal, Rajiv Kumar, Mahendra Singh Roj. and P K Mishra. Recent advancement in audio steganography. In 2014 International Conference on Parallel, Distributed and Grid Computing, pp. 402-405, 2014. doi: 10.1 10.1009PPDG.2204.703339
Benedikt Bochm. Stegexpose-a tool for detecting lsb steganography.
arXiv preprint
arXhv:1410.6656, 2014.
Mehdi Boroumand, Mo Chen, and Jessica Fridrich. Deep residual network for steganalysis of digital images. IEEE Transactions Information Forensics and Security, 14(5): 181-1193, 2018.
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium Security and Privacy, pp. 39-57, 2017.
Chi-Kwong Chan and Lee-Ming Cheng. Hiding data in images by simple Isb substitution. Pattern recognition, 37(3):469 4744, 2004.
Mo Chen, Vahid Sedighi. Mehdi Boroumand, and Jessica Fridrich. ppeg-phase-aware convolutional neural network for steganalysis of jpeg images. In Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security, pp. 75-84, 2017.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1220 2014.
Ingemar Cox, Matthew Miller, Jeffrey Bloom, Jessica Fridrich, and Ton Kalker. Digital watermark- ing and steganography. Morgan kaufmann, 2007.
Ron Crandall. Some notes on steganography. Posted steganography mailing list, pp. 1-6, 1998.
10

---

Published as a conference paper at ICLR 2023
Shiqi Dong, Ru Zhang, and Jianyi Liu. Invisible steganography via generative adversarial network. arXiv preprint arXiv: 111111107.81111 4, 2018.
Sorina Dumitrescu, Xiaolin Wu, and Nasir Memon. On steganalysis of random Isb embedding in continuous-tone images. In Proceedings. International Conference on Image Processing, vol- ume 3, pp. 641-644. IEEE, 2002a.
Sorina Dumitrescu, Xiaolin Wu, and Zhe Wang. Detection of Isb steganography via sample pair analysis. In International workshop on information hiding, pp. 355-372. Springer, 2002b.
Chi-Hung Fan, Hui-Y Yu Huang, and Wen-Hsing Hsu. A robust watermarking technique resistant jpeg compression. J. Inf. Sci. Eng., 27(1):163-180, 2011.
Roger Fletcher. Practical methods of optimization. John Wiley & Sons, 2013.
John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2367-22 2376, 2019.
Jessica Fridrich, Miroslav Goljan, and Rui Du. Detecting Isb steganography in color, and gray-scale images. IEEE multimedia, 8(4):22 28, 2001.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference international conference on machine learning, pp. 399-406, 2010.
Jamie Hayes and George Danezis. Generating steganographic images via adversarial training. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017a. URL htttps://prooeeeddngg. neuriig...euripsp. cc/paperr2017/Eile) pdf.
Jamie Hayes and George Danezis. Generating steganographic images via adversarial training. Ad- vances in neural information processing systems, 30, 2017b.
Vojtëch Holub and Jessica Fridrich. Designing steganographic distortion using directional filters. In 2012 IEEE International workshop on information forensics and security (WIFS), pp. 234-239. IEEE, 2012.
Vojtëch Holub, Jessica Fridrich, and Tomáš Denemark. Universal distortion function for steganog- raphy in an arbitrary domain. EURASIP Journal on Information Security, 2014(1):1-13, 2014.
Junpeng Jing, Xin Deng, Mai Xu, Jianyi Wang, and Zhenyu Guan. Hinet: Deep image hiding by invertible network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4732-4742 2021.
Varsha Kishore, Xiangyu Chen, Yan Wang, Boyi Li, and Kilian Q Weinberger. Fixed neural network steganography: Train the images, not the network. In International Conference Learning Representations, 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Dong C Liu and Jorge Nocedal, On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1):503-522) 1989.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(201 11; 2018.
Shao-Ping Lu, Rong Wang, Tao Zhong, and Paul L Rosin. Large-capacity image steganography based on invertible neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10816 10816-10825, 2021.

---

Published as a conference paper at ICLR 2023
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.060S, 2017.
Carlos Munuera. Steganography and error-correcting codes. Signal Processing. 87(6):1528-1555 2007.
Tomáš Pevný, Tomáš Filler, and Patrick Bas. Using high-dimensional image models to perform highly undetectable steganography. In International Workshop on Information Hiding, pp. 161- 177. Springer, 2010.
Yinlong Qian, Jing Dong, Wei Wang, and Tieniu Tan. Deep learning for steganalysis via convolu- tional neural networks. In Media Watermarking, Security, and Forensics 2015, volume 9409, pp. 94090J. International Society for Optics and Photonics, 2015.
Yinlong Qian, Jing Dong, Wei Wang, and Tieniu Tan. Learning and transferring representations for image steganalysis using convolutional neural network. In 2016 IEEE international conference on image processing (ICIP), pp. 2752-2756. IEEE, 2016.
Yinlong Qian, Jing Dong, Wei Wang, and Tieniu Tan. Feature learning for steganalysis using volutional neural networks. Multimedia Tools and Applications, 77(15):19333-196 19657, 2018.
Rafia Rahim, Shahroz Nadeem, et al. End-to-end trained cnn encoder-decoder networks for im- age steganography. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 0-0, 2018
Yueyun Shang. Shunzhi Jiang, Dengpan Ye, and Jiaqing Huang. Enhancing the security of deep learning steganography via adversarial examples. Mathematics, 8(9):1446, 2020.
Yeshwanth Srinivasan, Brian Nutter, Sunanda Mitra, Benny Phillips, and Daron Ferris. Secure transmission of medical records using high capacity steganography. In Proceedings. 17th IEEE Symposium Computer- Based Medical Systems, pp. pp. 122-127. IEBE, 2004.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: arXiv:1312.66 131266199, 2013.
Jingxuan Tan, Xin Liao, Jiate Liu, Yun Cao, and Hongbo Jiang. Channel attention image steganog- raphy with generative adversarial networks. IEEE Transactions on Network Science and Engi- neering, 9(2):888-90) 2021.
Matthew Tancik, Ben Mildenhall, and Ren Ng. Stegastamp: Invisible hyperlinks physical pho- tographs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 2117-2126, 2020.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pp. 402-419. Springer, 2020.
Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Weinberger. Deep feature interpolation for image content changes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7064-7073, 2017.
Gregory K Wallace. The jpeg still picture compression standard. JEEE transactions on consumer electronics, :xvixxxvii, 1992.
Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. SATNet: Bridging deep learning and logical reasoning using differentiable satisfiability solver. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3545-6554. PMLR, 09-15 Jun 2019. URL https:/ pttppppooonn....eooe www.s/sm...p..ss...mmmmmmmm wwtppppssssssseeeeeeeep 1eraanglee. html.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4))600- 612, 2004.
12

---

Published as a conference paper at ICLR 2023
Eric Wengrowski and Kristin Dana. Light field messaging with deep photographic steganography. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1515-1524, 2019.
Andreas Westfeld and Andreas Pfitzmann. Attacks on steganographic systems. In International workshop on information hiding, pp. 61-76. Springer, 1999.
Da-Chun Wu and Wen-Hsiang Tsai. A steganographic method for images by pixel-value differenc- ing. Pattern recognition letters, 2409-10):6113-666, 2003.
Songtao Wu, Sheng-hua Zhong, and Yan Liu. A novel convolutional neural network for image
steganalysis with shared normalization. IEEE Transactions Multimedia, 22(1):256-270, 2019.
Guanshuo Xu, Han-Zhou Wu, and Yun-Qing Shi. Structural design of convolutional neural networks for steganalysis. IEEE Signal Processing Letters, 2365):00--712, 2016.
Weike You, Hong Zhang, and Xianfeng Zhao. A siamese enn for image steganalysis. IEEE Trans- actions on Information Forensics and Security, 16:291-306, 2020.
Chong Yu. Attention based data hiding with generative adversarial networks. In Proceedings of the
AAAI Conference Artificial Intelligence, volume 34, pp. 1120-1128 2020.
Chaoning Zhang, Philipp Benz, Adil Karjauv, Geng Sun, and In So Kweon. Udh: Universal deep hiding for steganography, watermarking, and light field messaging. Advances in Neural Informa- fion Processing Systems, 13:10223-10234 2020.
Kevin Alex Zhang, Alfredo Infata-Infante, Lei Xu, and Kalyan Veeramachaneni. Steganogan: High 2019.
capacity image steganography with gans, arXiv preprint
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018
Yi Zhang, Xiangyang Luo, Chunfang Yang. Dengpan Ye, and Fenlin Liu. A jpeg-compression resistant adaptive steganography based on relative relationship between det coefficients. In 2015 10th International Conference on Availability, Reliability and Security, pp. 461-466. IEEE, 2015.
Yi Zhang, Xiangyang Luo, Chunfang Yang, Dengpan Ye, and Fenlin Liu. framework adaptive steganography resisting jpeg compression and detection. Security and Communication Networks, (15):2957-2971, 2016.
Jiren Zhu, Russell Kaplan, Justin Johnson, and Li Fei-Fei. Hidden: Hiding data with deep networks. In Proceedings of the European conference computer vision (ECCV), pp. 657-672, 2018.
Jun-Yan Zhu, Philipp Krähenbühh,, Eli Shechtman, and Alexei A Efros. Generative visual manipu- lation on the natural image manifold. In European conference computer vision, pp. 597-613. Springer, 2016.
13

---

Published as a conference paper at ICLR 2023
A GRU OPERATIONS
o (ConVaxa ([ [c), Wz))
F: o (Convax3 (f1--1: Ftl, W+))
h 1 tanh (Convax3 ([rr 11-1, Rr,, Wx))
h: (1 2t) h4-1 Zt h
As described in the paper, the Gated Recurrent Unit (GRU) is integral part of the LISO encoder. The operations of the GRU used in LISO are defined in Equation 3, where 2; is the input to the GRU in 1th iteration, h, the GRU's hidden state, and Wz, W,, W,. are GRU's weight matrices. For LISO, 2t is the concatenation features extracted from the input image, the gradient from the loss, and the perturbation.
B LPIPS RESULTS
In Table 3 of the main paper, used PSNR and SSIM evaluate image similarity between the cover images and the corresponding steganographic images. In Table 6 we show the results of using (LPIPS) (Zhang al., 2018), newer perceptual image similarity metric, evaluate image similarity between cover and steganographic images. We observe similar trends as with PSNR and SSIM.
LPIPS 1
Dataset
Method
bit
bits
bits
bits
SteganoGAN
0.13
0.13
0.14
0.14
LISO
0.04
0.05
0.06
0.08
Div2k
FNNS-D
0.01
0.02
0.08
0.07
LISO+L-BFGS
0.04
0.05
0.07
0.08
SteganoGAN
0.15
0.15
0.14
0.15
LISO
0.09
0.08
0.12
0.15
CelebA
FNNS-D
0.07
0.08
0.15
0.15
LISO+L-BFGS
0.06
0.06
0.11
0.14
SteganoGAN
0.12
0.13
0.13
0.13
LISO
0.04
0.05
0.09
0.14
MS COCO
FNNS-D
0.01
0.01
0.05
0.05
LISO+L-BFGS
0.07
0.06
0.09
0.15
Table 6: Steganographie image quality measured with the LPIPS metric (Zhang et 2018). Lower
numbers indicate better image quality.
C ERROR VS ITERATION
Figure 6 shows how the recovery сπоо rate decreases every iteration. As we see, the error rate monotonically decreases and this implies that LISO leams good descent direction. We also that 15-20 iterations are enough for the error rate to converge.
D LOss FOR DIFFERENT OPTIMIZATION METHODS
Figure analogous to figure 4 in the main paper. However, instead of seeing how the error rate decreases for different optimization methods, see how the loss decreases. Note that the first data point we plot for each curve is after iteration and we do so to improve the visualization. Similar to figure 4, we that the loss for LISO optimization decreases much faster than any other method.
14

---

Published as a conference paper at ICLR 2023
10"
104
PGD
Decoding Error Rate (16
LBFds
10
PGD- Pretrredrdded
LEFGG-Pruttranned
10"2
USO
107
LISO+PGD
10-3
L
bits
20-4
bits
10"
bits
10-1
4 bits
10 Number of Iterations" 7
15
200
400
600
800
1000
Number of Iterations
Figure 6: LISO recovery error rate with regard to number of iterations I under different bit rates, evaluated on Div2k's validation set. Y-axis is shown in the log scale.
Figure Loss-iteration curves for different optimization methods (at 4 bpp). The pre- trained network is SteganoGAN.
E COMPARISON WITH CHAT-GAN
Tan al. (2021) present steganography results for hiding 1-4 bpp messages in images from the COCO dataset using CHAT-GAN, We train LISO using different À weights to match the error rates of CHAT-GAN
listed in their paper) as closely as possible and the results are presented in Table 7; we use À 100 for 1-2 bpp and X 20 for 3-4 bpp for training LISO. From the table, we see that error rates and SSIM values are comparable for both methods, CHAT-GAN obtains better PSNR numbers. This is likely due to the fact that CHAT-GAN has specially designed feature attention module to hide bits imperceptibly. However, as evidenced by both the SSIM values and the qualitative images below, steganographic images from LISO visually look indistinguishable from the cover images despite having lower PSNR values (PSNR absolute mean squared difference in pixel values and two images can look similar despite have low PSNR). Moreover, note that the contributions of our paper are orthogonal to the contributions of Tan al. (2021). We also combine our iterative LISO method with the CHAT-GAN encoder network architecture (Tan et al., 2021) to take ad advantage of both methods and to perform steganography with higher payloads and harder images.
local statistic and it measures the
Capacity
Method
Error (%)
PSNR f
SSIM
LISO
0.34
38.02
0.99
1 bpp
CHAT-GAN
0.93
46.42
0.99
LISO
1.36
34.25
0.98
2 bpp
CHAT-GAN
1.20
43.17
0.99
LISO
0.60
30.06
0.93
3 bpp
CHAT-GAN
3.82
41.84
0.99
LISO
4.87
26.16
0.83
4 bpp
CHAT-GAN
5.44
38.92
0.95
Table 7: Comparing the performance of LISO with CHAT-GAN
F COMPARISON OF DIFFERENT STEGANOGRAPHY METHODS
Table compares LISO with both learned methods like SteganoGAN (Zhang et al., 2019) and CHAT-GAN (Tan et al., 2021), and optimization based methods methods like FNNS (Kishore al., 2021). We see that LISO has a trained encoder like learned methods, but its learned encoder is iterative and emulates the optimization algorithm in optimization-hasee steganography methods.
15

---

Published as a conference paper at ICLR 2023
Method
Type Leurned
Encoder
Speed
Decoder
Learned Methods
Learned 1-step Network
Fast
Learned Network
FNNS-R (Kishore et al., 2021)
Optimization Optimization Hybrid
L-BFGS/ PGD
Slow
Random Network
FNNS-D (Kishore et al., 2021)
L-BFGS PGD
Slow
Fixed Pre-trained Network
LISO (ours)
Learned Iterative Network
Fast
Learned Network
Table 8: A comparison of how different learned and optimization based steganography methods differ from each other.
G LISO RESULTS WITH MSE LOSS WEIGHT À = 10 FOR HIGHER IMAGE QUALITY
If higher image quality is desired, higher mse loss weight À can be used. Consequently, the image quality will be better but the eгrоr rate will be worse. We repeat the experiments in Table I of the main paper with mse loss weight A = 10, and show the results in Table 9. As seen the table, the error rate is slightly worse (it's still less than 1 1.5%), but the image quality is better. We can further
increase the value of to obtain even better image quality.
Rate
PSNR
SSIM
Dataset
Method
bii
bits
3 bits
4 bits
bit
2 bits
3 bits
4 hits
bit
bits
3 bits
4 hiis
Div2k
LISO
5.1E-
1.0E-2
6.5E-2
3.8E-1
36.40
36.62
32.85
30.88
0.95
0.95
0.91
0.87
LISO
0
3.8E-6
5.8E-2
34.86
35.49
32.41
29.45
0.92
0.93
0.82
CelebA
LISO
1,5E3
6.4F33
4.5F-2
1.1E40
35.81
36.80
35.19
32,94
0.90
0.92
0.89
0.84
LISO
0
1.3E-6
8.4E-1
35.13
35,91
34.35
32.89
0.89
0.91
0.88
0.84
COCO
LISO
18E-3
7.6E-3
16E-T
1.4440
34.32
36.43
31.46
30.14
0.91
0.86
0.81
LISO
0
0
1.2E-6
3.2E40
33.20
35.23
30.35
29.57
0,90
0.93
0.83
0.78
Table 9: Image steganography results of LISO with loss weight À LISO+L-BFGS and was shortened due to space constraints.
10.
Note that LISO* is
H ADDITIONAL STEGANALYSIS
In addition to SiaStegNet (You et al., 2020), we attempt to avoid detection from two other state-of- the-art steganalysis systems: SRNet (Boroumand et al., 2018) and XuNet (Xu et are reported in Table 10. For most cases, the results are similar to SiaStegNet. However, with XuNet very high detection accuracy in the "w/o defense" scenario; this is because XuNet uses handcrafted kernels in addition to using a convolutional network to detect steganographic images. Despite this, adding the loss XuNet allows defense" scenario). We can also achieve lower detection accuracy in the "wwo defense" scenario method like XuNet (with hand-crafted kernels) is used as a discriminator/critic during LISO training.
et al., 2016). Results
evade detection from it (as seen from the "w/
Error Rate (%)
PSNR 1
Detection Accuracy Rate (%)
Method
bit
bits
3 bits
hits
hits
bits
SRNet (wlo defense)
TE-04
3E-04
56-03
4F-02
33.83
34.15
30.08
24.58
51
40
SRNet (w/ defense)
6E-04
1E-04
1E-03
2E:01
33.43
32,74
28.51
24.89
0
0
XuNet (wlo defense)
IE-04
3E-04
SE-03
4E-02
33.83
34.15
30.08
24.58
100
98
100
100
XuNet (w/ defense)
2E-04
3E-03
1E-02
4E-02
34.16
32.98
28.40
25.31
Table 10: Steganalysis results with SRNet and XuNet. All experimental configurations follow Table 3 in main paper.
I EXPERIMENT RESULTS WITH HIGHER PAYLOADS
Extending the results from the main paper, in Table 11, we evaluate LISO with 5-6 bits encoded per pixel. We see that we are able to get low error rates (< 3%) even for messages hiding 5-6bpp of information. However, the image quality is not very good; as described in Appendix G we can get images with better image quality but a slightly worse enror rate we increase the weight on the MSE loss term.
16

---

Published as a conference paper at ICLR 2023
Error Rate (%) 4
PSNR f
SSIM
1
Dataset
Method
5 bits
6 bits
bits
bits
bits
bits
SteganoGAN
31.44
35.35
20.05
20.34
0.79
0.80
LISO
2.37
9.08
22.58
23.29
0.53
0.56
Div2k
FNNS-D
18.12
19.67
12.34
12.3
0.13
0.14
LISO+L-BFGS
2.13
9.17
20.86
23.24
0.44
0.55
SteganoGAN
32.15
31.16
19.51
21.82
0.74
0.79
LISO
0.48
8.73
24.88
25.15
0.48
0.47
CelebA
FNNS-D
15.3
18.41
12.94
12.99
0.07
0.07
LISO+L-BFGS
0.16
8.31
24.68
24.69
0.46
0.45
SteganoGAN
33.20
35.67
22.74
23.07
0.86
0.85
LISO
1.09
8.66
21.32
22.63
0.36
0.45
MS COCO
FNNS-D
16.41
17.44
15.63
15.78
0.19
0.20
LISO+L-BFGS
1.52
8.50
21.08
22.55
0.35
0.45
Table 11: Image steganography results with 5-6 bits encoded per pixel.
J ADDITIONAL QUALITATIVE RESULTS
A few qualitative examples of steganographic images were presented in the main paper. Additional steganographic images produced by LISO with 1-4bpp hidden information are shown in Figure 8. Stenographic images obtained with an increased mse weight À 10 are shown in Figure 9.
As explained the paper, we can use L-BFGS with LISO to further reduce the error rate obtained from LISO, Figure 10 shows steganographic images obtained from LISO+L-BFGS and there is no difference in visual quality even after the additional optimization.
that
In section 5 of the main paper show that LISO avoid being detected by SiaStegNet (You et al., 2020) by using the gradient from SiaStegNet in LISO's optimization network. In Figure 11, show sample images from the Div2k dataset that are generated in this way avoid SiaStegNet detection, Again we see no noticeable difference in visual quality.
17

---

Published as a conference paper at ICLR 2023
Cover image
Bits Per Pixel
2 Bits Per Pixel
Bits Per Pixel
Bits Per Pixe!
Figure 8: Cover images with corresponding steganographic images under different payloads. The
first 2 images from Div2k, the following are from CelebA, and the last are from XS-COCO.
18

---

Published as a conference paper at ICLR 2023
Cover image
Bits Per Pixel
2 Bits Per Pixel
Bits Per Pixel
Bits Per Pixe!
Figure 9: Cover images with corresponding steganographic images under different payloads trained
with À = 10.
19

---

Published as a conference paper at ICLR 2023
Cover image
Bits Per Pixel
2 Bits Per Pixel
Bits Per Pixel
Bits Per Pixe!
Figure 10: Cover images with corresponding steganographic images under different payloads using LISO+L-BFGS.
20

---

Published as a conference paper at ICLR 2023
Cover
Bits Per Pixel
2 Bits Per Pixel
Bits Per Pixel
Bits Per Pixe!
Figure 11: Steganographic produced by LISO with additional detection loss. These images can all avoid detection by SiaStegNet.
21


Reviews:
[{"review_id": "uXGpqYj0GXX", "reviewer": "Reviewer_3A9y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes Learned Iterative Steganography Optimization (LISO), a steganography approach that leverages deep neural networks and iterative optimization to hide a secret message in an image. LISO is faster than previous optimization-based approaches and has low message recovery error. ", "summary_of_the_review": "The presented LISO is a novel steganography approach that outperforms previous optimization-based steganography approaches. The authors provide convincing results and analysis regarding their approach. From my judgment, this work holds sufficient novelty for a subgroup in the ML community. \n", "strength_and_weaknesses": "## Strengths \n(+) Deep-learning-based steganography approaches are an active research field, gaining more and more popularity. This work is one of a few, that investigates the optimization-based approach.  \n(+) This work presents convincing results, with a relatively low error rate and average time (outperforming FNNS).  \n(+) I highly appreciate the analysis regarding steganalysis, runtime, and JPEG compression.  \n\n## Weaknesses \n(-) I might have missed it, but I would be interested regarding the hidden messages. Were the bit string messages chosen randomly? I am also interested if the authors encountered “adversarial” bit strings, meaning bit strings that are difficult to encode.  \n(-) I appreciate the evaluation in Appendix I. What are the limitations of the algorithm regarding the length of bit strings? Or in other words, at what payload lengths, does the algorithm not produce sufficient results?  \n(-) I am curious about a study regarding the cover image. Are there cover images, for which it is easier or more difficult to encode messages, e.g. random noise, black-and-white images, or single-colored images?   \n(-) The authors strongly argue that “the steganographic image remains on the natural image manifold with the help of a critic”. This argument is repeated multiple times throughout the paper, but no concrete evidence or proof is provided. Hence, this claim remains an assumption.  \n(-) Will the code be provided? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rUWkl6Ohh9M", "reviewer": "Reviewer_PqKU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new method for image steganography based on an iterative optimisation method (gradient-descendent) able to achieve zero recovery error rate for encodings up to 3bpp. The method is able to keep steganography output images close to (or on) the manifold created by natural images. The experimental results show that the paper outperforms the state-of-the-art in terms of recovery (decoding) error rate and in terms of computational cost.", "summary_of_the_review": "The paper is interesting and well written. It would benefit of some improvements in terms of comparison of the recovery error rate and, mainly, in terms of resulting image quality. ", "strength_and_weaknesses": "\nStrengths:\n\n1) the paper is well written and it is very clear\n2) the novel ideas of the paper explores two separate previous approaches to the problem, namely the neural learning and the iterative optimisation. The method presented by the paper is able to jointly explore these two approaches by creating a learning process for the optimisation problem, able to learn the gradient descent direction.\n3) the experiments are thorough and show that the method outperforms the state-of-the-art\n\nWeaknesses:\n1) when comparing the method and the cited methods of the state-of-the-art, the authors usually state the recovery error rate of the latter ones, by 5-25% for 4bpp, and state that the presented method can achieve 0% recovery error rate for up to 3bpp. In my opinion, it would be more fair and clear to compare in the text both the presented method and SOTA in both 3bpp and 4bpp. notice that the results are stated in Table 1, but the text is not so clear mentioning it.\n2) the paper does not explores sufficiently the resulting image quality in terms of being natural (indirectly measured by PNSR and SSIM). The results of Table 1 show that the proposed method does not always achieve better results in terms of image quality, so I suggest a better discussion in terms of image quality and maybe including some subjective evaluation of images by humans.\n3) In section 4, in the Training paragraph, it is not very clear whether the end-to-end training is image-specific or uses several different images. \n4) There are a few typos. I suggest a carefully revision of the text.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UcDjteRlGe", "reviewer": "Reviewer_cBgs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method which combines the benefits of iterative optimization and a learned encoder-decoder for the task of image steganography. Similar to the style of LISTA, the paper learns individual optimization blocks in an E2E fashion while applying the blocks iteratively at inference time. Experiment results show superior performance in terms of error rate and PSNR compared to previous baselines. \n\n", "summary_of_the_review": "Overall I believe this paper has enough novelty and strong enough experiment results to recommend acceptance. ", "strength_and_weaknesses": "Strengths: \n* The paper is well-written and easy to follow. \n* The proposed method is intuitive and novel. The rationale behind applying learnable blocks iteratively is explained clearly. \n* The experiment results are strong and convincing. \n\nWeaknesses. \nMinor points: \n* Figure 1 could include a scaled version of the diff if space allows. \n* Figure 4 CelebA sample has a very visible color shift a 1bit per pixel. Given the relatively high average PSNR reported, are the shown results correct or an outlier in the dataset? From the supplementary material samples, other images do not show this color shift.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "IDnE573HRop", "reviewer": "Reviewer_UMSE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method called Learned Iterative Steganography Optimization (LISO) that performs image steganography, i.e. embedding arbitrary information in image data (not metadata) in a way that is difficult to see or detect through statistical analysis. LISO uses iterative optimization based on a gated recurrent unit (GRU) to determine how to perturb image pixels to minimize visibility and maximize recovery of an arbitrary message. The authors show that compared to existing steganography methods, LISO is fast (Table 4), leads to low error rates in recovering the hidden message (Table 1), and can be adapted to more resilient to JPEG compression and more resistant to automated detection.\n", "summary_of_the_review": "I'm recommending that this paper be accepted based on the interesting use of GRU for fast (low iteration count) optimization, and the strong empirical results presented.\n\nInitially, I wasn't sure that steganography applications were a good fit for ICLR, at least without further analysis of the actual representation that's learned, but the main baseline (FNNS) was published at ICLR 2021. This work appears to be an interesting and significant improvement over FNNS.", "strength_and_weaknesses": "The primary strength of the paper is the interesting approach and solid empirical results. The approach is based on iterative optimization using GRU where each iteration takes the previous estimate of the perturbation and a gradient, and the hidden state allows the method to minimize the loss more quickly and more effectively than methods that only take a gradient (see Fig 3).\n\nThe empirical evaluation involves three components: the error rate in recovering the hidden message, the image quality after embedding the message, and runtime. Table 1 shows that LISO (coupled with L-BFGS) outperforms SteganoGAN by a significant margin. Compared to FNNS-D, both methods reach zero rates (or very close to it) up to 3 bits per pixel (bpp) but only LISO has near-zero error rates at 4bpp. On the other hand, FNNS-D does have higher PSNR and SSIM for two out of the three data sets (MS-COCO and CelebA but not Div2k). It's not clear to me how big of a problem this is, though, since LISO uses a GAN-like loss to keep perturbed images near the manifold of natural images. I would expect this loss to make the images look more realistic at the expense of fidelity (this is often discussed as high perceptual quality but also high distortion). Since PSNR and SSIM are more closely aligned with distortion than perception/realism, they may not be good measures of image quality, or, at least, they're at odds with the GAN-like loss used by LISO (see work on the perception-distortion tradeoff by Blau & Michaeli for more information).\n\nFurthermore, LISO runs much more quickly than FNNS-D and LISO+L-BFGS has a significant speed advantage over FNNS-D too.\n\nIn practice, the main weakness of LISO is that performance drops with lossy compression, however LISO-JPEG (a variant of LISO that incorporates JPEG compression into the training procedure) fairs better than the baseline methods. Since natural images are almost always compressed using a lossy codec, the main results (\"LISO-PNG\" that uses *lossless* compression) has relatively little real-world impact.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster