SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiwen  $\mathbf{Y}\mathbf{u}^{1}$  Xuanyu Zhang<sup>1</sup> Youmin  $\mathbf{X}\mathbf{u}^{1,2}$  Jian Zhang<sup>1†</sup>
1 Peking University Shenzhen Graduate School 2 Peng Cheng Laboratory
Current image steganography techniques are mainly focused on cover-based methods, which commonly have the risk of leaking secret images and poor robustness against degraded container images. Inspired by recent developments in diffusion models, we discovered that two properties of diffusion models, the ability to achieve translation between two images without training, and robustness to noisy data, can be used to improve security and natural robustness in image steganography tasks. For the choice of diffusion model, we selected Stable Diffusion, a type of conditional diffusion model, and fully utilized the latest tools from open-source communities, such as LoRAs and ControlNets, to improve the controllability and diversity of container images. In summary, we propose a novel image steganography framework, named Controllable, Robust and Secure Image Steganography (CRoSS), which has significant advantages in controllability, robustness, and security compared to cover-based image steganography methods. These benefits are obtained without additional training. To our knowledge, this is the first work to introduce diffusion models to the field of image steganography. In the experimental section, we conducted detailed experiments to demonstrate the advantages of our proposed CRoSS framework in controllability, robustness, and security. Code is available at https://github.com/vvictoryuki/CroSS.
With the explosive development of digital communication and AIGC (AI-generated content), the privacy, security, and protection of data have aroused significant concerns. As a widely studied technique, steganography [10] aims to hide messages like audio, image, and text into the container image in an undetected manner. In its reveal process, it is only possible for the receivers with predefined revealing operations to reconstruct secret information from the container image. It has a wide range of applications, such as copyright protection [4], digital watermarking [15], e-commerce [11], anti-visual detection [34], spoken language understanding [12, 13] and cloud computing [76].
For image steganography, traditional methods tend to transform the secret messages in the spatial or adaptive domains [27], such as fewer significant bits [9] or indistinguishable parts. With the development of deep neural networks, researchers begin to use auto-encoder networks [5, 6] or invertible neural networks (INN) [35, 26] to hide data, namely deep steganography.
The essential targets of image steganography are security, reconstruction quality, and robustness [9, 45, 77]. Since most previous methods use cover images to hide secret images, they tend to explicitly retain some secret information as artifacts or local details in the container image, which poses a risk of information leakage and reduces the security of transmission. Meanwhile, although previous works can maintain well reconstruction fidelity of the revealed images, they tend to train models in a noise-free simulation environment and can not withstand noise, compression artifacts, and non-linear transformations in practice, which severely hampers their practical values and robustness [30, 44, 25].
†Corresponding author. This work was supported by National Natural Science Foundation of China under Grant 62372016.
To address security and robustness concerns, researchers have shifted their focus toward coverless steganography. This approach aims to create a container image that bears no relation to the secret information, thereby enhancing its security. Current coverless steganography methods frequently employ frameworks such as CycleGAN [78] and encoder-decoder models [76], leveraging the principle of cycle consistency. However, the controllability of the container images generated by existing coverless methods remains limited. Their container images are only randomly sampled from the generative model and cannot be determined by the user. Moreover, existing approaches [47] tend to only involve hiding bits into container images, ignoring the more complex hiding of secret images. Overall, current methods, whether cover-based or coverless, have not been able to achieve good unity in terms of security, controllability, and robustness. Thus, our focus is to propose a new framework that can simultaneously improve existing methods in these three aspects.
Recently, research on diffusion-based generative models [22, 55, 56] has been very popular, with various unique properties such as the ability to perform many tasks in a zero-shot manner [36, 28, 64, 63, 72, 37, 20], strong control over the generation process [16, 49, 74, 40, 19, 50], natural robustness to noise in images [64, 28, 14, 65], and the ability to achieve image-to-image translation [75, 8, 20, 39, 57, 14, 29, 37]. We were pleasantly surprised to find that these properties perfectly match the goals we mentioned above for image steganography: (1) Security: By utilizing the DDIM Inversion technique [54] for diffusion-based image translation, we ensure the invertibility of the translation process. This invertible translation process enables a coverless steganography framework, ensuring the security of the hidden image. (2) Controllability: The powerful control capabilities of conditional diffusion models make the container image highly controllable, and its visual quality is guaranteed by the generative prior of the diffusion model; (3) Robustness: Diffusion models are essentially Gaussian denoisers and have natural robustness to noise and perturbations. Even if the container image is degraded during transmission, we can still reveal the main content of the secret image.
We believe that the fusion of diffusion models and image steganography is not simply a matter of mechanically combining them, but rather an elegant and instructive integration that takes into account the real concerns of image steganography. Based on these ideas, we propose the Controllable, Robust and Secure Image Steganography (CROSS) framework, a new image steganography framework that aims to simultaneously achieve gains in security, controllability, and robustness.
- We identify the limitations of existing image steganography methods and propose a unified goal of achieving security, controllability, and robustness. We also demonstrate that the diffusion model can seamlessly integrate with image steganography to achieve these goals using diffusion-based invertible image translation technique without requiring any additional training.
- We propose a new image steganography framework: Controllable, Robust and Secure Image Steganography (CRoSS). To the best of our knowledge, this is the first attempt to apply the diffusion model to the field of image steganography and gain better performance.
- We leveraged the progress of the rapidly growing Stable Diffusion community to propose variants of CRoSS using prompts, LoRAs, and ControlNets, enhancing its controllability and diversity.
- We conducted comprehensive experiments focusing on the three targets of security, controllability, and robustness, demonstrating the advantages of CRoSS compared to existing methods.
2.1 Steganography Methods
Cover-based Image Steganography. Unlike cryptography, steganography aims to hide secret data in a host to produce an information container. For image steganography, a cover image is required to hide the secret image in it [5]. Traditionally, spatial-based [24, 41, 43, 46] methods utilize the Least Significant Bits (LSB), pixel value differencing (PVD) [43], histogram shifting [60], multiple bit-planes [41] and palettes [24, 42] to hide images, which may arise statistical suspicion and are vulnerable to steganalysis methods. Adaptive methods [45, 31] decompose the steganography into embedding distortion minimization and data coding, which is indistinguishable by appearance but limited in capacity. Various transform-based schemes [10, 27] including JSteg [46] and DCT steganography [21] also fail to offer high payload capacity. Recently, various deep learning-based
schemes have been proposed to solve image steganography. Baluja [5] proposed the first deep-learning method to hide a full-size image into another image. Generative adversarial networks (GANs) [53] are introduced to synthesize container images. Probability map methods focus on generating various cost functions satisfying minimal-distortion embedding [45, 59]. [69] proposes a generator based on U-Net. [58] presents an adversarial scheme under distortion minimization. Three-player game methods like SteganoGAN [73] and HiDDeN [77] learn information embedding and recovery by auto-encoder architecture to adversarily resist steganalysis. Recent attempts [66] to introduce invertible neural networks (INN) into low-level inverse problems like denoising, rescaling, and colorization show impressive potential over auto-encoder, GAN [3], and other learning-based architectures. Recently, [35, 26] proposed designing the steganography model as an invertible neural network (INN) [17, 18] to perform image hiding and recovering with a single INN model.
Coverless Steganography. Coverless steganography is an emerging technique in the field of information hiding, which aims to embed secret information within a medium without modifying the cover object [47]. Unlike traditional steganography methods that require a cover medium (e.g., an image or audio file) to be altered for hiding information, coverless steganography seeks to achieve secure communication without introducing any changes to the cover object [33]. This makes it more challenging for adversaries to detect the presence of hidden data, as there are no observable changes in the medium's properties [38]. To the best of our knowledge, existing coverless steganography methods [34] still focus on hiding bits into container images, and few explorations involve hiding images without resorting to cover images.
Diffusion models [22, 55, 56] are a type of generative model that is trained to learn the target image distribution from a noise distribution. Recently, due to their powerful generative capabilities, diffusion models have been widely used in various image applications, including image generation [16, 48, 51, 49], restoration [52, 28, 64], translation [14, 29, 37, 75], and more. Large-scale diffusion model communities have also emerged on the Internet, with the aim of promoting the development of AIGC(AI-generated content)-related fields by applying the latest advanced techniques.
In these communities, the Stable Diffusion [49] community is currently one of the most popular and thriving ones, with a large number of open-source tools available for free, including model checkpoints finetuned on various specialized datasets. Additionally, various LoRAs [23] and ControlNets [74] are available in these communities for efficient control over the results generated by Stable Diffusion. LoRAs achieve control by efficiently modifying some network parameters in a low-rank way, while ControlNets introduce an additional network to modify the intermediate features of Stable Diffusion for control. These mentioned recent developments have enhanced our CRoSS framework.
3.1 Definition of Image Steganography
Before introducing our specific method, we first define the image steganography task as consisting of three images and two processes (as shown in Fig. 1): the three images refer to the secret image  $\mathbf{x}_{\text{sec}}$ , container image  $\mathbf{x}_{\text{cont}}$ , and revealed image  $\mathbf{x}_{\text{rev}}$ , while the two processes are the hide process and reveal process. The secret image  $\mathbf{x}_{\text{sec}}$  is the image we want to hide and is hidden in the container image  $\mathbf{x}_{\text{cont}}$  through the hide process. After transmission over the Internet, the container image  $\mathbf{x}_{\text{cont}}$  may become degraded, resulting in a degraded container image  $\mathbf{x}_{\text{cont}}^{\prime}$  from which we extract the revealed image  $\mathbf{x}_{\text{rev}}$  through
the reveal process. Our goal is to make our proposed framework have the following properties: (1) Security: even if the container image  $\mathbf{x}_{cont}$  is intercepted by other receivers, the hidden secret image  $\mathbf{x}_{sec}$  cannot be leaked. (2) Controllability: the content in the container image  $\mathbf{x}_{cont}$  can be controlled by the user, and its visual quality is high. (3) Robustness: the reveal process can still generate semantically consistent results ( $\mathbf{x}_{rev} \approx \mathbf{x}_{sec}$ ) even if there is deviation in the  $\mathbf{x}_{cont}'$  compared to the  $\mathbf{x}_{cont}$  ( $\mathbf{x}_{cont}' = d(\mathbf{x}_{cont}), d(\cdot)$  denotes the degradation process). According to the above definition,
Figure 1: Illustration used to show the definition of image steganography.
(a) Image Translation using Different Conditions
Figure 2: In part (a), Conditional diffusion models can be used with different conditions to perform image translation. In this example, we use two different prompts ("cat" and "tiger") to translate a cat image into a tiger image. However, a critical challenge for coverless image steganography is whether we can reveal the original image from the translated image. The answer is yes, and we can use DDIM Inversion (shown in part (b)) to achieve dual-direction translation between the image distribution and noise distribution, allowing for invertible image translation.
(b) DDIM Inversion
we can consider the hide process as a translation between the secret image  $\mathbf{x}_{\text{sec}}$  and the container image  $\mathbf{x}_{\text{cont}}$ , and the reveal process as the inverse process of the hide process. In Sec. 3.2, we will introduce how to use diffusion models to implement these ideas, and in Sec. 3.3, we will provide a detailed description of our proposed framework CRoSS for coverless image steganography.
3.2 Invertible Image Translation using Diffusion Model
Diffusion Model Defined by DDIM. A complete diffusion model process consists of two stages: the forward phase adds noise to a clean image, while the backward sampling phase denoises it step by step. In DDIM [54], the formula for the forward process is given by:
$$
\mathbf {x} _ {t} = \sqrt {\alpha_ {t}} \mathbf {x} _ {t - 1} + \sqrt {1 - \alpha_ {t}} \boldsymbol {\epsilon}, \quad \boldsymbol {\epsilon} \sim \mathcal {N} (\mathbf {0}, \mathbf {I}), \tag {1}
$$
where  $\mathbf{x}_t$  denotes the noisy image in the  $t$ -th step,  $\epsilon$  denotes the randomly sampled Gaussian noise,  $\alpha_t$  is a predefined parameter and the range of time step  $t$  is  $[1,T]$ . The formula of DDIM for the backward sampling process is given by:
$$
\mathbf {x} _ {s} = \sqrt {\bar {\alpha} _ {s}} \mathbf {f} _ {\boldsymbol {\theta}} (\mathbf {x} _ {t}, t) + \sqrt {1 - \bar {\alpha} _ {s} - \sigma_ {s} ^ {2}} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} (\mathbf {x} _ {t}, t) + \sigma_ {s} \boldsymbol {\epsilon}, \quad \mathbf {f} _ {\boldsymbol {\theta}} (\mathbf {x} _ {t}, t) = \frac {\mathbf {x} _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon} _ {\boldsymbol {\theta}} (\mathbf {x} _ {t} , t)}{\sqrt {\bar {\alpha} _ {t}}}, (2)
$$
where  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  is a randomly sampled Gaussian noise with  $\sigma_s^2$  as the noise variance,  $\mathbf{f}_{\theta}(\cdot ,t)$  is a denoising function based on the pre-trained noise estimator  $\epsilon_{\theta}(\cdot ,t)$ , and  $\bar{\alpha}_t = \prod_{i = 1}^t\alpha_i$ . DDIM does not require the two steps in its sampling formula to be adjacent (i.e.,  $t = s + 1$ ). Therefore,  $s$  and  $t$  can be any two steps that satisfy  $s < t$ . This makes DDIM a popular algorithm for accelerating sampling. Furthermore, if  $\sigma_s$  in Eq.2 is set to 0, the DDIM sampling process becomes deterministic. In this case, the sampling result is solely determined by the initial value  $\mathbf{x}_T$ , which can be considered as a latent code. The sampling process can also be equivalently described as solving an Ordinary Differential Equation (ODE) using an ODE solver [54]. In our work, we choose deterministic DDIM to implement the diffusion model and use the following formula:
$$
\mathbf {x} _ {0} = \operatorname {O D E S o l v e} \left(\mathbf {x} _ {T}; \boldsymbol {\epsilon} _ {\boldsymbol {\theta}}, T, 0\right) \tag {3}
$$
to represent the process of sampling from  $\mathbf{x}_T$  to  $\mathbf{x}_0$  using a pretrained noise estimator  $\epsilon_{\theta}$ .
Image Translation using Diffusion Model. A large number of image translation methods [75, 8, 20, 39, 57, 14, 29, 37] based on diffusion models have been proposed. In our method, we will adopt a simple approach. First, we assume that the diffusion models used in our work are all conditional diffusion models that support condition  $\mathbf{c}$  as input to control the generated results. Taking the example shown in Fig. 2 (a), suppose we want to transform an image of a cat into an image of a tiger. We add noise to the cat image using the forward process (Eq. 1) to obtain the intermediate noise, and then control the backward sampling process (Eq. 2) from noise by inputting a condition (prompt="tiger"), resulting in a new tiger image. In general, if the sampling condition is set to  $\mathbf{c}$ , our conditional sampling process can be expressed based on Eq. 3 as follows:
$$
\mathbf {x} _ {0} = \operatorname {O D E S o l v e} \left(\mathbf {x} _ {T}; \boldsymbol {\epsilon} _ {\boldsymbol {\theta}}, \mathbf {c}, T, 0\right). \tag {4}
$$
For image translation, there are two properties that need to be considered: the structural consistency of the two images before and after the translation, and whether the translation process is invertible.
Figure 3: Our coverless image steganography framework CRoSS. The diffusion model we choose is a conditional diffusion model, which supports conditional inputs to control the generation results. We choose the deterministic DDIM as the sampling strategy and use the two different conditions  $(\mathbf{k}_{pri}$  and  $\mathbf{k}_{pub})$  given to the model as the private key and the public key.
Algorithm 1 The Hide Process of CRoSS.
Input: The secret image  $\mathbf{x}_{\mathrm{sec}}$  which will be hidden, a pre-trained conditional diffusion model with noise estimator  $\epsilon_{\theta}$ , the number  $T$  of time steps for sampling and two different conditions  $\mathbf{k}_{pri}$  and  $\mathbf{k}_{pub}$  which serve as the private and public keys.
Output: The container image  $\mathbf{x}_{cont}$  used to hide the secret image  $\mathbf{x}_{sec}$ .
$$
\mathbf {x} _ {\text {n o i s e}} = \operatorname {O D E S o l v e} \left(\mathbf {x} _ {\text {s e c}}; \boldsymbol {\epsilon} _ {\boldsymbol {\theta}}, \mathbf {k} _ {\text {p r i}}, 0, T\right)
$$
$$
\mathbf {x} _ {\text {c o n t}} = \operatorname {O D E S o l v e} \left(\mathbf {x} _ {\text {n o i s e}}; \boldsymbol {\epsilon} _ {\boldsymbol {\theta}}, \mathbf {k} _ {\text {p u b}}, T, 0\right)
$$
$$
\mathbf {\Delta} \mathrm {r e t u r n} \mathbf {x} _ {\text {c o n t}}
$$
Structural consistency is crucial for most applications related to image translation, but for coverless image steganography, ensuring the invertibility of the translation process is the more important goal. To achieve invertible image translation, we utilize DDIM Inversion based on deterministic DDIM.
DDIM Inversion Makes an Invertible Image Translation. DDIM Inversion (shown in Fig. 2 (b)), as the name implies, refers to the process of using DDIM to achieve the conversion from an image to a latent noise and back to the original image. The idea is based on the approximation of forward and backward differentials in solving ordinary differential equations [54, 29]. Intuitively, in the case of deterministic DDIM, it allows  $s$  and  $t$  in Eq. 2 to be any two steps (i.e., allowing  $s < t$  and  $s > t$ ). When  $s < t$ , Eq. 2 performs the backward process, and when  $s > t$ , Eq. 2 performs the forward process. As the trajectories of forward and backward processes are similar, the input and output images are very close, and the intermediate noise  $\mathbf{x}_T$  can be considered as the latent variable of the inversion. In our work, we use the following formulas:
$$
\mathbf {x} _ {T} = \operatorname {O D E S o l v e} \left(\mathbf {x} _ {0}; \boldsymbol {\epsilon} _ {\boldsymbol {\theta}}, \mathbf {c}, 0, T\right), \quad \mathbf {x} _ {0} ^ {\prime} = \operatorname {O D E S o l v e} \left(\mathbf {x} _ {T}; \boldsymbol {\epsilon} _ {\boldsymbol {\theta}}, \mathbf {c}, T, 0\right), \tag {5}
$$
to represent the DDIM Inversion process from the original image  $\mathbf{x}_0$  to the latent code  $\mathbf{x}_T$  and from the latent code  $\mathbf{x}_T$  back to the original image  $\mathbf{x}_0$  (the output image is denoted as  $\mathbf{x}_0'$  and  $\mathbf{x}_0' \approx \mathbf{x}_0$ ). Based on DDIM Inversion, we have achieved the invertible relationship between images and latent noises. As long as we use deterministic DDIM to construct the image translation framework, the entire framework can achieve invertibility with two DDIM Inversion loops. It is the basis of our coverless image steganography framework, which will be described in detail in the next subsection.
3.3 The Coverless Image Steganography Framework CRoSS
Our basic framework CRoSS is based on a conditional diffusion model, whose noise estimator is represented by  $\epsilon_{\theta}$ , and two different conditions that serve as inputs to the diffusion model. In our work, these two conditions can serve as the private key and public key (denoted as  $\mathbf{k}_{pri}$  and  $\mathbf{k}_{pub}$ ), as shown in Fig.3, with detailed workflow described in Algo.1 and Algo.2. We will introduce the entire CRoSS framework in two parts: the hide process and the reveal process.
The Process of Hide Stage. In the hide stage, we attempt to perform translation between the secret image  $\mathbf{x}_{\text{sec}}$  and the container image  $\mathbf{x}_{\text{cont}}$  using the forward and backward processes of deterministic DDIM. In order to make the images before and after the translation different, we use the pre-trained conditional diffusion model with different conditions in the forward and backward processes respectively. These two different conditions also serve as private and public keys in the CRoSS framework. Specifically, the private key  $\mathbf{k}_{pri}$  is used for the forward process, while the
Algorithm 2 The Reveal Process of CRoSS.
Input: The container image  $\mathbf{x}_{cont}^{\prime}$  that has been transmitted over the Internet (may be degraded from  $\mathbf{x}_{cont}$ ), the pre-trained conditional diffusion model with noise estimator  $\epsilon_{\theta}$ , the number  $T$  of time steps for sampling, the private key  $\mathbf{k}_{pri}$  and public key  $\mathbf{k}_{pub}$ .
Output: The revealed image  $\mathbf{x}_{rev}$
$$
\begin{array}{l} \mathbf {x} _ {\text {n o i s e}} ^ {\prime} = \operatorname {O D E S o l v e} \left(\mathbf {x} _ {\text {c o n t}} ^ {\prime}, \boldsymbol {\epsilon} _ {\boldsymbol {\theta}}, \mathbf {k} _ {\text {p u b}}, 0, T\right) \\ \mathbf {x} _ {r e v} = \operatorname {O D E S o l v e} \left(\mathbf {x} _ {\text {n o i s e}} ^ {\prime}; \boldsymbol {\epsilon} _ {\boldsymbol {\theta}}, \mathbf {k} _ {\text {p r i}}, T, 0\right) \\ \mathbf {\Delta} \mathrm {r e t u r n} \mathbf {x} _ {r e v} \\ \end{array}
$$
Figure 4: Further explanation of the CRoSS framework. We simulated the possible problems that a receiver may encounter in three different scenarios during the reveal process.
public key  $\mathbf{k}_{pub}$  is used for the backward process. After getting the container image  $\mathbf{x}_{cont}$ , it will be transmitted over the Internet and publicly accessible to all potential receivers.