Stego Networks: Information Hiding on Deep Neural Networks
Authors: Youngwoo Cho
Beomsoo Kim
Jaegul Choo
Venue:  (2021)
Keywords: Steganography
Information Hiding
Security

Abstract: The best way of keeping a secret is to pretend there is not one. In this spirit, a class of techniques called steganography aims to hide secret messages on various media leaving as little detectable trace as possible. This paper considers neural networks as novel steganographic cover media, which we call stego networks, that can be used to hide one's secret messages. Although there have been numerous attempts to hide information in the output of neural networks, techniques for hiding information in the neural network parameters themselves have not been actively studied in the literature. The widespread use of deep learning models in various cloud computing platforms and millions of mobile devices as of today implies the importance of safety issues regarding stego networks among deep learning researchers and practitioners. In response, this paper presents the advantages of stego networks over other types of stego media in terms of security and capacity. We provide observations that the fraction bits of some typical network parameters in a floating-point representation tend to follow uniform distributions and explain how it can help a secret sender to encrypt messages that are indistinguishable from the original content. We demonstrate that network parameters can embed a large amount of secret information. Even the most significant fraction bits can be used for hiding secrets without inducing noticeable performance degradation while making it significantly hard to remove secrets by perturbing insignificant bits. Finally, we discuss possible use cases of stego networks and methods to detect or remove secrets from stego networks.

Under review as a conference paper at ICLR 2021
STEGO NETWORKS: INFORMATION HIDING ON DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
The best way of keeping a secret is to pretend there is not one. In this spirit, a class of techniques called steganography aims hide secret messages on various media leaving as little detectable trace possible. This paper considers neural networks as novel steganographic cover media, which we call stego networks, that can be used to hide one's secret messages. Although there have been numerous attempts to hide information in the output neural networks, techniques for hiding in- formation in the neural network parameters themselves have not been actively studied in the literature. The widespread use of deep learning models in various cloud computing platforms and millions of mobile devices of today implies the importance of safety issues regarding stego networks among deep learning researchers and practitioners. In response, this paper presents the advantages of stego networks over other types of stego media in termss of security and capacity. We provide observations that the fraction bits of some typical network parameters in a floating-pon point representation tend to follow uniform distributions and explain how it can help a secret sender to enerypt messages that are indistinguishable from the original content. We demonstrate that network parameters can embed a large amount of secret information. Even the most significant fraction bits can be used for hiding secrets without inducing noticeable performance degradation while making it significantly hard to remove secrets by perturbing insignificant bits. Finally, we discuss possible cases of stego networks and methods to de- tect or remove secrets from stego networks.
INTRODUCTION
As much as it goes without saying knowledge is power, inventing methods for keeping and selee- tively conveying secret messages has been crucial mission throughout the history of humanity. Among various methods to protect secrets, an effective approach called steganography makes it dif- ficult to detect the very existence of the secrets in object looking innocuous. The object containing the secrets is called a slego medium in the context of steganography.
Starting with the case of hiding secret message in the form of engraved tattoos hidden parts of a human body in an ancient greek period, numerous methods (e.g.. using invisible inks, writ- ing tiny-sized letters) were employed to transmit information without leaving detectable footprints Kahn 1996). Most recently, digital steganography, which embeds secret messages in digital images or audio files, has been actively developed. Traditional steganography is typically used in commu- nication between two individuals, but steganography in digital media enables its brand-new usage by conveying secrets in multitude of devices and unknowingly influencing their behavior when accompanied with small decoding code. The secrets in this scenario are often called stegomalware Nagaraja er al. 2011 Suarez-Tangil et al. 2014).
Meanwhile, deep neural networks (DNNs) have shown remarkable success in various areas over the years, and are now beginning to be applied to industry and the consumer sector as well as to the academic. DNNs have been deployed in variety of computing systems ranging from large- scale cloud computing systems to millions of mobile devices (Howard et al., 2017). More and more mobile devices running application programs that include deep learning models with numerous camera filter and speech recognition applications being good examples. Furthermore, building upon existing large pre-trained models, such ResNet (He al. 2016), BERT (Devlin et al. 2019). and

---

Under review as a conference paper at ICLR 2021
0.012
0.0042
0.0042
Froquency
0.010
Density
60
0.0040
0.0040
40
0.008
0.0038
0.0038
0.006
20:
64
128
192 256
0
64
128
192
256
32
64
96
128
-0.05 0.00 0.05 Parameter
value
Fraction
Fraction 16-22
0.025
100%
0.0042
0.020
Froquency
Probubility
75%
0.0040
0.010
0.0038
0.005
0%
192 Byte value
128 Byte value
24
Index of FP32 bit
JPEG (encrypted)
JPG (onginnal
ResNet: 18 DenseNet-200
AlexNet
MP3 (encrypted)
MP3 (originall
ResNat-155 152
VGG-19
PDF (encrypted)
(original)
Figure Fraction bit distributions of image classification model parameters and distributions of evaluated bytes of various file formats (JPEG, MP3, and PDF). We visualized the distributions of fraction bits of parameters into three different plots. Three subsequences of 23 fraction bits are eval- uated at their decimal values (top row, first three). The distributions of the entire parameters of each model is also presented (top row, last one). Next, we plotted byte distributions the original and the encrypted files (bottom row, first two). The plotted values are averaged over 100 files for each file format. Finally, the probabilities of each bit in floating-point representation to be one is visualized (bottom row, last one). An interesting observation is that the fraction bits of five archi- tectures follow nearly identical distributions, but the shape of parameter distributions are noticeably different. Exponent bits of floating- point numbers mainly responsible for the discrepancy between parameter distributions.
GPT-3 (Brown al. 2020), rather than training complete neural networks from scratch has become a trend in deep learning research. Accordingly, various files containing neural network parameters uploaded on the source code repositories, and FTP servers, and they are frequently exchanged among individuals and organizations.
The fraction bits, which sometimes called the significand or the mantissa, of a floating-point number follows distribution close to uniform distribution as shown in Figure [ A secret sender can easily embed encrypted messages, which also typically follows uniform distribution, without causing much suspicion from static analysis tools. Also, as the sizes of pre-trained neural network parameters are often large being more than hundreds of megabytes in size, neural network parame- ters are suitable media to exchange a nontrivial amount of secrets.
In this paper, we analyze the distribution of fraction bits of typical neural network parameters. Frac- tion bits, which contain the least significant information of a floating-pon number, can conveniently be used to embed secret messages. We experimented with a special kind of weight perturbation, which simulate general cases of hiding secrets in network parameters and explored several methods to inject arbitrary data into neural network parameters without noticeable performance degradation. We empirically showed that steganography in the least significant fraction bits is readily applicable and even steganography in the most significant fraction bits is also possible. Our main contributions are follows:
We demonstrate suitability of neural network parameters as a novel steganographic medium.
We propose novel approaches to effectively embed secrets in neural network parameters.
We give comprehensive analysis of stego networks in terms of security and capacity, and discuss the advantages of stego networks over the conventional stego media.
2

---

Under review as a conference paper at ICLR 2021
2 BACKGROUND
This section covers materials to facilitate understanding of this paper, which includes the fundamen- tals of steganography and related work.
2.1 STEGANOGRAPHY
Steganography is technique to conceal information from unauthorized persons or eavesdroppers. It is a sub-field of information hiding. The goal of steganography is to hide the presence of the secret by embedding it in a camouflage medium, which is called cover medium. There exist three aspects in steganography that decide its effectiveness: security, capacity, and robustness (Provos & Honeyman 20033 Cox et al., 2007). Among them, security is often considered the most important factor. Security means that the presence of the secret must not be revealed under any situation. The next one, capacity, means the amount of information that the steganographic system can carry. Usually the risk of being detected increases as capacity increases, Robustness means the durability from external noise or interruption to eliminate the embedded information. In the context of stego networks, robustness can be degraded if re-training fine-tuning is applied.
Among various cover media, we mainly compare stego networks with images because it is one of the most actively studied media of secret information and image files are widely-used file formats among deep learning researchers and practitioners. Usually, each pixel of a three-channel RGB image is represented by total of 24 bits, eight bits per channel. A trivial approach to hide secret image in a cover image is to replace the least significant bits in the pixel values of the cover image with the most significant bits of the secret image. However, this approach is vulnerable because the secret image can be seen in the stego image due to humans' highly sensitive visual perception. The process that reveals the existence of steganography by visual characteristic is called visual attack. Also, digital images follow a certain distribution of pixel values that can be easily distinguished.
Steganalysis refers to the process of detecting the steganography by capturing the statistical anomaly of data. Several techniques have been proposed such as a subtractive pixel adjacency matrix (SPAM) (Pevny et al. 2010) to neutralize the steganographic systems. Currently, well-known steganography algorithms include HUGO, WOW, and S-UNIWARD (Pevný et al. 2010 Holub & Fridrich 2012 Holub et al. 2014).
2.2
RELATED WORK
There have been numerous approaches in the literature that applied deep neural network to steganal- ysis. The objective of some of these studies was to construct neural networks, which can determine whether hidden contents is embedded in an input image (Xu et al. 2016 Yedroudj et al. 2018). On the other hand, Baluja (017)) and Zhu et al. (2018) proposed methods to generate stego images. Our work is clearly different from these studies in that our work considers the neural network itself as a cover medium of steganography.
To the best of our knowledge, there has not been a principled study of neural networks as cover media of steganography so far. One work relevant to of ours LSB embedding attack proposed by Song et al. (2017). It the work, a scenario is suggested where an adversary intercepts training data with compromised training algorithm which hides the data in the LSFBs of the trained neural network parameters. The data embedding method can be seen as a special form of LSFB steganography which we explain in Section B However, [Song et al. (2017) does not include discussion of neural network parameters in terms of steganographic security, capacity, and robustness,, which are the three main aspects of steganography. Therefore, can hardly be seen as study of properteis of neural networks as steganographic cover media.
Our work is closely related with research on sensitivity and precision of neural network parameters. Cheney et al. (0017) conducted sensitivity analysis on the convolutional neural networks (CNNs) and reported that AlexNet (Krizhevsky et al. 2012) layers close to the input showed relatively high sensitivity to weight perturbation than those layers close to the output. Weight quantization (McK- linstry et al. 20118) Wang et al., 2018; Zairir et al. 2019)) Jung et al. 2019) is also an actively studied research area for energy and storage efficiencies. Studies in this field showed that neural networks
3

---

Under review as a conference paper at ICLR 2021
1) Secret information
(e.g., image, audio, docs)
1
0
0
0
Compress / Encrypt
0
PDF
4) Replace parameters
3) Replace fraction bits
0 0 31 30 Exponent (8 bits) 23 22 Sign
Fraction (23 bits)
F(4 MSFBs)
LSFBs)
MP3
L16 (16 LSFB))
Figure 2: Hiding information in neural network parameters. Before embedding, secrets should be compressed and encrypted to provide additional protection. A hiding procedure is to simply replace fraction bits of the parameter with the encoded message. In this paper, we convey information either in 1) the most significant 4 fraction bits (denoted by M4), or in 2) the least significant 8 or 16 fraction bits (denoted by L8 and L16,, respectively) of fraction bits.
can preserve their performance although a half or less number of bits are used to represent parame- ters.
In terms of DNN security, weight poisoning Gu et al., 2017) and adversarial attack (Goodfellow it al. 2013) also have been frequently studied. Both methods aim to manipulate the output of neu- ral networks, but the difference between the two methods is that weight poisoning perturbs neu- ral network parameters while adversarial attack perturbs the input of the network. In the case of adversarial attack, its counter-measures called adversarial defense (Metzen et al. (017) and their counter-counter-messne have also been continuously studied (Ghiasi et al. 2020).
Another closely related field to our work is watermarking. Both watermarking and steganography belong to the area of information hiding. While the top priority of steganography is security, water- marking focuses on robustness, in order to prevent attempts to removing information stored in cover media. Watermarking
neural networks the network. Adi et al. (018) prepared a trigger dataset, which is completely unrelated to the origi- nal task, and train fine-tune model yield 100% accuracy the trigger set. Then, the auditor can determine whether the model is watermarked or not by checking the accuracy of the trigger set. Similarly, Le Merrer al. 2020) adjusted the decision boundary of the model for specific inputs. Instead of examining the output, there exists method that probe the activation values from the networks for a private input (Darvish Rouhani al. 2019). Lastly,[Uchida |Uchida al. (2017) added a reg- ularization loss on the network parameters to embed watermark. The watermark is form of a binary vector, which can be recovered by multiplying an embedding matrix with the flattened parameters. In the case of watermarking, it usually leaves the noticeable trace the network parameters, thus cannot transmit information confidentially.
be used integrity check and copyright protection of
STEGO NETWORKS
This section presents the analysis on the sample distributions of fraction bits of typical neural net- work parameters. Afterwards, we propose two of our novel approaches to embed secret messages in neural networks.
3.1 FRACTION BITS OF NEURAL NETWORK PARAMETERS
Neural network parameters are usually in floating-poit types. A floating number consists of three parts for sign, exponent and fraction bit For example, a single-precision (FP32) format has one sign bit, eight exponent bits, and 23 fraction bits. Since FP32 is one of the most widely used formats deep learning research, we use the format throughout this paper. The least significant fraction bit has the lowest index, i.e., zero Since embedding messages in sign exponent bits
Details of floating- point number descrited in IEFE Standard for Binary Floating-Poiie Point Arithmetic
4

---

Under review as a conference paper at ICLR 2021
can induce relatively significant perturbation to the original value, only consider the case of embedding bits in fraction bits.
The fraction bit distributions of the parameters of few models commonly used in computer vision are provided in Figure [ We divide the 23 bits of fraction bits into three parts indexed as 0-7, 8- 15, and 16-22, to ease the visualization. We represent sequence of bits of each part simply by their corresponding decimal value. For example, a bit sequence 10000010 with the right-most bit denoting the bit of the lowest index is represented 130 in Figure
When it comes to randomness of fraction bits, we particularly considered two issues, which are uniformity of random variable and independent and identically distributed (i.i.d.) property of sequence of random variables. We can denote the part of FP32 fraction bits indexed 0-7 as X; with |.X, 256, and i is the index of the corresponding FP32 number in the entire parameters. Then one may think X; is uniform random variable if she observes each outcome happens with almost the same frequency. On the other hand, if one often observes the same outcomes consecutively, she may think the sequence of X; is not i.i.d. If X; is uniform and sequence of X; are i..dd, then the sequence has the maximum entropy.
Note that message embedder usually encrypt secret messages to provide additional protection. Also, encrypting messages makes it difficult for a steganalysis system to detect the existence of the message since much of the traces disappears. Encryption algorithms such AES and RSA are often involved before message embedding. In the rest of this section, we assume message embedders always encrypt secret messages.
Uniformity of a random variable is directly related with the capacity of stego networks and an secret embedder may need sacrifice the bit rate if the bit distribution of target medium is not uniform. Figure [ shows distributions of the three fraction parts of five popular models in computer vision. The two least significant parts show nearly uniform distributions across all the models. The most significant seven bits seemingly followed non-uniform distributions. In this case, secret sender may want to add dummy bits in the original secret message to avoid discrepancy between the fraction bit distribution of the original neural network and the resulting stego network to ensure security.
Another important property relevant to steganographic security is the i.i.d. property. This property what various steganalysis algorithms are based on to detect anomaly stego media. For example, SPAM algorithm (Pevny et al., 2010) utilizes the fact that pixels of natural images have high spatial correlation such that the colors of neighhoring pixels have similar values. Therefore, the usual bits of cover medium steganalysis systems. Note that establishing thcoretical independence between the significands of parameters is out of scope in work. Instead, we mainly provide empirical results that has practical implications in the experiment section. From the experimental results, we suspect that significands of stego networks exhibit considerable low dependency between them compared to the storages of other popular steganographic media.
purely i.i.d. it automatically wards off a large number of sophisticated
3.2 HIDING SECRETS IN THE LEAST SIGNIFICANT FRACTION BITS
We first propose embedding in neural network parameters by putting secrets in the least significant fraction bits (LSFBs). We recommend to measure the sensitivity of target neural network parameters before embedding messages by perturbing different amounts of fraction bits of the parameters to de- cide the precise number of bits that can be used for steganography. As can be observed Section 4 a large portion of LSFBs in popular image classification models can be perturbed without noticeably affecting the validation accuracy.
An obvious drawback of this approach is that secrets can be easily threatened by an active warden, an agent who aims to eliminate the chance of the existence of secrets, by simply perturbing the LSFBs. Also, applying this method can be tricky if targeted network parameters have significands with low- enough precision such that messages cannot be injected without inducing significant performance degradation.
5

---

Under review as a conference paper at ICLR 2021
Per param, sensitivity
Gradient sensitivity
BN layers
(log-sca (e)
(log-sca
Conv layers
-6-
-3
-4-
Input
Output
Input
Output
Figure 3: Validation accuracy sensitivity and gradient sensitivity of the layers of ResNet-18. The more layer is to the left, the closer it is to input. Due to significant scale differences between BN and convolutional layers, y-axes are represented in base-10 10 log scale. Overall, layer closer to the output has larger sensitivity in either metric.
3.3 HIDING SECRETS IN THE MOST SIGNIFICANT FRACTION BITS
We also propose an opposite- opposite- directional approach from the above which to hide secrets in the most significant fraction bits (MSFBs) of floating-point number. This method perturbs the original parameter much more than when hiding secrets the LSFBs. For example, when perturbing the most significant bit the value of the perturbed parameter can become 3/2 2/3 times of the original value.
Since changing the MSFBs result in significant performance degradation of model we suggest several remedies. First, since each layer a neural network has different sensitivities to random perturbations recommend one conduct sensitivity analysis to choose proper parameters. Also, fine-tuning can be followed to compensate performance degradation after embedding the secret. In Section we partitioned the parameters of model into two where the former is used hide secrets into one part while the latter is fine-tuned.
4
Another due consideration is the distribution of the most significant fraction bits. If they do follow uniform distribution, a message sender may need to add dummy bits as described earlier. However, adding non-uniform dummy bits violates i.i.d. property of the encrypted message and may increase the risk of being detected, assuming the original distribution also has the i.i.d. property.
Although this approach has more challenges than the other approach, also has clear advantage that the message cannot be simply removed by perturbing insignificant bits since it can significantly degrade the model performance.
4 EXPERIMENTS
This section provides experimental results and demonstrate the usefulness DNNs as cover media. First, we conducted preliminary experiment by measuring sensitivity (Cheney al. 2017 Widrow & Lehr 1990) of each layer of the ResNet-18 18 architecture. Next, we applied the secret embedding methods proposed in the previous section for larger models simulate real-world scenarios. All the values reported in the table are the averaged ones across ten different runs.
Throughout the experiments, we perturbed network parameters by injecting independent random bits which has the equal probability to be or 1. Although it is just a simple setting, it can actually represent general cases since it is common to encrypt messages before embedding using encoding algorithms generating uniformly random bit patterns.
4.1 SENSITIVITY ANALYSIS
The sensitivity of each layer of ResNet-18 was measured after perturbing all the fraction bits of the corresponding layer's parameters. We defined and used two different sensitivity metrics discover parameters causing minimum effects after a message is embedded. The first one measures the de- crease validation accuracy due to the perturbation to a specific layer and divides that value by the number of the parameters of that layer. The latter one computes the parameter gradients with respect to the loss function over large-sized input. The gradients are multiplied by the magnitudes of the parameters to reflect the fact that parameters are perturbed by relatively large amount when the

---

Under review as a conference paper at ICLR 2021
Avg. per param. sensitivity
Avg. gradient sensitivity
Layer type
Ratio
Conv
Weight
95.53
1.139 10
4.005 10-
Weight Bias
4.38
7.000
10-
3.299
10-4
FC
0.01
4.000
10-
6.961
10
Weight
0.04
1.495 x 10-4
2.883
10-2
BN
Bias
0.04
8.408
10-5
9.381
10
Table 1: Summary of sensitivity analysis on ResNet-18 18 architecture. We measured the sensitivity of all layers the model. Then, we categorized and averaged the result by the type of layers. Batch normalization layers, which accounts for 0.08% of the total number of parameters are particularly more sensitive to the output.
Accuracy
Model
Original
L8
L16
M4
M4 (tuned)
Top-1 Top-5
56.522
56.524 (.001)
56.501 (.018)
56.388 (.040)
56.388 (.072)
AlexNet
79.066
79.066 (.002)
79.090 (.012)
79.047 (.024)
79.047 (.040)
Top-1 Top-5
72.376
72.376 (.001)
72.368 (.013)
72.379 (.026)
72.379 (.046)
VGG-19
90.876
90.876 (.000)
90.876 (.009)
90.871 (.018)
90.871 (.033)
Top-1 78.312
78.311 (.001)
78.309 (.018)
76.836 (.074)
78.368 (.061)
ResNet-152
Top-5 94.046
94.046 (.000)
94.032 (.008)
93.277 (.068)
94.034 (.033)
Top-1 76.894
76.898 (.002)
76.896 (.021)
70.660 (.226)
76.796 (.075)
DenseNet-201
Top-5
93.370
93.371 (.001)
93.384 (.016)
90.406 (.139)
93.375 (.040)
Table 2: Accuracy the original and the perturbed image classification models (Krizhevsky al. 2012 Simonyan & Zisserman, [215)] He et al. [2016] Huang et al. (017). We measured accuracies of various model architectures with four perturbing strategies on ImageNet validation dataset.
values of the parameters are are larger than those of other parameters in our setting. The rectified gradients are averaged in each layer. Finally, we changed the signs of both metrics so that they al- ways have positive values. ILSVRC2012 (Russakovsky et al. 2015), also known as ImageNet, was used to compute validation accuracy and average gradients. The average sensitivity values of same layer type reported in Table I We found that the sensitivity of batch normalization (BN) layer is 10 to 100 times larger than other layers in both metrics. Since the number of BN parameters only account for less than 1% of the entire parameters, which indicates low capacity, we excluded BN parameters in later experiments. In addition, we plotted the trend of the sensitivity along the lay- of a network. FigureB]showe that, in general, the layers closer the output have lower sensitivity than the layers close to the input. A similar tendency is also reported in Cheney et al. (011)).
4.2 L.SBB AND MSFB PERTURBATIONS
We compared four perturbation approaches by assessing the validation accuracy of the resulting models. We took four pre-trained ImageNet classification models (AlexNet, VGG-19, ResNet-155 152, and DenseNet-201) and perturbed their parameters. The result described in Table shows that the least significant 8 fraction bits (L8) and the least significant 16 fraction bits (L16) perturbation do not significantly affect the model accuracy in all cases. On the other hand, ResNet-152 and DenseNet-201 showed remarkable degradation in accuracy in the case of the most significant 4 bits (M4) perturbation (we perturbed only 50% of the parameters for M4 experiment). We fine-tuned the models (M4) for one epoch, and ResNet- 152 and DenseNet-200 largely recovered the original accuracy.

---

Under review as a conference paper at ICLR 2021
Max. Z-score
Avg. Z-score
Layer type
# of layers
Frac. 0-7
Frac. 8-15
Frac. 16-22
Image
Conv
Weight
20
3.877
4.528
3.943
Weight Bias
1.053
0.653
2.083
FC
626.747
0.744
1.645
0.327
Weight
20
2.109
1.992
2.478
BN
Bias
20
2.147
2.004
2.048
Table 3: Result of rums tests ResNet-18 parameters and Imagenet validation images. A high Z- score can interpreted as an evidence that the corresponding sequence of the variables are not i..d.
4.3 PARAMETER DISTRIBUTION
We observed the parameter distribution after perturbation and fine-tuning (see Appendix Aj. In the case of M4, clearly left the trace in the bit distribution while L8 kept the resulting bit distribution nearly indistinguishable from the original one and L16 left few footprints. Those traces stayed even after fine-tuning, thus appending dummy bits may be needed avoid detection especially when applying M4.
Also, check the i.d property of statistical procedure that determines whether a sequence of observations are mutually independent, was conducted on the parameters of ResNet-18 (He al. 2016) in simple setting. Fraction bits of a floating-poii number were converted into the usual three decimal values as before and only the mutual independence between one type of decimal values were tested as shown Table We also measured the Z-scores of the runs test in ImageNet validation data for comparison. The Z-score of the parameters
fraction bits another, runs test (Bradley 1968).
layer was separately calculated and only the maximum Z-scores among the same type of layers were reported. In deciding positiveness in the testing. of a sequence as a reference. The resulting Z-scores for the parameters are still too high in the usual statistical sense accept the null hypothesis that the sequence variables are independent. However, there was a significance difference between Z-scores from the neural network parameters and the image data. We suspect this relatively little dependency among parameter fraction bits makes fundamentally harder to detect anomaly from neural network parameters than from image data.
used the mean value
4.4 DEEP LEARNING STEGANALYSIS TO DETECT STEGO NETWORKS
We trained deep learning models to distin- guish stego network parameters from nocuous neural network parameters. Al- though there cxist a few benchmark datasets for image steganography such as BOSS (Bas jet al. 2011). no suitable dataset for the task of detecting stego networks seem to exist at the moment. Hence, we used the parameters of the fully connected layer of four pre-trained ImageNet classification models (AlexNet, VGG-19, ResNet-152, and DenscNet-201) as our training and test datasets and split them into the ratio of to 2. In particular, we further divided the datasets into two types, one consisting only of the most significant fraction bits and the other consisting of the least significant frac- tion bits. Windows of size 32 x 32 cropped from the parameter matrix of size 1000 x 512 were fed the models trained. Both the original parameters and parameters with perturbation in
Best accuracy L,6
Model
Method
M4
MLP
51.063
50.485
AlexNet
Yedroudj-Net
50.917
54.780
MLP
50.686
50.539
VGG-19
Yedroudj-Net
52.229
54.776
MLP 50.748
50.710
ResNet- 152
Yedroudj-Net
51.646
55.449
MLP
50.434
50.271
DenseNet-201
Yedroudj-Net
51.421
54.506
Table 4: Performance of two deep learning steganaly- sis models on stego networks classification. Note that dummy bits were used in M4.
8

---

Under review as a conference paper at ICLR 2021
the entire fraction bits were included as negative and positive (stego) input data, respectively. More details about the experiment can be found at Appendix A
We experimented with custom multilayer perceptron (MLP) and popular image steganalysis model named Yedroudj-Net (Yerroudj et al. 2018). As can be seen in Table both models generated output close random predictions. The result indicates that the significands of the fully connected layers do not follow a probability distribution that can be easily distinguished from a random bit sequence. Please note that this experiment is intended be interpreted as a definite evidence for security of stego networks. We expect more specialized methods for stego networks are developed in the future works.
preliminary one and, therefore, cannot
DISCUSSION & CONCLUSION
In this paper, we explored stego networks, a novel application of neural networks in steganography. First, we analyzed parameters of various image classification models and discovered that the fraction bits of the parameters tend to follow a uniform distribution. We also observed that the sequence of the fraction bits exhibit degree of independence in the sense of the Z-score of runs test. These two fundamental characteristics are what, we suspect, give stego networks outstanding advantages over the other conventional stego media.
In the experiment section, we applied our suggested methods to widely used deep learning models to simulate embedding secrets neural networks. We confirmed that there was not significant per- formance degradation even when one half of the total bits of a neural network were perturbed. In addition, we suggested best practices in embedding messages in MSFBs. Sensitivity analysis was particularly helpful by exhibiting relatively high sensitivity of batch normalization layers and lay- ers close to input. With additional help of fine-tuning, we showed that a neural network can large maintain its original performance after perturbations in MSFBs. Finally, we reported our experi- ment where an existing deep learning based steganalysis technique could not be effectively applied distinguish compromised neural network parameters from normal neural network parameters.
A potential application of steganography is to embed malware into innocuous-looking media turning them into stegomalware (Nagaraja et al., 2011: Suarez-Tangii et al., 2014). A typical stegomalware consists of an actual stegomalware and its decoding program of a tiny size. It is generally more involved to identify stego malware by static analysis than identifying usual obfuscated malware. Currently, there exist tremendous amount of pre-trained models and open-source code on the Internet. According to Verdi al. (2019) Zimmerrannn al. (2019), a large number of publicly available source code repositories contain vulnerabilities, but numerous software developers still employ the codes without doubts. the worst case, pre-trained models can be a prevalent hacking vuinerability enjoying the obscurity of stego networks at the moment. Also, more and more mobile devices are running application programs that include deep learning models with numerous camera filter and speech recognition applications being good examples. This implies that the general public can be a target of malicious stego networks. We attached a practical demonstration of a malicious stego network Supplementary Material for proof-of-coonept..
Finally, we provide a few useful counter-measures to either detect or eliminate secret messages. First, perturbing the entirety or a part of the parameters of a model by a small amount can effective eliminate hidden messages in the LSFBs without significant performance degradation. In the case of hiding messages in MSFBs, one may conduct static analysis and and detect the existence of abnormal information by observing frequencies of certain bit patterns or dependencies among a series of significands.
REFERENCES
Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weak- ness into a strength: Watermarking deep neural networks by backdooring. In USENIX Security Symposium, 2018
Shumeet Baluja. Hiding images in plain sight: Deep steganography. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2017.
9

---

Under review as a conference paper at ICLR 2021
Patrick Bas, Tomáš Filler, and Tomáš Pevný. break steganographic system": the ins and outs of organizing boss. In International workshop on information hiding. 2011.
James V Bradley. Distributiion-fre statistical tests. Citeseer, 1968.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Nicholas Cheney, Martin Schrimpf, and Gabriel Kreiman. On the robustness of convolutional neural networks to internal architecture and weight perturbations. arXiv preprint arXiv: 2017.
Ingemar Cox, Matthew Miller, Jeffrey Bloom, Jessica Fridrich, and Ton Kalker. Digital watermark- ing and steganography. Morgan kaufmann, 2007.
Bita Darvish Rouhani, Huili Chen, and Farinaz Koushanfar. Deepsigns: an end-to-end watermarking framework for ownership protection of deep neural networks. In Proc. the International Confer- ence on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2019.
Jacob Devlin, Ming- bidirectional transformers for language understanding. In Proc. of The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
Amin Ghiasi, Ali Shafahi, and Tom Goldstein. Breaking certified defenses: Semantic adversarial examples with spoofed robustness certificates. In Proc. the International Conference on Learning Representations (ICLR), 2020.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Proc. the International Conference Learning Representations (ICLR), 2015.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. In arXiv preprint 1rXXvv1708.0611 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. of the IEEE conference on computer vision and pattern recognition (CVPR), 2016.
Vojtech Holub and Jessica Fridrich. Designing steganographic distortion using directional filters. In 2012 IEEE International workshop information forensics forenssic and security (WIFS), 2012.
Vojtëch Holub, Jessica Fridrich, and Tomáš Denemark. Universal distortion function for steganog- raphy in an arbitrary domain. EURASIP Journal on Information Security, 2014.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04861, 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proc. of the IEEE conference on computer vision and pattern recog- nition (CVPR), 2017.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization tervals with task loss. In Proc. of the IEEE conference computer vision and pattern recognition (CVPR), 2019.
David Kahn. The history of steganography. In Information Hiding, pp. 1-5, 1996.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. In Proc. the Advances in Neural Information Processing Systems (NeurIPS). 2012.
10

---

Under review as a conference paper at ICLR 2021
Erwan Le Merrer, Patrick Perez, and Gilles Trédan. Adversarial frontier stitching for remote neural network watermarking. Neural Computing and Applications, 2020.
Jeffrey L McKinstry, Steven K Esser, Rathinakumar Appuswamy, Deepika Bablani, John V Arthur, Izzet B Yildiz, and Dharmendra S Modha. Discovering low-precision networks close full- precision networks for efficient embedded inference. arXiv preprint 1809::800.044 2018.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In Proc. the International Conference Learning Representations (ICLR), 2017.
Shishir Nagaraja, Amir Houmansadr, Pratch Piyawongwisal, Vijit Singh, Pragya Agarwal, and Nikita Borisov. Stegobot: a covert social network botnet. In International Workshop on In- formation Hiding, 2011.
Tomáš Pevny, Patrick Bas, and Jessica Fridrich. Steganalysis by subtractive pixel adjacency matrix.
IEEE Transactions on information Forensics and Security, 2010.
Tomáš Pevný, Tomáš Filler, and Patrick Bas. Using high-dimensional image models to perform highly undetectable steganography. In International Workshop Information Hiding, 2010.
Niels Provos and Peter Honeyman. Hide and seek: An introduction to steganography. IEEE security & privacy, 2003.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Proc. the International Conference on Learning Representations (ICLR), 2015.
Congzheng Song. Thomas Ristenpart, and Vitaly Shmatikov. Machine learning models that remem- ber too much. In Proc. the ACM SIGSAC Conference on Computer and Communications Security (CCS), pp. 587-601, 2017.
Guillermo Suarez- Suarez-Tangil, Juan E Tapiador, and Pedro Peris-Lopez. Stegomalware: Playing hide and seek with malicious components in smartphone apps. In International conference on information security and cryptology, 2014.
Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin'ichi Satoh. Embedding watermarks into deep neural networks. In Proc. the ACM International Conference on Multimedia Retrieval (ICMR), 2017.
Morteza Verdi, Ashkan Sami, Jafar Akhondali, Foutse Khomh, Gias Uddin, and Alireza Karami Motlagh. An empirical study of c++ vulnerabilities in crowd-sourced code examples. arXiv preprint arX0v:1910.01321, 2019.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2018.
Bernard Widrow and Michael A Lehr. 30 years of adaptive neural networks: perceptron, madaline, and backpropagation. Proceedings of the IEEE, 1990.
Guanshuo Xu, -hn- Wu, and Yun-Qing Shi. Structural design of convolutional neural networks for steganalysis. IEEE Signal Processing Letters, 2016.
Mehdi Yedroudj, Frédéric Comby, and Mare Chaumont. Yedroudj-net. An efficient enn for spatial steganalysis. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.
Ofr Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8hert: Quantized 8bit bert. arXiv preprint 19Xiv:19/0.06/4/ 2019.
Jiren Zhu, Russell Kaplan, Justin Johnson, and Li Fei-Fei. Hidden: Hiding data with deep networks. In Proc. of the European Conference Computer Vision (ECCV), 2018.

---

Under review as a conference paper at ICLR 2021
Markus Zimmermann. Crixtian-Alexandru Staieu, Cam Tenny, and Michael Pradel. Small world with high risks; A study of security threats in the npm ecosystem. In USENIX Security Synposium, 2019.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
This section provides additional materials, which are not described the main manuscript.
A.1 DEEP LEARNING BASED STEGANALYSIS
We used a 3-layer multilayer perceptron (MLP) with 1024 and 128 hidden features and two dropout layers of the dropout rate of 0.3. We trained both our model and Yedroudj-Net for 30 epochs using stochastic gradient descent with the learning rate 0.01 and the weight decay of 0.0001.
When processing the input data, we made few peculiar design choices. Given a floating-poit paiit number we discarded the sign and exponent bits and only kept the fraction bits. The fraction bits were again divided into two parts as in the previous experiments, the four most significant 4 bits (M4) and the least significant 16 bits (L16). L16 was treated as two 8-bit sequences rather than one 16-bit sequence. Finally, evaluated them integers and composed two types of samples, one consisting of evaluated MSFBs of the size 32 x 32 x and the other consisting of two evaluated less significant bit sequences of the size 32 32 2. The values of the elements of MSFB samples range from 0 to 15, and those of LSFB sample from 0 to 255.
When simulating secret embedding (actually, simply perturbing bits) in M4, also added dummy bits to make the embedded bits indistinguishable from normal MSFBs. We first randomly generate finite number of dummy intervals. They are used in rotation access next parameter. When dummy intervals are one, three and two, for example, parameters at indices 1, 6, 7, 10, 12, will store dummy bits. In this experiment, we generated 1024 random dummy intervals and used the same intervals for both train and test datasets. We added randomly chosen dummy bits into M4 parts of stego network such that the evaluted M4 integers of stego network has the same integer frequencies that of corresponding cover network. Note that this is unnatural setting since, in practice, a warden does not have information about the dummy intervals a message embedder used. In more realistic and complex setting, the number of dummy intervals be much less while retaining the equivalent performance.
13

---

Under review as a conference paper at ICLR 2021
A.2 PARAMETER DISTRIBUTION
The two plots below illustrate the change of the marginal probability distributions of each bit of parameter before and after fine-tuning- The most significant four fraction bits (M4) are mostly likely to be detected by steganalysis due to their larger difference from the original networks marginal probabilities. Fine-tuning does seem help close the discrepancy between the original and stego network marginal bit distributions although the difference is still significant enough to be detectable by a warden.
AlexNet (no fine-tune)
DenseNeet-201 (no fine-tune) fine-tune)
Prob. difference
fine-tune)
4.0%
2.0%
0.0%
24
Index of FP32 bit
6.0%
AlexNet (fine-tuned)
DenseNet-201 tuine)tuned) VGG-19 (fine tuned)
Frob. difference
(ine-tuned)
4.0%
2.0%
0.0%
0.. ...............0..55
Index of FP32 bit
Figure 4: Change in the probabilities of each bit of a floating-pon poiit number to be one. The plot in the top shows the difference probabilities between perturbed model distribution and the original model. The plot in the top shows the difference of probabilities between a perturbed and fine-tuned model and the original model.
14


Reviews:
[{"review_id": "xYUmpn5Iut6", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates if neural network parameters, trained for a standard task such as image recognition, can be used as a cover medium in steganography. The authors contend that neural networks are a good choice of cover medium, mainly because the less important fractional bits of parameters follow a uniform distribution. This is empirically demonstrated in Figure 1 where the probability of least significant bits taking values 0 or 1 are 50% for VGG, ResNet, DenseNet (I assume these are ImageNet models judging from later experiments). The authors experiment with replacing either the least or most significant bits from the fractional part of parameter values. The motivation behind replacing the most significant bits is that it will be more difficult for an attacker to remove secret information; bits of lower significance could be removed or replaced with little effect on test accuracy, while removing more significant fractional bits will have a larger impact on accuracy. The downside of this approach is that replacing more important fractional bits with secrets will also have a larger impact on accuracy, but the authors show fine-tuning can somewhat alleviate this drawback. The authors then show simple MLPs trained to distinguish between standard and stego networks are only slightly better than a random coin flip. Overall, I thought the paper was generally well-written and contains some interesting ideas, but these positives are also accompanied by an unclear motivation, lack of positioning with respect to related work, and poor experimental evaluation.\n\nMy primary concern is the practical motivation for this idea. In steganography, it is normally assumed that both the cover and steg image/audio/network cannot both be revealed, otherwise a trivial comparison will reveal secret information is present thus breaking security of the scheme. Does this mean a new model needs to be trained each time two parties want to communicate? If so, this seems to represent a serious limitation, since training these models requires a non-trivial amount of effort, in comparison to say, generating an image or audio file as the cover. Furthermore, a problematic scaling law then appears, where larger models need to be trained (on presumably more complex tasks to recover some plausible deniability e.g. it would be suspicious to see a ResNet-50 trained on MNIST), to hide larger messages. These larger models are much more expensive to train, and again generating high-resolution images is a much cheaper option. Sharing large resolution images will generally consume less memory than very large neural networks. I wonder how the authors view the practicality of this work? It would have been great to compare this scheme with some standard steganography schemes on images or audio across desideratum such as bandwidth, robustness etc.\n\nThe authors may not be aware, but there has been work on information hiding in previous work. Song et al. (2017) [1] also investigate how information can be imperceptibly embedded and recovered from neural networks. In addition to experiments on embedding secrets in fractional bits, they have experiments in black-box settings. Could the authors comment on relationship between this work and Song et al. (2017) [1]? As far as I can tell, they are quite similar ideas.\n\nI was a little disappointed that the authors didn't include any practical demonstrations, such as successful recovery of secrets as shown in Song et al. (2017) [1]? Steganographic scheme are evaluated by security, capacity and robustness, as the authors comment in Section 2.1, but most of the experiments in section 4 concentrate on capacity. How does the scheme stand-up against robustness attacks such as down-stream task fine-tuning, parameter pruning etc.? I was also confused by the experiment set-up in Section 4.4. For a fair experiment of distinguishability, I had expected a large number of cover and stego models to be trained (for some notion of confidence), and then use a steganalysis tool to distinguish between the two. I'm not sure training a steganalysis model on a single stego network allows for a fair interpretation of performance. \n\n\n[1] Song, Congzheng, Thomas Ristenpart, and Vitaly Shmatikov. \"Machine learning models that remember too much.\" Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 2017.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UcStQC1ibsu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a method to hide information in the parameters of neural network models. To avoid significant perturbation, the paper only considers embed the information in the fraction bits of the parameters. The paper considers hiding the information in either the least significant bits of the most significant bits. Hiding in the least significant bits is harder to be detected but the message can also be easily removed without much degradation in the model performance. On the other hand, information  hiding in the most significant fraction bits will be very hard to remove without model performance degradation but also harder to embed the message for the same reason. Sensitivity analysis to select least sensitive parameters to use and fine-tuning after embedding to recover the model performance can be two remedies.\n\nOverall the paper is clearly written and the proposed method is well supported by the experiments. Some questions / comments below:\n- How many files in each format are used to generate the plots in figure 1? Is it statistically significant?\n- Sec 3.3 \"In the experiment section, we partitioned the parameters of a model into two where the former is used to hide secrets into one part while the latter is fine-tuned.\" It's not clear to me what the criteria is for partitioning the parameters. Is it purely based on sensitivity analysis?\n- Sec 4. \"All the values reported in the table are the averaged ones across ten different runs.\". Can the standard deviation be added to the results as well?\n- Sec 4.1 \"Since the number of BN parameters only account for less than 1% of the entire parameters, which indicates low capacity, we excluded BN parameters in later experiments.\" Does it mean all models used do not have BN layers?\n- Re. Table 2: It's interesting to see some models are not sensitive to even MSFB perturbations. Any insight on the specific reason?", "extra_scores": {}, "extra_text": {}}, {"review_id": "C40k1yKra6T", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper highlights and studies the interesting possibility of hiding information within neural network weights, which is a form of steganography. The sensitivity of different neural network layers to perturbations is evaluated, and based on this a technique for hiding information is proposed and demonstrated. It is shown that it is possible to hide information in the weights of a number of standard baseline neural networks without being easily detectable.\n\nOverall this is an interesting paper, which highlighted a possibility which I was not aware of (although I do have a little familiarity with steganography). My only significant criticism is that I would really like to see a more thorough exploration and discussion of the information hiding capacity of neural networks. In particular it would be great if the authors could explore the relationship between the quantity of information that can be hidden and the size/number of parameters. As far as I can see, the sizes of the weights are only mentioned in passing once (on the first page of the paper), where they are said to be 'in the hundreds of megabytes' (an understatement these days), and the paper does not say how much information they were able to hide in weights. As a bare minimum I would ask that the authors state how much information they were able to hide (as in, how many megabytes), as well as the actual sizes of the weights for the neural networks used (rather than the vague 'hundreds of megabytes'). A more thorough exploration of the relationship between neural net size and stego capacity would be even better. Does it follow the 'square root law' typical in other settings (see e.g. http://www.cs.ox.ac.uk/andrew.ker/docs/ADK71B.pdf)? Perhaps this more thorough exploration can be deferred to future work.\n\nApart from this I have a number of more minor criticisms and suggested improvements:\n - Multiple issues with Figure 1.\n   - In the first sentence of the caption: 'Fraction bit distributions', you should probably replace 'distributions' with 'histograms'.\n   - The y-axis of at least one of the histogram plots should be labelled (I guess with 'frequency') so that it's clear what the numerical values represent.\n   - Instead of 'top left three' you should be more specific and say 'top row, first three'.\n - Generally there were a lot of grammatical errors and spelling mistakes, and you should spend more time carefully proof-reading the paper. A (probably not exhaustive) list of examples:\n   - The first sentence of the second paragraph on page 2 beginning 'Since the fraction bits' is not a valid sentence.\n   - 'Neural' is misspelled in the first line of the last paragraph of page 3.\n   - The title of Section 2 should be 'Background', not 'Backgrounds'\n   - In first sentence of Section 4, what does 'their' refer to? Probably should replace 'their usefulness' with 'the usefulness of DNNs'\n   - About two thirds of the way through the first paragraph in Section 4.4, in the sentence beginning 'In particular, ...'; 'fraction bits an...' that 'an' should be an 'and'. Also in 'we made further divided them into two types', delete 'made'.\n   - The third sentence of the second paragraph of Section 5, beginning 'In particular, our experiment...' is not a valid sentence.\n - Don't use double parentheses for citations as you do on the last line of page 1. You can use semicolons to separate the citations from the other text inside the parentheses.\n - In the third paragraph of page 3, you say 'A trivial approach to hide... replace the least significant bits in the pixel values of the **secret** image...' I'm pretty sure you mean cover image there, not secret image.\n - In the second paragraph of page 4, could you explain the difference between watermarking and steganography?\n - In the fifth paragraph on page 5 you say '...a load of...', this is far too colloquial for an academic paper.\n - The investigation represented by Table 1 could have been more detailed. For example you could have tested (as others have done previously) whether outputs are more sensitive to weights closer to the input/output of the NN. You could also note that it's pretty obvious that a NN would be more sensitive to batchnorm parameters than others since these affect the scale and position of all of the outputs of a layer.\n - The way you structure the experiments section, with two short introductory paragraphs followed by subsections, could be streamlined. Merge the informative parts of the intro paragraphs into the subsections and then delete the intro paragraphs.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents an original analysis of using the weights of a neural network as a medium on which to hide information. Although the paper offers a novel perspective, its motivation and applicability remain unclear. As reviewer 3 points out, the proposed method does not seem very practical for any particular application, and the authors do not give a practical demonstration that shows the usefulness of the approach. It's not clear how the paper should be positioned with respect to previous work, and the proposed method is not directly compared with standard steganography schemes on metrics such as bandwidth, robustness etc, making it difficult to assess the value of the contribution. For these reasons I recommend rejecting the paper in its current form.
Decision: Reject