Stego Networks: Information Hiding on Deep Neural Networks

1 INTRODUCTION . As much as it goes without saying knowledge is power , inventing methods for keeping and selectively conveying secret messages has been a crucial mission throughout the history of humanity . Among various methods to protect secrets , an effective approach called steganography makes it difficult to detect the very existence of the secrets in an object looking innocuous . The object containing the secrets is called a stego medium in the context of steganography . Starting with the case of hiding a secret message in the form of engraved tattoos on hidden parts of a human body in an ancient greek period , numerous methods ( e.g. , using invisible inks , writing tiny-sized letters ) were employed to transmit information without leaving detectable footprints ( Kahn , 1996 ) . Most recently , digital steganography , which embeds secret messages in digital images or audio files , has been actively developed . Traditional steganography is typically used in communication between two individuals , but steganography in digital media enables its brand-new usage by conveying secrets in a multitude of devices and unknowingly influencing their behavior when accompanied with a small decoding code . The secrets in this scenario are often called stegomalware ( Nagaraja et al. , 2011 ; Suarez-Tangil et al. , 2014 ) . Meanwhile , deep neural networks ( DNNs ) have shown remarkable success in various areas over the years , and are now beginning to be applied to industry and the consumer sector as well as to the academic . DNNs have been deployed in a variety of computing systems ranging from largescale cloud computing systems to millions of mobile devices ( Howard et al. , 2017 ) . More and more mobile devices are running application programs that include deep learning models with numerous camera filter and speech recognition applications being good examples . Furthermore , building upon existing large pre-trained models , such as ResNet ( He et al. , 2016 ) , BERT ( Devlin et al. , 2019 ) , and GPT-3 ( Brown et al. , 2020 ) , rather than training complete neural networks from scratch has become a trend in deep learning research . Accordingly , various files containing neural network parameters are uploaded on the source code repositories , and FTP servers , and they are frequently exchanged among individuals and organizations . The fraction bits , which are sometimes called the significand or the mantissa , of a floating-point number follows a distribution close to uniform distribution as shown in Figure 1 . A secret sender can easily embed encrypted messages , which also typically follows a uniform distribution , without causing much suspicion from static analysis tools . Also , as the sizes of pre-trained neural network parameters are often large being more than hundreds of megabytes in size , neural network parameters are suitable media to exchange a nontrivial amount of secrets . In this paper , we analyze the distribution of fraction bits of typical neural network parameters . Fraction bits , which contain the least significant information of a floating-point number , can conveniently be used to embed secret messages . We experimented with a special kind of weight perturbation , which simulate general cases of hiding secrets in network parameters and explored several methods to inject arbitrary data into neural network parameters without noticeable performance degradation . We empirically showed that steganography in the least significant fraction bits is readily applicable and even steganography in the most significant fraction bits is also possible . Our main contributions are as follows : • We demonstrate suitability of neural network parameters as a novel steganographic medium . • We propose novel approaches to effectively embed secrets in neural network parameters . • We give comprehensive analysis of stego networks in terms of security and capacity , and discuss the advantages of stego networks over the conventional stego media . 2 BACKGROUND . This section covers materials to facilitate understanding of this paper , which includes the fundamentals of steganography and related work . 2.1 STEGANOGRAPHY . Steganography is a technique to conceal information from unauthorized persons or eavesdroppers . It is a sub-field of information hiding . The goal of steganography is to hide the presence of the secret by embedding it in a camouflage medium , which is called a cover medium . There exist three aspects in steganography that decide its effectiveness : security , capacity , and robustness ( Provos & Honeyman , 2003 ; Cox et al. , 2007 ) . Among them , security is often considered the most important factor . Security means that the presence of the secret must not be revealed under any situation . The next one , capacity , means the amount of information that the steganographic system can carry . Usually the risk of being detected increases as capacity increases . Robustness means the durability from external noise or interruption to eliminate the embedded information . In the context of stego networks , robustness can be degraded if re-training or fine-tuning is applied . Among various cover media , we mainly compare stego networks with images because it is one of the most actively studied media of secret information and image files are widely-used file formats among deep learning researchers and practitioners . Usually , each pixel of a three-channel RGB image is represented by a total of 24 bits , eight bits per channel . A trivial approach to hide a secret image in a cover image is to replace the least significant bits in the pixel values of the cover image with the most significant bits of the secret image . However , this approach is vulnerable because the secret image can be seen in the stego image due to humans ’ highly sensitive visual perception . The process that reveals the existence of steganography by visual characteristic is called visual attack . Also , digital images follow a certain distribution of pixel values that can be easily distinguished . Steganalysis refers to the process of detecting the steganography by capturing the statistical anomaly of data . Several techniques have been proposed such as a subtractive pixel adjacency matrix ( SPAM ) ( Pevny et al. , 2010 ) to neutralize the steganographic systems . Currently , well-known steganography algorithms include HUGO , WOW , and S-UNIWARD ( Pevnỳ et al. , 2010 ; Holub & Fridrich , 2012 ; Holub et al. , 2014 ) . 2.2 RELATED WORK . There have been numerous approaches in the literature that applied deep neural network to steganalysis . The objective of some of these studies was to construct neural networks , which can determine whether hidden contents is embedded in an input image ( Xu et al. , 2016 ; Yedroudj et al. , 2018 ) . On the other hand , Baluja ( 2017 ) and Zhu et al . ( 2018 ) proposed methods to generate stego images . Our work is clearly different from these studies in that our work considers the neural network itself as a cover medium of steganography . To the best of our knowledge , there has not been a principled study of neural networks as cover media of steganography so far . One work relevant to of ours is LSB embedding attack proposed by Song et al . ( 2017 ) . It the work , a scenario is suggested where an adversary intercepts training data with compromised training algorithm which hides the data in the LSFBs of the trained neural network parameters . The data embedding method can be seen as a special form of LSFB steganography which we explain in Section 3 . However , Song et al . ( 2017 ) does not include discussion of neural network parameters in terms of steganographic security , capacity , and robustness , which are the three main aspects of steganography . Therefore , it can hardly be seen as study of properteis of neural networks as steganographic cover media . Our work is closely related with research on sensitivity and precision of neural network parameters . Cheney et al . ( 2017 ) conducted a sensitivity analysis on the convolutional neural networks ( CNNs ) and reported that AlexNet ( Krizhevsky et al. , 2012 ) layers close to the input showed relatively high sensitivity to weight perturbation than those layers close to the output . Weight quantization ( McKinstry et al. , 2018 ; Wang et al. , 2018 ; Zafrir et al. , 2019 ; Jung et al. , 2019 ) is also an actively studied research area for energy and storage efficiencies . Studies in this field showed that neural networks can preserve their performance although a half or less number of bits are used to represent parameters . In terms of DNN security , weight poisoning ( Gu et al. , 2017 ) and adversarial attack ( Goodfellow et al. , 2015 ) also have been frequently studied . Both methods aim to manipulate the output of neural networks , but the difference between the two methods is that weight poisoning perturbs neural network parameters while adversarial attack perturbs the input of the network . In the case of adversarial attack , its counter-measures called adversarial defense ( Metzen et al. , 2017 ) and their counter-counter-measures have also been continuously studied ( Ghiasi et al. , 2020 ) . Another closely related field to our work is watermarking . Both watermarking and steganography belong to the area of information hiding . While the top priority of steganography is security , watermarking focuses on robustness , in order to prevent attempts to removing information stored in cover media . Watermarking on neural networks can be used to integrity check and copyright protection of the network . Adi et al . ( 2018 ) prepared a trigger dataset , which is completely unrelated to the original task , and train or fine-tune a model to yield 100 % accuracy on the trigger set . Then , the auditor can determine whether the model is watermarked or not by checking the accuracy of the trigger set . Similarly , Le Merrer et al . ( 2020 ) adjusted the decision boundary of the model for specific inputs . Instead of examining the output , there exists a method that probe the activation values from the networks for a private input ( Darvish Rouhani et al. , 2019 ) . Lastly , Uchida et al . ( 2017 ) added a regularization loss on the network parameters to embed watermark . The watermark is a form of a binary vector , which can be recovered by multiplying an embedding matrix with the flattened parameters . In the case of watermarking , it usually leaves the noticeable trace on the network parameters , thus it can not transmit information confidentially . 3 STEGO NETWORKS . This section presents the analysis on the sample distributions of fraction bits of typical neural network parameters . Afterwards , we propose two of our novel approaches to embed secret messages in neural networks . 3.1 FRACTION BITS OF NEURAL NETWORK PARAMETERS . Neural network parameters are usually in floating-point types . A floating-point number consists of three parts for sign , exponent and fraction bits1 . For example , a single-precision ( FP32 ) format has one sign bit , eight exponent bits , and 23 fraction bits . Since FP32 is one of the most widely used formats in deep learning research , we use the format throughout this paper . The least significant fraction bit has the lowest index , i.e. , zero . Since embedding messages in sign or exponent bits 1Details of floating-point number are described in IEEE Standard for Binary Floating-Point Arithmetic . can induce a relatively significant perturbation to the original value , we only consider the case of embedding bits in fraction bits . The fraction bit distributions of the parameters of a few models commonly used in computer vision are provided in Figure 1 . We divide the 23 bits of fraction bits into three parts indexed as 0-7 , 8- 15 , and 16-22 , to ease the visualization . We represent a sequence of bits of each part simply by their corresponding decimal value . For example , a bit sequence 10000010 with the right-most bit denoting the bit of the lowest index is represented as 130 in Figure 1 . When it comes to randomness of fraction bits , we particularly considered two issues , which are uniformity of a random variable and independent and identically distributed ( i.i.d . ) property of a sequence of random variables . We can denote the part of FP32 fraction bits indexed 0-7 as Xi with |Xi| = 256 , and i is the index of the corresponding FP32 number in the entire parameters . Then one may think Xi is a uniform random variable if she observes each outcome happens with almost the same frequency . On the other hand , if one often observes the same outcomes consecutively , she may think the sequence of Xi is not i.i.d . If Xi is uniform and a sequence of Xi are i.i.d. , then the sequence has the maximum entropy . Note that a message embedder usually encrypt secret messages to provide additional protection . Also , encrypting messages makes it difficult for a steganalysis system to detect the existence of the message since much of the traces disappears . Encryption algorithms such as AES and RSA are often involved before message embedding . In the rest of this section , we assume message embedders always encrypt secret messages . Uniformity of a random variable is directly related with the capacity of stego networks and an secret embedder may need to sacrifice the bit rate if the bit distribution of a target medium is not uniform . Figure 1 shows distributions of the three fraction parts of five popular models in computer vision . The two least significant parts show nearly uniform distributions across all the models . The most significant seven bits seemingly followed non-uniform distributions . In this case , a secret sender may want to add dummy bits in the original secret message to avoid a discrepancy between the fraction bit distribution of the original neural network and the resulting stego network to ensure security . Another important property relevant to steganographic security is the i.i.d . property . This property is what various steganalysis algorithms are based on to detect anomaly in stego media . For example , SPAM algorithm ( Pevny et al. , 2010 ) utilizes the fact that pixels of natural images have high spatial correlation such that the colors of neighboring pixels have similar values . Therefore , if the usual bits of a cover medium are purely i.i.d. , it automatically wards off a large number of sophisticated steganalysis systems . Note that establishing theoretical independence between the significands of parameters is out of scope in our work . Instead , we mainly provide empirical results that has practical implications in the experiment section . From the experimental results , we suspect that significands of stego networks exhibit considerable low dependency between them compared to the storages of other popular steganographic media .

Summary: This paper highlights and studies the interesting possibility of hiding information within neural network weights, which is a form of steganography. The sensitivity of different neural network layers to perturbations is evaluated, and based on this a technique for hiding information is proposed and demonstrated. It is shown that it is possible to hide information in the weights of a number of standard baseline neural networks without being easily detectable.