Stego Networks: Information Hiding on Deep Neural Networks

1 INTRODUCTION . As much as it goes without saying knowledge is power , inventing methods for keeping and selectively conveying secret messages has been a crucial mission throughout the history of humanity . Among various methods to protect secrets , an effective approach called steganography makes it difficult to detect the very existence of the secrets in an object looking innocuous . The object containing the secrets is called a stego medium in the context of steganography . Starting with the case of hiding a secret message in the form of engraved tattoos on hidden parts of a human body in an ancient greek period , numerous methods ( e.g. , using invisible inks , writing tiny-sized letters ) were employed to transmit information without leaving detectable footprints ( Kahn , 1996 ) . Most recently , digital steganography , which embeds secret messages in digital images or audio files , has been actively developed . Traditional steganography is typically used in communication between two individuals , but steganography in digital media enables its brand-new usage by conveying secrets in a multitude of devices and unknowingly influencing their behavior when accompanied with a small decoding code . The secrets in this scenario are often called stegomalware ( Nagaraja et al. , 2011 ; Suarez-Tangil et al. , 2014 ) . Meanwhile , deep neural networks ( DNNs ) have shown remarkable success in various areas over the years , and are now beginning to be applied to industry and the consumer sector as well as to the academic . DNNs have been deployed in a variety of computing systems ranging from largescale cloud computing systems to millions of mobile devices ( Howard et al. , 2017 ) . More and more mobile devices are running application programs that include deep learning models with numerous camera filter and speech recognition applications being good examples . Furthermore , building upon existing large pre-trained models , such as ResNet ( He et al. , 2016 ) , BERT ( Devlin et al. , 2019 ) , and GPT-3 ( Brown et al. , 2020 ) , rather than training complete neural networks from scratch has become a trend in deep learning research . Accordingly , various files containing neural network parameters are uploaded on the source code repositories , and FTP servers , and they are frequently exchanged among individuals and organizations . The fraction bits , which are sometimes called the significand or the mantissa , of a floating-point number follows a distribution close to uniform distribution as shown in Figure 1 . A secret sender can easily embed encrypted messages , which also typically follows a uniform distribution , without causing much suspicion from static analysis tools . Also , as the sizes of pre-trained neural network parameters are often large being more than hundreds of megabytes in size , neural network parameters are suitable media to exchange a nontrivial amount of secrets . In this paper , we analyze the distribution of fraction bits of typical neural network parameters . Fraction bits , which contain the least significant information of a floating-point number , can conveniently be used to embed secret messages . We experimented with a special kind of weight perturbation , which simulate general cases of hiding secrets in network parameters and explored several methods to inject arbitrary data into neural network parameters without noticeable performance degradation . We empirically showed that steganography in the least significant fraction bits is readily applicable and even steganography in the most significant fraction bits is also possible . Our main contributions are as follows : • We demonstrate suitability of neural network parameters as a novel steganographic medium . • We propose novel approaches to effectively embed secrets in neural network parameters . • We give comprehensive analysis of stego networks in terms of security and capacity , and discuss the advantages of stego networks over the conventional stego media . 2 BACKGROUND . This section covers materials to facilitate understanding of this paper , which includes the fundamentals of steganography and related work . 2.1 STEGANOGRAPHY . Steganography is a technique to conceal information from unauthorized persons or eavesdroppers . It is a sub-field of information hiding . The goal of steganography is to hide the presence of the secret by embedding it in a camouflage medium , which is called a cover medium . There exist three aspects in steganography that decide its effectiveness : security , capacity , and robustness ( Provos & Honeyman , 2003 ; Cox et al. , 2007 ) . Among them , security is often considered the most important factor . Security means that the presence of the secret must not be revealed under any situation . The next one , capacity , means the amount of information that the steganographic system can carry . Usually the risk of being detected increases as capacity increases . Robustness means the durability from external noise or interruption to eliminate the embedded information . In the context of stego networks , robustness can be degraded if re-training or fine-tuning is applied . Among various cover media , we mainly compare stego networks with images because it is one of the most actively studied media of secret information and image files are widely-used file formats among deep learning researchers and practitioners . Usually , each pixel of a three-channel RGB image is represented by a total of 24 bits , eight bits per channel . A trivial approach to hide a secret image in a cover image is to replace the least significant bits in the pixel values of the cover image with the most significant bits of the secret image . However , this approach is vulnerable because the secret image can be seen in the stego image due to humans ’ highly sensitive visual perception . The process that reveals the existence of steganography by visual characteristic is called visual attack . Also , digital images follow a certain distribution of pixel values that can be easily distinguished . Steganalysis refers to the process of detecting the steganography by capturing the statistical anomaly of data . Several techniques have been proposed such as a subtractive pixel adjacency matrix ( SPAM ) ( Pevny et al. , 2010 ) to neutralize the steganographic systems . Currently , well-known steganography algorithms include HUGO , WOW , and S-UNIWARD ( Pevnỳ et al. , 2010 ; Holub & Fridrich , 2012 ; Holub et al. , 2014 ) . 2.2 RELATED WORK . There have been numerous approaches in the literature that applied deep neural network to steganalysis . The objective of some of these studies was to construct neural networks , which can determine whether hidden contents is embedded in an input image ( Xu et al. , 2016 ; Yedroudj et al. , 2018 ) . On the other hand , Baluja ( 2017 ) and Zhu et al . ( 2018 ) proposed methods to generate stego images . Our work is clearly different from these studies in that our work considers the neural network itself as a cover medium of steganography . To the best of our knowledge , there has not been a principled study of neural networks as cover media of steganography so far . One work relevant to of ours is LSB embedding attack proposed by Song et al . ( 2017 ) . It the work , a scenario is suggested where an adversary intercepts training data with compromised training algorithm which hides the data in the LSFBs of the trained neural network parameters . The data embedding method can be seen as a special form of LSFB steganography which we explain in Section 3 . However , Song et al . ( 2017 ) does not include discussion of neural network parameters in terms of steganographic security , capacity , and robustness , which are the three main aspects of steganography . Therefore , it can hardly be seen as study of properteis of neural networks as steganographic cover media . Our work is closely related with research on sensitivity and precision of neural network parameters . Cheney et al . ( 2017 ) conducted a sensitivity analysis on the convolutional neural networks ( CNNs ) and reported that AlexNet ( Krizhevsky et al. , 2012 ) layers close to the input showed relatively high sensitivity to weight perturbation than those layers close to the output . Weight quantization ( McKinstry et al. , 2018 ; Wang et al. , 2018 ; Zafrir et al. , 2019 ; Jung et al. , 2019 ) is also an actively studied research area for energy and storage efficiencies . Studies in this field showed that neural networks can preserve their performance although a half or less number of bits are used to represent parameters . In terms of DNN security , weight poisoning ( Gu et al. , 2017 ) and adversarial attack ( Goodfellow et al. , 2015 ) also have been frequently studied . Both methods aim to manipulate the output of neural networks , but the difference between the two methods is that weight poisoning perturbs neural network parameters while adversarial attack perturbs the input of the network . In the case of adversarial attack , its counter-measures called adversarial defense ( Metzen et al. , 2017 ) and their counter-counter-measures have also been continuously studied ( Ghiasi et al. , 2020 ) . Another closely related field to our work is watermarking . Both watermarking and steganography belong to the area of information hiding . While the top priority of steganography is security , watermarking focuses on robustness , in order to prevent attempts to removing information stored in cover media . Watermarking on neural networks can be used to integrity check and copyright protection of the network . Adi et al . ( 2018 ) prepared a trigger dataset , which is completely unrelated to the original task , and train or fine-tune a model to yield 100 % accuracy on the trigger set . Then , the auditor can determine whether the model is watermarked or not by checking the accuracy of the trigger set . Similarly , Le Merrer et al . ( 2020 ) adjusted the decision boundary of the model for specific inputs . Instead of examining the output , there exists a method that probe the activation values from the networks for a private input ( Darvish Rouhani et al. , 2019 ) . Lastly , Uchida et al . ( 2017 ) added a regularization loss on the network parameters to embed watermark . The watermark is a form of a binary vector , which can be recovered by multiplying an embedding matrix with the flattened parameters . In the case of watermarking , it usually leaves the noticeable trace on the network parameters , thus it can not transmit information confidentially . 3 STEGO NETWORKS . This section presents the analysis on the sample distributions of fraction bits of typical neural network parameters . Afterwards , we propose two of our novel approaches to embed secret messages in neural networks . 3.1 FRACTION BITS OF NEURAL NETWORK PARAMETERS . Neural network parameters are usually in floating-point types . A floating-point number consists of three parts for sign , exponent and fraction bits1 . For example , a single-precision ( FP32 ) format has one sign bit , eight exponent bits , and 23 fraction bits . Since FP32 is one of the most widely used formats in deep learning research , we use the format throughout this paper . The least significant fraction bit has the lowest index , i.e. , zero . Since embedding messages in sign or exponent bits 1Details of floating-point number are described in IEEE Standard for Binary Floating-Point Arithmetic . can induce a relatively significant perturbation to the original value , we only consider the case of embedding bits in fraction bits . The fraction bit distributions of the parameters of a few models commonly used in computer vision are provided in Figure 1 . We divide the 23 bits of fraction bits into three parts indexed as 0-7 , 8- 15 , and 16-22 , to ease the visualization . We represent a sequence of bits of each part simply by their corresponding decimal value . For example , a bit sequence 10000010 with the right-most bit denoting the bit of the lowest index is represented as 130 in Figure 1 . When it comes to randomness of fraction bits , we particularly considered two issues , which are uniformity of a random variable and independent and identically distributed ( i.i.d . ) property of a sequence of random variables . We can denote the part of FP32 fraction bits indexed 0-7 as Xi with |Xi| = 256 , and i is the index of the corresponding FP32 number in the entire parameters . Then one may think Xi is a uniform random variable if she observes each outcome happens with almost the same frequency . On the other hand , if one often observes the same outcomes consecutively , she may think the sequence of Xi is not i.i.d . If Xi is uniform and a sequence of Xi are i.i.d. , then the sequence has the maximum entropy . Note that a message embedder usually encrypt secret messages to provide additional protection . Also , encrypting messages makes it difficult for a steganalysis system to detect the existence of the message since much of the traces disappears . Encryption algorithms such as AES and RSA are often involved before message embedding . In the rest of this section , we assume message embedders always encrypt secret messages . Uniformity of a random variable is directly related with the capacity of stego networks and an secret embedder may need to sacrifice the bit rate if the bit distribution of a target medium is not uniform . Figure 1 shows distributions of the three fraction parts of five popular models in computer vision . The two least significant parts show nearly uniform distributions across all the models . The most significant seven bits seemingly followed non-uniform distributions . In this case , a secret sender may want to add dummy bits in the original secret message to avoid a discrepancy between the fraction bit distribution of the original neural network and the resulting stego network to ensure security . Another important property relevant to steganographic security is the i.i.d . property . This property is what various steganalysis algorithms are based on to detect anomaly in stego media . For example , SPAM algorithm ( Pevny et al. , 2010 ) utilizes the fact that pixels of natural images have high spatial correlation such that the colors of neighboring pixels have similar values . Therefore , if the usual bits of a cover medium are purely i.i.d. , it automatically wards off a large number of sophisticated steganalysis systems . Note that establishing theoretical independence between the significands of parameters is out of scope in our work . Instead , we mainly provide empirical results that has practical implications in the experiment section . From the experimental results , we suspect that significands of stego networks exhibit considerable low dependency between them compared to the storages of other popular steganographic media .

Summary: This paper proposed a method to hide information in the parameters of neural network models. To avoid significant perturbation, the paper only considers embed the information in the fraction bits of the parameters. The paper considers hiding the information in either the least significant bits of the most significant bits. Hiding in the least significant bits is harder to be detected but the message can also be easily removed without much degradation in the model performance. On the other hand, information  hiding in the most significant fraction bits will be very hard to remove without model performance degradation but also harder to embed the message for the same reason. Sensitivity analysis to select least sensitive parameters to use and fine-tuning after embedding to recover the model performance can be two remedies.