# Autoregressive Linguistic Steganography Based on BERT and Consistency Coding

Authors: Xiaoyan Zheng; Hanzhou Wu
Journal: Security and Communication Networks
 (2022)

---

## Abstract

Linguistic steganography (LS) conceals the presence of communication by embedding secret information into a text. How to generate a high-quality text carrying secret information is a key problem. With the widespread application of deep learning in natural language processing, recent algorithms use a language model (LM) to generate the steganographic text, which provides a higher payload compared with many previous arts. However, the security still needs to be enhanced. To tackle this problem, we propose a novel autoregressive LS algorithm based on BERT and consistency coding, which achieves a better trade-off between embedding payload and system security. In the proposed work, based on the introduction of the masked LM, given a text, we use consistency coding to make up for the shortcomings of block coding used in the previous work so that we can encode arbitrary-size candidate token set and take advantage of the probability distribution for information hiding. The masked positions to be embedded are filled with tokens determined by an autoregressive manner to enhance the connection between contexts and therefore maintain the quality of the text. Experimental results have shown that compared with related works, the proposed work improves the fluency of the steganographic text while guaranteeing security and also increases the embedding payload to a certain extent.


---

## Body

## 1. Introduction

With the rapid development of wireless communication and social networking services, a lot of people are instantly sharing daily-life feelings and media data by integrating into social networks through mobile terminal devices. This has brought great convenience to covert communication which aims to reliably convey a secret message to the data receiver without arousing the suspicion of the monitor. Steganography, as a means to covert communication [1], is an art of embedding secret information in an innocent digital carrier such as image and video by modifying the carrier [2–4]. Since the resulting modified carrier containing secret information will not introduce noticeable artifacts, by sending the modified carrier via an insecure public channel such as social network, the receiver is able to fully reconstruct the secret information. Steganography secures communication using its concealment, becoming more and more important in information security [5, 6].Regardless of the past or the present, text is one of the most important information exchange media in everyday life, and communication and conversation between each other need to rely on text to convey a large amount of information. At the same time, natural language has high robustness in the process of information transmission; even in the case of interference, text can still keep secret transmission in the public channel without distortion. It can be seen from the above two points that it is reasonable and necessary to study using text as the carrier of steganography. However, due to the low redundancy of the text itself, compared with image, audio, video, and other digital media [7–9], the research on linguistic steganography (LS) is more complex and challenging.The basic framework of LS is based on the scene description of the prisoners’ problem [1]. In order to ensure that Alice and Bob communicate secretly without being discovered by the guardian Wendy, a steganographic system (stego system) was designed based on natural language processing (NLP), the key problems of which are how to embed secret information in a text (i.e., data embedding for Alice) and how to extract secret information from the text containing secret information (i.e., data extraction for Bob). The most important purpose for a LS system is to conceal the existence of communication to ensure the unsuspiciousness of steganographic texts (stego texts). It means that it should not reveal any evidence of hidden information when the stego texts are being analyzed by humans or computers. In other words, it should be impossible for both the human perceptual system and a linguistic steganalyzer to separate the stego texts from natural ones. Under the condition of good concealment, it is generally quite desirable to embed as much information as possible in a text. To count the amount of the embedded information, “bits per word (BPW)” have been widely used in LS methods. Mainstream LS methods focus on improving the imperceptibility (corresponding to concealment) and the embedding payload. However, the relationship between them is mutually competitive and contradictory [10–12]. Enhanced imperceptibility will reduce the embedding payload, and the increase of the embedding payload will require more modifications to the original text, reducing the imperceptibility (which leads to low security). How to achieve a good balance between them is thus a core problem for LS.LS can generally be divided to two categories: modification based and generation based. Traditional stego systems such as [13–15] embed secret information in a given text (typically also called cover) by modifying the cover, which is referred to as the former category. It is required that the modification should not significantly impair the cover text. Early methods mainly alter the format attributes to embed secret information such as character spacing, font size, and text line spacing [16, 17]. Modifying the file structures has also been investigated since it leaves the content of the text unchanged and thus has good concealment but cannot resist against re-editing attacks [18, 19]. Researchers have also tried to use the transformation of text sentence characteristics [20, 21] or vocabulary [22, 23] to generate the stego text such as the most common synonym substitution [24, 25]. In particular, when using word (or say token) replacement to embed secret information, one should well design the synonym dictionary as well as the information encoding strategy. Although modification-based LS has high semantic concealment, the payload is not high. So, it is necessary to find a good trade-off between them.Generation-based LS is the process of directly generating a stego text based on secret information and a trained language model (LM). During stego text generation, the trained LM will assign a prediction probability to each candidate word in the word pool such that the present output is the “most suitable” word maintaining the text quality well while matching secret information. Compared with traditional modification-based LS methods, generation-based LS methods can provide a higher embedding payload [26], but the security cannot be well guaranteed [27–32]. Even if the modern LM is combined with arithmetic coding, the anti-statistical analysis ability has not been improved, and there is still a poor anti-staganalysis ability [33]. So, the security needs to be enhanced.Recently, Ueoka et al. [34] presented a novel masked language model (masked LM) based on BERT (short for Bidirectional Encoder Representations from Transformers) [35] for realizing LS, which simplifies the rules of thesaurus construction and provides a new perspective for modification-based LS. Though the stego texts generated by this method can carry a relatively high payload, the word prediction in masked positions of this method is parallel. It means that the words corresponding to masked positions have a high degree of independence. It is known that every word in a text is closely related to the fluency of the text. If the words have a high degree of independence, it will be easily recognized by the human perceptual system, inspiring the adversary to develop advanced steganalyzers that reduce the security. Therefore, we urgently need to find a better strategy to generate higher-quality stego texts.In this paper, we propose a novel autoregressive LS algorithm based on the well-known BERT, which uses an autoregressive strategy to generate the stego texts based on the secret information to be hidden. Our main contribution is that we use an autoregressive strategy for the candidate words obtained from the masked LM, fill the masked positions with replaceable words in turn, and then input them to the masked LM to make predictions. We have also improved the information coding strategy, which imitates the mechanism of word selection by native speakers and improves the readability and the authenticity of sentences. Experimental results have shown that compared with the non-autoregressive method, the proposed method improves the fluency of the stego text while guaranteeing the security and increases the embedding payload to a certain extent, which verifies the superiority.The rest of this paper is organized as follows. We review the related work in Section2. Then, we introduce the proposed work in detail in Section 3, followed by convinced experimental results and analysis in Section 4. We conclude this work and provide valuable discussion in Section 5. The preprint version of this work can be found in [36].


## 2. Related Work

Recently, Ueoka et al. [34] presented a novel masked language model (masked LM) based on BERT for realizing LS, which successfully extends LM to modification-based LS. Mathematically, given a text x=x1,x2,…,xn⊂Vn, where xi is the i-th word sampled from a large vocabulary V and n is the total number of words in the text, the method aims to generate such a stego text y=y1,y2,…,yn⊂Vn that a secret message b⊂0,1L can be extracted from y. The data embedding process can be described as follows. First, a certain number of word positions (also called “masked positions”) are selected from x according to the secret key k. Then, by feeding the words in the non-masked positions to the masked LM, a list of candidate words associated with a prediction probability can be determined for each masked position. Thereafter, based on b and the block coding strategy introduced in [34], y can be generated by replacing the words in masked positions in x with the suitable words in the corresponding candidate sets.In order to successfully extractb from y, the data hider and the data receiver should share the secret key k and the masked LM ℳ. The secret key ensures that the data hider and the data receiver can identify the same masked positions. The masked LM ℳ ensures that the data hider and the data receiver obtain the same prediction probability of each candidate word for each masked position so that the data hider knows how to encode a secret stream to a word and the data receiver knows how to decode a word to the corresponding secret stream.In summary, this method significantly increases the embedding payload compared with the previous modification-based LS methods and provides satisfactory security. It also brings the relationship between modification-based LS and generation-based LS closer to each other. However, there are two shortcomings in this method which can be easily exploited by the adversary to reveal the existence of hidden information. The first one is that the word prediction in masked positions in the method is parallel. In other words, the word prediction of different masked positions will be intuitively independent of each other, i.e., predicting the word in a specific position will not affect the prediction of another one. However, every word is closely related to the text fluency. If the words have a high degree of independence, it will be easily recognized by the human perceptual system, allowing the adversary to develop new detectors reducing the security. Therefore, we urgently need to design a more efficient strategy for word prediction.On the other hand, the information encoding strategy (which is a kind of block coding technique) used in this method simply maps a certain number of candidate words to a binary stream with a fixed length for each masked position. For example, for a specific masked position, assuming that there arem words whose prediction probabilities are larger than a threshold tp, the largest l satisfying 2l≤m can be determined. The above method uses the 2l words with the largest probabilities as the candidate words, each of which carries exactly l secret bits. In this way, during data embedding, the candidate word matching the secret stream with a length of l is used to fill the masked position. It is seen that though the length l can be controlled by tuning the threshold tp, it ignores the probabilistic associations between different words. For example, assuming that l=2 and the candidate words are “wonderful (0.6),” “decent (0.2),” “fine (0.1),” and “great (0.1),” where “wonderful (0.6)” means that the prediction probability of “wonderful” is 0.6, the above method maps each word to a binary stream with a length of 2, e.g., the words can be, respectively, mapped to “00,” “01,” “10,” and “11.” Obviously, during data embedding, all the candidate words have the identical probability (i.e., 1/2l) to be chosen to fill the masked position. However, according to the masked LM, during data embedding, instead of randomly choosing a word, choosing a word with a prediction probability as high as possible is more desirable for achieving the better quality of the stego text. It means that we urgently need to design a more efficient strategy for information encoding as well.Word prediction and information encoding have direct impact on the system security. It strongly motivates us to propose a new LS algorithm in this paper, which significantly enhances the security and, more importantly, presents a new perspective to modification-based LS. We show the details in Section3.


## 3. Proposed Method

### 3.1. Overview

As shown in Figure1, the proposed autoregressive LS method involves three participants: the data hider Alice, the adversary (attacker) Wendy, and the data receiver Bob. The goal of Alice is to hide a secret message (in the form of a binary stream) into a cover text. The resulting stego text carrying the secret message will be sent to Bob via an insecure channel such as the Internet, which is being monitored by the adversary Wendy whose purpose is to determine whether a text being transferred contains secret information or not.Figure 1
An intuitive explanation for the proposed method.Alice embeds secret bits in a given cover text by applying the proposed autoregressive LS algorithm based on masked LM (BERT) and consistency coding. Wendy detects whether the stego text contains secret bits or not. Bob reconstructs the secret bits from the stego text by the operation similar to Alice. Alice and Bob should share some side information including the threshold controlling the number of candidate words, the secret key, masked LM, and information encoding strategy in advance, which is required for mainstream LM-based LS algorithms as well.Specifically, according to the secret key and the secret message to be embedded,Alice first determines a set of masked positions in the cover text. For each masked position to be processed, a word will be determined according to the masked LM, the present temporary text (which may contain processed and unprocessed masked positions), and the consistency coding technique so that the word will substitute the masked position and meanwhile match the secret bits to be embedded. When all masked positions are processed, a stego text can be generated and will be sent to Bob, who performs the operation similar to Alice on the stego text so that the entire secret message can be extracted from the stego text. In order to ensure that the data extraction procedure can be successfully executed, Alice and Bob should share the necessary side information in advance including the secret key, the threshold controlling the selection of candidate words, the masked LM, and the consistency coding strategy. Algorithm 1 shows the pseudocode of data embedding. For the receiver, a similar process is performed to reconstruct the secret bits. We provide the details below.


### 3.2. Masking Strategy and Masked Language Model (LM)

The goal of the masking strategy is to replace some words (tokens) in the cover text with the special token “[MASK].” These masked positions will be filled with tokens generated by the masked LM based on the context. The new tokens not only fit into the context but also carry secret information. It is free for us to design the masking strategy. Though the masking strategy can be carefully crafted, it is not the main contribution of this paper. For fair comparison in experiments, we follow the simple but effective masking strategy in [34]. In brief summary, a masking interval f (which is an integer and can be considered as a secret key) is used to control the total number of masked positions. A higher f means less tokens are masked, which reduces the embedding capacity but increases the difficulty of detection.Algorithm 1: Pseudocode for the data embedding procedure.
Input: cover text x, masked LM ℳ, secret bits b, parameters tp, f, and secret key.Output: stego text y.(1)
DeterminexS according to (1), f and secret key(2)
fori=1,2,…,ndo(3)
ifxi′ is [MASK] then(4)
Setxi=xS(5)
Determinepj for every vj∈V with ℳ and xi(6)
Determine a set of candidate wordsV′⊂V with tp(7)
Normalize probabilities of words inV′ by (3)(8)
Map words inV′ to bit streams by Huffman coding(9)
Determinexi∗ which matches a prefix of b(10)
UpdatexS by replacing xi′ with xi∗(11)
Remove the embedded prefix fromb(12)
end if(13)
end for(14)
Determiney according to xS(15)
returnyThe masked strategy has been previously introduced along with BERT [35] as an efficient pretraining strategy for transformer-based nets [37]. By using the masked strategy, BERT overcomes the limitations of unidirectional training and realizes the common dependence on context for prediction. Generally speaking, fine-tuning the downstream tasks with the pretrained BERT according to heir specific tasks such as sentence classification, sequence tagging, and question answering can greatly reduce the expense of designing their own architecture. However, fine-tuning is not required in this paper. Different from recurrent neural networks (RNNs) [38] processing texts by a word-wise manner and generative pretrained transformer (GPT) [39] obscuring future markers, the masked LM provides superior prediction performance due to its bidirectionality. Therefore, we use the pretrained BERT as the masked LM.Letℳ and ℐ=i1,i2,…,is be the masked LM and the index set for the masked positions determined with the secret key. For each i∈ℐ, the corresponding word xi in the cover text x=x1,x2,…,xn will be replaced with “[MASK].” To express it mathematically, for any S⊂ℐ, we define xS as(1)xS=x1′,x2′,…,xn′,where for 1≤i≤n, we have(2)xi′=MASK,ifi∈S,xi,otherwise.Obviously, we havex∅=x and xℐ is the text after replacing all the words in the masked positions with “[MASK].” To embed secret information in the masked positions, we orderly process the masked positions. For each masked position, ℳ will output a prediction probability for each word in the vocabulary according to the temporary text generated previously. A higher probability implies that the corresponding word is more suitable to replace the “[MASK]” to fit into the context. Next, we determine the temporary text for each masked position.Without the loss of generalization, leti1<i2<…<is. The temporary text for the masked index ij∈ℐ is determined as follows. First, the temporary text for i1∈ℐ, denoted by xi1, is xℐ. By feeding xi1 to ℳ, we can generate a prediction probability pi∈0,1 for each vi∈V=v1,v2,…,vz and ∑i=1zpi=1. According to the secret bits to be embedded and the proposed consistency coding technique, we select a word xi1∗∈V to replace the “[MASK]” in the i1-th position in x. For i2∈ℐ, the only one difference between xi1 and xi2 is that in the i1-th position, xi1 uses “[MASK]” but xi2 uses xi1∗. Generally, the only one difference between xij−1 and xij is that in the ij−1-th position, xij−1 uses “[MASK]” but xij uses xij−1∗ for any 2≤j≤s.Taking Figure1 for explanation, the cover text is “Midshire is a wonderful little city” and there are two masked positions. For the first masked position (from left to right), the temporary text to be fed into the masked LM is “Midshire is a [MASK] little [MASK].” After the first masked position was replaced with the stego word “nice,” the temporary text to be fed into the masked LM for the second masked position is “Midshire is a nice little [MASK].” It is seen that the temporary text for the present masked position depends on the generation of the previous temporary text. We deem it an autoregressive method.


### 3.3. Consistency Coding

A most important problem is how to build the mapping relationship between words and secret bits. To deal with this problem, the authors in [34] used a threshold 0≤tp≤1 to collect a list of candidate words whose prediction probabilities are higher than tp and then mapped each of the candidate words to a fixed-length binary stream. In other words, the collected candidate words have the same probability of being selected to fill in the present masked position, which does not take into account the probability distribution of the candidate words and may not fit into the context well.As mentioned previously, it is quite desirable to select such a word that it not only matches the secret bits to be embedded but also has a prediction probability obtained from the masked LM as high as possible. Because the secret bits are evenly distributed, the probability of selecting a specific word as the output actually depends on the number of secret bits carried by the word. In other words, for any two words mapped to a binary stream with the same length, they often have the same probability of matching the secret stream to be embedded. For example, in [34], if all candidate words are mapped to a binary stream with a length of l>0, the probability of choosing any one among 2l candidate words as the present output is 1/2l. It indicates that once the mapping relationship is finished, we are no longer free to choose the word. Therefore, it is necessary that the construction of mapping relationship itself has taken into account the prediction probability distribution so that for those words with a high prediction probability, the probability of choosing any one of them is high as well.In summary, we expect to find such an information encoding strategy that the probability of choosing a word as the present output is proportional to its prediction probability obtained by the masked LM. We treat such information encoding strategy asconsistency coding. Compared with [34], consistency coding has two significant advantages: the number of candidate words can be any integer, rather than a power of two which is required in [34], and it is based on the statistical probability distribution of each word in the vocabulary, which takes into account the frequency of words and makes coding more conducive to the regularization of the text. During each iteration, the model selects the word with the highest prediction probability while matching the secret information as the output to guarantee the quality of the sentence. Such a mechanism imitates the priority of native speakers in choosing words and improves the security accordingly.It is free for us to design the consistency coding. However, many classical entropy encoding methods in information theory can be used to realize consistency coding such as Huffman coding, arithmetic coding, and Shannon–Fano coding [40]. The main implementation steps of consistency coding are in lines 6–8 in Algorithm 1, and the specific process can be described as follows. For simplicity, we use Huffman coding to provide the off-the-shelf solution. Formally, for compactness, for each of the specific masked positions, let p1,p2,…,pz be the prediction probabilities for the words in the vocabulary V=v1,v2,…,vz, where p1≥p2≥…≥pz and ∑i=1zpi=1. We first collect all the words whose prediction probabilities are higher than a predetermined threshold tp. The collected words are then regarded as the candidate words to be encoded. Then, by normalizing the prediction probabilities of the candidate words, we further apply Huffman coding to map each candidate word to a stream. For example, let V′=v1,v2,…,vw⊂V,w≤z, include the candidate words, and the normalization operation is(3)pi←pi∑j=1wpj,∀1≤i≤w,which enables us to thereafter directly apply Huffman coding.Remark 1.
When the entire secret message cannot be carried by a single cover text, multiple cover texts can be used. When the entire secret message can be carried by a single cover text, some masked positions in the cover text will be embedded with secret bits while the other masked positions (if any) are filled with the original words without data embedding.Remark 2.
It is known that Huffman coding does not result in the unique mapping. For example, in Figure1, the two words “city” and “town” are mapped to “0” and “1,” respectively. It is also possible that “city” is mapped to “1” and the other is mapped to “0.” As long as the data hider and the data receiver use the same secret key to control the Huffman coding process, they can obtain the same mapping result.


## 3.1. Overview

As shown in Figure1, the proposed autoregressive LS method involves three participants: the data hider Alice, the adversary (attacker) Wendy, and the data receiver Bob. The goal of Alice is to hide a secret message (in the form of a binary stream) into a cover text. The resulting stego text carrying the secret message will be sent to Bob via an insecure channel such as the Internet, which is being monitored by the adversary Wendy whose purpose is to determine whether a text being transferred contains secret information or not.Figure 1
An intuitive explanation for the proposed method.Alice embeds secret bits in a given cover text by applying the proposed autoregressive LS algorithm based on masked LM (BERT) and consistency coding. Wendy detects whether the stego text contains secret bits or not. Bob reconstructs the secret bits from the stego text by the operation similar to Alice. Alice and Bob should share some side information including the threshold controlling the number of candidate words, the secret key, masked LM, and information encoding strategy in advance, which is required for mainstream LM-based LS algorithms as well.Specifically, according to the secret key and the secret message to be embedded,Alice first determines a set of masked positions in the cover text. For each masked position to be processed, a word will be determined according to the masked LM, the present temporary text (which may contain processed and unprocessed masked positions), and the consistency coding technique so that the word will substitute the masked position and meanwhile match the secret bits to be embedded. When all masked positions are processed, a stego text can be generated and will be sent to Bob, who performs the operation similar to Alice on the stego text so that the entire secret message can be extracted from the stego text. In order to ensure that the data extraction procedure can be successfully executed, Alice and Bob should share the necessary side information in advance including the secret key, the threshold controlling the selection of candidate words, the masked LM, and the consistency coding strategy. Algorithm 1 shows the pseudocode of data embedding. For the receiver, a similar process is performed to reconstruct the secret bits. We provide the details below.


## 3.2. Masking Strategy and Masked Language Model (LM)

The goal of the masking strategy is to replace some words (tokens) in the cover text with the special token “[MASK].” These masked positions will be filled with tokens generated by the masked LM based on the context. The new tokens not only fit into the context but also carry secret information. It is free for us to design the masking strategy. Though the masking strategy can be carefully crafted, it is not the main contribution of this paper. For fair comparison in experiments, we follow the simple but effective masking strategy in [34]. In brief summary, a masking interval f (which is an integer and can be considered as a secret key) is used to control the total number of masked positions. A higher f means less tokens are masked, which reduces the embedding capacity but increases the difficulty of detection.Algorithm 1: Pseudocode for the data embedding procedure.
Input: cover text x, masked LM ℳ, secret bits b, parameters tp, f, and secret key.Output: stego text y.(1)
DeterminexS according to (1), f and secret key(2)
fori=1,2,…,ndo(3)
ifxi′ is [MASK] then(4)
Setxi=xS(5)
Determinepj for every vj∈V with ℳ and xi(6)
Determine a set of candidate wordsV′⊂V with tp(7)
Normalize probabilities of words inV′ by (3)(8)
Map words inV′ to bit streams by Huffman coding(9)
Determinexi∗ which matches a prefix of b(10)
UpdatexS by replacing xi′ with xi∗(11)
Remove the embedded prefix fromb(12)
end if(13)
end for(14)
Determiney according to xS(15)
returnyThe masked strategy has been previously introduced along with BERT [35] as an efficient pretraining strategy for transformer-based nets [37]. By using the masked strategy, BERT overcomes the limitations of unidirectional training and realizes the common dependence on context for prediction. Generally speaking, fine-tuning the downstream tasks with the pretrained BERT according to heir specific tasks such as sentence classification, sequence tagging, and question answering can greatly reduce the expense of designing their own architecture. However, fine-tuning is not required in this paper. Different from recurrent neural networks (RNNs) [38] processing texts by a word-wise manner and generative pretrained transformer (GPT) [39] obscuring future markers, the masked LM provides superior prediction performance due to its bidirectionality. Therefore, we use the pretrained BERT as the masked LM.Letℳ and ℐ=i1,i2,…,is be the masked LM and the index set for the masked positions determined with the secret key. For each i∈ℐ, the corresponding word xi in the cover text x=x1,x2,…,xn will be replaced with “[MASK].” To express it mathematically, for any S⊂ℐ, we define xS as(1)xS=x1′,x2′,…,xn′,where for 1≤i≤n, we have(2)xi′=MASK,ifi∈S,xi,otherwise.Obviously, we havex∅=x and xℐ is the text after replacing all the words in the masked positions with “[MASK].” To embed secret information in the masked positions, we orderly process the masked positions. For each masked position, ℳ will output a prediction probability for each word in the vocabulary according to the temporary text generated previously. A higher probability implies that the corresponding word is more suitable to replace the “[MASK]” to fit into the context. Next, we determine the temporary text for each masked position.Without the loss of generalization, leti1<i2<…<is. The temporary text for the masked index ij∈ℐ is determined as follows. First, the temporary text for i1∈ℐ, denoted by xi1, is xℐ. By feeding xi1 to ℳ, we can generate a prediction probability pi∈0,1 for each vi∈V=v1,v2,…,vz and ∑i=1zpi=1. According to the secret bits to be embedded and the proposed consistency coding technique, we select a word xi1∗∈V to replace the “[MASK]” in the i1-th position in x. For i2∈ℐ, the only one difference between xi1 and xi2 is that in the i1-th position, xi1 uses “[MASK]” but xi2 uses xi1∗. Generally, the only one difference between xij−1 and xij is that in the ij−1-th position, xij−1 uses “[MASK]” but xij uses xij−1∗ for any 2≤j≤s.Taking Figure1 for explanation, the cover text is “Midshire is a wonderful little city” and there are two masked positions. For the first masked position (from left to right), the temporary text to be fed into the masked LM is “Midshire is a [MASK] little [MASK].” After the first masked position was replaced with the stego word “nice,” the temporary text to be fed into the masked LM for the second masked position is “Midshire is a nice little [MASK].” It is seen that the temporary text for the present masked position depends on the generation of the previous temporary text. We deem it an autoregressive method.


## 3.3. Consistency Coding

A most important problem is how to build the mapping relationship between words and secret bits. To deal with this problem, the authors in [34] used a threshold 0≤tp≤1 to collect a list of candidate words whose prediction probabilities are higher than tp and then mapped each of the candidate words to a fixed-length binary stream. In other words, the collected candidate words have the same probability of being selected to fill in the present masked position, which does not take into account the probability distribution of the candidate words and may not fit into the context well.As mentioned previously, it is quite desirable to select such a word that it not only matches the secret bits to be embedded but also has a prediction probability obtained from the masked LM as high as possible. Because the secret bits are evenly distributed, the probability of selecting a specific word as the output actually depends on the number of secret bits carried by the word. In other words, for any two words mapped to a binary stream with the same length, they often have the same probability of matching the secret stream to be embedded. For example, in [34], if all candidate words are mapped to a binary stream with a length of l>0, the probability of choosing any one among 2l candidate words as the present output is 1/2l. It indicates that once the mapping relationship is finished, we are no longer free to choose the word. Therefore, it is necessary that the construction of mapping relationship itself has taken into account the prediction probability distribution so that for those words with a high prediction probability, the probability of choosing any one of them is high as well.In summary, we expect to find such an information encoding strategy that the probability of choosing a word as the present output is proportional to its prediction probability obtained by the masked LM. We treat such information encoding strategy asconsistency coding. Compared with [34], consistency coding has two significant advantages: the number of candidate words can be any integer, rather than a power of two which is required in [34], and it is based on the statistical probability distribution of each word in the vocabulary, which takes into account the frequency of words and makes coding more conducive to the regularization of the text. During each iteration, the model selects the word with the highest prediction probability while matching the secret information as the output to guarantee the quality of the sentence. Such a mechanism imitates the priority of native speakers in choosing words and improves the security accordingly.It is free for us to design the consistency coding. However, many classical entropy encoding methods in information theory can be used to realize consistency coding such as Huffman coding, arithmetic coding, and Shannon–Fano coding [40]. The main implementation steps of consistency coding are in lines 6–8 in Algorithm 1, and the specific process can be described as follows. For simplicity, we use Huffman coding to provide the off-the-shelf solution. Formally, for compactness, for each of the specific masked positions, let p1,p2,…,pz be the prediction probabilities for the words in the vocabulary V=v1,v2,…,vz, where p1≥p2≥…≥pz and ∑i=1zpi=1. We first collect all the words whose prediction probabilities are higher than a predetermined threshold tp. The collected words are then regarded as the candidate words to be encoded. Then, by normalizing the prediction probabilities of the candidate words, we further apply Huffman coding to map each candidate word to a stream. For example, let V′=v1,v2,…,vw⊂V,w≤z, include the candidate words, and the normalization operation is(3)pi←pi∑j=1wpj,∀1≤i≤w,which enables us to thereafter directly apply Huffman coding.Remark 1.
When the entire secret message cannot be carried by a single cover text, multiple cover texts can be used. When the entire secret message can be carried by a single cover text, some masked positions in the cover text will be embedded with secret bits while the other masked positions (if any) are filled with the original words without data embedding.Remark 2.
It is known that Huffman coding does not result in the unique mapping. For example, in Figure1, the two words “city” and “town” are mapped to “0” and “1,” respectively. It is also possible that “city” is mapped to “1” and the other is mapped to “0.” As long as the data hider and the data receiver use the same secret key to control the Huffman coding process, they can obtain the same mapping result.


## 4. Experimental Results and Analysis

In the experiments, we use the large-scale CC-100 dataset [42] consisting of high-quality data obtained through web crawling and containing monolingual data in more than 100 languages. A total of 10,000 texts are randomly selected from the English part of the CC-100 dataset, which guarantees the randomness and universality and therefore better proves the reliability of the proposed method. One thing to note is that the length of each cover text is required to be larger than 20 so that a sufficient payload can be embedded. The cover texts are embedded with a randomly generated binary stream to generate the stego texts. As mentioned in Section 3, we use BERT as the masked LM. For comparison, following [34], we use the Google’s BERTBase,Cased model and Hugging Face’s transformers package [43] with the default settings. In other words, our setup closely matches [34] for fair comparison. In addition, we use the commonly used indicator perplexity (PPL) [29] to evaluate the text quality and measure the security by steganalysis. In general, a lower PPL indicates that the quality of the text is better.


### 4.1. Qualitative Results

We first show some examples to evaluate the quality of the stego texts generated by the proposed method. Table1 shows the experimental examples, in which we compare the proposed method with [34] due to the close correlation mentioned in Section 2. The threshold tp defined in Sections 2 and 3 is empirically set to 0.02 by default. The embedding parameter f controls the number of the masked positions. The higher f is, the lower the number of the masked positions is. The payload is measured by bits per word (BPW). Since the masking strategy always skips punctuation, the punctuation marks never carry data. For simplicity, the punctuation marks are excluded to determine the total number of words in a text, e.g., in Table 1, when f=2, the payload of the stego text generated by [34] is 13/31 = 0.42, rather than 13/32 = 0.41. In Table 1, “Wa,b” indicates that the word “W” is mapped to a binary stream b and the number of the candidate words for the present masked position is a during data embedding. For example, the term “ways (2,1)” means that the word “ways” carries only one secret bit “1” and there are 2 candidate words for the present position. The term “exploring (0, −)” means that there are no candidate words for choice and the original word “exploring” is used for the present position.Table 1
Example comparison between different linguistic steganographic methods due to different parameters.
fTypeTextPPLPayload2CoverGerson and Walter contend there are multiple paths to solving a mathematical problem and students should be encouraged to chart their own way by exploring problems drawn from the real world——[34]Gerson and Walter contend there are multiple ways (2, 1) to solving a particular (8, 001) problem and students (4, 10) should be encouraged to go (4, 00) their own way by exploring (0, −) problems different (8, 001) from the real one (4, 10)63.52610.42ProposedGerson and Walter contend there are multiple approaches (3, 1) to solving a given (7, 001) problem and users (7,100) should be encouraged to find (4,000) their own way by solving (3, 1) problems different (4, 1) from the real one (5, 000)56.45650.483CoverAn unlucky accident befell my servant, Stevens, in falling from the coach and being dragged by the foot upon the pavement. Edition: Current; page: [186] He was in great danger, but happily was not essentially hurt--[34]An unlucky accident befell my friend (4, 00), Stevens, in falling from the roof (8, 101) and being dragged by the foot onto (2, 1) the pavement. Edition: Current; page: [186] He was in great shape (4, 01), but happily was not essentially happy (4, 11)105.59820.27ProposedAn unlucky accident befell my nephew (7, 0010), Stevens, in falling from the window (9, 1101) and being dragged by the foot across (3, 11) the pavement. Edition: Current; page: [186] He was in great debt (8, 000), but happily was not essentially rich (7, 1000)96.98070.464CoverAccountability is a major key to BILSTEIN’s success. We hold our team members, line employees, frontline leaders, managers, and company leaders responsible for thinking above the line in addressing both success and failure, taking direct responsibility for situations through personal steps to solve and overcome issues and not becoming a victim for individual or collective results--[34]Accountability is a major key (4, 01) to BILSTEIN’s success. We hold our team leaders (2, 1), line employees, frontline leaders, managers (4,00), and company leaders responsible for working (8, 010) above the line in addressing both success and loss (4, 01), taking direct responsibility for situations (0, −) through personal steps to solve and address (4, 10) issues and not becoming a victim for individual (2, 0) or collective results64.45650.23ProposedAccountability is a major key (5, 011) to BILSTEIN’s success. We hold our team managers (3, 00), line employees, frontline leaders, executives (5, 010), and company leaders responsible for staying (11, 01) above the line in addressing both success and failure (6, 1), taking direct responsibility for moving (6, 0010) through personal steps to solve and resolve (5, 0) issues and not becoming a victim for mistakes (3, 01) or collective results57.08320.32It can be inferred from Table1 that compared with [34], the proposed method not only improves the semantic quality but also increases the payload for all the three cases, which have verified the effectiveness of the proposed method. This can be explained from two aspects. On the one hand, compared with the non-autoregressive prediction strategy in [34], the proposed autoregressive prediction strategy enhances the semantic connection between contexts so that those words better fitting into contexts can be collected for data embedding. On the other hand, compared with block coding [34], the proposed consistency coding strategy is based on the prediction probability distribution of each word in the vocabulary, which enlarges the candidate set and makes the choice of stego words more conducive to maintaining the text quality.


### 4.2. Quantitative Results

To evaluate the performance against steganalysis, following [34], for each experiment, we first split the above-mentioned 10,000 natural texts and their stego versions to three disjoint subsets, i.e., training set (60%), validation set (10%), and testing set (30%), and then fine-tune the BERTbase−cased model with the training set and the validation set. The fine-tuned model is used to detect whether a text to be tested is stego or not. The accuracy is evaluated on the testing set with the model of the highest validation accuracy. We compare the proposed work with four state-of-the-art methods, i.e., [27, 29, 33, 34]. Table 2 shows the experimental results. In Table 2, the parametric settings are described as follows. Reference [34] and our work use the same f=3 and tp=0.02 for fair comparison. The explanation for f and tp has been described in Section 4.1. τ is a system parameter introduced in [33] and we use the recommended value in [33], i.e., τ=0.7. The authors in [29] propose two methods called fixed-length coding and variable-length coding for information encoding. We use the variable-length coding strategy for simulation since it has the better performance. For [27, 29], the truncation length L and the block size B are set to 22 for fair comparison as well. As shown in Table 2, different methods have different performances. In terms of payload, the methods in [27, 29, 33] outperform the method in [34] and the proposed method. The reason is that the three methods are actually generation based while the method in [34] and the proposed method are actually modification based. Generation-based methods often provide a high payload since each word can be used to carry secret information. However, in Table 2, the detection accuracies of the three generation-based methods are high, meaning that the security is not satisfied. By comparing with [34], we can find that the proposed work not only achieves a higher payload but also has a lower detection accuracy, indicating that the proposed work achieves a better trade-off between payload and security.Table 2
Detection accuracy for detecting different steganographic methods with different payloads.
MethodParametersMean payloadAccuracyReference [34]f=3,tp=0.020.24790.5570Reference [33]τ=0.72.29340.8967Reference [29]L=221.80720.9580Reference [27]B=222.00000.9615Proposedf=3,tp=0.020.25060.5498To further demonstrate the superiority of the proposed work, we visualize the statistical distribution of the stego texts and the natural (cover) texts by applying t-SNE [41]. It is observed from Figure 2 that when the payload size decreases, more points of the two colors overlap, indicating that the stego texts are more approximate to the natural ones and therefore indeed have higher security. For the different parameters given in Figure 2, we also show the corresponding mean payload and detection accuracy for [34] and the proposed work. The experimental results are shown in Tables 3 and 4. We use tp=0.02 for Table 3 and tp=0.03 for Table 4. It can be inferred from Tables 3 and 4 that the proposed work outperforms [34] in terms of payload and detection accuracy under different parameter settings, which has verified the superiority of the proposed work. We also determine the mean PPL of stego texts due to different parameters. The experimental results are plotted in Figure 3. It can be inferred that the proposed work also generates stego texts with better quality than [34], indicating that it is more difficult to distinguish between stego texts generated by the proposed work and natural texts compared with [34], which reveals that our work has good concealment.Figure 2
Visualization for stego texts and natural ones by applying t-SNE [41]. The blue dots represent natural texts and the others are stego texts.Table 3
Detection accuracy for detecting different steganographic methods with different masking intervalstp=0.02.
fReference [34]ProposedMean payloadAccuracyMean payloadAccuracy20.38180.61070.39060.604530.24790.55700.25060.549840.19030.54030.19370.5337Table 4
Detection accuracy for detecting different steganographic methods with different masking intervalstp=0.03.
fReference [34]ProposedMean payloadAccuracyMean payloadAccuracy20.29160.60500.31450.596830.19190.55320.20570.546540.14740.54170.15780.5407Figure 3
The mean PPLs for the stego texts generated by different methods. (a)tp = 0.02. (b) tp = 0.03.
(a)(b)


### 4.3. Ablation Study

The proposed work achieves superior performance by using an autoregressive strategy and a consistency coding technique. In order to show that both the autoregressive strategy and the consistency coding technique have the ability to improve the security or the payload, we further conduct ablation study for evaluation. The method in [34] uses the non-autoregressive strategy and block coding technique for steganography. By substituting non-autoregressive with autoregressive, we can develop a new steganographic method named as “autoregressive only.” Similarly, by substituting block coding with consistency coding, a new steganographic method named as “consistency coding only” can be simulated. Table 5 shows the experimental results. In terms of payload, “consistency coding only” and the proposed work are close to each other, but both are higher than [34] and “autoregressive only.” It indicates that the proposed work can benefit from the consistency coding technique so as to increase the payload. In terms of steganalysis accuracy, “autoregressive only” and the proposed work have similar performance, but both outperform [34] and “consistency coding only.” It means that the autoregressive strategy is more secure than the non-autoregressive strategy. By combining the autoregressive strategy and the consistency coding technique, the PPL value of the proposed work is the lowest in most cases. In summary, it can be concluded that the proposed work has achieved the best performance compared with related works.Table 5
Performance comparison between different steganographic strategies (the payload and PPL are using mean values).
ftpReference [34]Autoregressive onlyConsistency coding onlyProposedPayloadPPLAccuracyPayloadPPLAccuracyPayloadPPLAccuracyPayloadPPLAccuracy20.020.381886.05370.61070.381884.83110.60480.391683.18670.61230.390682.06090.604530.247981.77080.55700.247381.52830.54870.253279.76530.56120.250679.40730.549840.190380.62520.54030.189780.38750.53980.193878.92040.54230.193778.90920.533720.030.291682.74070.60500.293481.71110.60280.313481.39150.60450.314580.42900.596830.191979.48690.55320.192379.31320.55120.205878.54380.55200.205778.35080.546540.147478.83770.54170.147278.79710.54100.157878.22370.54330.157878.24370.5407In addition, though the proposed work uses BERT as the masked LM, it is actually free for us to set the masked LM. We replace BERT with RoBERTa [44] to further demonstrate the applicability of the proposed work. The experimental results due to different parameters are provided in Tables 6 and 7. In Tables 6 and 7, “RoBERTa” means to replace BERT used in [34] with RoBERTa, and “RoBERTa+Proposed” means to replace BERT used in this paper with RoBERTa. We use tp=0.02 for Table 6 and tp=0.03 for Table 7. It is inferred from Tables 6 and 7 that the proposed strategy does enhance the steganographic performance, which verifies the applicability and superiority.Table 6
Detection accuracy for detecting different steganographic methods with different masking intervals using RoBERTa as the masked LMtp=0.02.
fRoBERTaRoBERTa + proposedMean payloadPPLMean payloadPPL20.764994.23670.796674.034130.445472.64910.449867.449340.323569.22590.323965.9934Table 7
Detection accuracy for detecting different steganographic methods with different masking intervals using RoBERTa as the masked LMtp=0.03.
fRoBERTaRoBERTa + proposedMean payloadPPLMean payloadPPL20.635985.76420.649872.049930.365868.76360.380865.621240.264266.60710.276764.7378


## 4.1. Qualitative Results

We first show some examples to evaluate the quality of the stego texts generated by the proposed method. Table1 shows the experimental examples, in which we compare the proposed method with [34] due to the close correlation mentioned in Section 2. The threshold tp defined in Sections 2 and 3 is empirically set to 0.02 by default. The embedding parameter f controls the number of the masked positions. The higher f is, the lower the number of the masked positions is. The payload is measured by bits per word (BPW). Since the masking strategy always skips punctuation, the punctuation marks never carry data. For simplicity, the punctuation marks are excluded to determine the total number of words in a text, e.g., in Table 1, when f=2, the payload of the stego text generated by [34] is 13/31 = 0.42, rather than 13/32 = 0.41. In Table 1, “Wa,b” indicates that the word “W” is mapped to a binary stream b and the number of the candidate words for the present masked position is a during data embedding. For example, the term “ways (2,1)” means that the word “ways” carries only one secret bit “1” and there are 2 candidate words for the present position. The term “exploring (0, −)” means that there are no candidate words for choice and the original word “exploring” is used for the present position.Table 1
Example comparison between different linguistic steganographic methods due to different parameters.
fTypeTextPPLPayload2CoverGerson and Walter contend there are multiple paths to solving a mathematical problem and students should be encouraged to chart their own way by exploring problems drawn from the real world——[34]Gerson and Walter contend there are multiple ways (2, 1) to solving a particular (8, 001) problem and students (4, 10) should be encouraged to go (4, 00) their own way by exploring (0, −) problems different (8, 001) from the real one (4, 10)63.52610.42ProposedGerson and Walter contend there are multiple approaches (3, 1) to solving a given (7, 001) problem and users (7,100) should be encouraged to find (4,000) their own way by solving (3, 1) problems different (4, 1) from the real one (5, 000)56.45650.483CoverAn unlucky accident befell my servant, Stevens, in falling from the coach and being dragged by the foot upon the pavement. Edition: Current; page: [186] He was in great danger, but happily was not essentially hurt--[34]An unlucky accident befell my friend (4, 00), Stevens, in falling from the roof (8, 101) and being dragged by the foot onto (2, 1) the pavement. Edition: Current; page: [186] He was in great shape (4, 01), but happily was not essentially happy (4, 11)105.59820.27ProposedAn unlucky accident befell my nephew (7, 0010), Stevens, in falling from the window (9, 1101) and being dragged by the foot across (3, 11) the pavement. Edition: Current; page: [186] He was in great debt (8, 000), but happily was not essentially rich (7, 1000)96.98070.464CoverAccountability is a major key to BILSTEIN’s success. We hold our team members, line employees, frontline leaders, managers, and company leaders responsible for thinking above the line in addressing both success and failure, taking direct responsibility for situations through personal steps to solve and overcome issues and not becoming a victim for individual or collective results--[34]Accountability is a major key (4, 01) to BILSTEIN’s success. We hold our team leaders (2, 1), line employees, frontline leaders, managers (4,00), and company leaders responsible for working (8, 010) above the line in addressing both success and loss (4, 01), taking direct responsibility for situations (0, −) through personal steps to solve and address (4, 10) issues and not becoming a victim for individual (2, 0) or collective results64.45650.23ProposedAccountability is a major key (5, 011) to BILSTEIN’s success. We hold our team managers (3, 00), line employees, frontline leaders, executives (5, 010), and company leaders responsible for staying (11, 01) above the line in addressing both success and failure (6, 1), taking direct responsibility for moving (6, 0010) through personal steps to solve and resolve (5, 0) issues and not becoming a victim for mistakes (3, 01) or collective results57.08320.32It can be inferred from Table1 that compared with [34], the proposed method not only improves the semantic quality but also increases the payload for all the three cases, which have verified the effectiveness of the proposed method. This can be explained from two aspects. On the one hand, compared with the non-autoregressive prediction strategy in [34], the proposed autoregressive prediction strategy enhances the semantic connection between contexts so that those words better fitting into contexts can be collected for data embedding. On the other hand, compared with block coding [34], the proposed consistency coding strategy is based on the prediction probability distribution of each word in the vocabulary, which enlarges the candidate set and makes the choice of stego words more conducive to maintaining the text quality.


## 4.2. Quantitative Results

To evaluate the performance against steganalysis, following [34], for each experiment, we first split the above-mentioned 10,000 natural texts and their stego versions to three disjoint subsets, i.e., training set (60%), validation set (10%), and testing set (30%), and then fine-tune the BERTbase−cased model with the training set and the validation set. The fine-tuned model is used to detect whether a text to be tested is stego or not. The accuracy is evaluated on the testing set with the model of the highest validation accuracy. We compare the proposed work with four state-of-the-art methods, i.e., [27, 29, 33, 34]. Table 2 shows the experimental results. In Table 2, the parametric settings are described as follows. Reference [34] and our work use the same f=3 and tp=0.02 for fair comparison. The explanation for f and tp has been described in Section 4.1. τ is a system parameter introduced in [33] and we use the recommended value in [33], i.e., τ=0.7. The authors in [29] propose two methods called fixed-length coding and variable-length coding for information encoding. We use the variable-length coding strategy for simulation since it has the better performance. For [27, 29], the truncation length L and the block size B are set to 22 for fair comparison as well. As shown in Table 2, different methods have different performances. In terms of payload, the methods in [27, 29, 33] outperform the method in [34] and the proposed method. The reason is that the three methods are actually generation based while the method in [34] and the proposed method are actually modification based. Generation-based methods often provide a high payload since each word can be used to carry secret information. However, in Table 2, the detection accuracies of the three generation-based methods are high, meaning that the security is not satisfied. By comparing with [34], we can find that the proposed work not only achieves a higher payload but also has a lower detection accuracy, indicating that the proposed work achieves a better trade-off between payload and security.Table 2
Detection accuracy for detecting different steganographic methods with different payloads.
MethodParametersMean payloadAccuracyReference [34]f=3,tp=0.020.24790.5570Reference [33]τ=0.72.29340.8967Reference [29]L=221.80720.9580Reference [27]B=222.00000.9615Proposedf=3,tp=0.020.25060.5498To further demonstrate the superiority of the proposed work, we visualize the statistical distribution of the stego texts and the natural (cover) texts by applying t-SNE [41]. It is observed from Figure 2 that when the payload size decreases, more points of the two colors overlap, indicating that the stego texts are more approximate to the natural ones and therefore indeed have higher security. For the different parameters given in Figure 2, we also show the corresponding mean payload and detection accuracy for [34] and the proposed work. The experimental results are shown in Tables 3 and 4. We use tp=0.02 for Table 3 and tp=0.03 for Table 4. It can be inferred from Tables 3 and 4 that the proposed work outperforms [34] in terms of payload and detection accuracy under different parameter settings, which has verified the superiority of the proposed work. We also determine the mean PPL of stego texts due to different parameters. The experimental results are plotted in Figure 3. It can be inferred that the proposed work also generates stego texts with better quality than [34], indicating that it is more difficult to distinguish between stego texts generated by the proposed work and natural texts compared with [34], which reveals that our work has good concealment.Figure 2
Visualization for stego texts and natural ones by applying t-SNE [41]. The blue dots represent natural texts and the others are stego texts.Table 3
Detection accuracy for detecting different steganographic methods with different masking intervalstp=0.02.
fReference [34]ProposedMean payloadAccuracyMean payloadAccuracy20.38180.61070.39060.604530.24790.55700.25060.549840.19030.54030.19370.5337Table 4
Detection accuracy for detecting different steganographic methods with different masking intervalstp=0.03.
fReference [34]ProposedMean payloadAccuracyMean payloadAccuracy20.29160.60500.31450.596830.19190.55320.20570.546540.14740.54170.15780.5407Figure 3
The mean PPLs for the stego texts generated by different methods. (a)tp = 0.02. (b) tp = 0.03.
(a)(b)


## 4.3. Ablation Study

The proposed work achieves superior performance by using an autoregressive strategy and a consistency coding technique. In order to show that both the autoregressive strategy and the consistency coding technique have the ability to improve the security or the payload, we further conduct ablation study for evaluation. The method in [34] uses the non-autoregressive strategy and block coding technique for steganography. By substituting non-autoregressive with autoregressive, we can develop a new steganographic method named as “autoregressive only.” Similarly, by substituting block coding with consistency coding, a new steganographic method named as “consistency coding only” can be simulated. Table 5 shows the experimental results. In terms of payload, “consistency coding only” and the proposed work are close to each other, but both are higher than [34] and “autoregressive only.” It indicates that the proposed work can benefit from the consistency coding technique so as to increase the payload. In terms of steganalysis accuracy, “autoregressive only” and the proposed work have similar performance, but both outperform [34] and “consistency coding only.” It means that the autoregressive strategy is more secure than the non-autoregressive strategy. By combining the autoregressive strategy and the consistency coding technique, the PPL value of the proposed work is the lowest in most cases. In summary, it can be concluded that the proposed work has achieved the best performance compared with related works.Table 5
Performance comparison between different steganographic strategies (the payload and PPL are using mean values).
ftpReference [34]Autoregressive onlyConsistency coding onlyProposedPayloadPPLAccuracyPayloadPPLAccuracyPayloadPPLAccuracyPayloadPPLAccuracy20.020.381886.05370.61070.381884.83110.60480.391683.18670.61230.390682.06090.604530.247981.77080.55700.247381.52830.54870.253279.76530.56120.250679.40730.549840.190380.62520.54030.189780.38750.53980.193878.92040.54230.193778.90920.533720.030.291682.74070.60500.293481.71110.60280.313481.39150.60450.314580.42900.596830.191979.48690.55320.192379.31320.55120.205878.54380.55200.205778.35080.546540.147478.83770.54170.147278.79710.54100.157878.22370.54330.157878.24370.5407In addition, though the proposed work uses BERT as the masked LM, it is actually free for us to set the masked LM. We replace BERT with RoBERTa [44] to further demonstrate the applicability of the proposed work. The experimental results due to different parameters are provided in Tables 6 and 7. In Tables 6 and 7, “RoBERTa” means to replace BERT used in [34] with RoBERTa, and “RoBERTa+Proposed” means to replace BERT used in this paper with RoBERTa. We use tp=0.02 for Table 6 and tp=0.03 for Table 7. It is inferred from Tables 6 and 7 that the proposed strategy does enhance the steganographic performance, which verifies the applicability and superiority.Table 6
Detection accuracy for detecting different steganographic methods with different masking intervals using RoBERTa as the masked LMtp=0.02.
fRoBERTaRoBERTa + proposedMean payloadPPLMean payloadPPL20.764994.23670.796674.034130.445472.64910.449867.449340.323569.22590.323965.9934Table 7
Detection accuracy for detecting different steganographic methods with different masking intervals using RoBERTa as the masked LMtp=0.03.
fRoBERTaRoBERTa + proposedMean payloadPPLMean payloadPPL20.635985.76420.649872.049930.365868.76360.380865.621240.264266.60710.276764.7378


## 5. Conclusion and Discussion

In this paper, we propose a novel autoregressive linguistic steganographic algorithm based on the BERT model equipped with consistency coding. By applying the autoregressive word-prediction strategy, the generated stego words better fit into the context, which improves the readability and the authenticity of the generated stego text. Meanwhile, the consistency coding technique enlarges the number of candidate words for LS and makes use of the statistical probability distribution of candidate words, which not only enhances the payload but also makes the stego text seemingly more natural. Experiments show that the proposed work has achieved the best performance compared with related works, verifying the superiority and applicability.From a practical perspective, with the widespread use of text over social networks, LS will become more and more popular. By conveying a stego text through a social network platform, the steganographic behavior can be easily concealed by the huge number of ordinary social activities. More importantly, once the receiver observes the stego text, he can keep silent and extract the secrets without making any suspicious interaction, which conceals the real identify of the receiver. Though there are increasing LS methods that are reported in the literature in recent years, how to achieve a good trade-off between payload, text quality, and security is still a challenging problem. On the one hand, though modification-based LS well maintains the semantic quality of the text, the embeddable payload is low. On the other hand, though generation-based LS allows us to embed a high payload, we need to pay the high price of uncontrollable semantic information and security. In this paper, by exploiting a language model for word prediction and making use of the probability distribution for information encoding, the proposed work improves the fluency of the stego text while guaranteeing security compared with modification-based methods and improves the security compared with generation-based methods. The payload is increased as well to a certain extent, which brings modification-based LS closer to generation-based LS. In the future, we will explore efficient strategies to increase the embeddable payload to build a bridge between modification-based LS and generation-based LS.



---

*Source: 9092785-2022-05-20.xml*