SlowGuess/NAACL_2021_OCR huggingface dataset:
Jiaming Shen, Heng Ji, and Jiawei Han. 2020. Near-imperceptible neural linguistic steganography via self-adjusting arithmetic coding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 303-313, Online. Association for Computational Linguistics.
Gustavus J Simmons. 1984. The prisoners' problem and the subliminal channel. In Advances in Cryptology, pages 51-67. Springer.
Cuneyt M. Taskiran, Umut Topkara, Mercan Topkara, and Edward J. Delp. 2006. Attacks on lexical natural language steganography systems. In Security, Steganography, and Watermarking of Multimedia Contents VIII, volume 6072, pages 97-105. International Society for Optics and Photonics, SPIE.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998-6008. Curran Associates, Inc.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv:2006.04768.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Association.
Alex Wilson and Andrew D. Ker. 2016. Avoiding detection on twitter: embedding strategies for linguistic steganography. *Electronic Imaging*, 2016(8):1-9.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Zhong-Liang Yang, Xiao-Qing Guo, Zi-Ming Chen, Yong-Feng Huang, and Yu-Jin Zhang. 2019. RNN-Stega: Linguistic steganography based on recurrent neural networks. IEEE Transactions on Information Forensics and Security, 14(5):1280-1295.
Zachary Ziegler, Yuntian Deng, and Alexander Rush. 2019. Neural linguistic steganography. In Proceedings of the 2019 Conference on Empirical Methods
in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1210-1215, Hong Kong, China. Association for Computational Linguistics.
A Details of Automatic Detection
The edit-based and generation-based methods differ considerably in the amount of text needed to encode a secret message because the latter has a higher payload capacity by design. For a fair comparison, we chose to encode bit sequences of equal size, meaning that the proposed method yielded longer stego texts. Specifically, we used random 32-bit sequences. The sequence of sentences was given to the discriminator at once. For GPT-2, we discarded the contexts and only fed stego texts into the discriminator.
For the binary classification task, we prepared the training, development and test sets with 8,000, 1,000, and 1,000 paragraphs, respectively. Each set had the same amount of stego and real texts, and they were comparable with respect to length. The discriminators were trained for 20 epochs, with the batch size of 32. We applied early stopping if the validation loss did not drop five times in a row. The model snapshots were saved for every 250 steps, and the one with the lowest validation loss was chosen. We used Adam (Kingma and Ba (2017)) as the optimizer with the learning late of  $10^{-6}$ .
For the proposed edit-based method, we used the same masking and encoding strategies used for automatic detection. The hyperparameters were as follows:  $p = 0.02$ , and  $f = 4$ , stopwords and subwords were skipped. We used random 6-bit sequences as secret messages.
For the generation-based method, the temperature parameter  $\tau$  was set to 0.7. We used random 45-bit sequences as secret messages.
We designed a MTurk HIT (human intelligence task) following Abdelnabi and Fritz (2020). Workers were asked to rate texts (each question had 5 texts) with a Likert scale from 1 (lowest) to 5 (highest). As shown in Table 4, the ratings were described with the instructions ranging from "This sentence is completely understandable, natural, and grammatically correct" to "This sentence is completely not understandable, unnatural, and you cannot get its main idea". Each HIT had 5 texts, with stego texts from both methods and real texts
Rating	Description
5	The text is understandable, natural, and grammatically and structurally correct.
4	The text is understandable, but it contains minor mistakes.
3	The text is generally understandable, but some parts are ambiguous.
2	The text is mainly not understandable, but you can get the main ideas.
1	The text is completely not understandable, unnatural, and you cannot get the main ideas.
Table 4: Ratings explanations given in the human evaluation.
of comparable length appearing in a random order. The questions also had a simple attention check and if the answer to the attention check was wrong, the corresponding HIT was discarded. We have set the reward per assignment at $0.3.
We observed that human raters strongly favored shorter texts. To verify the observation, we performed linear regression analysis with the number of words as a parameter. We found that it indeed had a statistically significant negative impact on the ratings with  $p < 10^{-3}$  for the t-statistic.