SlowGuess/ACL_2020_OCR huggingface dataset:
Tom is a cat and Jerry is a mouse. “It’s very sad!”. The writers had wanted Tom to have “something big to tell it . . . and a fun place to get excited”. The writers believed that the “little animal” and the “little black dog” at the end of the episode would have attracted more attention from viewers, but it never took place. Tom’s first television role was that of the boy scout “Mr. Krabs” in the 1978 NBC Western comedy pilot, The Search for Mr. Krabs.
Figure 3: Unidirectional Text Generation with GPT
The results are shown in Table 4. The results show that u-PMLM costs roughly  $20.1\%$  more time than GPT for generating sentences, which is much less than the theoretical time complexity difference.
Comparison With GPT for Generation In the introduction section, we have shown an example showing the application of arbitrarily ordered text generation, where the tokens in the input sentences are distributed across the generated sentences. Indeed, the major difference with GPT is that the input text could be inserted anywhere in the generated text, which makes the generation process more controllable. Meanwhile, the output text contains certain predefined tokens.
Figure 3 and Figure 4 shows the generated paragraphs of GPT and u-PMLM, respectively. For GPT, the input text can only be placed in the beginning and the generation process become uncontrollable, resulting in generating sentences with topic drift. In contrast, u-PMLM allows manually placing anchor sentences in the middle or end of the generated text to guide the topic of the generated text. As shown in Figure 4, we place "Tom is a cat and Jerry is a mouse." and "Tom and Jerry become good friends in the end." at the beginning and end of the paragraph. The middle parts are generated by u-PMLM from left-to-right. Such generation method allows us to better retain the topic of the generated content.
Tom is a cat and Jerry is a mouse . However , the two have a common . The first part is a joke about Jerry and Tom fighting in the middle of the episode . The two get on the run from the restaurant , and Tom ’ s mother is shocked that they would have to do so . After a few minutes , Jerry arrives and decides to have a fight . The two go to the casino , where Jerry tries to fight them back by using a splint of grease and a bucket of wine in the bar . They reunite at a restaurant dance , and Tom and Jerry become good friends in the end .
Figure 4: Bidirectional Text Generation with u-PMLM
4.3 Natural Language Understanding
Besides evaluating the ability of u-PMLM for natural language generation, we also evaluate its performance on natural language understanding. Two widely adopted tasks, GLUE (Wang et al., 2018) and SQUAD 2.0 (Rajpurkar et al., 2018), are employed for evaluating u-PMLM. We have ensured that the evaluation for u-PMLM is influenced by as less model-irrelevant factors as possibles. For example, we do not tune the hyper-parameters and just follow the settings of BERT, including warming-up steps, learning rate, etc. In addition, since BERT employs absolute positional embeddings, the variant with absolute positional embeddings, u-PMLM-A, is intentionally trained for fairly evaluating the probabilistic masking scheme.
The results are shown in Table 5 and Table 6. u-PMLM-A general performs better than BERT, demonstrating that the probabilistic masking scheme is more effective than the fixed masking scheme. The reason could be that the probabilistic masking scheme covers more a wider range of masking patterns, which benefits pretraining for a masked language model. Moreover, u-PMLM-R performs better than u-PMLM-A consistently. The only difference between these two models is the way to handle positional embedding. Relative positional embedding emphasizes more on the relative positions between two tokens, which could be a better option to capture contextual representation. Recall that relative and absolute positional embedding do not make many differences regarding generation ability if the dataset is proper. Hence we conclude u-PMLM-R is a better model than u-PMLM-A considering both NLU and NLG tasks.
Model	COLA	SST2	MRPC	STSB	QQP	MNLI-m/mm	QNLI	RTE	AVG.
BERT(A)	52.1	93.5	88.9/84.8	87.1/85.8	71.2/89.2	84.6/83.4	90.5	66.4	78.3
u-PMLM-A	56.5	94.3	88.8/84.4	87.0/85.9	71.4/89.2	84.5/83.5	91.8	66.1	79.0
u-PMLM-R	58.0	94.0	89.7/85.8	87.7/86.8	71.2/89.2	85.0/84.1	92.3	69.8	80.0
u-PMLM-R*	56.9	94.2	90.7/87.7	89.7/89.1	72.2/89.4	86.1/85.4	92.1	78.5	81.3
Table 5: Evaluation on GLUE test set.
Model	F1	EM
BERT(A)	76.85	73.97
u-PMLM-A	78.31	74.62
u-PMLM-R	81.52	78.46
Table 6: Evaluation on SQUAD 2.0.
Model	SQUAD 2.0
F1/EM	MNLI
m/mm	SST2
XLNet (R)	81.33/78.46	85.84/85.43	92.66
u-PMLM-R	81.52/78.46	85.99/85.60	93.58
Table 7: Comparison with XLNet.
In addition, u-PMLM-R*, finetuned with a commonly used technique by sharing data from multiple tasks, is the state-of-the-art base model (with 110M parameters) trained on the BookCorpus and Wikipedia datasets on GLUE leaderboard on the date of paper submission.
Comparison with XLNet We also compare our proposed model with XLNet-base, which adopts relative positional embedding. As will be discussed in Section 5, XLNet is the most relevant model to u-PMLM. We are not able to train an XLNet using the same settings except that we make sure both u-PMLM-R and XLNet-base are of the same model size and are both trained using the same datasets. The comparison results shown in Table 7 demonstrate that the performance of our proposed u-PMLM-R is comparable to XLNet.
5.1 Non-traditional Text Generation
Conventionally, text is commonly generated autoregressively in the left-to-right direction. Recently, some research works have proposed several models for non-autoregressive text generation (Welleck et al., 2019; Gu et al., 2019). Stern et al. (2019) proposes insertion Transformer, where text are generated in an iterative and partially autoregressive manner based on insertion operations. Ma et al. (2019) design a latent variable based method to generate all the tokens in one pass. Ghazvinine
jad et al. (2019) and Wang and Cho (2019) employ masked language model for refinement-based non-autoregressive text generation, when a subset of tokens in a sequence are refined iteratively. Later, Mansimov et al. (2019) propose a generalized framework of sequence generation accommodating autoregressive, semi-autoregressive, and refinement-based non-autoregressive model. Strictly speaking, our proposed arbitrarily ordered autoregressive text generation is a special case of this generalized framework. We are the first work to address such kind of text generation, which enables a lot of new applications over tradition text generation.
UNILM (Dong et al., 2019) and MASS (Song et al., 2019) are another two works that combine masked language model and autoregressive language model. However, UNILM only combines the training objective of GPT and BERT. MASS employs mask mechanism to train sequence to sequence language model. Both models do not address arbitrarily ordered text generation.
5.2 XLNet
XLNet (Yang et al., 2019) is the most relevant pretrained language model to u-PMLM. Both of them can be treated as an autoregressive permutated language model. However, XLNet is trained by permutating only a small fraction of the sequences, which does not fully address the generation problem. Though, we suppose that the training method for XLNet is feasible to train a model for arbitrarily ordered text generation as well. The main difference between these two models is that XLNet employs unidirectional Transformer, while u-PMLM is based on bidirectional Transformer. Regarding the training algorithm, XLNet shuffles the attention matrix and introduce two-stream self-attention, which is a bit complex and memory consuming. On the other hand, PMLM takes the simple training objective of masked language model and approximates permutated language model.
<sup>1</sup>http://gluebenchmark.com/leaderboard/
We have proposed a probabilistically masked language model for autoregressive generation in arbitrary word order. The experiments show that the text generated in arbitrary order has comparable quality with GPT. Besides, the proposed probabilistic masking scheme also improves the NLU capability of a masked language model.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems 32, pages 13042-13054.
Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. 2015. Made: Masked autoencoder for distribution estimation. In Proceedings of the 32nd International Conference on Machine Learning, pages 881-889.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6114-6123.
Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019. Levenshtein transformer. In Advances in Neural Information Processing Systems, pages 11179-11189.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. 2019. FlowSeq: Non-autoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4281-4291.
Elman Mansimov, Alex Wang, and Kyunghyun Cho. 2019. A generalized framework of sequence generation with application to undirected sequence models. arXiv preprint arXiv:1905.12790.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2.amazon.com/openai-assetss/researchcovers/languageunsupervised/language understanding paper.pdf.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 464-468.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, pages 5926-5936.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. Insertion transformer: Flexible sequence generation via insertion operations. In Proceedings of the 36th International Conference on Machine Learning, pages 5976-5985.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223.
Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30-36.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Liwei Peng, and Luo Si. 2019. Structbert: Incorporating language structures into pre-training for deep language understanding. arXiv preprint arXiv:1908.04577.
Sean Welleck, Kianté Brantley, Hal Daumé III, and Kyunghyun Cho. 2019. Non-monotonic sequential text generation. In Proceedings of the 36th International Conference on Machine Learning.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32, pages 5754-5764.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2019. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations.
A Proof of Equivalence
We prove that PMLM with a continuous uniform distribution on the masking ratio, namely u-PMLM, is equivalent to an autoregressive permuted language model.
When  $p(r)$  is a continuous uniform distribution,
the probability  $p(M)$  is analytical, denoted as:
$$
\begin{array}{l} p (M) = \int r ^ {K} (1 - r) ^ {N - K} p (r) d (r) \\ = \int_ {0} ^ {1} r ^ {K} (1 - r) ^ {N - K} d (r) \\ = B (N - K + 1, K + 1) \tag {9} \\ = \frac {\Gamma (N - K + 1) \Gamma (K + 1)}{\Gamma (N + 2)} \\ = \frac {(N - K) ! K !}{(N + 1) !} \\ \end{array}
$$
where  $B(\cdot)$  is Beta function and  $\Gamma (\cdot)$  is Gamma function. Thus for u-PMLM, the expected loss-likelihood function denoted in Equation 6 becomes:
$$
\begin{array}{l} L _ {\mathrm {p l m l m}} \left(X _ {\Pi} | X; \theta\right) \\ = \sum_ {M} \left[ \frac {1}{K} \sum_ {k = 1} ^ {K} \log p \left(x _ {\pi_ {k}} \mid X _ {- \Pi}; \theta\right) \right] \frac {(N - K) ! K !}{(N + 1) !} \\ = \frac {\sum_ {M} \sum_ {k = 1} ^ {K} (N - K) ! (K - 1) ! \log p \left(x _ {\pi_ {k}} \mid X _ {- \Pi} ; \theta\right)}{(N + 1) !} \tag {10} \\ \end{array}
$$
On the other hand, we rewrite the formulation of an autoregressive permutated language model (APLM) denoted in Equation 8 as:
$$
\begin{array}{l} L _ {\mathrm {a p l m}} (X) = \mathbb {E} _ {\sigma} \left[ \sum_ {t = 1} ^ {N} \log p \left(x _ {\sigma_ {t}} \mid x _ {\sigma_ {1}}, \dots , x _ {\sigma_ {t - 1}}; \theta\right) \right] \\ = \frac {\sum_ {\sigma} \left[ \sum_ {t = 1} ^ {N} \log p \left(x _ {\sigma_ {t}} \mid x _ {\sigma_ {1}} , \dots , x _ {\sigma_ {t - 1}} ; \theta\right) \right]}{C} \tag {11} \\ \end{array}
$$
where the numerator sums over the log-likelihood for all the possible permutations and the denominator  $C$  is a constant. In fact, we can rewrite the term  $p(x_{\sigma_t}|x_{\sigma_1},\ldots ,x_{\sigma_{t - 1}};\theta)$  by  $p(x_{\sigma_t}|X_{-\Pi_t^\sigma};\theta)$  where  $\Pi_t^\sigma = X - \{\sigma_1,\sigma_2,\dots,\sigma_{t - 1}\}$ . Noted that  $K$  is the size of  $\Pi_t^\sigma$ . Thus the size of  $\Pi_t^\sigma$  is denoted as  $|\Pi_t^\sigma | = K = N - t + 1$ . Therefore we rewrite Equation 11 as:
$$
\begin{array}{l} L _ {\mathrm {a p l m}} (X) \\ = \frac {1}{C} \sum_ {\sigma} \left[ \log p \left(X _ {\Pi_ {t + 1} ^ {\sigma}} \mid X _ {- \Pi_ {t + 1} ^ {\sigma}}; \theta\right) + \log p \left(x _ {\sigma_ {t}} \mid X _ {- \Pi_ {t} ^ {\sigma}}; \theta\right) \right. \\ + \log p \left(X _ {- \Pi_ {t} ^ {\sigma}}; \theta\right) ] \tag {12} \\ \end{array}
$$
According to the above equation, we can derive the duplication factor for specific term  $\log p(x_{\sigma_t}|X_{-\Pi_t^\sigma})$  when summing over all the permutations, which is exactly the product of numbers of permutations for  $\Pi_{t + 1}^{\sigma}$  and  $-\Pi_t^\sigma$  in the first and last term respectively. Specifically, the number of
permutations for  $\Pi_{t + 1}^{\sigma}$  and  $-\Pi_t^\sigma$  are factorials of  $|\Pi_{t + 1}^{\sigma}|$  and  $|\textbf{-}\Pi_t^\sigma |$  , denoted as:
$$
\begin{array}{l} \operatorname {p e r m u t a t i o n} \left(\Pi_ {t + 1} ^ {\sigma}\right) = \left| \Pi_ {t + 1} ^ {\sigma} \right|! = (N - K)! \tag {13} \\ \operatorname {p e r m u t a t i o n} \left(- \Pi_ {t} ^ {\sigma}\right) = | - \Pi_ {t} ^ {\sigma} |! = (K - 1)! \\ \end{array}
$$
Hence the duplication factor for  $\log p(x_{\sigma_t}|X_{-\Pi_t^\sigma})$  is computed as  $(N - K)!(K - 1)!$ , which is the coefficient of the expected log-likelihood function of u-PMLM denoted in Equation 10. Thus we conclude that Equation 10 is equivalent to Equation 8, where the constant  $C = (N + 1)!$ .
B Generation Examples of u-PMLM and BERT
We show more examples of the text generated by u-PMLM-A and BERT respectively. Note that we do not manually select examples. Instead, these examples are picked randomly for fair comparison.
Below are texts generated by u-PMLM-A:
1. Tom is a cat and Jerry is a mouse. Though initially wary that he might steal the pie, Jerry eventually admits his fears. Tom's father, Selma, is upset when the cat is not present. She tries making a play about the pie but the girl ignores her and insists that she will be glad if he's able to steal the pie. Tom and Jerry become good friends in the end.
2. Tom is a cat and Jerry is a mouse. “Jerry and Tom meet again in a restaurant and in a confrontation Tom tries to call the house police. Jerry’s neighbor Tom Martin (Tom Lee) finds them flirting by taking Jerry, but Tom finds out. With Jerry’s help Tom gets their hand together in a car wash and the two have a date. Tom and Jerry become good friends in the end.
3. Tom is a cat and Jerry is a mouse. When the kids get their first game of cricket at his home, he has him as catcher. When all of the teams are on the field except the school cricket player, he takes part in every game, which is always the cricket. He later breaks his leg on the last day of cricket and falls down a ladder. Tom and Jerry become good friends in the end.
4. Tom is a cat and Jerry is a mouse. The two start off on a fishing trip together to see what their parents would think of them. The boys
first discover a bad fish on the ocean and decide to try the game. A boathouse is built in town where they see a big picture of Tom's grandfather. The boys fight it out by chance, and Tom and Jerry become good friends in the end.
5. Tom is a cat and Jerry is a mouse. This makes Tom very hard to understand why he and the other citizens like him, so he tries to have to get rid of the mouse. His best friend, Pete, tries to keep his dog quiet about this, even while Jerry and Marge insist they are just starting on a date due to Tom's dog. Tom and Jerry become good friends in the end.
Below are texts generated by BERT:
1. Tom is a cat and Jerry is a mouse. “That same month the two characters were portrayed as their characters at the NBC series The Office, with the character played by Critrean Figgins and the mouse played by the cat. The two characters make a romantic connection and meet in the end of the episode” When We Were Meeted Together, ” seeing why Tom and Jerry become good friends in the end.
2. Tom is a cat and Jerry is a mouse . . . And it ' s about when Tom is in trouble and he becomes frustrated when everything gets upsetting . I make my own movie . . . But I do not want to be a film so sad or about what I like about a kid ' s life . I want to make a movie that is more real and Tom and Jerry become good friends in the end .
3. Tom is a cat and Jerry is a mouse. “On August 7, 1999, DeLanto started a Twitter account, which included an online forum address on NBC’s show 30 Rock through his account of his life on stage and on the Internet. During 2008, he also posted on his personal blog a message saying “This world’s really getting bigger. Tom and Jerry become good friends in the end.
4. Tom is a cat and Jerry is a mouse. He is a cat and Tom is a mouse. At McKinley High School, Tom enters the Class 3A, and then is elected President of High School (H. F. R.), the district's popular high school. He becomes the principal and a student's
supervisor at the High School in 2004 . Tomand Jerry become good friends in the end .