SlowGuess/NAACL_2022_OCR huggingface dataset:
We adapted the way we split the dataset into two parts and train/test/Validation sets depending on the original dataset splits. Amazon_polarity is composed of a training set of 3600000 examples and a test set of 400000. We split both into two parts and kept  $20\%$  of each training set for validation. Emotion already comes with train, test and validation set, hence we just split each into two parts. Finally, CLS is composed of a train set and a test set of 6000 examples. We split the training set in two and split the test set twice so we got two validation and test sets. Thus, for each dataset, we end up with two training sets, two validation sets and two test sets.
The first train and validation sets are used to train and control the training of models used for the generation: the guiding classifier, the "vanilla" LM and the CC-LM. The test set serves to control their performance.
The second ones are used to train the LM oracle and the classifier used to measure the accuracy. The test set allows to verify that these models are trustworthy for accurate evaluation. Once all the models are trained, the constrained generation is evaluated on 900 samples generated from prompts never seen by models during training.
A.2 Complementary results
We tested three temperature values for each proposed method: 1.0, 1.1 and 1.2. As the temperature increases, the output distribution of the language model becomes more and more uniform. This means that high temperatures should result in high perplexities because the sampling deviates further from the original distribution.
For PPL-MCTS, we also studied the impact of  $c_{puct}$  by testing values 1.0, 3.0, 5.0 and 8.0 along with the different temperature values mentioned.  $c_{puct}$  defines the compromise between exploiting nodes that already have great scores and exploring less played but promising ones. A high  $c_{puct}$  encourages exploration. We remind that the repetition penalty  $I$  in Eqn. 2 has been set to 1.2 as defined
in CTRL.
In Section 'Results', for each method and dataset, we reported only the results obtained with the set of parameter values yielding the best accuracy. Hereafter, we report results with every tested set of parameters in Tables 3, 4 and 5 for respectively the emotion, CLS and amazon_polarity datasets.
Unsurprisingly, the perplexity of methods which sample on the LM logits explodes when  $\tau$  increases, without a noticeable gain in accuracy. Since the diversity is already high for low  $\tau$  values, it seems to be better to keep the temperature low with these approaches. Note that the couple  $c_{puct} = 3$ ,  $\tau = 1.0$  for PPL-MCTS always leads to the best result. Using  $c_{puct} = 8$  seems to yield slightly worse results, especially with a low temperature. However, the different parameters do not greatly affect the results of PPL-MCTS.
A.3 Examples of generation
We provide an example of generation for amazon_polarity and emotion datasets using PPLMCTS, PPLM, GeDi and Sampling - Argmax methods, respectively in Figures 5 and 6. Texts generated using Sampling - Argmax are rather different as suggested by the reported high perplexity results. Note that emotion texts are only one sentence while those of amazon_polarity are complete reviews. This difference motivated the choice of these datasets. Also, we preferred amazon_polarity over IMDb used in the GeDi and PPLM papers because of its bigger size, suitable format and because a French equivalent is available (CLS), which allows us to test another language with a similar dataset.
A.4 Constraint strength through  $\alpha$
As described in the model improvements section, a parameter  $\alpha$  can be defined to control the relative importance of the discriminator score and the language model likelihood. Thus, this parameter allows to control the constraint application strength as it helps to define a trade-off between staying close the original LM and satisfying the constraint. Note that in all of our experiments reported earlier, this parameter has been set to 1, focusing on the constraint application since the proposed methods already inherently provide legible texts.
Here, as a proof of concept, we test a range of values for  $\alpha$ , using the Sampling - Argmax method on the amazon_polarity dataset with the automatic
metrics. We chose this method and dataset since it yields the best accuracy, but also exhibits a very high perplexity. In this case, it seems interesting to trade a bit of accuracy for better written texts.
Results are roughly constant when  $\alpha$  is lower than 0.98, so it has an impact only for values between 0.98 and 1. This is due to the fact that, for a long enough sequence,  $p_{\theta}(x)$  is often relatively small compared to  $p_D(c \mid x)$ . This difference of scale annihilates the influence of  $\alpha$ . This [0.98-1] interval thus corresponds to values of  $\alpha$  that rescale  $p_D(c \mid x)^{\alpha}$  and  $p_{\theta}(x)^{1 - \alpha}$  on a same order of magnitude. As shown in Figure 7, within this regime, we can observe a linear dependency between  $\alpha$  and the accuracy as well as the perplexity. This illustrate that a trade-off can be obtained by tuning this parameter, allowing to define the strength of the constraint application which also defines how far the generation can be from the original LM distribution.
PPL-MCTS
realistic account [SEP] ive read this book and i think it is a very good account of the life of a young man. i would recommend to anyone who likes to know about the lives of people in america, or any other country that has been around for centuries.
PPLM
<startoftext> realistic account [SEP] I enjoyed this book. It was realistic and I enjoyed the way the author described the people and places. I would recommend this book to anyone who is interested in the history of the United States.<endoftext>
GeDi
realistic account [SEP] This is a real book. It's not just about the war, but also about the people who fought it. The author has a very realistic view of what happened in Vietnam and how it affected our country. I would recommend this book to anyone interested in learning more about Vietnam or any other war that took place during that time period
Sampling - Argmax
<startoftext> realistic account [SEP] ive seen many of these. they mostly in the 20's, 30's and 40's and the luscious ones i know - men like this - there's an old saying that farts, u makin' u sell it..this movie has a lot of realism to it too! and i was totally impressed on how good the kids and the predator was! will it be hard for them to make more like this? i think it will! i read that war is going to be much<endoftext>
Figure 5: Examples of constrained generation using PPL-MCTS, PPLM, GeDi and Sampling - Argmax methods (from top to bottom) on the positive class of amazon_polarity, using the same prompt (in bold).
<startoftext> i feel that working with a group of people who are so passionate about the same thing is really important<endoftext>
<startoftext> i feel that working hard and caring for someone i don t care for is a lot less selfish than i would be feeling for someone i<endoftext>
<startoftext> i feel that working with the ladies of the family is a wonderful thing and i am very fond of the way they look and feel<endoftext>
<startoftext> i feel that working at imgur for so many years is ill be devoted to it<endoftext>
Figure 6: Examples of constrained generation using PPL-MCTS, PPLM, GeDi and Sampling - Argmax methods (from top to bottom) on the 'love' class form 'emotion', using the same prompt (in bold).
Generation method	Accuracy ↑	5 - Self-Bleu ↓	Oracle perplexity ↓
Beam sampling - Argmax τ = 1.0	0,61	0,41	3,7
Beam sampling - Argmax τ = 1.1	0,65	0,48	3,72
Beam sampling - Argmax τ = 1.2	0,67	0,48	3,88
Beam sampling - First true τ = 1.0	0,58	0,4	3,68
Beam sampling - First true τ = 1.1	0,64	0,48	3,69
Beam sampling - First true τ = 1.2	0,66	0,49	3,85
Beam sampling - Sampling τ = 1.0	0,59	0,41	3,69
Beam sampling - Sampling τ = 1.1	0,64	0,49	3,69
Beam sampling - Sampling τ = 1.2	0,66	0,48	3,88
CC-LM - Greedy Search	0,51	0,1	17
CC-LM - Sampling τ = 1.0	0,52	0,13	11,1
CC-LM - Sampling τ = 1.1	0,51	0,1	15,8
CC-LM - Sampling τ = 1.2	0,47	0,08	31,4
CC-LM - Classloss - Greedy Search	0,89	0,65	3,72
CC-LM - Classloss - Sampling τ = 1.0	0,83	0,11	19,6
CC-LM - Classloss - Sampling τ = 1.1	0,79	0,07	33,2
CC-LM - Classloss - Sampling τ = 1.2	0,79	0,05	64,8
Sampling - Argmax τ = 1.0	0,87	0,13	11,7
Sampling - Argmax τ = 1.1	0,86	0,1	19,6
Sampling - Argmax τ = 1.2	0,86	0,07	47,5
Sampling - First true τ = 1.0	0,82	0,13	10,4
Sampling - First true τ = 1.1	0,81	0,11	16,2
Sampling - First true τ = 1.2	0,77	0,09	33,2
Sampling - Sampling τ = 1.0	0,81	0,13	10,4
Sampling - Sampling τ = 1.1	0,8	0,11	15
Sampling - Sampling τ = 1.2	0,79	0,08	25,7
PPL-MCTS - cpuct = 1.0, τ = 1.0	0,83	0,37	4,81
PPL-MCTS - cpuct = 1.0, τ = 1.1	0,8	0,36	4,9
PPL-MCTS - cpuct = 1.0, τ = 1.2	0,82	0,33	4,97
PPL-MCTS - cpuct = 3.0, τ = 1.0	0,84	0,37	4,82
PPL-MCTS - cpuct = 3.0, τ = 1.1	0,82	0,35	4,85
PPL-MCTS - cpuct = 3.0, τ = 1.2	0,84	0,35	4,9
PPL-MCTS - cpuct = 5.0, τ = 1.0	0,84	0,38	4,74
PPL-MCTS - cpuct = 5.0, τ = 1.1	0,84	0,34	4,79
PPL-MCTS - cpuct = 5.0, τ = 1.2	0,84	0,33	4,88
PPL-MCTS - cpuct = 8.0, τ = 1.0	0,81	0,38	4,71
PPL-MCTS - cpuct = 8.0, τ = 1.1	0,81	0,37	4,72
PPL-MCTS - cpuct = 8.0, τ = 1.2	0,82	0,35	4,79
Table 3: Results for every tested set of parameters on the proposed methods; emotion dataset. Results reported in the body of the paper are in italic.
A.5 Concurrent work
During the time of writing, two preprints using MCTS for NLP tasks have been released (Leblond
Figure 7: Accuracy (left) and perplexity (right) of the Sampling - Argmax method according to the  $\alpha$  parameter; amazon_polarity dataset
Generation method	Accuracy ↑	5 - Self-Bleu ↓	Oracle perplexity ↓
Beam sampling - Argmax τ = 1.0	0,87	0,71	3,85
Beam sampling - Argmax τ = 1.1	0,88	0,67	3,91
Beam sampling - Argmax τ = 1.2	0,88	0,63	4,12
Beam sampling - First true τ = 1.0	0,85	0,71	3,8
Beam sampling - First true τ = 1.1	0,84	0,68	3,87
Beam sampling - First true τ = 1.2	0,85	0,63	4,07
Beam sampling - Sampling τ = 1.0	0,74	0,71	3,82
Beam sampling - Sampling τ = 1.1	0,72	0,68	3,89
Beam sampling - Sampling τ = 1.2	0,76	0,63	4,07
CC-LM - Greedy Search	0,59	0,57	2,51
CC-LM - Sampling τ = 1.0	0,62	0,15	12,3
CC-LM - Sampling τ = 1.1	0,63	0,09	18,7
CC-LM - Sampling τ = 1.2	0,66	0,06	31,5
CC-LM - Classloss - Greedy Search	0,8	0,59	2,77
CC-LM - Classloss - Sampling τ = 1.0	0,85	0,13	17
CC-LM - Classloss - Sampling τ = 1.1	0,87	0,07	28
CC-LM - Classloss - Sampling τ = 1.2	0,89	0,04	50,6
Sampling - Argmax τ = 1.0	0,92	0,12	14,3
Sampling - Argmax τ = 1.1	0,92	0,08	20,7
Sampling - Argmax τ = 1.2	0,92	0,05	33,6
Sampling - First true τ = 1.0	0,87	0,14	13
Sampling - First true τ = 1.1	0,86	0,1	19,1
Sampling - First true τ = 1.2	0,86	0,06	33,1
Sampling - Sampling τ = 1.0	0,77	0,14	12,9
Sampling - Sampling τ = 1.1	0,78	0,09	18,8
Sampling - Sampling τ = 1.2	0,81	0,06	31,8
PPL-MCTS - cpuct = 1.0, τ = 1.0	0,88	0,54	4,98
PPL-MCTS - cpuct = 1.0, τ = 1.1	0,87	0,53	5
PPL-MCTS - cpuct = 1.0, τ = 1.2	0,87	0,53	5,02
PPL-MCTS - cpuct = 3.0, τ = 1.0	0,89	0,54	4,98
PPL-MCTS - cpuct = 3.0, τ = 1.1	0,89	0,54	4,81
PPL-MCTS - cpuct = 3.0, τ = 1.2	0,89	0,54	4,86
PPL-MCTS - cpuct = 5.0, τ = 1.0	0,88	0,55	4,9
PPL-MCTS - cpuct = 5.0, τ = 1.1	0,89	0,54	4,97
PPL-MCTS - cpuct = 5.0, τ = 1.2	0,89	0,54	4,91
PPL-MCTS - cpuct = 8.0, τ = 1.0	0,83	0,54	4,98
PPL-MCTS - cpuct = 8.0, τ = 1.1	0,86	0,54	4,95
PPL-MCTS - cpuct = 8.0, τ = 1.2	0,88	0,55	4,94
Table 4: Results for every tested set of parameters on the proposed methods; CLS dataset. Results reported in the body of the paper are in italic.
Generation method	Accuracy \( \uparrow \)	5 - Self-Bleu \( \downarrow \)	Oracle perplexity \( \downarrow \)
Beam sampling - Argmax \( \tau = {1.0} \)	0,94	0,79	3,55
Beam sampling - Argmax \( \tau = {1.1} \)	0,96	0,77	3,65
Beam sampling - Argmax \( \tau = {1.2} \)	0,97	0,73	3,82
Beam sampling - First true \( \tau = {1.0} \)	0,86	0,77	3,73
Beam sampling - First true \( \tau = {1.1} \)	0,89	0,77	3,68
Beam sampling - First true \( \tau = {1.2} \)	0,9	0,73	3,84
Beam sampling - Sampling \( \tau = {1.0} \)	0,87	0,77	3,7
Beam sampling - Sampling \( \tau = {1.1} \)	0,92	0,76	3,68
Beam sampling - Sampling \( \tau = {1.2} \)	0,89	0,73	3,83
CC-LM - Greedy Search	0,91	0,71	3,21
CC-LM - Sampling \( \tau = {1.0} \)	0,87	0,17	15,7
CC-LM - Sampling \( \tau = {1.1} \)	0,86	0,1	32,2
CC-LM - Sampling \( \tau = {1.2} \)	0,8	0,08	80,2
CC-LM - Classloss - Greedy Search	0,82	0,79	2,56
CC-LM - Classloss - Sampling \( \tau = {1.0} \)	0,81	0,16	18,4
CC-LM - Classloss - Sampling \( \tau = {1.1} \)	0,79	0,1	37,1
CC-LM - Classloss - Sampling \( \tau = {1.2} \)	0,74	0,07	95,4
Sampling - Argmax \( \tau = {1.0} \)	0,99	0,17	16,5
Sampling - Argmax \( \tau = {1.1} \)	0,99	0,11	31,8
Sampling - Argmax \( \tau = {1.2} \)	0,99	0,07	84,50
Sampling - First true \( \tau = {1.0} \)	0,88	0,16	16,4
Sampling - First true \( \tau = {1.1} \)	0,87	0,1	31,5
Sampling - First true \( \tau = {1.2} \)	0,89	0,07	85,9
Sampling - Sampling \( \tau = {1.0} \)	0,88	0,17	16,3
Sampling - Sampling \( \tau = {1.1} \)	0,87	0,1	30,8
Sampling - Sampling \( \tau = {1.2} \)	0,88	0,07	81
PPL-MCTS - \( {c}_{puct} = {1.0},\tau = {1.0} \)	0,96	0,62	5,61
PPL-MCTS - \( {c}_{puct} = {1.0},\tau = {1.1} \)	0,96	0,63	5,65
PPL-MCTS - \( {c}_{puct} = {1.0},\tau = {1.2} \)	0,96	0,62	5,66
PPL-MCTS - \( {c}_{puct} = {3.0},\tau = {1.0} \)	0,97	0,63	5,69
PPL-MCTS - \( {c}_{puct} = {3.0},\tau = {1.1} \)	0,97	0,62	5,77
PPL-MCTS - \( {c}_{puct} = {3.0},\tau = {1.2} \)	0,96	0,62	5,72
PPL-MCTS - \( {c}_{puct} = {5.0},\tau = {1.0} \)	0,95	0,63	5,6
PPL-MCTS - \( {c}_{puct} = {5.0},\tau = {1.1} \)	0,96	0,63	5,66
PPL-MCTS - \( {c}_{puct} = {5.0},\tau = {1.2} \)	0,96	0,63	5,63
PPL-MCTS - \( {c}_{puct} = {8.0},\tau = {1.0} \)	0,93	0,64	5,57
PPL-MCTS - \( {c}_{puct} = {8.0},\tau = {1.1} \)	0,93	0,64	5,57
PPL-MCTS - \( {c}_{puct} = {8.0},\tau = {1.2} \)	0,95	0,63	5,57
Table 5: Results for every tested set of parameters on the proposed methods; amazon_polarity dataset. Results reported in the body of the paper are in italic.
et al., 2021; Scialom et al., 2021). While we emphasize that these are concurrent studies, PPL-MCTS has some major differences. Indeed, these studies focus on improving the overall quality of generated texts rather than following a given constraint. While "being well written" can be seen as a constraint, PPL-MCTS rather explores how a constraint that is not present in the original language model (i.e. not a goal in the original training of the LM) can be added at generation time. Scialom
et al. (2021) train a discriminator to distinguish generated and real samples because their goal is ultimately to train the language model in a Generative Adversarial setup to create a better LM. This iterative training, in addition to not being possible in our task, is not wanted since we aim to be plug and play. Our goal is indeed to apply an additional constraint to an untouched original language model. Yet, even if goals are different and applying MCTS for constrained generation is not trivial, the "MLE
Coop-MCTS" is close to PPL-MCTS. However, focusing on MCTS as a decoding only strategy allowed an in-depth study that provided interesting results, in particular the effect of the roll-out size (the roll-out is totally omitted in their paper) and the  $\alpha$  parameter.
On the other hand, Leblond et al. (2021) also focus on MCTS as a decoding strategy but for the very specific case of machine translation. MCTS is used to optimize metrics for machine translation, which are known to not necessarily correlate with human judgement (Novikova et al., 2017). Again, the goal is different since these metrics are used as a proxy of the sample quality. In contrast, our work shows that MCTS can be used to optimize a given property, but instead of optimizing the quality of samples, we optimize for a given constraint while retaining the original quality of writing. The fact that MCTS also works in such cases was non-trivial since adding such constraints to the generation could lead to deteriorate texts.
Beside MCTS, we also proposed and explored simpler methods based on re-ranking for our task and showed that diversity allows to satisfy the constraint, often at the price of a lower quality, emphasizing the compromise between exploration and exploitation made by the MCTS.
Finally, these concurrent studies provide evidences that MCTS is promising for many different usage in NLP. We hope that the large amount of experiments, parameter analysis and the availability of our open-sourced code working out-of-the-box will help foster future research in this direction.