SlowGuess/NAACL_2021_OCR huggingface dataset:
Given the same discriminator  $g_{\theta}$ , the difference of KL-divergence between the true model  $p$  and residual language models  $\tilde{p}_{\theta}^{\prime}(\mathbf{x}) = p_0^{\prime}(\mathbf{x})\cdot \exp g_{\theta}(\mathbf{x})$  and the KL-divergence between the true model and  $\tilde{p}_{\theta}^{\prime \prime}(\mathbf{x}) = p_0^{\prime \prime}(\mathbf{x})\cdot \exp g_{\theta}(\mathbf{x})$ , defined with base models  $p_0^{\prime}$  and  $p_0^{\prime \prime}$  respectively, can be written as
$$
\begin{array}{l} \operatorname {K L} \left[ p \mid \mid p _ {\theta} ^ {\prime} \right] - \operatorname {K L} \left[ p \mid \mid p _ {\theta} ^ {\prime \prime} \right] \\ = \mathrm {K L} [ p | | p _ {0} ^ {\prime} ] - \mathrm {K L} [ p | | p _ {0} ^ {\prime \prime} ] + \log \frac {Z ^ {\prime}}{Z ^ {\prime \prime}}, \tag {4} \\ \end{array}
$$
where  $Z^{\prime} = \mathbb{E}_{\mathbf{x}\sim p_{0}^{\prime}}[\exp g_{\theta}(\mathbf{x})]$ , and  $Z''$  is similarly defined with  $p_0^{\prime \prime}$ . As a direct result of equation (4), we can see that finding  $p_0^{\prime \prime}$  where  $\mathrm{KL}[p||p_0^{\prime \prime}] < \mathrm{KL}[p||p_0^{\prime}]$  implies improvement in  $\mathrm{KL}[p||p_{\theta}^{\prime \prime}]$  over  $\mathrm{KL}[p||p_{\theta}^{\prime}]$ , under mild conditions:
Theorem B.1. If  $\exists k > 0$  such that  $\frac{\mathbb{E}_{\mathbf{x} \sim p_0'}[\exp g_{\boldsymbol{\theta}}(\mathbf{x})]}{\mathbb{E}_{\mathbf{x} \sim p_0''}[\exp g_{\boldsymbol{\theta}}(\mathbf{x})]} > \exp(-k)$  and  $\mathrm{KL}[p||p_0'] - \mathrm{KL}[p||p_0''] > k$  then  $\mathrm{KL}[p||p_{\boldsymbol{\theta}}'] > \mathrm{KL}[p||p_{\boldsymbol{\theta}}'']$ .
$$
\begin{array}{l} \mathrm {K L} [ p | | p _ {\theta} ^ {\prime} ] - \mathrm {K L} [ p | | p _ {\theta} ^ {\prime \prime} ] \\ = \underset {\mathbf {x} \sim p} {\mathbb {E}} \left[ \log p _ {\theta} ^ {\prime \prime} (\mathbf {x}) - \log p _ {\theta} ^ {\prime} (\mathbf {x}) \right] \\ = \underset {\mathbf {x} \sim p} {\mathbb {E}} \left[ \log \frac {p _ {0} ^ {\prime \prime} (\mathbf {x}) \exp g _ {\theta} (\mathbf {x})}{\sum_ {\mathbf {x} ^ {\prime} \in L _ {\leq T}} p _ {0} ^ {\prime \prime} (\mathbf {x}) \exp g _ {\theta} (\mathbf {x})} \right. \\ - \log \left. \frac {p _ {0} ^ {\prime} (\mathbf {x}) \exp g _ {\theta} (\mathbf {x})}{\sum_ {\mathbf {x} ^ {\prime} \in L _ {\leq T}} p _ {0} ^ {\prime} (\mathbf {x}) \exp g _ {\theta} (\mathbf {x})} \right] \\ = \underset {\mathbf {x} \sim p} {\mathbb {E}} \left[ \log \frac {p _ {0} ^ {\prime \prime} (\mathbf {x}) \exp g _ {\theta} (\mathbf {x})}{\mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p _ {0} ^ {\prime \prime}} [ \exp g _ {\theta} (\mathbf {x}) ]} \right. \\ - \log \frac {p _ {0} ^ {\prime} (\mathbf {x}) \exp g _ {\theta} (\mathbf {x})}{\mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p _ {0} ^ {\prime}} [ \exp g _ {\theta} (\mathbf {x}) ]} \\ = \underset {\mathbf {x} \sim p} {\mathbb {E}} \left[ \log p _ {0} ^ {\prime \prime} (\mathbf {x}) - \log p _ {0} ^ {\prime} (\mathbf {x}) \right] \\ + \underset {\mathbf {x} \sim p} {\mathbb {E}} \left[ \log \underset {\mathbf {x} ^ {\prime} \sim p _ {0} ^ {\prime}} {\mathbb {E}} \left[ \exp g _ {\boldsymbol {\theta}} (\mathbf {x}) \right] - \log \underset {\mathbf {x} ^ {\prime} \sim p _ {0} ^ {\prime \prime}} {\mathbb {E}} \left[ \exp g _ {\boldsymbol {\theta}} (\mathbf {x}) \right] \right] \\ = \mathrm {K L} [ p | | p _ {0} ^ {\prime} ] - \mathrm {K L} [ p | | p _ {0} ^ {\prime \prime} ] \\ + \log \frac {\mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p _ {0} ^ {\prime}} [ \exp g _ {\theta} (\mathbf {x}) ]}{\mathbb {E} _ {\mathbf {x} ^ {\prime} \sim p _ {0} ^ {\prime \prime}} [ \exp g _ {\theta} (\mathbf {x}) ]}. \tag {5} \\ \end{array}
$$
Plugging assumptions  $\frac{\mathbb{E}_{\mathbf{x}\sim p_0^{\prime}}[\exp g_{\boldsymbol{\theta}}(\mathbf{x})]}{\mathbb{E}_{\mathbf{x}\sim p_0^{\prime\prime}}[\exp g_{\boldsymbol{\theta}}(\mathbf{x})]} >\exp (-k)$  and KL[p||p'] - KL[p||p'] > k into equation (5), KL[p||p'] - KL[p||p'] > 0.
Theorem B.1 suggests a training strategy that we first train the base model  $p_0$ , then finetune  $g_{\theta}$  under a roughly uniform  $g_{\theta}$  (e.g. when  $\theta$  is newly initialized),  $\mathbb{E}_{\mathbf{x} \sim p_0'}[\exp g_{\theta}] / \mathbb{E}_{\mathbf{x} \sim p_0''}[\exp g_{\theta}] \approx \exp(0)$ ; so improvements on the inclusive KL-divergence of base model KL  $[p||p_0]$  will mostly translate to improvement in KL  $[p||\tilde{p}_{\theta}]$ . Optimizing the base model (i.e. finding  $p_0''$  such that KL  $[p||p_0''] < \mathrm{KL}[p||p_0'']$  is much easier than directly minimizing KL  $[p||p_{\theta}']$ : the former can be done by minimizing empirical cross entropy, which is computationally efficient, while the latter involves an intractable partition function  $\sum_{\mathbf{x} \in L_{\leq T}} \tilde{p}_{\theta}'(\mathbf{x})$ .
Pseudocode for fine-tuning  $g_{\theta}$  is listed in Algorithm 1.
B.5 Computing normalized probabilities
The unnormalized probability  $\tilde{p}_{\theta}(\mathbf{x})$  (in equation (1)) can be evaluated easily, and should suffice for (re)ranking purposes (e.g. for ASR and MT applications). However, the normalized probability  $q_{\theta}(\mathbf{x}) \triangleq \frac{\tilde{p}_{\theta}(\mathbf{x})}{\sum_{\mathbf{x}} \tilde{p}_{\theta}(\mathbf{x})}$  does require computing the partition function  $Z_{\theta}$ . An unbiased importance sampling
estimate of  $\sum_{\mathbf{x}\in L_{\leq T}}\tilde{p}_{\boldsymbol{\theta}}(\mathbf{x})$  is
$$
\begin{array}{l} Z _ {\theta} = \sum_ {\mathbf {x} \in L _ {\leq T}} \tilde {p} _ {\theta} (\mathbf {x}) \\ = \sum_ {\mathbf {x} \in L _ {<   T}} p _ {0} (\mathbf {x}) \exp g _ {\theta} (\mathbf {x}) \\ = \underset {\mathbf {x} \sim p _ {0}} {\mathbb {E}} \left[ \exp g _ {\boldsymbol {\theta}} (\mathbf {x}) \right] \\ \approx \sum_ {m = 1} ^ {M} \frac {\exp g _ {\theta} \left(\mathbf {x} ^ {(m)}\right)}{M} = \hat {Z} _ {\theta M}, \tag {6} \\ \end{array}
$$
where  $\mathbf{x}^{(1)}\ldots \mathbf{x}^{(M)}\sim q_0$
C Comparison between REBMs and autoregressive models
We evaluate the effectiveness of REBMs on two different neural architectures (GRU- and Transformer-based) and 3 datasets: WikiText (Merit et al., 2017), Yelp (Yelp), and RealNews (Zellers et al., 2019), on the task of modeling sequence probabilities. An REBM  $\tilde{p}_{\theta}$  has two components,  $g_{\theta}$  and  $p_0$ , and we would like to see how  $\tilde{p}_{\theta}$  competes against  $p_0$  itself. We do not further tune  $p_0$  while training  $p_{\theta}$ . As a fair comparison, we also see how  $p_0'$  compares against  $p_0$ , where  $p_0'$  is simply a version of  $p_0$  that has been trained as many additional epochs as were used to train  $p_{\theta}$ .
$p_0$  models are pretrained on moderately large corpora (in GRU cases) or a very large corpus (in the Transformer case).[27] We compare residual energy-based models  $\tilde{p}_{\theta}$  to further-fine-tuned base models  $p_0'$ ,
on conservatively estimated (at the low end of  $95\%$  confidence interval) token perplexity and bootstrap-sampled log likelihood improvements. The results are in Table 2. Residual energy-based models show consistent perplexity improvement compared to  $p_0'$  that are trained on the same data using the same maximum numbers of iterations. Although the improvement in log-likelihood of  $p_\theta$  over  $p_0$  is modest (especially for RealNews experiments, where  $p_0$  is a very strong baseline), we verify that these improvements are all statistically significant ( $p < 0.05$ ) using bootstrapped test datasets.
We experiment with different designs of the discriminator  $g_{\theta}$ , evaluating the effectiveness of bounding  $g_{\theta}$  and varying its number of parameters. We find that in Transformer-based experiments, bounding  $g_{\theta}$  considerably helps with performance; but the opposite happens for GRU-based models. We speculate that this is due to the base models' performance: the Transformer base models have high parameter count and were trained on a lot of data; and the true distribution  $p$  likely is relatively similar to  $p_0$ , and benefits from a small hypothesis space—even though we don't know if the at-most-  $\epsilon$  error assumption in §4.1 holds. On the other hand our GRU-based  $p_0$  has neither the capacity, nor the huge amount of training data. As a result, the unbounded variant  $g_{\theta}$  (and  $q_{\theta}$ ) may end up learning a better approximation of  $p$ .
Residual language model experiments are conducted on these datasets:
- Segmented WikiText: we take the standard WikiText-2 corpus (Merit et al., 2017), and segment it into sequences at new line breaks. We discard all empty lines, and any line that starts with the  $=$  token. In effect, we obtain sequences that are mostly entire paragraphs. We also only keep lines that are shorter than 800 tokens after BPE tokenization. Because of our preprocessing, Segmented WikiText loses much interparagraph context information, and doesn't have the 'simple' header sequences that were in the original WikiText corpus, and is much harder to language-model.
- Yelp: the Yelp dataset (Yelp) contains business reviews. As in Segmented WikiText, We keep
(Zellers et al., 2019) pretrained language model, which is based on the GPT-2 (Radford et al., 2019) architecture and performs competitively on news article generation.
27In the Transformer case we simply take  $p_0$  to be the Grover
Experiment (Architecture)	Model	Best configuration	log likelihood improvement (95% CI)	perplexity improvement
RealNews (Transformer)	pθ	4-layer, tanh	(-0.18, -0.13), μ = -0.15	.03%
RealNews (Transformer)	p'0	N/A	N/A	.00%
WikiText (GRU)	pθ	1-layer/500, softplus	(-1.85, -1.54), μ = -1.69	1.44%
WikiText (GRU)	p'0	N/A	N/A	.50%
Yelp (GRU)	pθ	2-layer/500, softplus	(-1.89, -1.67), μ = -1.80	1.82%
Yelp (GRU)	p'0	N/A	N/A	.49%
reviews shorter than 800 tokens.
- REALNEWS: we make use of the standard REAL-News corpus comes from (Zellers et al., 2019), which contains news articles that are up to 1,024 tokens long.
In all experiments we tokenize with BPE tokenizers derived from the GPT-2 language models: the GRU models use Huggingface's implementation $^{28}$  and the Transformers use Grover's $^{29}$ . Number of sequences in preprocessed datasets are listed in Table 3.
Table 2: Residual energy-based model  ${\widetilde{p}}_{\theta }$  improvements over autoregressive base models  ${p}_{0}$  . The perplexity numbers are per-token, and log likelihood improvements are per sequence (in nats). We only report each dataset's best model (according to validation data) in this table. See Appendix D for experimental details.
Train	Dev	Test
RealNews	3,855	1,533	6,158
WikiText	18,519	878	2,183
Yelp	10,951	9,964	994
D.2 Pretraining base models  $p_0$
We use a pretrained Grover model as the base model in RealNews experiments. For GRU-based experiments, we train base models on WikiText and Yelp datasets using separate training and validation splits than those of the discriminator  $g_{\theta}$  (Table 4). The base models are periodically (every 1,000 iterations) evaluated on the validation split for early stopping, where we stop if there is no improvement on validation perplexity for 10 consecutive evaluations. The base models  $q_{\theta}$  achieve 113.98 for Segmented WikiText, and 110.89 in test set perplexity, respectively. Note that these base models are further fine-tuned on additional datasets in our comparison against residual language models.
We evaluate the relative performance of residual language models against autoregressive models (i.e.
Table 3: Number of sequences in preprocessed datasets (for training and tuning the discriminators  $g_{\theta}$ , and evaluation).
Train	Dev
WikiText	17,556	1,841
Yelp	9,954	1,000
Table 4: Number of sequences in preprocessed datasets (for training and tuning the base model  $q$  ). Note that we do not train our own base models for RealNews, but use one of the pretrained models provided by (Zellers et al., 2019).
fine-tuned base models) on two metrics, log likelihood and perplexity improvement, which are approximated as follows:
- Log likelihood improvement: since  $p, p_{\theta}$  and  $q_{0}$  are all distributions over  $L_{\leq T}$ , we can quantitatively evaluate their difference in log likelihood. We measure the difference between KL  $[p||p_{\theta}]$  and KL  $[p||p_{0}]$ :<sup>30</sup>
$$
\begin{array}{l} \mathrm {K L} [ p | | p _ {\theta} ] - \mathrm {K L} [ p | | p _ {0} ] \\ = \underset {\mathbf {x} \sim p} {\mathbb {E}} \left[ \log p _ {\boldsymbol {\theta}} (\mathbf {x}) - \log p _ {0} (\mathbf {x}) \right] \\ = \underset {\mathbf {x} \sim p} {\mathbb {E}} \left[ \log \tilde {p} _ {\theta} (\mathbf {x}) - \log p _ {0} (\mathbf {x}) \right] - \log Z _ {\theta} \\ = \underset {\mathbf {x} \sim p} {\mathbb {E}} \left[ g _ {\boldsymbol {\theta}} (\mathbf {x}) \right] - \log Z _ {\boldsymbol {\theta}} \\ \approx \frac {\sum_ {\mathbf {x} \in \mathcal {D} _ {\text {t e s t}}} g _ {\boldsymbol {\theta}} (\mathbf {x})}{| \mathcal {D} _ {\text {t e s t}} |} - \log \hat {Z} _ {\boldsymbol {\theta} M}, \tag {7} \\ \end{array}
$$
where  $\hat{Z}_{\theta M}$  is estimated using equation (6). A negative value of log likelihood difference indicates that  $\tilde{q}_{\theta}$  approximates  $p$  better than  $p_0$  in terms of KL-divergence.
- Perplexity improvement: perplexity is a common language modeling metric. Following (Rosenfeld et al., 2001), we compute
$$
\begin{array}{l} \text {p e r p l e x i t y} p _ {\theta} \\ = \frac {\exp \frac {| \mathcal {D} | \log \hat {Z} _ {\boldsymbol {\theta} M}}{w (\mathcal {D} _ {\text {t e s t}})}}{\exp \frac {\sum_ {\mathbf {x} \in \mathcal {D} _ {\text {t e s t}}} g _ {\boldsymbol {\theta}} (\mathbf {x})}{w (\mathcal {D} _ {\text {t e s t}})}}, \tag {8} \\ \end{array}
$$
30Note that  $p_0$  here is the base model component of  $\tilde{p}_{\theta}$ . While comparing between residual language models and autoregressive models, we also finetune  $p_0$  on additional data to get a new model  $q_0'$ , which has different parameters than  $p_0$ .
28https://github.com/huggingface/ transformers
29https://github.com/rowanz/grover
where  $w(\mathcal{D})$  is the total token count of dataset  $\mathcal{D}$ , and  $|\mathcal{D}|$  is the number of sequences of  $\mathcal{D}$ .  $\hat{Z}_{\theta M}$  is computed in Appendix B.5
Both evaluation metrics involve estimating the partition function with  $\hat{Z}_{\theta M}$ . For the perplexity improvement metric, we obtain 32 estimates of  $\hat{Z}_{\theta M}^{31}$ , which are normally distributed, and compute equation (8) using  $\hat{Z}_{\theta M}$  the conservative end of a  $95\%$  confidence level. To account for variance in our test datasets, we further make use of bootstrapping estimation for log likelihood improvement: we bootstrap-sample 1,000 subsamples for each test dataset, and compute equation (7) for each datapoint in the Cartesian product  $(1,000\times 32$  in total). We then report results at the  $2.5\%$  and  $97.5\%$  percentiles.
D.4 Hyperparameters
Transformer experiments. We train our models on 64 GPUs across 8 nodes, with a total batch size of  $64 \times 8 \times 2 = 1,024$ , and with 1 noise sequence ( $K = 1$  in Appendix B.4) per batch. We use an initial learning rate of  $5e - 5$ . The rest of the hyperparameters largely follow settings in (Zellers et al., 2019). Optimization is done with the Grover implementation of AdaFactor.
GRU experiments. We train our models on 8 GPUs on a single node, with a total batch size of  $8 \times 2 = 16$ , and with 25 noise sequences ( $K = 25$  in Appendix B.4) per batch. We have an initial learning rate of  $1e - 4$ . Upon no improvement on validation data, we half the learning rate, with patience  $= 1$ . The model parameters are  $l_{2}$  regularized with a coefficient of  $1e - 5$ . We also apply dropout regularization with  $p = 0.5$ . Optimization is done with PyTorch-supplied Adam.
D.5 Configurations
We study the effects of these configurations:
- Bounding  $g_{\theta}$ : we note in §4.1 that with the strong hypothesis that the base model  $p_0$  has bounded error,  $g_{\theta}$  will have a bounded range, and leads to a much smaller hypothesis space. In this work we experiment with both bounded and unbounded  $g_{\theta}$ 's, with ranges  $(-\infty, 0)$  and  $(-2, 2)$  respectively. More details can be found in Appendix B.3.
- Model capability of  $g_{\theta}$ : we hypothesize that the expressiveness of  $g_{\theta}$  does not need to be as rich as the parametrization of  $p_0$ , since  $g_{\theta}$  essentially only has to tell whether the sequence  $\mathbf{x}$
comes from  $p$  or  $p_0$ . For the GRU + WikiText experiments, we experiment with {1, 2}-layer GRU models of  $g_{\theta}$ . For 1-layer models, we additionally experiment with a setup that has only 250 hidden units. For the Transformers/RealNews dataset, we experiment with {12, 4}-layer Transformer models.
D.6 Log likelihood improvements under different configurations
We also see in Table 5 that using tanh as the activation function  $f$  does better than softmax for Transformers; but performs very poorly for GRUs. We also observe degeneracy problems. We speculate that our Transformer-based base models  $q_{\theta}$  have already learned a good approximation of the true distribution; and limiting the model capacity of  $g_{\theta}$  in exchange of smaller variance results in a favorable trade-off, and vice versa for GRUs. Regarding discriminator capability: we see that performance is not sensitive to model size. Our best Transformers run actually is from the smaller-model runs. And the 1-layer 500-unit GRU models achieve best performance. Overall, results in Table 5 suggests that performance is sensitive to the choice of model configuration.
31We set  $M = 512$  in this paper.
Model Size	Activation	log likelihood improvement
95% CI	μ
RealNews (Transformers)
12-layer	softplus	(-0.13, 0.08)	-0.09
12-layer	tanh	(-0.14, -0.10)	-0.12
4-layer	softplus	(-0.15, 2.62)	-0.02
4-layer	tanh	(-0.18, -0.13)	-0.16
WikiText (GRUs)
2-layer / 500	tanh	(-0.00, 0.00)	-0.00
2-layer / 500	softplus	(-1.32, -0.85)	-1.18
1-layer / 500	tanh	(-0.79, -0.64)	-0.71
1-layer / 500	softplus	(-1.85, -1.54)	-1.69
1-layer / 250	tanh	(-0.02, 0.02)	-0.00
1-layer / 250	softplus	(-1.85, -1.46)	-1.67
Yelp (GRUs)
2-layer / 500	tanh	(-0.03, 0.01)	-0.02
2-layer / 500	softplus	(-1.89, -1.67)	-1.80
1-layer / 500	tanh	(-0.65, -0.57)	-0.61
1-layer / 500	softplus	(-2.62, -2.03)	-2.43
1-layer / 250	tanh	(-0.00, 0.00)	-0.00
1-layer / 250	softplus	(-2.25, -1.99)	-2.13
Table 5: Comparison of different configurations.