SlowGuess/ACL_2020_OCR huggingface dataset:
In the process of collecting human annotations (§4), we collect 3,600 scores in total from 185 Amazon MTurk workers (4 scores for each context-response pair). To assess the data's reliability, we use the Krippendorff's  $\alpha$  (Krippendorff, 2018) instead of commonly used Cohen's  $\kappa$  and Fleiss'  $\kappa$ , because Krippendorff's  $\alpha$  can handle 1) an arbitrary number of annotators, 2) various levels of measurement (e.g. nominal, interval), and 3) missing data.
The Krippendorff's  $\alpha$  of the original 3,600 annotations of response appropriateness is 0.431, which is considered not good according to the interpretation of the number in Table 5. Therefore, we decided to remove the outliers to improve the inter-annotator agreement. We detected outliers for each of the 900 four-annotation groups using the median absolute deviation (MAD) method (Leys et al., 2013). By setting the deviation threshold as 1.0, we identified 895 annotations as outliers. On the remaining 2,705 annotations (roughly 1 annotation is removed for each group), the Krippendorff's  $\alpha$  reaches 0.815, which suggests that the data is reliable for the subsequent experiments.
The ADEM and RUBER models use a 2-layer bidirectional gated recurrent unit (BiGRU) sentence encoder with 500 hidden units and a 2-layer BiGRU dialogue encoder with 500 hidden units. The encoders are initialized with the parameters of a pretrained HRED's encoders of the same architecture. To encode speaker information, we concatenate each sentence embedding with a 30-dimensional speaker embedding that indicates whether the sentence's speaker is identical to the response's speaker (Zhao and Kawahara, 2019). Principal component analysis (PCA) is applied to project response and context embeddings into low-dimensional vectors in ADEM. The number of principal components is 50. The RoBERTa evaluator is based on a pretrained RoBERTa-large model, and we finetune the entire model in our experiments.
Table 6 shows the hyper-parameters in unsupervised training and supervised training. Following the original paper, we freeze the ADEM's encoders and only finetune its parameters  $M$  and  $N$ , and thus a larger learning rate is used for ADEM. In all experiments, we decay the learning rate with a 0.1 decay rate when a model's validation loss does not improve and stop training early if the learning rate is less than 1e-7.
C Model Output Distributions
The distribution of human annotation scores on the 900 annotated responses has been given in Figure 1(a). To analyze the distribution of model outputs, we show the distributions of human annotation, ADEM's outputs, RUBER's outputs, and RoBERTa-eval's outputs on the test data of 90 responses in Figure 3. We found that: 1) The distribution of human score is similar to that in Figure 1(a). 2) The proposed RoBERTa evaluator's output has a flatter distribution than human scores. 3) The baseline RUBER and ADEM both have very peaky pseudo-Gaussian distributions whose means are around 3.
D Robustness to Changes in Input and Output
We conduct two sets of experiments to see whether the RoBERTa evaluator's performance would be affected by a slight change in its input and output.
Adding Gaussian Noise to Input. We added Gaussian noise  $(\mu = 0.0)$  to human annotations
Krippendorff's α	Interpretation
<0.67	not good
0.67~0.8	allowing tentative conclusions to be drawn
>0.8	good reliability
Table 5: Interpretation of Krippendorff's  $\alpha$  . (§A)
Hyper-parameter	ADEM	RUBER	RoBERTa-eval
Unsupervised Training
learning rate	1e-4	3e-6
batch size	30	3
epochs	30	2
Supervised Training
learning rate	1e-3	1e-4	3e-6
batch size	30	30	3
epochs	50	50	50
Table 6: Optimization hyper-parameters.
(a) Human
(b) RoBERTa-eval
(c) ADEM
Figure 3: Distributions of human annotations and model outputs on the test data (90 responses).
(d) RUBER
and ran 100 trials with random seeds from 1 to 100. With  $\sigma = 0.1$ , the RoBERTa evaluator's performance doesn't change much (Pearson's correlation from 0.64 to 0.64, Spearman's correlation from 0.66 to 0.65). With  $\sigma = 0.5$ , the performance degrades more (Pearson's correlation from 0.64 to 0.61, Spearman's correlation from 0.66 to 0.62). Considering that  $0.5\sigma$  is high and may skew the original human judgement, we believe the evaluator is not greatly affected by the noise.
Discretizing Output. We also tried discretizing the evaluator's outputs (from [1, 5] to  $\{1,2,3,4,5\}$ ) and observed a minimal improvement (Pearson's correlation from 0.64 to 0.65, Spearman's correlation from 0.66 to 0.66). Generally speaking, there is no dramatic change in the model's performance when we apply these transformations to the output scores. We believe this shows our model to be fairly robust.