SlowGuess/ACL_2020_OCR huggingface dataset:
Robert Tibshirani. 1996. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
Bianca Zadrozny and Charles Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, pages 609-616, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694-699. ACM.
A Details on Hyper-Parameters
All models are trained with equal hyperparameters: learning rate 2e-5, and BERT model size  $\mathrm{BERT}_{BASE}$ . Also, we set up an early stopping rule for train: we track the validation loss for every 50 steps and then halt to train if current validation loss is bigger than the averaged 10 prior validation losses (i.e., patience 10). For L1, we use the regularization weight value 1-e8. For PosCal, we set up another weight value  $\lambda$  for  $\mathcal{L}_{Cal}$ , and the number of updating empirical probability per epoch  $(u)$ . We tune these two hyper-parameters per each task. For more details, see Table 5. As a baseline of post-calibration method, we also report ECE with a temperature scaling (Guo et al., 2017), which is current state-of-the-art method.
xSLUE	u	λ	GLUE	u	λ
GYAFC	5	0.6	CoLA	5	0.2
SPolite	5	0.6	SST-2	10	1.0
SHumor	5	1.0	MRPC	10	1.0
SJoke	5	1.0	QQP	10	1.0
SarcGhosh	5	0.6	MNLI	2	0.2
SARC	5	0.6	MNLImm	2	0.2
SARC-pol	5	1.0	QNLI	1	0.6
VUA	2	1.0	RTE	10	1.0
TroFi	5	1.0	WNLI	2	0.2
CrowdFlower	5	0.6
DailyDialog	5	1.0
HateOffens	5	1.0
SRomance	5	1.0
SentiBank	5	1.0
PASTEL_gender	5	1.0
PASTEL_age	5	1.0
PASTEL_count	5	1.0
PASTEL_polit	5	1.0
PASTEL_educ	5	1.0
PASTEL_ethn	5	1.0
Table 5: Hyper-parameters for PosCal training across tasks : the number of updating empirical probabilities per epoch  $u$  and weight value  $\lambda$  for  ${\mathcal{L}}_{\text{Cal }}$  . We tune them using the validation set.
B Examples When MLE and PosCal Predicts Different Label
Table 6 shows some examples in RTE and Stanford-Politeness datasets with their predicted  $\hat{p}$  of true label from MLE and PosCal.
Data	Sentence	True label	MLE
p	PosCal
p
RTE	(S1) Charles de Gaulle died in 1970 at the age of eighty. He was thus fifty years old when, as an unknown officer recently promoted to the (temporary) rank of brigadier general, he made his famous broadcast from London rejecting the capitulation of France to the Nazis after the debacle of May-June 1940.
(S2) Charles de Gaulle died in 1970.	entail	34.9	58.9
INCOR → COR
(S1) Police in the Lower Austrian town of Amstetten have arrested a 73 year old man who is alleged to have kept his daughter, now aged 42, locked in the cellar of his house in Amstetten since 29th August 1984. The man, identified by police as Josef Fritzl, is alleged to have started sexually abusing his daughter, named as Elisabeth Fritzl, when she was eleven years old, and to have subsequently fathered seven children by her. One of the children, one of a set of twins born in 1996, died of neglect shortly after birth and the body was burned by the father.
(S2) Amstetten is located in Austria.	entail	45.5	57.3
INCOR → COR
(S1) Blair has sympathy for anyone who has lost their lives in Iraq.
(S2) Blair is sorry for anyone who has lost their lives in Iraq.	entail	31.3	50.1
INCOR → COR
(S1) Capital punishment acts as a deterrent.
(S2) Capital punishment is a deterrent to crime.	entail	41.6	64.5
INCOR → COR
(S1) The U.S. handed power on June 30 to Iraq's interim government chosen by the United Nations and Paul Bremer, former governor of Iraq.
(S2) The United Nations officially transferred power to Iraq.	not entail	59.2	44.9
COR → INCOR
Spolite.	I don't know what page you are talking about, as this is your only edit. Did you perhaps have another account?	impolite	47.3	65.4
INCOR → COR
Hi. Not complaining, but why did you remove the category "high schools in california" from this article?	impolite	1.2	91.7
INCOR → COR
Hi, sorry I think I'm missing something here. Why are you adding a red link to the vandalism page?	impolite	5.6	61.9
INCOR → COR
Huh, looks fine to me. Maybe this computer just lies to me to get me to shut up and stop complaining?	impolite	3.3	58.1
INCOR → COR
Can you put an NSLog to make sure it's being called only once? Also, can you show us where you are declaring your int?	polite	16.5	76.5
INCOR → COR
I don't understand the reason for <url>. Would you please explain it to me?	polite	91.5	37.1
COR → INCOR
Another question: Does "Senn" exist in Japanese? If it does, is it possible to render Sennin as Senn-in?	polite	88.8	45.5
COR → INCOR
@Smjg, thanks. But why did you also remove the categories I added?	impolite	78.3	45.7
COR → INCOR
You can place islands so there is no path between points. What should happen then?	impolite	91.7	35.8
COR → INCOR
Table 6: Predicted  $\hat{p}(\%)$  of true label from MLE and PosCal with corresponding sentences in RTE (top) and Stanford's politeness (bottom) dataset. True label is either entail or not entail for RTE, and polite or impolite for SPolite. We show the cases where two methods predict the label differently. The case with INCOR → COR means only PosCal predicts the true label correctly, while the case with COR → INCOR means only MLE predicts the true label correctly.