text: 442,43,1353066623045799939,818436810,Ehsan Hosseini-Asl,"New paper on, - Analyzing calibration in NLU models - How to improve it using noise contrastive estimation training Accepted to EACL 2021 (<LINK>) @SFResearch [1/6] <LINK> We explore joint energy-based model (EBM) training during the finetuning of pretrained text encoders (e.g., Roberta) for natural language understanding (NLU) tasks. [2/6] <LINK> In most tasks, all three EBM variants get substantial improvement in ECE with little or no loss in accuracy comparing to the (strong) baseline methods. [3/6] <LINK> We plot how test-set ECE changes during training. It is shown as the training reaches the high-accuracy area, the calibration for baseline model becomes worse, while EBM training is able to reach a better trade-off between accuracy and calibration. [4/6] <LINK> How does the model get better calibration? It is shown that models trained with the hidden and sharp-hidden variants tend to assign more conservative predictions (reflected by higher entropy) for higher energy (less likely) samples. [5/6] <LINK> We suspect this is due to the strong coupling between the energy function and the classification logits. We provide concrete examples here. However, we need to mention that we do not observe this interesting trend (Figure 4) in all datasets (e.g., QNLI) [6/6] <LINK>",https://arxiv.org/abs/2101.06829,"In this work, we explore joint energy-based model (EBM) training during the finetuning of pretrained text encoders (e.g., Roberta) for natural language understanding (NLU) tasks. Our experiments show that EBM training can help the model reach a better calibration that is competitive to strong baselines, with little or no loss in accuracy. We discuss three variants of energy functions (namely scalar, hidden, and sharp-hidden) that can be defined on top of a text encoder, and compare them in experiments. Due to the discreteness of text data, we adopt noise contrastive estimation (NCE) to train the energy-based model. To make NCE training more effective, we train an auto-regressive noise model with the masked language model (MLM) objective. ","Joint Energy-based Model Training for Better Calibrated Natural Language