Does Alignment Tuning Really Break LLMs’ Internal Confidence?

Authors: hongseok  oh
wonseok  hwang
jinze  bai
shuai  bai
yunfei  chu
zeyu  cui
kai  dang
xiaodong  deng
yang  fan
wenbin  ge
yu  han
fei  huang
binyuan  hui
luo  ji
mei  li
junyang  lin
runji  lin
dayiheng  liu
gao  liu
chengqiang  lu
keming  lu
jianxin  ma
rui  men
xingzhang  ren
xuancheng  ren
chuanqi  tan
sinan  tan
jianhong  tu
peng  wang
shijie  wang
wei  wang
shengguang  wu
benfeng  xu
jin  xu
an  yang
hao  yang
jian  yang
shusheng  yang
yang  yao
bowen  yu
hongyi  yuan
zheng  yuan
jianwei  zhang
xingxuan  zhang
yichang  zhang
zhenru  zhang
chang  zhou
jin- gren  zhou
xiaohuan  zhou
peter  clark
isaac  cowhey
oren  etzioni
tushar  khot
ashish  sabharwal
carissa  schoenick
oyvind 2018 tafjord
leo  gao
jonathan  tow
baber  abbasi
stella  biderman
sid  black
anthony  dipofi
charles  foster
laurence  golding
jeffrey  hsu
alain  le noac'h
haonan  li
kyle  mcdonell
niklas  muennighoff
chris  ociepa
jason  phang
laria  reynolds
hailey  schoelkopf
aviya  skowron
lintang  sutawika
eric  tang
an- ish  thite
ben  wang
kevin  wang
andy  zou
dirk  groeneveld
iz  beltagy
pete  walsh
akshita  bha- gia
rodney  kinney
ananya  harsh jha
hamish  ivison
ian  magnusson
yizhong  wang
shane  arora
david  atkinson
russell  authur
khy- athi raghavi chandu
arman  cohan
jennifer  du- mas
yanai  elazar
yuling  gu
jack  hessel
william  merrill
jacob  morrison
niklas  muen- nighoff
aakanksha  naik
crystal  nam
matthew e peters
valentina  pyatkin
abhilasha  ravichander
dustin  schwenk
saurabh  shah
will  smith
emma  strubell
nishant  subramani
mitchell  wortsman
pradeep  dasigi
nathan  lambert
kyle  richardson
albert q jiang
alexandre  sablayrolles
arthur  men- sch
chris  bamford
singh  devendra
diego  chaplot
jeremy  nixon
michael w dusenberry
linchuan  zhang
ghassen  jerfel
dustin 2019 tran
  measuring
josh  openai
steven  achiam
sandhini  adler
lama  agarwal
ilge  ahmad
florencia  akkaya
diogo  leoni
janko  almeida
sam  altenschmidt
shyamal  alt- man
red  anadkat
igor  avila
suchir  babuschkin
valerie  balaji
paul  balcom
haim- ing  baltescu
mohammad  bao
jeff  bavarian
ir- wan  belgum
jake  bello
gabriel  berdine
christopher  bernadett-shapiro
lenny  berner
oleg  bogdonoff
madelaine  boiko
anna-luisa  boyd
greg  brakman
tim  brock- man
miles  brooks
kevin  brundage
trevor  button
rosie  cai
andrew  campbell
brittany  cann
chelsea  carey
rory  carlson
brooke  carmichael
che  chan
fotis  chang
derek  chantzis
sully  chen
ruby  chen
jason  chen
mark  chen
ben  chen
chester  chess
casey  cho
won  chu
dave  chung
jeremiah  cummings
yunxing  currier
  dai
tarun  goel
gabriel  gogineni
rapha  goh
jonathan  gontijo- lopes
morgan  gordon
scott  grafstein
ryan  gray
joshua  greene
shane  gross
yufei  gu
chris  guo
jesse  hallacy
jeff  han
yuchen  harris
mike  he
johannes  heaton
chris  heidecke
alan  hesse
wade  hickey
peter  hickey
brandon  hoeschele
kenny  houghton
shengli  hsu
xin  hu
joost  hu
shantanu  huizinga
shawn  jain
joanne  jain
angela  jang
roger  jiang
haozhun  jiang
denny  jin
shino  jin
billie  jomoto
hee- woo  jonn
tomer  jun
łukasz  kaftan
ali  kaiser
ingmar  ka- mali
  kanitscheider
shirish  nitish
tabarak  keskar
logan  khan
jong wook kilpatrick
christina  kim
yongjik  kim
jan hendrik kim
jamie  kirch- ner
matt  kiros
daniel  knight
łukasz  kokotajlo
andrew  kondraciuk
aris  kondrich
kyle  kon- stantinidis
gretchen  kosic
vishal  krueger
michael  kuo
ikai  lampe
teddy  lan
jan  lee
jade  leike
daniel  leung
ming  levy
rachel  li
molly  lim
stephanie  lin
mateusz  lin
theresa  litwin
ryan  lopez
patricia  lowe
anna  lue
kim  makanju
sam  malfacini
todor  manning
yaniv  markov
bianca  markovski
katie  martin
andrew  mayer
bob  mayne
scott mayer mcgrew
christine  mckinney
paul  mcleavey
jake  mcmillan
david  mcneil
aalok  medina
jacob  mehta
luke  menick
andrey  metz
pamela  mishchenko
vinnie  mishkin
evan  monaco
daniel  morikawa
tong  mossing
mira  mu
oleg  murati
david  murk
ashvin  mély
reiichiro  nair
rajeev  nakano
arvind  nayak
richard  neelakantan
hyeonwoo  ngo
long  noh
cullen  ouyang
jakub  o'keefe
alex  pachocki
joe  paino
ashley  palermo
giambat- tista  pantuliano
joel  parascandolo
emy  parish
alex  parparita
mikhail  passos
andrew  pavlov
adam  peng
filipe  perel- man
belbute  de avila
michael  peres
henrique  petrov
  ponde
oliveira  pinto
michelle  pokrass
vitchyr h pong
tolly  pow- ell
alethea  power
boris  power
elizabeth  proehl
raul  puri
alec  radford
jack  rae
aditya  ramesh
cameron  raymond
francis  real
kendra  rimbach
carl  ross
bob  rotsted
henri  roussez
nick  ry- der
mario  saltarelli
ted  sanders
shibani  santurkar
girish  sastry
heather  schmidt
david  schnurr
john  schulman
daniel  selsam
kyla  sheppard
toki  sherbakov
jessica  shieh
sarah  shoker
pranav  shyam
szymon  sidor
eric  sigler
maddie  simens
jordan  sitkin
katarina  slama
ian  sohl
benjamin  sokolowsky
yang  song
natalie  staudacher
c j weinmann
akila  welihinda
peter  welinder
ji- ayi  weng
lilian  weng
matt  wiethoff
dave  willner
clemens  winter
samuel  wolrich
hannah  wong
lauren  workman
sherwin  wu
jeff  wu
michael  wu
kai  xiao
tao  xu
sarah  yoo
kevin  yu
qim- ing  yuan
wojciech  zaremba
rowan  zellers
chong  zhang
marvin  zhang
shengjia  zhao
gemma  team
thomas  mesnard
cassidy  hardin
robert  dadashi
surya  bhupatiraju
shreya  pathak
laurent  sifre
morgane  rivière
sanjay  kale
juliette  love
pouya  tafti
léonard  hussenot
giuseppe  sessa
aakanksha  chowdhery
adam  roberts
aditya  barua
alex  botev
alex  castro- ros
ambrose  slone
amélie  héliou
andrea  tac- chetti
anna  bulanova
antonia  paterson
beth  tsai
bobak  shahriari
charline le lan
christo- pher a choquette-choo
clément  crepy
daniel  cer
daphne  ippolito
david  reid
elena  buchatskaya
eric  ni
eric  noland
geng  yan
george  tucker
george-christian  muraru
grigory  rozhdestvenskiy
henryk  michalewski
ian  tenney
ivan  grishchenko
jacob  austin
james  keeling
jane  labanowski
jean-baptiste  lespiau
jeff  stanway
jenny  bren- nan
jeremy  chen
johan  ferret
justin  chiu
justin  mao-jones
katherine  lee
kathy  yu
katie  milli- can
lars lowe sjoesund
lisa  lee
lucas  dixon
machel  reid
maciej  mikuła
mateo  wirth
michael  sharman
nikolai  chinaev
nithum  thain
olivier  bachem
oscar  chang
oscar  wahltinez
paige  bai- ley
paul  michel
petko  yotov
rahma  chaabouni
ramona  comanescu
reena  jana
rohan  anil
ross  mcilroy
ruibo  liu
ryan  mullins
samuel l smith
sebastian  borgeaud
sertan  girgin
sholto  douglas
shree  pandya
siamak  shakeri
soham  de
ted  kli- menko
tom  hennigan
vlad  feinberg
wojciech  stokowiec
yu hui chen
zafarali  ahmed
zhitao  gong
tris  warkentin
ludovic  peran
minh  giang
clément  farabet
oriol  vinyals
jeff  dean
koray  kavukcuoglu
demis  hassabis
zoubin  ghahramani
douglas  eck
joelle  barral
fernando  pereira
eli  collins
armand  joulin
noah  fiedel
evan  senter
alek  andreev
kathleen 2024 kenealy
  gemma
Venue: EMNLP_2024_Workshop_BlackBoxNLP
Published: 2024-08-16 03:10:39

Abstract: Large Language Models (LLMs) have shown remarkable progress, but their real-world application necessitates reliable calibration. This study conducts a comprehensive analysis of calibration degradation across four dimensions: models, calibration metrics, tasks, and confidence extraction methods. Initial analysis showed that the relationship between alignment and calibration is not always a trade-off, but under stricter analysis conditions, we found the alignment process consistently harms calibration. This highlights the need for (1) a careful approach when measuring model confidences and calibration errors and (2) future research into algorithms that can help LLMs to achieve both instruction-following and calibration without sacrificing either.

Main paper: 
 introduction: 
 The impressive performance of LLMs has opened up unprecedented possibilities for their application in real-world. Calibration, a measure of model reliability, is crucial in such applications (Pakdaman Naeini et al., 2015; Nixon et al., 2019) . Recently, there have even been cases of legal repercussions for using fabricated case law generated by large language models (LLMs) in legal documents (Weiser and Schweber, 2023) . However, recent studies indicate that the alignment process, a necessary step to improve instruction-following capabilities of LLMs, harms their calibration (Ope-nAI et al., 2024; Zhu et al., 2023) . Prior study (Tian et al., 2023) has explored the calibration of blackbox models with respect to confidence extraction methods and calibration metrics, and they argued that the calibration of LLMs after alignment may not degrade the reliability depending on the confidence extraction method. However, by focusing on the black-box API, they have not fully explored diverse logit-based confidence extraction methods. We extend this work to open-source LLMs, by comprehensively analyzing the calibration degradation across models, metrics, tasks, and confidence extraction methods. 
experiment: 
 We investigate calibration changes across 1) various open LLMs; Llama2 (Touvron et al., 2023) , Llama3 (AI@Meta, 2024), Mistral (Jiang et al., 2023 ), Gemma (Team et al., 2024 ), OLMo (Groeneveld et al., 2024 ), Qwen (Bai et al., 2023) ; 2) different metrics: Expected Calibration Error (ECE) (Pakdaman Naeini et al., 2015) and Static Calibration Error (SCE) (Nixon et al., 2019) ; 3) various tasks: ARC-easy (Clark et al., 2018) , HellaSwag (Zellers et al., 2019) , MedM-CQA (Pal et al., 2022) , MMLU (Hendrycks et al., 2021) and PIQA (Bisk et al., 2020) ; 4) different methods to extract model confidence: continuation-sum, continuation-min, and choice. continuation-sum uses sum of logits for each choice's continuation token sequence given a context to extract confidence of model. We noticed that choices often have similar content (e.g., ["2.2x1011 kg", "2.2x1014kg", "2.2x1020kg", and "2.2x1016 kg"] in MMLU), so we introduce continuation-min, which uses the minimum logit within a continuation sequence, to mitigate the influence of this similarity and em-Figure 2 : The rates of change in calibration metrics between instruction-tuned and pre-trained models. sum, min, and choice correspond to continuation-sum, continuation-min, and choice methods, respectively. Humanities and STEM are from MMLU phasize differences. choice method uses the logit of the token corresponding to the capital letter of each choice in the prompt. This eliminates any overlap in the choices' content. We follow Yang et al. (2024) to design prompt for the choice method when continuation is the default. All tasks were performed in a zero-shot setting using lm-evaluationharness (Gao et al., 2023) with four A6000 GPUs. 
results and analysis: 
 We have identified model-task combinations where the ECE decreased after the alignment process (Figure 1 , left), which contradicts previous studies (OpenAI et al., 2024; Zhu et al., 2023) . Examples of these model-task combinations include OLMO-7B on HellaSwag (blue), Qwen-72B on HellaSwag (orange), and Llama-3-8B on MMLU Humanities (green). However, on the same combinations, when examining the SCE, we observed consistent increases or negligible changes in calibration error. Furthermore, when we changed the method for extracting the model's confidence, the rate of change increased significantly, as depicted in the right side of Figure 1 . Given the consistent increase of SCE with the alignment process in the continuation-sum method, we decided to focus Table 1 : Rank of the change rate of the SCE for the task and models, when extracting confidence using the choice. Higher ranks assigned to larger changes.ARC_Easy, MMLU_STEM,HellaSwag MedMCQA MMLU_Humanities, PIQA 1 OLMo-7B OLMo-7B OLMo-7B 2 Llama-2-7b Llama-2-7b gemma-7b 3 gemma-7b gemma-7b Llama-2-7b 4 Llama-3-8B Llama-3-8B Llama-3-70B 5 Mistral-7B Llama-3-70B Llama-3-8B 6 Llama-3-70B Mistral-7B Mistral-7B 7 Qwen2-72B Qwen2-72B Qwen2-72Bon the SCE for the remainder of our experiments.Next, we analyzed continuation-min method. Unlike the consistent increase of SCE observed in continuation-sum method, as shown in the third graph of Figure 2 , we observed SCE decreases for some combinations in this method. Specifically, the Llama-3-8B model on HellaSwag and PIQA tasks, and the Mistral-7B model on HellaSwag and PIQA tasks, showed SCE changes of -1.25%, -2.67%, -6.08%, and -6.28%, respectively.Finally, we examined the case of extracting model confidence using the choice method. As shown in the fourth graph of Figure 2 , the choice method consistently exhibited positive SCE changes across all tasks and model combinations. This shows that if we combining strict calibration metric and confidence extraction method, calibration, as influenced by the alignment process, is consistently harmed regardless of the task or model. Furthermore, as seen in Table 1 , the ranking of models' change rates remained remarkably robust across tasks. Since the models we considered do not have significant structural differences, we believe that the consistent model ranking in Table 1 is primarily attributable to the training data and algorithms used in the model's alignment process. Further experimental results are available in the following GitHub repository. https: //github.com/abzb1/alingment_calibration 
discussion and conclusion: 
 Our analysis has yielded several insights. Firstly, The calibration of LLMs can vary complexly and diversely depending on the combinations we've examined, so careful analysis is required. Secondly, in the choice-SCE combination, where redundance of choices is minimized and confidence of all choices are considered, all models exhibited positive calibration error change rates across all tasks. This implies that under stricter analysis conditions, the alignment process can still be seen as detrimental to calibration. 
 
 Appendix:  


Decision: Work on the relationship between instruction tuning and calibration, may be of interest to the workshop participants.