Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templates

Authors: yusuke  sakai
adam  nohejl
jiangnan  hang
hidetaka  kamigaito
taro  watanabe
jsts  jnli
jsquad  jcommonsenseqa
mcc  pearson
spearman  acc
em / f1
acc  model
marah  abdin
sam ade jacobs
ahmad  awan
jyoti  aneja
ahmed  awadallah
hany  awadalla
nguyen  bach
amit  bahree
arash  bakhtiari
jian- min  bao
harkirat  behl
alon  benhaim
misha  bilenko
johan  bjorck
sébastien  bubeck
qin  cai
martin  cai
caio  césar
teodoro  mendes
weizhu  chen
vishrav  chaudhary
dongdong  chen
yen-chun  chen
yi-ling  chen
parul  chopra
xiyang  dai
allie del giorno
gustavo  de rosa
matthew  dixon
ronen  eldan
victor  fragoso
dan  iter
mei  gao
min  gao
jianfeng  gao
amit  garg
abhishek  goswami
suriya  gunasekar
emman  haider
junheng  hao
russell j hewett
jamie  huynh
mojan  javaheripi
xin  jin
piero  kauff- mann
nikos  karampatziakis
dongwoo  kim
ma- houd  khademi
lev  kurilenko
james r lee
yin tat lee
yuanzhi  li
yunsheng  li
chen  liang
lars  li- den
ce  liu
mengchen  liu
weishung  liu
eric  lin
zeqi  lin
chong  luo
piyush  madan
matt  mazzola
arindam  mitra
hardik  modi
anh  nguyen
brandon  norick
barun  patra
daniel  perez-becker
thomas  portet
reid  pryzant
heyang  qin
marko  radmi- lac
corby  rosset
sambudha  roy
olatunji  ruwase
olli  saarikivi
amin  saied
adil  salim
michael  san- tacroce
shital  shah
ning  shang
hiteshi  sharma
jiahang  xu
weijian  xu
sonali  xu
fan  yadav
jianwei  yang
ziyi  yang
yifan  yang
donghan  yang
lu  yu
chengruidong  yuan
cyril  zhang
jian- wen  zhang
  zhang
lyna  li
yi  zhang
yue  zhang
yunan  zhang
xiren  zhang
  zhou
  phi
josh  openai
steven  achiam
sandhini  adler
lama  agarwal
ilge  ahmad
florencia  akkaya
diogo  leoni
janko  almeida
sam  altenschmidt
shyamal  alt- man
red  anadkat
igor  avila
suchir  babuschkin
valerie  balaji
paul  balcom
haim- ing  baltescu
mohammad  bao
jeff  bavarian
ir- wan  belgum
jake  bello
gabriel  berdine
christopher  bernadett-shapiro
lenny  berner
oleg  bogdonoff
madelaine  boiko
anna-luisa  boyd
greg  brakman
tim  brock- man
miles  brooks
kevin  brundage
trevor  button
rosie  cai
andrew  campbell
brittany  cann
chelsea  carey
rory  carlson
brooke  carmichael
che  chan
fotis  chang
derek  chantzis
sully  chen
ruby  chen
jason  chen
mark  chen
ben  chen
chester  chess
casey  cho
won  chu
dave  chung
jeremiah  cummings
yunxing  currier
  dai
tarun  goel
gabriel  gogineni
rapha  goh
jonathan  gontijo- lopes
morgan  gordon
scott  grafstein
ryan  gray
joshua  greene
shane  gross
yufei  gu
chris  guo
jesse  hallacy
jeff  han
yuchen  harris
mike  he
johannes  heaton
chris  heidecke
alan  hesse
wade  hickey
peter  hickey
brandon  hoeschele
kenny  houghton
shengli  hsu
xin  hu
joost  hu
shantanu  huizinga
shawn  jain
joanne  jain
angela  jang
roger  jiang
haozhun  jiang
denny  jin
shino  jin
billie  jomoto
hee- woo  jonn
tomer  jun
łukasz  kaftan
ali  kaiser
ingmar  ka- mali
  kanitscheider
shirish  nitish
tabarak  keskar
logan  khan
jong wook kilpatrick
christina  kim
yongjik  kim
jan hendrik kim
jamie  kirch- ner
matt  kiros
daniel  knight
łukasz  kokotajlo
andrew  kondraciuk
aris  kondrich
kyle  kon- stantinidis
gretchen  kosic
vishal  krueger
michael  kuo
ikai  lampe
teddy  lan
jan  lee
jade  leike
daniel  leung
ming  levy
rachel  li
molly  lim
stephanie  lin
mateusz  lin
theresa  litwin
ryan  lopez
patricia  lowe
anna  lue
kim  makanju
sam  malfacini
todor  manning
yaniv  markov
bianca  markovski
katie  martin
andrew  mayer
bob  mayne
scott mayer mcgrew
christine  mckinney
paul  mcleavey
jake  mcmillan
david  mcneil
aalok  medina
jacob  mehta
luke  menick
andrey  metz
pamela  mishchenko
vinnie  mishkin
evan  monaco
daniel  morikawa
tong  mossing
mira  mu
oleg  murati
david  murk
ashvin  mély
reiichiro  nair
rajeev  nakano
arvind  nayak
richard  neelakantan
hyeonwoo  ngo
long  noh
cullen  ouyang
jakub  o'keefe
alex  pachocki
joe  paino
ashley  palermo
giambat- tista  pantuliano
joel  parascandolo
emy  parish
alex  parparita
mikhail  passos
andrew  pavlov
adam  peng
filipe  perel- man
belbute  de avila
michael  peres
henrique  petrov
  ponde
oliveira  pinto
michelle  pokrass
vitchyr h pong
tolly  pow- ell
alethea  power
boris  power
elizabeth  proehl
raul  puri
alec  radford
jack w rae
aditya  ramesh
cameron  raymond
francis  real
kendra  rimbach
carl  ross
bob  rotsted
henri  roussez
nick  ry- der
mario  saltarelli
ted  sanders
shibani  santurkar
girish  sastry
heather  schmidt
david  schnurr
john  schulman
daniel  selsam
kyla  sheppard
toki  sherbakov
jessica  shieh
sarah  shoker
pranav  shyam
szymon  sidor
eric  sigler
maddie  simens
jordan  sitkin
katarina  slama
ian  sohl
benjamin  sokolowsky
yang  song
natalie  staudacher
c j weinmann
akila  welihinda
peter  welinder
ji- ayi  weng
lilian  weng
matt  wiethoff
dave  willner
clemens  winter
samuel  wolrich
hannah  wong
lauren  workman
sherwin  wu
jeff  wu
michael  wu
kai  xiao
tao  xu
sarah  yoo
kevin  yu
qim- ing  yuan
wojciech  zaremba
rowan  zellers
chong  zhang
marvin  zhang
shengjia  zhao
gemini  team
rohan  anil
sebastian  borgeaud
yonghui  wu
jean-baptiste  alayrac
jiahui  yu
radu  soricut
johan  schalkwyk
andrew m dai
anja  hauth
katie  millican
david  silver
slav  petrov
melvin  johnson
ioannis  antonoglou
julian  schrit- twieser
amelia  glaese
jilin  chen
emily  pitler
timothy  lillicrap
angeliki  lazaridou
orhan  fi- rat
james  molloy
michael  isard
paul r barham
tom  hennigan
benjamin  lee
fabio  viola
malcolm  reynolds
yuanzhong  xu
ryan  doherty
eli  collins
clemens  meyer
eliza  rutherford
erica  moreira
kareem  ayoub
megha  goel
george  tucker
en- rique  piqueras
maxim  krikun
iain  barr
nikolay  savinov
ivo  danihelka
becca  roelofs
anaïs  white
anders  andreassen
tamara  von glehn
lakshman  yagati
mehran  kazemi
lucas  gonzalez
misha  khalman
jakub  sygnowski
alexandre  frechette
charlotte  smith
laura  culp
lev  proleev
yi  luan
xi- anghai  chen
james  lottes
nathan  schucher
federico  lebron
alban  rrustemi
natalie  clay
phil  crone
tomas  kocisky
jeffrey  zhao
bartek  perz
dian  yu
heidi  howard
adam  bloniarz
han  lu
laurent  sifre
marcello  maggioni
fred  alcober
dan  garrette
megan  barnes
shantanu  thakoor
ja- cob  austin
gabriel  barth-maron
william  wong
rishabh  joshi
rahma  chaabouni
deeni  fatiha
arun  ahuja
ruibo  liu
yunxuan  li
sarah  cogan
jeremy  chen
chao  jia
chenjie  gu
qiao  zhang
jordan  grimstad
jakse  hartman
martin  chad- wick
gaurav  singh tomar
xavier  garcia
evan  senter
emanuel  taropa
thanumalayan  sankara- narayana pillai
jacob  devlin
michael  laskin
diego  de
las  casas
dasha  valter
connie  tao
lorenzo  blanco
adrià  puigdomènech badia
david  reitter
mianna  chen
jenny  brennan
clara  rivera
sergey  brin
shariq  iqbal
gabriela  surita
jane  labanowski
abhi  rao
stephanie  winkler
emilio  parisotto
yim- ing  gu
kate  olszewska
yujing  zhang
ravi  ad- danki
antoine  miech
annie  louis
laurent  el
denis  teplyashin
geoff  brown
elliot  catt
nithya  attaluri
jan  balaguer
jackie  xiang
pi- dong  wang
zoe  ashwood
anton  briukhov
al- bert  webson
sanjay  ganapathy
smit  sanghavi
ajay  kannan
ming-wei  chang
axel  stjerngren
josip  djolonga
yuting  sun
ankur  bapna
matthew  aitchison
pedram  pejman
henryk  michalewski
tianhe  yu
cindy  wang
juliette  love
junwhan  ahn
dawn  bloxwich
kehang  han
peter  humphreys
thibault  sellam
james  bradbury
varun  godbole
sina  samangooei
bogdan  damoc
alex  kaskasoli
sébastien m r arnold
vijay  vasudevan
shubham  agrawal
jason  riesa
dmitry  lepikhin
richard  tan- burn
srivatsan  srinivasan
hyeontaek  lim
sarah  hodkinson
johan  ferret
steven  hand
ankush  garg
tom  le paine
jian  li
yu- jia  li
minh  giang
alexander  neitz
zaheer  abbas
sarah  york
machel  reid
elizabeth  cole
aakanksha  chowdhery
dipanjan  das
dominika  rogozińska
vitaly  nikolaev
pablo  sprechmann
zachary  nado
lukas  zilka
flavien  prost
luheng  he
marianne  monteiro
gaurav  mishra
chris  welty
josh  newlan
dawei  jia
miltiadis  allamanis
clara huiyi hu
raoul  de liedekerke
justin  gilmer
carl  saroufim
shruti  rijhwani
shaobo  hou
disha  shrivastava
anirudh  baddepudi
alex  goldin
adnan  ozturel
albin  cassirer
yunhan  xu
daniel  sohn
deven- dra  sachan
reinald kim amplayo
craig  swan- son
dessie  petrova
shashi  narayan
arthur  guez
siddhartha  brahma
jessica  landon
miteyan  patel
ruizhe  zhao
kevin  villela
luyu  wang
wenhao  jia
matthew  rahtz
mai  giménez
legg  yeung
hanzhao  lin
james  keeling
petko  georgiev
di- ana  mincu
boxi  wu
salem  haykal
rachel  sapu- tro
kiran  vodrahalli
james  qin
zeynep  cankara
abhanshu  sharma
nick  fernando
will  hawkins
behnam  neyshabur
solomon  kim
adrian  hut- ter
priyanka  agrawal
alex  castro-ros
george  van den driessche
tao  wang
fan  yang
yiin  shuo
paul  chang
ross  komarek
mario  mcilroy
guodong  lučić
wael  zhang
michael  farhan
paul  sharman
paul  natsev
yong  michel
yamini  cheng
siyuan  bansal
kris  qiao
siamak  cao
christina  shakeri
justin  butterfield
paul kishan chung
shivani  rubenstein
arthur  agrawal
kedar  mensch
karel  soparkar
timothy  lenc
aedan  chung
loren  pope
jackie  maggiore
priya  kay
shibo  jhakra
joshua  wang
mary  maynez
taylor  phuong
andrea  tobin
maja  tacchetti
kevin  trebacz
yash  robinson
sebastian  katariya
paige  riedel
ke- fan  bailey
nimesh  xiao
lora  ghelani
ambrose  aroyo
neil  slone
xuehan  houlsby
zhen  xiong
elena  yang
jonas  gribovskaya
mateo  adler
lisa  wirth
thais  lee
jay  kagohara
so- phie  pavagadhi
anna  bridgers
sanjay  bortsova
zafarali  ghemawat
tianqi  ahmed
richard  liu
vijay  powell
mariko  bolina
polina  iinuma
james  zablotskaia
da-woon  besley
timothy  chung
ramona  dozat
xiance  comanescu
jeremy  si
guolong  greer
martin  su
raphaël lopez polacek
simon  kaufman
hexiang  tokumine
elena  hu
yingjie  buchatskaya
mohamed  miao
aditya  elhawaty
nenad  siddhant
jinwei  tomasev
christina  xing
helen  greer
shereen  miller
aurko  ashraf
zizhao  roy
ada  zhang
angelos  ma
milos  filos
rory  besta
ted  blevins
chih-kuan  kli- menko
soravit  yeh
jiaqi  changpinyo
oscar  mu
mantas  chang
carrie  pajarskas
vered  muir
charline  cohen
krishna  le lan
amit  haridasan
steven  marathe
sholto  hansen
ra- jkumar  douglas
mingqiu  samuel
sophia  wang
chang  austin
jiepu  lan
justin  jiang
jaime alonso chiu
lars lowe lorenzo
sébastien  sjösund
zach  cevey
thi  gleicher
anudhyan  avrahami
hansa  boral
vittorio  srinivasan
rhys  selo
kon- stantinos  may
léonard  aisopos
  hussenot
baldini  livio
kate  soares
michael b baumli
re- casens  chang
ben  caine
alexander  pritzel
filip  pavetic
fabio  pardo
anita  gergely
justin  frye
vinay  ramasesh
dan  horgan
kartikeya  badola
nora  kassner
subhrajit  roy
ethan  dyer
alex  tomala
yunhao  tang
dalia el badawy
elspeth  white
basil  mustafa
oran  lang
ab- hishek  jindal
sharad  vikram
zhitao  gong
sergi  caelles
ross  hemsley
gregory  thornton
fangxi- aoyu  feng
wojciech  stokowiec
ce  zheng
phoebe  thacker
çaglar  ünlü
zhishuai  zhang
moham- mad  saleh
james  svensson
max  bileschi
piyush  patil
ankesh  anand
roman  ring
katerina  tsihlas
arpi  vezer
marco  selvi
toby  shevlane
mikel  ro- driguez
tom  kwiatkowski
samira  daruki
keran  rong
allan  dafoe
nicholas  fitzgerald
keren  gu-lemberg
mina  khan
lisa anne hendricks
marie  pellat
vladimir  feinberg
james  cobon- kerr
tara  sainath
maribeth  rauh
hadi  sayed
richard  hashemi
yana  ives
yaguang  hasson
eric  li
yuan  noland
nathan  cao
le  byrd
qingze  hou
thibault  wang
michela  sottiaux
jean-baptiste  paganini
alexandre  lespiau
samer  moufarek
kaushik  hassan
joost  shivakumar
amol  van amers- foort
pratik  mandhane
anirudh  joshi
matthew  goyal
andrew  tung
hannah  brock
vedant  shea- han
cheng  misra
nemanja  li
mostafa  rakićević
fangyu  dehghani
sid  liu
junhyuk  mittal
seb  oh
eren  noury
fantine  sezener
matthew  huot
nicola  lamm
charlie  de cao
gamaleldin  chen
ed  elsayed
mahdis  chi
ian  mahdieh
nan  tenney
ivan  hua
patrick  petrychenko
dylan  kane
rishub  scand- inaro
jonathan  jain
romina  uesato
adam  datta
oskar  sadovsky
dominik  bunyan
shimu  rabiej
john  wu
gautam  zhang
edouard  vasudevan
mahmoud  leurent
ionut  alnahlawi
nan  georgescu
ivy  wei
betty  zheng
pam g chan
piotr  rabinovitch
ye  stanczyk
david  zhang
subhajit  steiner
michael  naskar
matthew  azzam
adam  johnson
chung-cheng  paszke
jaume  chiu
elias  sanchez
afroz  mohiuddin
faizan  muhammad
jin  miao
andrew  lee
nino  vieillard
sahitya  potluri
jane  park
elnaz  davoodi
jiageng  zhang
jeff  stanway
drew  garmon
abhijit  karmarkar
zhe  dong
jong  lee
aviral  kumar
luowei  zhou
jonathan  evens
william  isaac
zhe  chen
johnson  jia
anselm  levskaya
zhenkai  zhu
chris  gorgolewski
peter  grabowski
yu  mao
alberto  magni
kaisheng  yao
javier  snaider
norman  casagrande
paul  sugan- than
evan  palmer
geoffrey  irving
edward  loper
manaal  faruqui
isha  arkatkar
nanxin  chen
izhak  shafran
michael  fink
alfonso  castaño
irene  gian- noumis
wooyeol  kim
mikołaj  rybiński
ashwin  sreevatsa
jennifer  prendki
david  soergel
adrian  goedeckemeyer
willi  gierke
mohsen  jafari
meenu  gaba
jeremy  wiesner
diana  gage
yawen  wei
harsha  vashisht
yana  kulizhskaya
jay  hoover
maigo  le
lu  li
chimezie  iwuanyanwu
lu  liu
kevin  ramirez
andrey  khorlin
albert  cui
tian  lin
marin  georgiev
marcus  wu
ricardo  aguilar
keith  pallo
abhishek  chakladar
alena  repina
xi- hui  wu
tom  van der weide
priya  ponnapalli
car- oline  kaplan
jiri  simsa
shuangfeng  li
olivier  dousse
jeff  piper
nathan  ie
minnie  lui
rama  pasumarthi
nathan  lintz
vi- jayakumar  anitha
lam  nguyen thiet
daniel  andor
pedro  valenzuela
cosmin  paduraru
daiyi  peng
kather- ine  lee
shuyuan  zhang
somer  greene
dung  nguyen
paula  kurylowicz
sarmishta  velury
se- bastian  krause
cassidy  hardin
lucas  dixon
lili  janzer
kiam  choo
ziqiang  feng
biao  zhang
achintya  singhal
tejasi  latkar
mingyang  zhang
quoc  le
elena allica abellan
dayou  du
dan  mck- innon
natasha  antropova
tolga  bolukbasi
orgad  keller
david  reid
daniel  finchelstein
maria abi raad
remi  crocker
peter  hawkins
robert  dadashi
colin  gaffney
sid  lall
ken  franko
egor  filonov
anna  bulanova
rémi  leblond
vikas  yadav
shirley  chung
harry  askham
luis c cobo
kelvin  xu
felix  fischer
jun  xu
christina  sorokin
chris  al- berti
chu-cheng  lin
colin  evans
hao  zhou
alek  dimitriev
hannah  forbes
dylan  banarse
zora  tung
jeremiah  liu
mark  omernick
colton  bishop
chintu  kumar
rachel  sterneck
ryan  foley
rohan  jain
swaroop  mishra
jiawei  xia
taylor  bos
ge- offrey  cideron
ehsan  amid
francesco  piccinno
xingyu  wang
praseem  banzal
petru  gurita
hila  noga
premal  shah
daniel j mankowitz
alex  polozov
nate  kushman
victoria  krakovna
sasha  brown
mohammadhossein  bateni
dennis  duan
vlad  firoiu
meghana  thotakuri
tom  natan
an- had  mohananey
matthieu  geist
sidharth  mudgal
sertan  girgin
hui  li
jiayu  ye
ofir  roval
reiko  tojo
michael  kwong
james  lee-thorp
christo- pher  yew
quan  yuan
sumit  bagri
danila  sinopal- nikov
sabela  ramos
john  mellor
abhishek  sharma
aliaksei  severyn
jonathan  lai
kathy  wu
heng- tze  cheng
david  miller
nicolas  sonnerat
denis  vnukov
rory  greig
jennifer  beattie
emily  cave- ness
libin  bai
julian  eisenschlos
alex  korchem- niy
tomy  tsai
mimi  jasarevic
weize  kong
phuong  dao
zeyu  zheng
frederick  liu
rui  zhu
mark  geller
huey  teh
jason  sanmiya
evgeny  gladchenko
nejc  trdin
andrei  sozanschi
daniel  toyama
evan  rosen
sasan  tavakkol
lint- ing  xue
chen  elkind
oliver  woodman
john  car- penter
george  papamakarios
rupert  kemp
sushant  kafle
tanya  grunina
rishika  sinha
alice  tal- bert
abhimanyu  goyal
diane  wu
denese  owusu- afriyie
cosmo  du
chloe  thornton
jordi  pont- tuset
pradyumna  narayana
jing  li
sabaer  fatehi
john  wieting
omar  ajmeri
benigno  uria
tao  zhu
kyle  levin
raphael  hoffmann
dan  holtmann- rice
olivier  bachem
summer  yue
sho  arora
eric  malmi
daniil  mirylenka
qijun  tan
christy  koh
hassas  soheil
siim  yeganeh
steven  põder
francesco  zheng
mukarram  pongetti
yan- hua  tariq
lucian  sun
mojtaba  ionita
pouya  seyedhosseini
ragha  tafti
zhiyu  kotikalapudi
an- mol  liu
jasmine  gulati
xinyu  liu
bart  ye
lily  chrzaszcz
nikhil  wang
tianrun  sethi
ben  li
shreya  brown
wei  singh
aaron  fan
joe  parisi
chenkai  stanton
vinod  kuang
christopher a koverkathu
yunjie  choquette-choo
t j li
abe  lu
prakash  ittycheriah
pei  shroff
mani  sun
  varadarajan
ba- hargam  sanaz
rob  willoughby
david  gaddy
ishita  das- gupta
guillaume  desjardins
marco  cornero
brona  robenek
bhavishya  mittal
ben  albrecht
ashish  shenoy
fedor  moiseev
henrik  jacobsson
alireza  ghaffarkhah
morgane  rivière
alanna  walton
clé- ment  crepy
alicia  parrish
yuan  liu
zongwei  zhou
clement  farabet
carey  radebaugh
praveen  srinivasan
claudia  van der salm
andreas  fidje- land
salvatore  scellato
eri  latorre-chimoto
hanna  klimczak-plucińska
david  bridson
dario  de ce- sare
tom  hudson
piermaria  mendolicchio
lexi  walker
alex  morris
ivo  penchev
matthew  mauger
alexey  guseynov
alison  reid
seth  odoom
lucia  loher
victor  cotruta
madhavi  yenugula
dominik  grewe
anastasia  petrushkina
tom  duerig
antonio  sanchez
steve  yadlowsky
amy  shen
amir  glober- son
adam  kurzrok
lynette  webb
sahil  dua
dong  li
preethi  lahoti
surya  bhupatiraju
dan  hurt
ha- roon  qureshi
ananth  agarwal
tomer  shani
matan  eyal
anuj  khare
rammohan  shreyas
lei  belle
chetan  wang
sanjay  tekur
jinliang  kale
ruoxin  wei
brennan  sang
tyler  saeta
yi  liechty
yao  sun
stephan  zhao
pandu  lee
doug  nayak
manish  fritz
john  reddy vuyyuru
nidhi  aslanides
martin  vyas
xiao  wicke
taylan  ma
ev- genii  bilal
daniel  eltyshev
nina  balle
hardie  martin
james  cate
keyvan  manyika
yelin  amiri
xi  kim
kai  xiong
florian  kang
nilesh  luisier
david  tripu- raneni
mandy  madras
austin  guo
oliver  waters
jason  ainslie
han  baldridge
garima  zhang
jakob  pruthi
feng  bauer
ri- ham  yang
jason  mansour
yang  gelman
george  xu
ji  polovets
honglong  liu
warren  cai
emily  sheng
sherjil  xue
adams  ozair
christof  yu
xiaowei  angermueller
weiren  li
ju- lia  wang
emmanouil  wiesinger
yuan  koukoumidis
anand  tian
madhu  iyer
mark  gurumurthy
parashar  goldenson
m k shah
hongkun  blake
anthony  yu
jennimaria  urbanowicz
chrisantha  palomaki
kevin  fer- nando
ken  brooks
harsh  durden
nikola  mehta
elahe  momchev
maria  rahimtoroghi
amit  geor- gaki
sebastian  raul
  ruder
red- shaw  morgan
jinhyuk  lee
komal  jalan
dinghua  li
ginger  perng
blake  hechtman
parker  schuh
milad  nasr
mia  chen
kieran  milan
vladimir  mikulik
trevor  strohman
juliana  franco
tim  green
Venue: EMNLP_2024_Workshop_BlackBoxNLP
Published: 2024-08-20 11:43:47

Abstract: The natural language understanding (NLU) performance of large language models (LLMs) has been evaluated across various tasks and datasets. The existing evaluation methods, however, do not take into account the variance in scores due to differences in prompts, which leads to unfair evaluation and comparison of NLU performance. Moreover, evaluation designed for specific prompts is inappropriate for instruction tuning, which aims to perform well with any prompt. It is therefore necessary to find a way to measure NLU performance in a fair manner, considering score variance between different instruction templates. In this study, we provide English and Japanese cross-lingual datasets for evaluating the NLU performance of LLMs, which include multiple instruction templates for fair evaluation of each task, along with regular expressions to constrain the output format. Furthermore, we propose the Sharpe score as an evaluation metric that takes into account the variance in scores between templates. Comprehensive analysis of English and Japanese LLMs reveals that the high variance among templates has a significant impact on the fair evaluation of LLMs.

Main paper: 
 introduction: 
 Decoder-based large language models (LLMs) have become foundational resources in the field of natural language processing, demonstrating superior natural language understanding (NLU) abilities and high pre-trained knowledge capacity in a wide variety of downstream tasks. Recently, LLMs can produce more human-like responses through instruction tuning (Wei et al., 2022a) , which involves training the LLMs to respond appropriately to user instructions for various tasks.Although LLM performance has been evaluated across various NLU tasks, the evaluation processes lack standardization in terms of prompts and output formats. This lack of standardization leads to differences in evaluation outcomes that cannot be attributed solely to the differences among LLMs. Moreover, the differences in prompts used for evaluation affect the evaluation results in NLU tasks (Zheng et al., 2023; Lu et al., 2022; Pezeshkpour and Hruschka, 2024; Zhao et al., 2021; Hou et al., 2024; Li et al., 2024; Sclar et al., 2024; Elazar et al., 2021; Madaan et al., 2024) . In the specific case of instruction tuning, the goal is a prompt-independent generalization, though it is questionable to measure such generalization performance using prompts designed for specific targets.For fair evaluation and comparison of the NLU performance of LLMs, we created benchmark datasets comprising multiple evaluation instruction templates for each NLU task based on the FLAN templates (Wei et al., 2022a) , using five English NLU tasks and their corresponding Japanese tasks based on JGLUE (Kurihara et al., 2022) . Additionally, we proposed a new evaluation metric, the Sharpe score, which accounts for the variance in LLM outputs due to template differences, inspired by the Sharpe ratio (Sharpe, 1966) used in finance to assess investment efficiency.We demonstrated its effectiveness for the evaluation of template-based NLU capability, as well as for analysis of the NLU performance of multiple LLMs in various experimental scenarios, such as zero-shot versus fine-tuning settings and English versus Japanese settings. We examined how factors such as continuous training, instruction tuning, and language-specific knowledge affect knowledgetransfer capability. In order to enforce output generation in line with the expected response format, we accompanied each instruction template with a regular expression of the expected output for each task. The regular expressions are employed in constrained decoding methods as implemented in Outlines (Willard and Louf, 2023) . We experimented with both constrained decoding and greedy decoding, demonstrating that constrained decoding with regular expressions is effective for zero-shot evaluation. Our datasets and evaluation scripts are available at https://github.com/naist-nlp/vite. 
background and related work: 
 The evaluation of the NLU capability of LLMs has mostly been based on benchmark datasets that combine several NLU tasks, such as GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) . Furthermore, NLU datasets that include domainspecific knowledge such as medical, economic, and mathematical knowledge (Jin et al., 2019; Baker et al., 2015; Pal et al., 2022; Shah et al., 2022; Chen et al., 2022; Amini et al., 2019; Hendrycks et al., 2021; Lin et al., 2022; Zhong et al., 2023b; bench authors, 2023; Suzgun et al., 2023; Liang et al., 2023) have been proposed for testing domain specific knowledge in LLMs. These benchmark datasets generally use automatic evaluation metrics, such as accuracy and F1 score.These datasets are typically constructed in a concise format relevant to the particular task, providing only minimal information, such as questions and their answers. Therefore, the standard practices in evaluating LLMs employ instruction templates to make the datasets easy for LLMs to understand the instructions. The data instances are instantiated with instruction templates to yield natural language sentences, from which LLMs infer answers in an autoregressive manner.Benchmark datasets for several languages other than English are available as well. Datasets for Japanese, a language we focus on in this study, include llm-jp-eval1 , JP Language Model Evaluation Harness2 , and Nejumi3 , all of which employ Japanese NLU datasets centered around JGLUE (Kurihara et al., 2022) . The JP Language Model Evaluation Harness uses LLMs as classifiers by combining each question with corresponding answer choices and selecting the one with the lowest perplexity when all choices are ranked. Llm-jp-eval and Nejumi perform automatic evaluation by postprocessing the generated text. In the evaluation used by Nejumi, if an answer cannot be obtained from the generated text, it assigns an arbitrary label, whereas the llm-jp-eval treats it as incorrect4 .However, benchmarks for evaluating LLMs report results using only specific prompts, completely ignoring the performance variance of LLMs caused by different prompts. To mitigate the performance variance of LLMs due to different prompts, some LLMs such as FLAN (Wei et al., 2022a; Chung et al., 2024; Longpre et al., 2023) , WizardLM (Xu et al., 2024) , OpenAssistant (Köpf et al., 2023) , and T0 (Sanh et al., 2022) enhance their generalization capabilities by instruction tuning with diverse templates, enabling robust responses to diverse inputs.Prompt engineering (Wei et al., 2022b; Kojima et al., 2022; Zhong et al., 2023a; Yang et al., 2024; Zhou et al., 2023; Chen et al., 2024; Yao et al., 2023; Chen et al., 2023) has improved downstream task performance by converting input sentences into optimal prompts for LLMs. It focuses, however, on finding the best prompts for particular LLMs, making the engineered prompts unsuitable for evaluating the LLMs' NLU performance considering generalization capability.While these approaches ensure the robustness of inputs, existing evaluation frameworks typically examine only a single template and ignore performance variance across multiple instruction templates. Consequently, to evaluate models' performance while taking into account their generalization ability, we need to find an evaluation method that incorporates variance across multiple instruction templates. 
evaluation method: 
 In our evaluation, we focus on the variance in results caused by differences in templates. To this end, we propose datasets and methods for evaluating the NLU performance of LLMs using multiple instruction templates. We evaluate performance in zero-shot and fine-tuning settings, but omit incontext learning, i.e., few-shot learning, settings. The prior studies (Mosbach et al., 2023; Zhang et al., 2024) have shown that the few-shot setting merely represents the exploration for optimal input prompts, capped by the performance of fine-tuning under the same number of examples.As shown in Table 1 , we employ five English NLU tasks and their corresponding Japanese tasks 5 to Figure 1 : Examples of the dataset creation process for the MNLI task. We manually modified the original FLAN templates for evaluation, as highlighted in green. A regular expression (RE) shown in the purple area is attached to the expected answer format. We translated this template to create the Japanese templates described in Appendix E. evaluate cross-lingual transfer capability and performance of multilingual LLMs. Appendix A provides details of each task and dataset.We created the instruction templates for evaluation based on the FLAN templates (Wei et al., 2022a) by modifying them for the English tasks and then manually translating them into Japanese for the Japanese tasks. These instruction templates consist of structured prompts designed to guide the LLMs in performing specific tasks. Figure 1 shows examples of the dataset creation process for MNLI tasks. For each data instance, MNLI provides pairs of sentences, a premise, and a hypothesis. We then apply each instruction template to these sentence pairs to create natural language sentences to be used as input sequences. The expected output English datasets, ensuring dataset alignment. Therefore, we can capture cross-lingual transfer performance that includes language-specific knowledge as noted by Sakai et al. (2024) .format for answers follows FLAN. We convert the answer labels to conversational text and instruct the LLMs to generate only the corresponding number or letter. We apply this procedure to other tasks to construct the entire benchmark dataset. All instruction templates are shown in Appendix F. Table 1 shows the number of templates and instances in the dataset. Furthermore, regular expressions for the expected answer format accompany each template, e.g., [0-2] in template 0-0 in Figure 1 . By using regular expression-based constrained decoding methods, such as Guidance6 or Outlines7 (Willard and Louf, 2023) , it is possible to ensure generation in the expected format without any post-processing. This allows the outputs to be used directly for evaluation, making the evaluation and comparison between LLMs fairer and simpler.Table 2 shows the LLMs evaluated in our experiments. We report the results for both zero-shot and fine-tuning settings. For the fine-tuning setting, we use QLoRA (Dettmers et al., 2023) 8 to train the LLMs on each dataset. The detailed experimental settings of the parameters are described in Appendix B. We conduct greedy decoding and constrained decoding using regular expressions with Outlines (Willard and Louf, 2023) . In greedy decoding, since the generated text may not follow the expected answer format, we referred to the post-processing method used by Ne- jumi 9 . We evaluate JCoLA and CoLA using accuracy (Acc) and the Matthews correlation coefficient (MCC) (Matthews, 1975) ; JSTS and STS-B using the Pearson and Spearman correlation coefficients; JNLI and MNLI using accuracy; JSQuAD and SQuAD using the exact match (EM) rate and F1 score; and JCommonsenseQA and Common-senseQA using accuracy. These are the standard evaluation methods for each task.-ja-7B lightblue/japanese-mpt-7b ELYZA-Llama-2-7B elyza/ELYZA-japanese-Llama-2-7b ELYZA-Llama-2-7B-inst elyza/ELYZA-japanese-Llama-2-7b-instruct English LLMs Llama-2-7B meta-llama/Llama-2-7b-hf Llama-2-7B-inst meta-llama/Llama-2-7b-chat-hf Llama-2-13B meta-llama/Llama-2-13b-hf Llama-2-13B-inst meta-llama/Llama-2-13b-chat-hfThe detailed post-processing methods and evaluation methods are described in Appendix B. 
experimental results and discussions: 
 Results on the Japanese benchmark dataset are shown in Tables 3 and 4 for the zero-shot and fine-tuning setting, respectively. Similarly, results on the English benchmark dataset are shown in Tables 5 and 6 for the zero-shot and fine-tuning setting, respectively. Note that the results for the English benchmark dataset exclude the Japanese LLMs listed in Table 2 . We will focus on important aspects in the following sections and defer more 9 https://github.com/wandb/llm-jp discussions to Appendix D.Linguistic acceptability In the JCoLA task in Table 3 , even the best-performing LLM has accuracy equal to the chance rate, and MCC score is close to zero, indicating that none of the LLMs can perform the task successfully in the zero-shot setting. Table 5 shows the same tendency in the CoLA task, suggesting that linguistic acceptability judgment is a challenging task in the zero-shot setting. The low performance could be explained by the fact that JCoLA and CoLA employ answer labels annotated by linguists, in which their judgement might differ from non-experts in terms of acceptability since linguists prioritize grammaticality (Hu et al., 2023) . Since LLMs are usually trained on general-domain corpora collected from the web, this difference may have an impact.In terms of zeroshot performance, shown in Table 5 , Llama-2-13Binst achieves high performance on the STS-B task in the English dataset. Furthermore, Table 3 shows that it also achieves high performance on the JSTS task in the Japanese dataset. This suggests that the LLM has a sufficient cross-lingual transfer capability for semantic textual similarity.Reading comprehension From the JSQuAD task results shown in Table 3 , the exact match rate improves after instruction tuning for Weblab-10B, LLM-jp-13B, ELYZA-Llama-2-7B, Llama-2-7B, and Llama-2-13B. However, no improvements are observed for StableLM-ja-7B after instruction tuning. This suggests that the quality of the instruction tuning data is important in the zero-shot setting.Commonsense reasoning In CommonsenseQA and JCommonsenseQA results shown in Table 3 and Table 5 , the Llama-2-7B-inst and Llama-2-13B-inst demonstrate a degree of language-transfer capability, although we would expect certain cultural differences embedded in commonsense knowledge of English and Japanese. However, if we focus at ELYZA-Llama-2-7B-inst10 , we observe a decrease in zero-shot performance compared to Llama-2-7B-inst. Nevertheless, in the results of the fine-tuning setting shown in Llama-2-7B-inst. This suggests that while the model has acquired knowledge through continuous training on Japanese data, it may have forgotten how to utilize it, leading to drop in accuracy in the zero-shot setting. At the same time, as shown in Table 5 , ELYZA-Llama-2-7B and ELYZA-Llama-2-7B-inst achieve higher scores than Llama-2-7B in the zero-shot setting. This indicates that even with continuous training on Japanese data, the knowledge from the previous instruction tuning is preserved to some extent.In the fine-tuning setting shown in Table 6 , Llama2-13B is either the best or second-best model in most cases on the English dataset. Moreover, the pretrained-only model achieves better results than its instruction-tuned version of Llama2-13B-inst. This demonstrates that instruction tuning does not guarantee better evaluation performance on the benchmark datasets, likely because instruction tuning aims to generalize the model for diverse queries.As shown in Table 4 , Llama2-13B achieves the highest or nearly the highest evaluation scores in JSTS, JNLI, and JSQuAD. In JCoLA, Weblab-10B achieves a particularly high score, and in JCom-monsenseQA, StableLM-ja-7B-inst stands out with high scores. Comparison of these results with the results on English datasets suggests that LLMs can handle tasks such as JSTS, JNLI, and JSQuAD by leveraging their cross-lingual transfer capabilities. However, in the case of natural language inference (NLI, represented by MNLI and JNLI in our data), it has been pointed out that models might make predictions based solely on superficial features due to overfitting (Kavumba et al., 2022; McCoy et al., 2019; Wang et al., 2022; Tang et al., 2023; Du et al., 2023) . Thus, further investigation is necessary to justify whether these results are truly due to crosslingual transfer, or not. In JCoLA and JCommonsenseQA, ELYZA-Llama-2-7B-inst, which is the continuously trained model from Llama-2-7B-inst, achieves higher scores compared to Llama-2-7B-inst in both accuracy and MCC in JCoLA, as well as improved scores in JCommonsenseQA. This suggests that continuous training with Japanese data contributes to improvement in language acceptability tasks and commonsense reasoning tasks, and cross-lingual transfer through continuous training is effective.As we can see in Table 4 and Table 6 , the scores in JCoLA and CoLA decrease after instruction tuning for some LLMs. One possible factor is that instruction tuning involves training to improve the models' ability to respond to diverse inputs, enabling them to accept even linguistically incorrect input sentences. As a result, the instruction-tuned LLMs may have become more lenient in their judg-ment of acceptability, leading to errors in this task.In the zero-shot setting shown in Table 3 and Table 5, constrained decoding with regular expressions generally achieves higher performance than greedy decoding. However, in the fine-tuning setting shown in Table 4 and Table 6 , greedy decoding generally achieves higher performance than constrained decoding. Therefore, especially when evaluating the zero-shot setting, it is reasonable to use constrained decoding to eliminate errors due to differences in output formats.Additionally, in Table 3 , we can see that LLMjp-13B-inst shows a significant difference in scores between greedy and constrained decoding. One possible reason for this is the influence of the instruction data, specifically the Jaster 11 dataset created, which is based on the JGLUE datasets. We hypothesize that due to instruction tuning with Jaster, higher generation probabilities are assigned to certain words, which may have worked well with greedy decoding but not with constrained decoding 11 https://github.com/llm-jp/llm-jp-eval (Jain et al., 2024 makes a similar observation about instruction tuning).Adequate Evaluation in the Fine-Tuning Setting?We investigated the number of sentences required for the fine-tuning setting to evaluate the NLU performance of LLMs. Figure 2 shows the evaluation scores when fine-tuning StableLM-ja-7B-inst and ELYZA-Llama-2-7B-inst with 1, 10, 100, 500, 1000, and 10000 sentences12 . Thin lines represent the results of fine-tuning with each number of sentences only once, while thick lines represent the results of repeated fine-tuning with the respective number of sentences to achieve a total of 1000 sentences, e.g., training with 10 sentences 100 times or with 500 sentences 2 times to achieve a total of 1000 sentences.Figure 2 shows that for the four datasets other than JCoLA, the difference in evaluation scores between training with 1000 and 10000 sentences is only marginal. Furthermore, for JSTS, training with 100 sentences repeated 10 times achieves sufficient inference accuracy. For JSQuAD, repeated training with a small number of sentences, such as 1 or 10, improves evaluation scores.The reason why JCoLA does not show the same tendency as the other datasets is unclear. It may be due to the difficulty of the task itself or due to the complexity of the dataset. In conclusion, to adapt the output, we only need to train with a small number of examples. Around 1000 sentences are generally sufficient to fine-tune the model adequately for evaluation of its NLU capabilities.5 Analysis Considering Variance Among Templates 
necessity of evaluation using multiple templates: 
 Figure 3 shows the evaluation results in the finetuning setting with only a single template on the Japanese dataset. The accuracy of each template varies greatly for JNLI and JCommonsenseQA, depending on whether the template's answer format uses letters or numbers. Moreover, in JSTS and JCoLA, certain templates result in lower scores. On the other hand, when constrained decoding is applied, some models and tasks produce more stable outputs. This suggests that while the models can respond to the input sentences, they fail to faithfully follow the correct output format. In other words, although we can observe generalization to some extent when a model is fine-tuned with a single template, the performance often varies due to a mismatch between the trained template and the answer format expected at inference time. Evaluation using a single template should, therefore, be avoided.It is instead necessary to use multiple templates for evaluation and to assess the variance among them in order to measure the generalization performance properly. This finding also confirms the results of studies that employed multiple templates for training (Wei et al., 2022a; Xu et al., 2024; Köpf et al., 2023; Sanh et al., 2022) , suggesting that model generalization and its language transfer performance improve by exposing the model to diverse input formats through the use of multiple templates. 0-0 0-1 1-1 2-0 2-1 3-0 3-1 4-0 5-0 5-1 6-0 6-1 0 0.2 0.4 0.6 0.8 1 0-0 0-1 1-0 1-1 2-0 2-1 3-0 3-1 4-0 4-1 5-0 5-1 6-0 6-1 -1 -0.5 0 0.5 1 0-0 1-0 2-0 3-0 4-0 5-0 6-0 7-0 -1 -0.5 0 0.5 1 0-0 1-0 2-0 3-0 4-0 5-0 6-0 7-0 -1 -0.5 0 0.5 1 0-0 0-1 1-0 1-1 2-0 2-1 3-0 3-1 4-0 4-1 5-0 5-1 6-0 6-1 7-0 7-1 8-0 8-1 0 0.2 0.4 0.6 0.8 1 0-0 1-0 2-0 3-0 4-0 5-0 6-0 7-0 0 0.2 0.4 0.6 0.8 1 0-0 1-0 2-0 3-0 4-0 5-0 6-0 7-0 0 0.2 0.4 0.6 0.8 1 0-0 0-1 1-0 1-1 2-0 2-1 3-0 3-1 4-0 4-1 5-0 5-1 0 0.2 0.4 0.6 0.8 1Performance Variance Among Templates Sharpe score LLMs are expected to provide correct answers to diverse prompts, rather than only responding to specific prompts. Therefore, we propose the Sharpe score, an evaluation metric designed to evaluate both the robustness and accuracy of outputs by considering different instruction templates. The Sharpe score is based on the Sharpe ratio (Sharpe, 1966) , which is used in finance to assess investment efficiency. The Sharpe ratio is used as a measure of the risk-adjusted return of an investment. The Sharpe ratio can be expressed as follows:EQUATIONwhere R p is the return of the portfolio, R f is the risk-free rate, and σ p is the standard deviation of the portfolio return.When applying this concept to our evaluation, the return of the portfolio R p corresponds to the average of the evaluation scores µ score , the riskfree rate R f corresponds to the chance rate, and the standard deviation of the portfolio return σ p corresponds to the standard deviation of the evaluation scores for each template σ score . Since the chance rate is constant for each task, we can ignore it.We define the Sharpe score as follows:EQUATION)where α is a parameter that controls the impact of variance in scores among templates. We add 1 to the denominator as a smoothing term to avoid the zero-division issue. When α is 0, the score is reduced to an average of performance evaluation metrics. When α is 1, the Sharpe score is computed analogously to the Sharpe ratio. For values greater than 1, the variance in results across templates leads to a proportionally larger penalty. The default parameter of α is set to 1.0. The Sharpe score can be applied to any evaluation metric as it adjusts based on the average result while considering variance. The more detail experimental results with the Sharpe score are discussed in Appendix C.Ranking Figure 4 shows the changes in the rankings among the models, using the Sharpe score by incrementing the hyperparameter α from 0 to 2 by steps of 0.1 in the Japanese dataset. Appendix C shows the results for the English dataset sharing a similar tendency. While the mean and variance values are constant for each model, the change in the hyperparameter α reflects the degree of impact of variance, resulting in the final score being underestimated. Moreover, when there are fluctuations in the rankings between models, a model that has moved down in rank might perform well in the overall score but exhibit large variations in scores for each template. This indicates that a model that has moved up in rank can produce more stable outputs.OpenCALM-7B StableLM-ja-7B StableLM-ja-7B-inst PLaMO-13B Weblab-10B Weblab-10B-inst LLM-jp-13B LLM-jp-13B-inst MPT-ja-7B ELYZA-Llama-2-7B ELYZA-Llama-2-7B-inst Llama-2-7B Llama-2-7B-inst Llama-2-13B Llama-2-13B-inst α α α α α α α α α α α α α αIn Figure 4 , we observe that the rankings of the models in JSQuAD and JCommonsenseQA show little change when the parameter α is varied. However, for other datasets such as JNLI, the rankings frequently change with the variation of α, indicating a larger variance in evaluation scores among the templates. These results suggest that, while there is generally a correlation between low variance in evaluation scores among templates and high performance when considering only the average of instruction templates, the trend of improvement in performance and variance does not necessarily align for all tasks. Therefore, it was found that the Sharpe score, which considers variance, is an effective performance evaluation metric. 
conclusion: 
 In this paper, we focused on the variance in the evaluation results of LLMs caused by the variations in instruction templates. We proposed a cross-lingual benchmark dataset based on multiple instruction templates, and reported the evaluation results of models trained on varied data. We also proposed the Sharpe score, which considers the variance in evaluation scores among templates, and demonstrated that it is necessary to consider variance when evaluating LLM performance.Based on a comparison of diverse LLMs using our dataset and an analysis of the results, we focused on the tasks where cross-lingual knowledge is effective and the effectiveness of LLMs created for specific languages such as Japanese. An issue closely related to what we touched upon in Section 4.1, i.e., the catastrophic forgetting due to continuous training and instruction tuning, is already being studied (Wang et al., 2023; Luo et al., 2023; Kotha et al., 2024) , and our dataset may help in analyzing the knowledge and cross-lingual capability of LLMs in more detail. Future LLM development would also benefit from a study verifying the extent of knowledge acquisition and the effects of instruction tuning after different sequences of pre-training, instruction tuning, and continuous training. As a future work, we intend to conduct further analyses and to create a comprehensive evaluation framework for analyzing the NLU capabilities of LLMs by expanding the proposed dataset. 
ethical considerations: 
 Our evaluation templates are based on the FLAN templates, which are released under the Apache License 2.0, allowing modification and redistribution. We have made modifications, including translations, to these templates. While the original templates were created by the authors of FLAN, we have adapted and extended them for our purposes. The extended templates will be released under the same Apache License 2.0. Moreover, we will only be distributing our modified templates and will not distribute any datasets such as JGLUE, ensuring that there are no licensing issues. Other parameters were set to their default values. We used the Transformers (Wolf et al., 2020) , peft (Mangrulkar et al., 2022) , and bitsandbytes (Dettmers et al., 2022) libraries.(MCC). The score range of accuracy is 0 to 1, while the range of MCC is -1 to 1.  
 
 Appendix: This study covered a limited number of tasks, templates, and languages. We conducted a comprehensive validation to demonstrate that evaluation results diverge depending on the variations in instruction templates, highlighting the necessity of evaluations using multiple templates. For the instruction templates used in the evaluation, we utilized the prompt templates from the FLAN dataset, modifying them to create the English evaluation templates and then translating those into the Japanese evaluation templates. In terms of tasks, our study is comprehensive as it covers all the currently accessible tasks in JGLUE, the Japanese standard NLU benchmark dataset, as well as data from comparable English tasks. Although increasing the number of tasks and languages is a direction for future research, obtaining completely aligned data is challenging. Therefore, creating such aligned multilingual datasets and developing evaluation prompt templates for other tasks to increase the number of corresponding tasks will also be future challenges. Moreover, the evaluation prompts were manually created from the FLAN templates. However, a future direction could involve automatically generating evaluation prompts using LLMs such as GPT-4 (OpenAI et al., 2024 ), phi (Abdin et al., 2024) or Gemini (Team et al., 2023) , potentially expanding the range of applicable tasks.In this study, we evaluated a total of 15 types of LLMs, categorized into four types of language models. Due to the rapid development of LLMs, the number of models continues to increase dramatically, making it impractical to include all results in this study. Therefore, we focused our evaluation on selected language models that cover various training procedures and training data. As discussed in Section 4, we conducted a comprehensive investigation into factors such as transfer performance, the impact of instruction-tuning, continuous training for each language, and the number of parameters. Furthermore, the Sharpe score revealed that the stability of outputs varied across models when considering the variance. Consequently, we believe that the number and quality of language models used in this study are sufficient to demonstrate the necessity of considering output stability in the evaluation of LLMs. To accommodate various future language models, one of the directions we are considering is to create leaderboards and other tools.Zero-shot evaluation of language models trained on similar data, such as FLAN-T5 (Chung et al., 2024) and FLACUNA (Ghosal et al., 2023) , would lead to unfair evaluations as discussed in Section 4.1. Therefore, it would not be appropriate to evaluate such models trained on FLAN data using the evaluation instruction templates created in this study. In contrast, in the fine-tuning setting we used, it is possible to conduct a fair evaluation without considering the effects of pre-training or instructiontuning data sources, assuming there was no leakage of test data. While we recommend evaluation after fine-tuning, this approach incurs a high computational cost, and therefore developing a mechanism to evaluate zero-shot performance in such models is also desirable and remains a future challenge due to the higher cost of fine-tuning compared to inference.Other evaluation paradigms The performance of LLMs is broadly evaluated along two axes: human-likeness and NLU capabilities. Zheng et al. (2023) ; Chiang and Lee (2023) ; Li et al. (2023) ; Wang et al. (2024) proposed methods that involve evaluating texts generated by LLMs using other LLMs, such as GPT-4 (OpenAI et al., 2024) . These evaluation methods focus on human-like dialogue capabilities, emphasizing the models' ability to follow given instructions. Although this study focused solely on NLU capabilities, the stability of outputs is also important for human-like dialogue abilities. We believe that the analysis methods used in this study can be applied to these new evaluation paradigms as well.Results Table 8 and 9 show the results considering the variance among templates using the Sharpe score for the fine-tuning setting on Japanese and English datasets, respectively. Note that α is set to 1, and the corresponding raw results in the same settings are shown in Tables 4 and 6 , respectively. Compared to the raw results in Table 4 , the evaluation results adjusted by the Sharpe score in Table 8 result in changes in the model ranking. For example, in JNLI with greedy decoding, ELYZA-Llama-2-7B achieves the best evaluation result after adjusting by the Sharpe score in Table 8 . Similar change in the model rankings occurs in other cases as well when we use the Sharpe score to consider the variance among instruction templates.Ranking on the English dataset Figure 6 shows the changes in the rankings among the models, considering the variance among instruction templates, as the Sharpe score parameter α is incremented from 0 to 2 by steps of 0.1 in the English dataset.In Figure 4 , we observe that the rankings of the models in JSQuAD and JCommonsenseQA show little change when the parameter α is varied. However, for other datasets such as JNLI, the rankings frequently change with the variation of α, indicating a larger variance in evaluation scores among the templates. This tendency is also observed in the English dataset shown in Figure 6 . Specifically, with greedy decoding, evaluation results have a significant variance due to the types of instruction templates in STS-B and MNLI.Based on the comparison of the 13B model group with the 7B model group, it cannot be concluded that an increase in parameters necessarily affects NLU performance. However, if we compare models based solely on the number of parameters within the Llama-2 series, the increase in evaluation scores relative to the increase in parameters is minimal.On the other hand, when comparing PLaMO-13B with StableLM-ja-7B, despite the difference in the number of parameters, StableLM-ja-7B achieves higher performance. This suggests that improvements in NLU performance are more significantly influenced by the training data than by the number of parameters. These results are in line with recent studies (Hoffmann et al., 2024; Xue et al., 2023) that indicate that the quantity of training data is more effective than the number of parameters.When discussing the cross-lingual transfer capability in Sections 4.1 and 4.2, we noted that LLMjp-13B-inst (results in Table 5 ), trained with the instruction-tuning dataset Jaster, which is based on JGLUE, can make certain inferences even in the zero-shot setting through cross-lingual transfer, despite not being trained on the corresponding English data for STS-B and CommonsenseQA. For STS-B, the results are comparable to those discussed for Llama2-13B in Section 4.1, demonstrating similar transfer performance from Japanese to English. For CommonsenseQA, the model could likely make correct inferences because some commonsense knowledge is shared between Japanese and English. This indicates that when NLU tasks are explicitly learned for a specific language, the performance can be transferred to some extent to other languages. It remains a future challenge, however, to identify the domains where cross-lingual transfer is possible.Figure 7 shows examples of the dataset creation process for JNLI tasks. We created Japanese JNLI templates by manually translating the MNLI templates corresponding to the English tasks, as shown in Figure 1 . For instance, JNLI provides pairs of sentences, a premise, and a hypothesis. We then apply each instruction template to these sentence pairs to create natural language sentences to be used as input sequences. The expected output format for answers follows FLAN. We convert the answer labels to conversational text and instruct the LLMs to generate only the corresponding number or letter.The evaluation templates are presented as follows:CoLA and JCoLA in Tables 10 and 11 ; STS-B and JSTS in Tables 12, and 13; MNLI and JNLI in Tables 14, 15 and 16; SQuAD and JSQuAD in Table 17 ; and CommonsenseQA and JCommon-senseQA in Tables 18 and 19 . The elements inside the curly brackets are replaced with questions, sentences, or contexts provided as minimal information for each task. Please refer to each task for the specific elements.JCoLA Constraint 0-0 Sentence: "{sentence}" Would a linguist rate this sentence to be acceptable linguistically?Answer using only the leading number of one of the following options. 0: Yes, 1: No Answer:文: 「{sentence}」 言語学者がこの文を言語学的に受け入れると思い ますか? 次の選択肢から先頭の数字のみを選択して回答し てください。 0: はい、1: いいえ 回答: [0-1] 0-1 Sentence: "{sentence}"Would a linguist rate this sentence to be acceptable linguistically?Answer [ab] 2-0 Test sentence: "{sentence}" Is this test sentence a correct grammatical English sentence?Answer using only the leading number of one of the following options. 0: Yes, 1: No Answer:テスト文: 「{sentence}」 このテスト文は日本語の文法を満たす正しい文で すか? 次の選択肢から先頭の数字のみを選択して回答し てください。 0: はい、1: いいえ 回答: [0-1]2-1 Test sentence: "{sentence}" Is this test sentence a correct grammatical English sentence?Answer using only the leading letter of one of the following options. a: Yes, b: No Answer:テスト文: 「{sentence}」 このテスト文は日本語の文法を満たす正しい文で すか?次の選択肢から先頭の英字のみを選択して回答し てください。 a: はい、b: いいえ 回答:[ab]3-0 Is the following sentence linguistically acceptable? {sentence} Answer using only the leading number of one of the following options. 0: Yes, 1: No Answer: Rate the textual similarity of these two sentences on a scale from 0 to 5, where 0 is "no meaning overlap" and 5 is "means the same thing".次の文は言語学的に受け入れられますか? {sentence} 次の選択肢から先頭の数字のみを選択して回答し てください。 0: はい、1: いいえ 回答: [0-1]Answer on a scale from 0.000 to 5.000 with 0.001 increments. Answer:{sentence1} {sentence2} この2つの文の類似度を0.0から5.0までの スコアで評価してください。なお、0.0を 「意味が重複していない」、5.0を「同じ 意味である」とします。 0.0から5.0までのスコアを0.1刻みで回答し てください。 回答: ([0 -4]\.[0 -9]{3}|5.0) 1-0 {sentence1} {sentence2}On a scale from 0 to 5, where 0 is "no meaning overlap" and 5 is "means the same thing", how closely does the first sentence resemble the second one?Answer on a scale from 0.000 to 5.000 with 0.001 increments. Answer:{sentence1} {sentence2} 0.0か ら5.0ま で の ス コ ア で0.0を 「 意 味 が 重複していない」、5.0を「同じ意味であ る」としたとき、最初の文は二つ目の文 にどれだけ似ていますか？ 0.0から5.0までのスコアを0.1刻みで回答し てください。 回答: ([0 -4]\.[0 -9]{3}|5.0)2-0 Sentence 1: {sentence1} Sentence 2: {sentence2}From 0 to 5 (0="no meaning overlap" and 5="means the same thing"), how similar are the two sentences?Answer on a scale from 0.000 to 5.000 with 0.001 increments. Answer:文1: {sentence1} 文2: {sentence2} 0.0から5.0までのスコアによる評価(0.0=意 味が重複しない、5.0=同じ意味である)に おいて、この二つの文はどれだけ似てい ますか？ 0.0から5.0までのスコアを0.1刻みで回答し てください。 回答:([0 -4]\.[0 -9]{3}|5.0)3-0 How similar are the following two sentences? {sentence1} {sentence2}Give the answer on a scale from 0 -5, where 0 is "not similar at all" and 5 is "means the same thing".Answer on a scale from 0.000 to 5.000 with 0.001 increments. Answer: Return your answer on a scale from 0 to 5, where 0 is "not similar" and 5 is "very similar".次の二つの文はどれだけ似ていますか？ {sentence1} {sentence2} 0.0から5.0までのスコアで評価してくださ い。0.0は「全く似ていない」、5.0は「同 じ 意 味 で あ る 」 を そ れ ぞ れ 表 し て い ま す。 0.0から5.0までのスコアを0.1刻みで回答し てください。 回答: ([0 -4]\.[0 -9]{3}|5.0)Answer on a scale from 0.000 to 5.000 with 0.001 increments. Answer:次の二つの文は同じ内容を表しています か？ {sentence1} {sentence2} あなたの回答を0.0から5.0までのスコアで 評価してください。0.0は「全く似ていな い」、5.0は「とても似ている」をそれぞ れ表しています。 0.0から5.0までのスコアを0.1刻みで回答し てください。 回答: ([0 -4]\.[0 -9]{3}|5.0)5-0 Rate the similarity of the following two sentences on a scale from 0 to 5, where 0 is "no meaning overlap" and 5 is "means the same thing"? {sentence1} {sentence2}Answer on a scale from 0.000 to 5.000 with 0.001 increments. Answer:次の二つの文の類似度を0.0から5.0までの スコアで評価してください。0.0は「意味 に被りがない」、5.0は「同じ意味を表し ている」をそれぞれ表しています。 {sentence1} {sentence2} 0.0から5.0までのスコアを0.1刻みで回答し てください。 回答: ([0 -4]\.[0 -9]{3}|5.0)6-0 On a scale from 0-5, where 0 is "not similar" and 5 is "very similar", how similar is the sentence "{sentence1}" to the sentence "{sentence2}"?Answer on a scale from 0.000 to 5.000 with 0.001 increments. Answer: 0.0から5.0までのスコアで、0.0を「似てい ない」、5.0を「似ている」とするとき、 文「{sentence1}」と文「{sentence2}」はど れだけ似ていますか? 0.0から5.0までのスコアを0.1刻みで回答し てください。 回答:([0 -4]\.[0 -9]{3}|5.0)7-0 How similar are these two sentences, on a scale from 0-5 (0 is "not similar" and 5 is "very similar")?{sentence1} {sentence2}Answer on a scale from 0.000 to 5.000 with 0.001 increments. Answer: .+ 次 の 二 つ の 文 は0.0か ら5.0ま で の ス コ ア (0.0は「似ていない」、5.0は「非常に似 ている」)で、どれだけ似ていますか？ {sentence1} {sentence2} 0.0から5.0までのスコアを0.1刻みで回答し てください。 回答: ([0 -4]\.[0 -9]{3}|5.0)https://github.com/llm-jp/llm-jp-evalhttps://github.com/Stability-AI/ lm-evaluation-harness/tree/jp-stablehttps://wandb.me/nejumiWe confirmed the behavior in the source code.We selected tasks based on the JGLUE(Kurihara et al., 2022) datasets, excluding MARC(Keung et al., 2020) as it is currently unavailable. The JGLUE datasets were created from scratch based on the methodology used for the correspondinghttps://github.com/guidance-ai/guidancehttps://github.com/outlines-dev/outlinesThe performance differences between QLoRA and full fine-tuning are minimal(Dettmers et al., 2023;Liu et al., 2024; Dettmers and Zettlemoyer, 2023).ELYZA-Llama-2-7B is continually trained from Llama-2-7B-inst, and ELYZA-Llama-2-7B-inst is instruction-tuned from ELYZA-Llama-2-7B.For JCoLA and JCommonsenseQA, as the training data is less than 10000 sentences, we report the results using all available training data instead.We excluded MARC(Keung et al., 2020) because it is currently unavailable. 


Decision: The reviewers appreciate the topic of the paper, and particularly reviewer A8j8 considers it a strong contribution to BlackboxNLP. We encourage the authors to take this reviewers comments into account for the camera ready version of the paper.

Reviews: review: paper_summary: paper_summary: The paper investigates the LLMs' sensitivity to prompt variation in NLU. While instruction tuning over diverse prompt templates and tasks can reduce this variability, current evaluation frameworks typically assess models using only a single template, ignoring performance variation across different prompts. Authors’ goal is thus to better evaluate models' generalization capabilities by taking into account performance across multiple instruction templates. For this, a new dataset (containing multiple evaluation instructions) and a new metric (Sharpe score, which accounts for the variability across prompts) are proposed and experimented with in zero-shot and full-fine-tuning (NLU) settings.
strength_weakness: strength_weakness: Fig 1 is not crystal clear: could be improved and more commented, more generally unclear if the generation of mutiple prompt templates is done manually or automatically (for isntance using gpt-4 or anotehr strong model)

Would have been good to see the topline performance of a strong model in zero-shot here (gpt-4o for instance)
questions: questions: Q: in figure 1 i see minor changes in the different evaluation templates (numbers replaced by letters for the option list ofr instance). Could more important changes be considered as well ?
Q: what is Matthews correlation coefficient (MCC) ?
Q: What about generating the multiple tempates automatically (for instance suing GPYT-4) ?
paper_summary: paper_summary: This paper essentially investigates the robustness of evaluation from the perspective of sensitivity to prompt perturbations, i.e., how LLMs' performance scores are sensitive to different prompts. However, the authors spend 6.5 pages discussing the fine-tuning setup, English/Japanese language setup, and even HuggingFace model names (Fig. 2). It's fair to say that putting this information into an appendix would be a more appropriate choice. Overall, I believe this work is currently not suitable for the conference and has much room for improvement.
strength_weakness: strength_weakness: The primary strength of this paper is that it addresses an interesting and important research question. However, the author fails to design effective experiments to answer this question. The critical RQ for this paper lies with the metrics people commonly used to evaluate IT. The first question to address is: how fragile are these traditional accuracy-based metrics, and what is the scale of variance for them? Based on that, we could identify the potential flaws when we do not consider prompt differences. The research community would likely not be overly concerned with whether the model used to prove the previous assumptions is in English, Japanese, or other languages, as long as a reasonable variety is included. Therefore, I would encourage the authors to reorganize the structure of the project, highlighting the exhaustiveness of the prompt perturbation experiments conducted, and validate the new metrics used (based on $\mu$ and $\sigma^2$ in this paper).
questions: questions: - From lines 71 to 74, I couldn't follow the statements about examining the effect of knowledge transfer. Why jump to this? I believe the sentences above are mostly about how you develop new metrics based on various evaluation templates.
 - For line 85, I can't open the anonymous link - https://anonymized_for_review/
(minor one) 
- I think this question is not only restricted to one language. Why did you choose Japanese as the complement to that? Is it due to the convenience of existing Japanese datasets like JGLUE, which you can easily use for experiments?