Has Your Pretrained Model Improved? A Multi-head Posterior Based
  Approach
Authors: name: Prince Aboagye; name: Yan Zheng; name: Junpeng Wang; name: Uday Singh Saini; name: Xin Dai; name: Michael Yeh; name: Yujie Fan; name: Zhongfang Zhuang; name: Shubham Jain; name: Liang Wang; name: Wei Zhang
Published: 2024-01-02 17:08:26

Abstract: The emergence of pretrained models has significantly impacted from Natural
Language Processing (NLP) and Computer Vision to relational datasets.
Traditionally, these models are assessed through fine-tuned downstream tasks.
However, this raises the question of how to evaluate these models more
efficiently and more effectively. In this study, we explore a novel approach
where we leverage the meta features associated with each entity as a source of
worldly knowledge and employ entity representations from the models. We propose
using the consistency between these representations and the meta features as a
metric for evaluating pretrained models. Our method's effectiveness is
demonstrated across various domains, including models with relational datasets,
large language models and images models.