How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary
  Investigation
Authors: name: Zhongyi Han; name: Guanglin Zhou; name: Rundong He; name: Jindong Wang; name: Xing Xie; name: Tailin Wu; name: Yilong Yin; name: Salman Khan; name: Lina Yao; name: Tongliang Liu; name: Kun Zhang
Published: 2023-12-12 16:48:07

Abstract: In machine learning, generalization against distribution shifts -- where
deployment conditions diverge from the training scenarios -- is crucial,
particularly in fields like climate modeling, biomedicine, and autonomous
driving. The emergence of foundation models, distinguished by their extensive
pretraining and task versatility, has led to an increased interest in their
adaptability to distribution shifts. GPT-4V(ision) acts as the most advanced
publicly accessible multimodal foundation model, with extensive applications
across various domains, including anomaly detection, video understanding, image
generation, and medical diagnosis. However, its robustness against data
distributions remains largely underexplored. Addressing this gap, this study
rigorously evaluates GPT-4V's adaptability and generalization capabilities in
dynamic environments, benchmarking against prominent models like CLIP and
LLaVA. We delve into GPT-4V's zero-shot generalization across 13 diverse
datasets spanning natural, medical, and molecular domains. We further
investigate its adaptability to controlled data perturbations and examine the
efficacy of in-context learning as a tool to enhance its adaptation. Our
findings delineate GPT-4V's capability boundaries in distribution shifts,
shedding light on its strengths and limitations across various scenarios.
Importantly, this investigation contributes to our understanding of how AI
foundation models generalize to distribution shifts, offering pivotal insights
into their adaptability and robustness. Code is publicly available at
https://github.com/jameszhou-gl/gpt-4v-distribution-shift.