Dissecting In-Context Learning of Translations in GPTs
Authors: name: Vikas Raunak, name: Hany Hassan Awadalla, name: Arul Menezes

Most of the recent work in leveraging Large Language Models (LLMs) such as
GPT-3 for Machine Translation (MT) has focused on selecting the few-shot
samples for prompting. In this work, we try to better understand the role of
demonstration attributes for the in-context learning of translations through
perturbations of high-quality, in-domain demonstrations. We find that
asymmetric perturbation of the source-target mappings yield vastly different
results. We show that the perturbation of the source side has surprisingly
little impact, while target perturbation can drastically reduce translation
quality, suggesting that it is the output text distribution that provides the
most important learning signal during in-context learning of translations. We
propose a method named Zero-Shot-Context to add this signal automatically in
Zero-Shot prompting. We demonstrate that it improves upon the zero-shot
translation performance of GPT-3, even making it competitive with few-shot
prompted translations.