Symbol tuning improves in-context learning in language models
Authors: name: Jerry Wei, name: Le Hou, name: Andrew Lampinen, name: Xiangning Chen, name: Da Huang, name: Yi Tay, name: Xinyun Chen, name: Yifeng Lu, name: Denny Zhou, name: Tengyu Ma, name: Quoc V. Le

We present symbol tuning - finetuning language models on in-context
input-label pairs where natural language labels (e.g., "positive/negative
sentiment") are replaced with arbitrary symbols (e.g., "foo/bar"). Symbol
tuning leverages the intuition that when a model cannot use instructions or
natural language labels to figure out a task, it must instead do so by learning
the input-label mappings.
  We experiment with symbol tuning across Flan-PaLM models up to 540B
parameters and observe benefits across various settings. First, symbol tuning
boosts performance on unseen in-context learning tasks and is much more robust
to underspecified prompts, such as those without instructions or without
natural language labels. Second, symbol-tuned models are much stronger at
algorithmic reasoning tasks, with up to 18.2% better performance on the List
Functions benchmark and up to 15.3% better performance on the Simple Turing
Concepts benchmark. Finally, symbol-tuned models show large improvements in
following flipped-labels presented in-context, meaning that they are more
capable of using in-context information to override prior semantic knowledge.