Symbol tuning improves in-context learning in language models
Authors: Jerry Wei
Le Hou
Andrew Kyle Lampinen
Xiangning Chen
Da Huang
Yi Tay
Xinyun Chen
Yifeng Lu
Denny Zhou
Tengyu Ma
Quoc V Le
Venue: EMNLP 2023 Main (2023)
Keywords: in-context learning
large language models
natural language processing
TL;DR: Symbol tuning is a simple method that emphasizing input–label mappings to improve robustness to prompts without instructions/relevant labels and boost performance on algorithmic reasoning tasks.

Abstract: We present symbol tuning - finetuning language models on in-context input-label pairs where natural language labels (e.g., "positive/negative sentiment") are replaced with arbitrary symbols (e.g., "foo/bar"). Symbol tuning leverages the intuition that when a model cannot use instructions or natural language labels to figure out a task, it must instead do so by learning the input-label mappings.

We experiment with symbol tuning across PaLM models up to 540B parameters and observe benefits across various settings. First, symbol tuning boosts performance on unseen in-context learning tasks and is much more robust to underspecified prompts, such as those without instructions or without natural language labels. Second, symbol-tuned models are much stronger at algorithmic reasoning tasks, with up to 18.2% better performance on the List Functions benchmark and up to 15.3% better performance on the Simple Turing Concepts benchmark. Finally, symbol-tuned models show large improvements in following flipped-labels presented in-context, meaning that they are more capable of using in-context information to override prior knowledge.

Symbol tuning improves in-context learning in language models
Jerry Wei
Le Hou-
Andrew Lampinen'
Xiangning Chen 1.*
Da Huang
Yi Tay! Xinyun Chen'
Yifeng Lu
Denny Zhou'
Tengyu Ma
Quoe Le
Google
Stanford University
Abstract
prompt engineering (Brown al., 2020; Reynolds and McDonell, 2021) or phrasing tasks as instruc- tions (Wei et al., 2022a; Ouyang et al., 2022; Sanh et al., 2022, inter alia), and they exhibit unexpected behaviors such as performance on tasks being un- affected even when shown exemplars with random labels (Min et al., 2022b) flipped labels (Wei et al., 2023).
We present symbol tuning-finetuning lan- guage models in-context input-label pairs where natural language labels (c.g.. "posi- tive/negative sentiment") are replaced with arbi- trary symbols (e.g.. "foo/bar"). Symbol tuning leverages the intuition that when model can- not use instructions or natural language labels to figure out task, it must instead do so by learning the imput-labe mappings.
In this paper, we propose a simple finetuning procedure that we call symbol tuming, which signif- icantly improves the ability of language models to reason with and learn from input-label mappings presented in-context. In the symbol-tuning proce- dure, finetune language models input-label pairs presented in-context where natural language labels are remapped to arbitrary symbols." The tuition is that when models cannot rely on instruc- tions or relevant natural language labels to figure out a given task, they must instead do so by reason- ing with input-label mappings presented in in-context in order to learn the mappings that reveal the task. We perform symbol tuning using mixture of 22 NLP datasets with various arbitrary symbols la- bels and experiment using instruction-tuned PaLM models "flan-aaLM with several sizes (8B, 62B, 62B-cont, 540B).
We experiment with symbol tuning across PaLM models up 540B parameters and ob- serve benefits across various settings. First, symbol tuning boosts performance on unseen in-context learning tasks and is much more ro- bust to underspecified prompts, such as those without instructions without natural lan- guage labels. Second, symbol-tuned tuned models are much stronger at algorithmic reasoning tasks, with up 18.2% better performance on the List Functions benchmark and up to 15.3% better performance on the Simple Turing Concepts benchmark. Finally, symbol-tuned models show large improvements in following flipped-labed presented in-context, meaning that they are more capable of using in-context information to override prior knowledge.
1 Introduction
A key feature of human intelligence is that humans can learn to perform new tasks by reasoning us- ing only few examples. Scaling up language models has unlocked range of new applications and paradigms in machine learning, including the ability to perform challenging reasoning tasks via few-shot examples given in-context (Brown et al., 2020; Chowdhery et al., 2022; OpenAI, 2023, inter alia). Language models, however, are still sen- sitive to the way that prompts are given, indicat- ing that they are not reasoning in robust manner. For instance, language models often require heavy Work done as Student Researcher at Google. Work done as a Visiting Researcher at Google.
First, symbol tuning improves performance of bascline models unseen in-context learning tasks across various settings (with/without instruc- tions, with/without relevant labels), with larger per- formance gains when instructions or natural lan- guage labels are not given in the prompt. For ex- ample, when prompts do not contain instructions or relevant labels, symbol tuning yields a +11 1.1%% average performance improvement across eleven evaluation tasks for Flan-PaLMM66BB-ooml
We call our method symbol tuning because urbitrary desig- nation is a key property of symbols (Newell and Simon, 1976) and using symbols is a crucial part of intelligence (Newell, 1980; Santoro et al., 2021).

---

Instruction Tuning
Symbol Tuning
In-context exemplars not needed to leam the
Must use in cantext exemplars to leamn the task
Input
Input
Instruction
What sentiment of this?
[None]
Exemplar
This movie great
This is great
Label
Answer: Positive
Answer: Foo
Unrolated
Instruction
What is the sentiment of this?
[None]
Exemplar
Worst film I've
Worst film I've
Label
Answer: Negative
Relevant
Answer: Bar
Unrelated
[more exemplars)
[more exemplars]
Evaluation Example
What the sentiment of this?
[None]
This movie temilee
This movie is ferrible
Answer:
Answer:
Output
Output
Negative
Bar
We tune models tasks where natural language labels replaced with arbitrary symbols (symbol tuning). Symbol tuning relies on the intuition that when instructions and relevant labels are not available, models must use in-context exemplars to learn the task.
Figure 1:
Second, symbol-tuned models are better at algo- rithmic reasoning tasks, striking result since sym- bol tuning only includes natural language data and did not have any numerical algorithmic data. On a set of reasoning evaluation suites for list functions (e.g., remove the last element a list), symbol- tuned models experience performance improve- ments of +18.2% for Flan-PaLM-8B, +11.1 % for Man-PaLL-622B, and +3.6% % for 5aM---BB. On a set of turing concept tasks (e.g., swapping Os and Is in string), symbol-tuned models also improve by +15.3% for Flan-PaL.M-88 and Flan- PaLM-62B and +4.7% % for P5L--...
Reynolds and McDonell, 2021; Wei et al., 2022a), suggesting that they are not reasoning in robust way. Instruction tuning has been shown to improve performance and allow models to better follow in- context exemplars (Mishra et al., 2022; Min et al., 2022a; Wei et al., 2022a; Ye et al., 2021; Chung et al., 2022). One shortcoming, however, that models are not forced to learn to use the exemplars because the task is redundantly defined in the evalu- ation example via instructions and natural language labels. For example, in the left-hand side of Fig- ure 1, although the exemplars understand the task, they are not strictly necessary since the model could ignore the exemplars and just read the instruction.
help the model
Finally, we experiment on an in-context learn- ing setting where inputs have flipped labels, which forces the model to override its prior knowledge when presented with contradictory information in- context. Pretrained language models have the abil- ity to somewhat follow flipped labels this ability is lost during instruction tuning but can be restored via symbol tuning. Overall, we hope that the strong empirical results from symbol tuning encourage further work in allowing language models to rea- son over arbitrary symbols given in-context.
To make the model better at in-context learning, propose symbol tuning, in which the model is finetuned exemplars where the instructions are removed and natural language labels are re- placed with semantically-unreltedd labels (e.g.. "Foo," "Bar," etc.). In this setup, the task un- clear without looking at the in-context exemplars. For example, if the prompt from the previous para- graph was changed to "<sentence>. Answer: (Foo, Bar/" (as shown in the right-hand side of Figure 1), multiple in-context exemplars would be needed in order figure out the task. Because symbol tuning teaches the model to reason over in-context ex- emplars, symbol-tuned models should have better performance on unseen tasks that require reasoning between in-context exemplars and their labels.
2 Symbol tuning
Despite their ability to perform some reason- ing tasks after being shown in-context exemplars (Chowdhery et al., 2022; OpenAI, 2023), language models are still sensitive to the way in which these tasks are presented in prompts (Brown et al., 2020;

---

Sentiment Analysis
Paraphrase Detection
Miscellaneous
Natural Language
RT
SST2
QQP
MRPC
TEO
TEI
Inference
RTE
WNU
TES
COLA
MNLI
Comman Sense
Topic Classification
Coreference
SNLI
CB
COPA
PIQA
AGN
TREC
WSC
WING
Figure 2: Datasets and task types used for symbol tuning. See Appendix D.1 for dataset details.
Integers
Characters
Words
Finetuning (-30k symbols)
(1-4 digits)
(1 (1-3 letter combinations)
(MIT list of 10,000 words
Evaluation
Integers (5 digits)
Characters
Words
(-270k symbols)
(3-4 letter combinations)
(MIT list of 100,000 words)
Figure 3: We set of ~300 arbitrary symbols from three categories (integers, character combinations, and words), ~~30 symbols are used during tuning and the rest are held out for evaluation. See Appendix E. for more details on the symbols that we used.
3 Experimental setup
used in symbol tuning (22 datasets) or used during instruction tuning (1.8k tasks). Hence, we choose 11 NLP datasets from HuggingFace (Lhoest et al., 2021) that were not used in either stage of fine- tuning (details are shown in Appendix D.2): (Con- neau and Kiela, 2018, SUBJ); (Basile et al., 2019, TEH); (Mohammad et al., 2016, TEAB); (Moham- mad et al., 2016, TEAT); (Mohammad et al., 2016, TEFE); (Mohammad et al., 2016, TEHI); (Alex et al., 2021, ADEC); (Alex et al., 2021, OR); (Alex et al., 2021, SOT); (Alex et al., 2021, TOS); and (Alex et al., 2021, TC). We the validation split of each dataset to generate evaluation prompts. For each dataset, we randomly select maximum of 100 examples to use during evaluation. Each evalu- ation prompt uses a randomly-selectee input-label format following Section 3.1, though we fix the number of in-context exemplars per class at k = 4 (we ablate this parameter in Appendix C.4).
3.1 Tuning tasks & prompt formatting
Figure 2 shows the 22 publicly-available NLP datasets from HuggingFace (Lhoest et al., 2021) (see Appendix D.1 for dataset details) that we use for our symbol- procedure (we ablate the number of datasets used for symbol tuning in Ap- pendix B.4). We selected NL.P tasks that have been widely used in the literature (Wang al., 2018, 2019). Each dataset is categorized into one of seven task types-we only selected classification- type tasks because symbol tuning requires dis- crete labels. For each dataset, we use examples from the training split to compose prompts that we use for tuning. Each prompt uses a randomly- selected input-label format (formats are shown in Appendix E.2) and contains randomly-selected number between 2 and 10 of in-context exem- plars per class. We remap labels to a randomly- selected label from set of ~30k labels from three label types as shown in Figure (we ablate the number of labels in Appendix B.5 and the label types in Appendix B.6). Examples of generated tuning prompts for each task are shown in Ap- pendix G.1. Code for generating arbitrary sym- bols can be found at https: : /gthhb....// com/ JerryWeiAI/see wwweeiiA/ssmmoo-ttuume
We generate prompts for the four different in- context learning (ICL) settings described in Fig- ure 4; each setting either contains or does not contain instructions describing the task (see Ap- pendix D.2 for the instructions we use for each task) and does or does not contain relevant natural language labels. For settings that do not rel- evant natural language labels, we remap original labels to a randomly-selecte label from set of 270k semantically-unrelaan
labels as shown in Figure (we removed labels that were seen during symbol tuning). Examples of generated evaluation prompts for each task are shown in Appendix G.2.
3.2 Evaluation tasks
We want to evaluate a model's ability to perform on unseen tasks, so we cannot evaluate on tasks

---

Relevant Label: <
Relevant Label: /
Relevant Label: X
Relevant Label: X
Instructions: V/
Instructions: X
Instructions: o
Instructions: X
Input
Input
Input
Input
Instruction
What is the sentiment of thi??
[None]]
What is the sentiment this? great
[lone]
Exempiar
This great
is groar
Label
Answer: Poetive
Rodcwort
Answer: Positive
Answer: Foo Unvoated
Answer: Foo
Unedaasd
Instruction
What the sentiment this?
What the senliment this?
Exemplar
I've over 2000
Worst I've soon
I've ever soon
I've ewar seon
Label
Answer: Negative
Roinwart
Answer: Negathve
Reiwon
Answer: Bar Unisond
Answer: Bar Unilow
(nare csomptans)
|nore oxemppior)
[nore exompland)
Jnore csomptond
Evaluation
sentiment
[Nonol
sontiment
Nonne
Example
The movie is temble
This movie is terrible Answer:
This movie is tembile Answer.
This movie is temrille
Output
Output
Output
Output
Negallve
Negalive
Bar
Bar
Figure 4: Depending on the availability of instructions and relevant natural language labels, models may need to do varying amounts of reasoning with in-context exemplars. When these features are not available, models must reason with the given in-context exemplars in order to successfully perform the task. When they are available, reasoning with exemplars can help but is not necessary.
3.3 Models & finetuning procedure
and for 540B model evaluations, report results from the checkpoint after tuning for 1 steps (we ablate the number of tuning steps in Appendix B.2). See Appendix E.3 for the number of finetuning steps, learning rate, batch size, and dropout used for each model. As a baseline, we compare symbol- tuned models against Flan-PaLM models, and we also compare symbol tuning against continued in- struction tuning in Appendix B.1.
For our experiments, we tune Flan-PaLM,, the instruction-tuned variants of PaLM. We use the instruction-tuner variants in order to reduce the number of steps needed for tuning, since symbol tuning instruction-turee model does not require relearning the information learned during the origi- nal round of instruction tuning. We use three dif- ferent sizes of Flan-PaL.A models: Flan-PaLMM-80 Flan-PaLMM-62 and Man-PaLM-5400.. We also tested Ma (-11MMM-BBcoo (PaL.M62BB at 1.3T
Symbol-tuned models are better in-context learners
tokens instcad of 780B tokens); we abbreviate this
model size 62B-c.
During symbol tuning, models must learn to rea- son with in-context exemplars in order to success- fully perform tasks because prompts are modified to ensure that tasks cannot be learned from natu- ral language labels or instructions. Symbol-tuned models should thus perform better in settings where tasks are unclear and require reasoning between in- context exemplars and their labels. Additionally, since symbol tuning is meant to improve the abil- ity follow in-context exemplars, should not modify prior knowledge and should thus retain the same performance in settings where exemplars are not as necessary to complete the task.
Our symbol-tuning pipeline mixes all datasets and randomly samples from each dataset. To ensure that the dataset sizes are balanced (i.c., no dataset gets completely overshadowed), we limit the num- ber of training examples per dataset to maximum of 25k randomly-selede amples are combined into a single sequence using packing (Raffel al., 2020), and inputs are sepa- rated from labels using an end-of-sequence (EOS) token. We tune all models using a batch size of 32 and the Adafactor optimizer (Shazcer and Stern, 2018). For 8B and 62B models, we tune with a learning rate 3 x 10-3, and we tune Flan-PaLM- 540B with learning rate of x 10-3, We use 2048 and 512, respectively, the input and target sequence lengths during tuning.
examples. Training ex-
To explore these settings, define four ICL settings that vary the amount of reasoning required between inputs and labels in order learn the task (based on the availability of instructions/rellvaatt labels), as shown in Figure 4. The casiest of these settings uses prompts where both instructions and
For 8B and 62B model evaluations, we report re- sults from the checkpoint after tuning for 4k steps,

---

Average performance on eleven tasks
Relevant labels:
X
X
Task instructions:
V
X
Random Guessing
42.4
42.4
42.4
42.4
Flan-PaLLM-88
63.9
61.6
42.4
44.2
+ Symbol tuning (ours)
57.6 (-6.3)
54.3 (-7.3)
58.2 (+15.8)
52.8 (+8.6)
Plan-PaLMM-G2
74.3
70.0
57.0
50.5
+ Symbol tuning (ours)
75.5 (+1.2)
70.8 (+0.8)
71.4 (+14.4)
60.3 (+9.8)
Flan Saan-aPaM--2--ooo
77.3
70.3
56.3
51.0
+ Symbol luning (ours)
78.9 (+1.6)
74.5 (+4.2)
71.8 (+15.5)
62.1 (+11.1)
Mlan-PaLM-540B
82.2
77.4
70.7
58.1
+ Symbol luming (ours)
84.4 (+2.2)
78.8 (+1.4)
80.0 (+9.3)
63.6 (+5.5)
Table I: Large-enough symbol-tuned models are better at in-context learning than baselines, especially in settings where relevant labels are not available. Performance shown as average model accuracy (%) across eleven tasks (per-task results are shown in Appendix F.2).
relevant labels are available (as in in-context exem- plars are not necessary to learn the task), while the hardest sctting uses prompts where instructions and relevant labels are both unavailable.
edge due overfitting. Overall, the improvements demonstrate the strong potential of symbol tuning to improve model performance, especially when tasks require learning from in-context exemplars.
In Table 1, evaluate model performance be- fore and after symbol tuning in each of these tings. We find that symbol tuning improves per- formance across all ICL settings for models 62B and larger, with small improvements settings with relevant natural language labels (+0.8% to +4.2%) and substantial improvements in settings without relevant natural language labels (+5.5% to +15.5%). Strikingly, when relevant labels unavailable, symbol-tuned Flan-PaLMM8DD outper- forms Flan-PaLMM-622 and symbol-tuned Flan- PaLM-62B outperforms Flan-PaLM-5547 performance difference suggests that symbol tun- ing can allow much smaller models to perform as well as large models on learning input-label map- ping from exemplars (effectively saving ~ ~0x in- ference compute).
Symbol tuning improves algorithmic reasoning
Symbol tuming is designed to force the model to learn from input-label mappings in the in-context exemplars because the symbols the task and no instructions are provided (and thus the model cannot rely on any other guidance determine the task). For this reason, we posit that symbol tuning should not only improve the model's ability to map natural language inputs to arbitrary symbols, but also its ability to learn other forms of input-label mappings such as algorithms.
unrelated
This
To test this, we experiment on algorithmic rea- soning tasks from BIG- Bench (Srivastava et al., 2022). We first experiment on set of list func- tion tasks (Rule et al., 2020; Srivastava et al., 2022) where the model needs to identify a transforma- tion function (e.g.. remove the last element in a list) between input and output lists containing negative integers. These tasks evaluated in four-shot setting, following our evaluation setup in Section 3.2. Additionally, we test models on set of simple turing concepts (Telle et al., 2019; Sri- vastava et al., 2022) where models need to reason
Symbol-tuned models also perform somewhat- comparably settings with only relevant labels or only instructions, unlike baseline models whose performance in settings with only relevant labels is always better than in settings with only instruc- tions. Performance in settings with relevant labels actually decreases for Flan-PaLM-80 after symbol- tuning, however, which may suggest that symbol tuning a small model can override its prior knowl-

---

(A) Remove elements
(B) Modify the list
(C) Input-independent
100
100
100
Accuracy (%)
80
80
80
60
40
40
40
20
20
20
0
(D) Add elements
E
Miscellaneous
(F) Simple turing concepts
100
100
100
Accuracy (%)
80
60
60
60
40
40
40
20
20
20
8B
62B
62B-c
540B
8B
62B
628-c
540B
SB
62B
62B-c
540B
Flan-PaLM
Flan-PaLM Symbol tuning (ours)
Figure 5: Symbol-tuned models achieve higher performance on list function tasks and simple turing concept tasks.
(A-E); categories of list functions tasks (Rule al., 2020; Srivastava al., 2022). (F): simple turing concepts task (Telle et al., 2019; Srivastava et al., 2022). Accuracy per list function category is averaged across all subtasks (categories and per-task results are shown Appendix F.1).
with binary strings to learn the concept that maps input to output (e.g., swapping Os and Is in string). These tasks have predetermined shots for each evaluation example. We selected these algorithmic tasks because they lest the model's ability to generalize to different task types (the symbol-tuning tasks were classification problems with discrete labels, while these tasks are more open-ended generation problems²) and do not re- quire world knowledge (symbol tuning does not increase model's prior knowledge).
improvement across all tasks of 18.2% for Flan- PaLM-8B, 11.1 1% for Flan-PaLMM-62B, 15.5% for Flan-PaLLM-6BB-oont and 3.6% for Flan-PaLM-- 540B. On the turing concept tasks, symbol tuning results in a performance improvement of 15.3% for Maan-PaLM-8B and Flan-PPLLM-622 14.1% % for Flan-PaLM-62B--om and 4.7% for Flan-PaLM- 540B.
TPL.M-62B-oo with symbol tuning outperforms Flan- M---M on the list function tasks (in terms of average accuracy across tasks), which is equal to ~ 10x reduction in inference compute. These improvements on an unseen task type suggest that symbol tuning indeed strength- ens the model's ability to learn in-context, as the symbol-tuning procedure did not have algorithmic data and only used natural language data.
In Figure 5, we show model performance on the twenty list function tasks with the highest human accuracy baselines (Rule, 2020) separated into five categories (category details are described in Appendix F.1) and the turing concepts containing 3 fewer instructions in the AS Il subset of the sim- ple turing concepts task. On the list function tasks, symbol tuning results an average performance
Symbol-tuned models can override priors via flipped labels
'Although thain-of-thee th-thugght (Wei et al., 2022b, CoT) can viewed as an open-ended generation problem, Ap- not significantly affect model's CoT reasoning abilities, possibly because our symbol-tuning data not incorporate any CoT prompts.
Wei al. (2023) showed that while pretrained lan- guage models (without instruction tuning) could, to some extent, follow flipped labels presented in- context, instruction tuning degraded this ability. Symbol tuning, on the other hand, forces models to
pendix C.2, we found that symbol-tuning
"We do not directly compare with the human baselines hecause our evaluation format was different.

---

Average
ADEC
OR
SUBJ
100
100
100
100
90
Accuracy (%)
80
75
75
75
70
50
50
50
60
50
25
25
25
40
30
0
20
TC
TEH
TOS
100
100
10
8B
62B
62B-c
540B
PaLM (base model)
Flan-Pal.M
88
628
628
$408
628
62B-c
540Bi
6.8
628-t
SAUB
Flan-PaLM Symbol tuning (ours)
Figure 6: Symbol-tuned models are much better at following flipped labels presented in in-context than instruction- tuned models are for model sizes. Instruction tumed models cannot flip predictions to follow flipped labels (performance is well below random guessing), while symbol-tuned models can do this more often (performance matches or is slightly above random guessing). Ground-truth labels for evaluation examples are flipped, so if a model learns follow flipped labels, its accuracy should be above random guessing (e.g., perfect model that can evaluations).
follow flipped labels should get 100% accuracy
consider the label presented in-context as an arbi- trary symbol, which should reduce the model's 's us- age of prior knowledge that contradicts the flipped labels. For this reason, we expect that symbol tun- ing would be able to improve and restore the ability to follow flipped labels in-context.
ment of 34.0%. For some datasets (e.g., OR, SUBJ, TC), symbol-tuned models can now override pri- ors and follow flipped labels (i.e., achieve much better performance than random guessing). despite instruction-tunel models not being able to do so for any datasets. Additionally, symbol-tuned mubbol-tuned models match or beat pretraining-only models in terms of average performance, indicating that symbol tuming has, to some extent, restored the model's original ability to follow flipped labels.
To test this, we flip the labels of both in-context exemplars and the evaluation example for the tasks described in Section 3.2 (we remove tasks with more than two labels from this experiment since it is unclear how to best "flip" more than two labels). For example, for the SST2 dataset, all exemplars that are labeled having "positive" sentiment will now be labeled as having "negative" sentiment. A perfect model that can follow these flipped labels should achieve 100% accuracy on these tasks its accuracy in the standard ICL. setting is also 100%.
These results further indicate another type of generalized in-context learning capability, we did not include any flipped labels during symbol tuning. Although the performance improvement from symbol tuning large, we note that more work should be done this area since performance on the flipped-labels settings is, on average, not significantly better than random guessing.
As shown in Figure 6, symbol tuning restores the ability follow flipped labels that was lost during instruction tuning. We see that there similar trend across all model sizes innsssuciittttnnn tuned models are generally unable to follow flipped labels (as demonstrated by their performance be- ing far below random guessing), but symbol-tuned models are much more capable of doing so. We found that after symbol tuning, Flan-PaLM-8B sees average improvement across all datasets of 26.5% Flan-PaLM-62B sees an improvement of 33.7%, and Tan-Pal.M-544 sees an improve-
Related work
7.1 In-context learning via semantic prior knowledge
Recent studies on in-context learning suggest that prior knowledge plays significant role in how models learn in-context. For example, Wei et al. (2023) showed that some small models and instruction-tuned models cannot follow flipped la- bels presented in-context, suggesting that these models primarily utilize prior knowledge for in-

---

context learning. Min al. (2022b) found simi- lar result that using random ground-truth labels in in-context exemplars does not significantly affect performance, meaning that performance may be driven by other factors such as the label space.
2022: Dai et al., 2023). Furthermore, Webson and Pavlick (2022) demonstrated that, in natural lan- guage setting, language models can learn at the same rate during finetuning even when given irrel- evant or misleading prompts. On a broader level, Rajendran et al. (2020) and Yin et al. (2020) found that adding noise to, shuffling, or regularizing the label space can make systems better at learning and adapting to new tasks.
Reynolds and McDonell (2021) also showed that cleverly-construeted prompts in a zero-shot setting could outperform prompts in few-shot setting, implying that, for some tasks, models can achieve better performance by leveraging their ex- isting knowledge than from attempting to learn the task from in- in-context exemplars. Additionally, in chain-of-thought prompting (Wei al., 2022b), Madaan and Yazdanbakhsh (2022) and Wang et al. (2022) showed that performance on multi-step rea- soning tasks does not decrease when models are provided with logically-incorreet prompts. Raghu et al. (2020) also demonstrated that systems such as MAML can effectively "memorize" labels when trained in a way where all labels can be memorized, which further illustrates that, when possible, mod- els may attempt to use prior knowledge rather than adapt each new task.
In this paper, we attempt to improve the degree to which language models are able to learn tasks via input-label mappings. Our can be seen as a form of label augmentation and is thus similar to the proposed methods from Ra- jendran al. (2020) and Yin et al. (2020), though differs crucially in that we apply them to tume large language models. Additionally, We found that symbol-tunet models saw significant improve- ments in their ability to learn in-context (e.g., on algorithmic tasks or settings with underspecified prompts), which supports the idea that langauge models have the ability to learn in-context using the given exemplars.
method
Our findings do not dispute the idea that seman- tic prior knowledge can provide significant ben- efits to in-context learning. Indeed, we showed that instruction-tuned models cannot follow flipped labels in-context, which is consistent with the find- ings from Wei al. (2023). We instead aim to demonstrate that through symbol tuming, language models can retain the benefits of utilizing prior knowledge while also improving their ability to learn from input-label pairs shown in-context.
7.3 Tuning language models
Our work presented symbol tuning, a form of finetuning on input-label pairs where labels are remapped to arbitrary symbols. Symbol tuning relates to broader body of work showing that finetuning language models can significantly alter their behavior and performance in different settings. For example, Wei al. (2022a) first presented in- struction tuning (finetuning on tasks phrased as instructions) and showed that this finetuning pro- cedure substantially improves model performance in zero-shot settings. Chung et al. (2022) further scaled this procedure by adding more tasks, in- creasing model sizes, and adding thain-of-thouge data, demonstrating that, with these changes, tuned models are significantly better at chain-of-thougee reasoning, open-ended generation, and several eval- uation benchmarks.
7.2 In-context learning via in-context exemplars
At the same time, however, other recent work has suggested that language models can, in fact, learn in-context using the given exemplars. This abil- ity may be more useful than the ability to use se- mantic prior knowledge because it would allow models to perform tasks that are not seen in or contradict pretraining data. Garg et al. (2022), for instance, showed that transformers trained from scratch can perform in-context learning linear- regression tasks similar performance level as the least-squares estimator. This capability was shown to result from transformers implementing standard learning algorithms such as gradient de- scent (Akyürek et al., 2023; von Oswald et al.,
Our experimental findings match these results in terms of showing that finetuning can signifi- cantly alter model performance. Our work differs, however, by not only focusing on settings with in-context exemplars and underspecified prompts. but also by modifying the finetuning procedure make tasks harder to learn and require additional reasoning with in in-context exemplars.

---

8 Limitations
Conclusions
While our study presents a simple yet effective method of improving in-context learning for lan- guage models, there are several limitations to work. An open question how apply symbol tuning in a generative setting-ww symbol tuned models on range of classification tasks with dis- crete labels so that we can remap labels to arbitrary symbols, but we did not tune on generation tasks since it is unclear how to remap outputs to symbols in those settings. Future work could thus explore whether symbol tuning can be applied in genera- tive setting.
In this paper, we presented symbol tuning, a new method of tuning models tasks where natural language labels are remapped arbitrary symbols. Symbol tuning is based off of the intuition that when models cannot use instructions or relevant labels to determine a presented task, must do so by instead learning from in-context exemplars. We tuned four language models (Flan-PaLM--P)) Flan- PaLM-62B, Flan-PaLM--622 62B-cont, and Flan- PaLM-540BB using our symbol-tuning procedure, utilizing tuning mixture of 22 datasets and ap- proximately 30k arbitrary symbols labels.
Additionally, our symbol-tuning procedure used 22 NLP latasets-whil we ablated the number of datasets in Appendix B.4 and saw that increas- ing the number of datasets used for symbol tun- ing generally improves performance, we did not experiment with adding more tasks. Prior work, however, has demonstrated that scaling up finetun- ing methods can improve their impact on language models (Chung ct al., 2022), so natural extension would be to examine whether further scaling up the symbol-tuning method would have similar result,
Experimentally, we showed that symbol tuning can significantly improve a model's ability to learn from in in-context exemplars in not only natural lan- guage settings, but also algorithmic tasks. First, showed that symbol tuning improves perfor- mance on unseen in-context learning tasks, espe- cially when prompts do not contain instructions or relevant labels. We also found that symbol- -uudel models were much better at algorithmic reasoning tasks, despite the lack of numerical or algorithmic data in the symbol-tuning procedure. Finally, in an in-context learning setting where inputs have flipped labels, symbol tuming (for some datasets) reunlocks the ability to follow flipped labels that was lost during instruction tuning.
Furthermore, we applied symbol tuning fam- ily of instruction-tunet language models. un- known, however, whether the effects of symbol tuning that we showed may be affected by changes to the pretraining objective, model architecture, or training process. Similarly, symbol tuning may have different effects on language models that are did not specifically
Through symbol tuning, we aim to have in- creased the degree to which models can exam- ine and learn from input-label mappings during in-context learning. We hope that our results en- courage further work towards improving language models' ability to reason over symbols presented in-context.
not instruction tuned, periment on this factor. For this reason, future work may investigate how these factors impact the effec- tiveness of symbol tuning for improving in-context learning abilities in language models.
Because we only experimented with one fam- ily of language models, it is still unclear whether symbol tuning is effective for other models. Apply- ing symbol tuning to other language models would likely require adjustments to the finetuning proce- dure to be successful (e.g., number of finetuning steps, mixing with previous data, number datasets), but a definitive conclusion about these factors can- not be drawn without further experimentation. We thus note that crucial direction for future work is to explore how well symbol tuning translates to other language models.

---

References
Foundation Models at the International Conference on Learning Representations.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2023. What learn- ing algorithm is in-context learning? Investigations with linear models. In International Conference on Learning Representations.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gre- gory Valiant. 2022. What can transformers learn in-context? A case study of simple function classes. In Conference on Neural Information Processing Sys- tems.
Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Car- lier, Michael Noetel, and Andreas Stuhlmüller. 2021. RAFT: A real-world fho-shot text classification benchmark. In Conference on Neural Information Processing Systems.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Mcasuring massive multitask language under- standing. In International Conference on Learning Reprexentations.
Sakaguchi Keisuke. Le Bras Ronan, Bhagavatula Chan- dra, and Choi Yejin. 2021. WinoGrande: An adver- sarial winograd schema challenge at scale. Communi-- cations of the Association for Computing Machinery.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo. Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 Task 5: Multilingual detec- tion of hate speech against immigrants and women in Twitter. In International Workshop on Semantic Evaluation.
Hector Levesque, Ernest Davis, and Leora Morgenstem. 2012. The winograd schema challenge. In Interna- rional Conference on the Principles of Knowledge Representation and Reasoning.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about physical commonsense in natural language. In Con- ference of the Association for the Advancement of Artificial Intelligence.
Aitor Lewkowyez. Anders Andreassen. David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantita- tive reasoning problems with language models. In Conference on Neural Information Processing Sys- tems.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Mamning. 2015. A large anno- tated corpus for learning natural language inference. Conference on Empirical Methods Natural Lan- guage Processing.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis. Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger. Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing, In Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Conference Neural Information Pro- cessing Systems.
Zihang Chen, Hongho Zhang, Xiaoji Zhang, and Leqi Zhao. 2017. Quora question pairs.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, et al. 2022. PaIM: Scaling language modeling with Pathways.
Xin Li and Dan Roth. 2002. Learning question classi- fiers. In Conference on Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling mnstructionffieetuned language models.
Aman Madaan and Amir Yazdanbakhsh. 2022. Text and patterns: For effective chain thought, it takes two to tango.
Alexis Conneau and Douwe Kicla. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Conference on Language Resources and Evaluation.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2022a. MetalCL: Learning to lear in context. Conference of the North American Chapter of the Association for Computational Lin- guistics.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2023. Why can GPT learn in-context? Language models secretly perform gradient descent as meta-optimizers. In Workshop on Understanding
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022b. Rethinking the role of demonstrations:

---

What makes in-context learning work? In Confer- ence on Empirical Methods in Natural Language Processing.
Sara Rosenthal, Noura Farra, and Preslav Nakov, 2017. SemBval-2017 Task 4: Sentiment analysis in twitter. In International Workshop on Semantic Evaluation.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsoureing instructions. In Proceedings of the Association for Computational Linguistics.
Joshua S. Rule, Joshua B. Tenenbaum, and Steven T. Piantadosi. 2020. The child as hacker. Trends in Cognitive Sciences.
Joshua Stewart Rule. 2020. The child as hacker: huild- ing more human-llkke models of learning. Ph.D. the- sis, Massachusetts Institute of Technology.
Saif Mohammad, Svetlana Kiritchenko. Parinaz Sob- hani, Xiaodan Zhu, and Colin Cherry. 2016. SemFval-2016 Task 6: Detecting stance in tweets. In International Workshop on Semantic Evaluation.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler. Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shamya Sharma Sharma, Eliza Szezechla, Taewoon Kim. Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan. Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Allen Newell. 1980. Physical symbol systems. Cogni- five Science.
Allen Newell and Herbert A. Simon. 1976, Computer science as empirical inquiry: Symbols and search. In Communications of the Association for Computing Machinery.
OpenAl. 2023. GPT-4 technical report.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Conference on Neural Informa- tion Processing Systems.
Adam Santoro, Andrew K. Lampinen, Kory W. Mathew- son, Timothy P. Lillicrap, and David Raposo. 2021. Symholic behaviour in artificial intelligence.
Noam Shazeer and Mitchell Stem. 2018. Adafactor:
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proceedings of the Association for Computational Linguistics.
Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013, Recursive deep models for semantic compositionality over sentiment treebank. In Conference on Empirical Methods in Natural Lan- guage Processing.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text text trans- former. Journal of Machine Learning Research.
Aarohi Srivastava. Abhinav Rastogi, Abhishek Rao. Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. 2020. Rapid learning or feature reuse? towards understanding the effectiveness of MAML. In International Conference on Learning Representa- tions.
Janarthanan Rajendran, Alexander Irpan, and Eric Jang. 2020. Meta-learning requires audd--mmmmtnnion... In Conference on Neural Information Processing Sys- tems.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2022. Challenging BIG-Bench tasks and whether chain-of of-tthught can solve them.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and
Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing.
Jan Ame Telle, José Hernández-Orallo, and Cèsar Ferri. 2019. The teaching size: computable teachers and learners for universal languages. Machine Learning.
Laria Reynolds and Kyle McDonell. 2021. Prompt pro- gramming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the Conference on Human Factors in Computing Sys- lems.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 Task 3: Irony detection in en- glish tweets. In International Workshop on Semantic Evaluation.

---

Johannes von Oswald, Eyvind Niklasson, Ettore Ran- dazzo, João Sacramento, Alexander Mordvintsew. An- drey Zhmoginov, and Max Vladymyrov. 2022. Trans- formers learn in-context by gradient descent.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun, 2015. Character-level convolutional networks for text clas- sification. In Conference on Newral Information Pro- cessing Systems.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understand- ing systems. In Conference on Neural Information Processing Systems.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase Adversaries from Word Scram- bling. In Proceedings of the North American Chapter of the Association for Computational Linguistics.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Black- Conference on Empirical Methods in Natural Language Processing.
boxNLP Workshop
Boshi Wang. Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2022. Towards understanding chain-of-thougee prompting: An empirical study of what matters.
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Conference of the North American Chapter of the Association for Computational Lin- guistics.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du An- drew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners. In Interna- tional Conference on Learning Representations.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed H. Chi, Quae V Le, and Denny Zhou. 2022b. Chain of thought prompt- ing elicits reasoning in large language models. In Conference on Neural Information Processing Sys- tems.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023, Larger language models do in-context learning dif- ferently.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. CrossFit: A few-shot learning challenge for cross- task generalization in NLP. In Conference on Empir- ical Methods in Natural Language Processing.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. 2020. Meta-
learning without memorization. In International Con- ference on Learning Representations.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. SemEval-2019 Task 6: Identifying and catego- rizing offensive language in social media (offenseval). In International Workshop on Semantic Evaluation.


Reviews:
[{"review_id": "FXVQiFCttH", "reviewer": "Reviewer_43HS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and feedback. Our work proposes a simple finetuning procedure that can significantly improve a language model’s in-context learning and algorithmic reasoning abilities, and we are glad that you found our approach to be novel yet simple and effective.\n\nPlease let us know if you have any further suggestions—we would be happy to continue working to improve our paper.\n\n> Reproducibility is an issue due to the models tested being closed-source\n\nThanks for this important comment on the reproducibility of our results. Because our models were internal, we could not release our symbol-tuned model checkpoints, though we made sure to describe the symbol-tuning procedure in as much detail as possible, as you mentioned. Additionally, we made sure that all of our finetuning and evaluation datasets were publicly-available on HuggingFace. \n\n> Although the authors hypothesize about why symbol tuning allows models to perform better in the tested in-context learning and algorithmic reasoning tasks, the research lacks clear evidence backing these model’s performance\n\nThank you for providing this discussion on why symbol tuning improves in-context learning. Our idea leverages the intuition that traditional instruction tuning overspecifies tasks and does not necessarily require models to even look at in-context exemplars, and we proposed symbol tuning as a way to force model to use in-context exemplars to learn the task by underspecifying the task via removing instructions and replacing labels with arbitrary symbols. Experiments in Appendix A.1, A.7, B.2, and B.3 are consistent with this. For even more evidence, experiments on analyzing specific datasets or applying symbol tuning in generative settings could shed further light, which we discussed in Section 8 (our limitations/future work section).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Novelty: the authors provide a novel approach to finetuning models, drawing from the intuition that when LMs struggle using instructions or labels to determine the task at hand, they instead resort to learning from in-context examples. This simple but effective approach provides both unique findings and a compelling explanation for model’s improvement with symbol tuning. \n* Robustness: the authors support their methodology with clear and logical analysis on the impact of symbol tuning in a variety of different contexts–providing robust findings while also demonstrating the need for further research on the impact of symbol tuning for LLMs; this work provides promise for future work exploring more creative uses of input-label pairs during in-context learning. \n", "reasons_to_reject": "Minor weaknesses/nits\n* Reproducibility is an issue due to the models tested being closed-source (although the process of symbol-tuning is described in detail)\n* Although the authors hypothesize about why symbol tuning allows models to perform better in the tested in-context learning and algorithmic reasoning tasks, the research lacks clear evidence backing these model’s performance (but this may be better kept for future work)\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors provide a novel finetuning approach for LLM via “symbol tuning”--finetuning LMs on input-label pairs for which the natural language labels are replaced with nonce words (“foo”,”bar”)--and conduct experimental validation of the proposed approach for in-context learning and algorithmic reasoning tasks. The authors find that symbol tuned models demonstrate superior performance on in-context learning tasks and with underspecified prompts. They further show that symbol tuning improves performance on multiple algorithmic reasoning benchmarks (BIG-Bench, Turing Concepts). The authors also conduct tests with prompts that have flipped labels, necessitating models’ overriding of prior knowledge to correctly answer prompts, showing that symbol tuning improves model performance in this domain as well. ", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "ln. 60: it → they "}, "extra_text": {}}, {"review_id": "7KCEi5zbr7", "reviewer": "Reviewer_F3QV", "summary": "", "questions": "1. Will the authors make their code once accepted? \n2. Why does symbol tuning performs worse for smaller LMs like Flan-PaLM-8B?\n3. In Table-1 we see for the IT-LLM-62B-cont that without instructions performed better than with instructions? So, in this case even without instructions the model was able to learn from in-context examples. Does it mean we don't need symbol tuning in all the cases?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments and important questions. We presented symbol tuning, a new finetuning method that helps models learn in-context by forcing them to use in-context exemplars to learn a task. We are happy that you found the method to be effective and our results to be strong, useful, and comprehensive.\n\nBased on your feedback, we have added revisions to the manuscript (we cannot revise the submission during the review process but will do so as soon as possible)—please feel free to let us know if you have additional comments or questions.\n\n> Will the authors make their code [available] once accepted?\n\nThanks for this nice suggestion to make code available. While we are unable to release our entire pipeline for generating symbol-tuning data, we released the script for generating arbitrary symbols, as the use of symbols as labels is the primary contribution of our paper (the rest of the pipeline includes collecting datasets from HuggingFace, replacing labels with the generated symbols, and placing input–label pairs into prompts). Note that we provide example prompts and implementation details in the Appendix so that generating symbol-tuning data should be a straightforward implementation.\n\nThe anonymized repository can be found here: \n\nhttps://anonymous.4open.science/r/symbol-tuning-2FD5/ \n\nWe also added a mention of this code in Section 3 (we will replace the anonymized repository with the non-anonymized version after the reviewing process):\n\n“Code for generating arbitrary symbols can be found at https://anonymous.4open.science/r/symbol-tuning-2FD5/.” \n\n> Why does symbol tuning perform worse for smaller LMs like [IT-LLM-8B]?\n\nThank you for this comment on how the effects of symbol tuning can vary with respect to model size. While symbol tuning improves performance for both small and large language models on algorithmic reasoning and flipped label settings, in Section 4 we indeed found that symbol tuning may hurt performance when relevant natural language labels are available. In this setting, symbol tuning a large-enough model still improves performance, but symbol tuning the smallest model (IT-LLM-8B) resulted in a performance drop.\n\nIn our manuscript, we hypothesized that this could have been a result of the smallest model overfitting to the symbol-tuning data. This hypothesis is supported by the finding that holding the availability of task instructions constant, the performance with and without relevant labels is very similar (e.g., with task instructions: no relevant labels = 58.2%, with relevant labels = 57.6%). This may suggest that the model does not draw any additional semantic information from the relevant natural language labels that are included and may instead treat them as an arbitrary symbol, which could explain why performance with irrelevant labels matches performance with relevant labels.\n\nWe added further discussion on this question in the “frequently asked questions” section in the Appendix:\n\n“In Section 4, we found that for our smallest model (IT-LLM-8B), symbol tuning may result in performance drops in in-context learning settings where relevant natural language labels are available. Our primary hypothesis for this behavior is that via the symbol-tuning procedure, the model learns to override its prior knowledge and overfit to the symbol-tuning data. This could result in the model learning to treat any label as an arbitrary symbol to learn, regardless of whether the label is relevant to the task or not. A key finding that supports this hypothesis is that the model achieves similar performance whether the labels are relevant to the task or not—as shown in Table 1, when task instructions are available, the model achieves 57.6% accuracy when the labels are relevant to the task and 58.2% accuracy when the labels are not relevant to the task, and when task instructions are not available, the model achieves 54.3% accuracy when the labels are relevant and 52.8% accuracy when the labels are not relevant. This finding implies that the model does not consider the semantics of the relevant labels and instead treats them just like any other arbitrary symbol, which could further suggest that the model has overfit to the symbol-tuning data by learning to treat *any* label as a symbol to learn.”\n\n> In Table-1 we see IT-LLM-62B-cont without instructions performed better than with instructions? [...] Does it mean we don't need symbol tuning in all cases?\n\nThanks for this important comment on whether there are cases in which symbol tuning is not needed. In Table 1, we indeed found that the performance improvement from adding symbol tuning is much larger for settings where relevant labels or task instructions are not available. \n\nThe most important factor is whether relevant natural language labels are available. In settings with relevant labels, symbol tuning improves performance by up to 4.2%, but in settings without relevant labels, symbol tuning improves performance by up to 15.5%. The large difference in improvements from symbol tuning suggests that symbol tuning is most effective when the task is underspecified. For tasks that are well-defined (e.g., instructions and relevant labels are both included), symbol tuning does not have a significant effect on performance (though it may still slightly improve performance by up to 2.2%).\n\nWe have added discussion about this question in a new “frequently asked questions” section in the Appendix:\n\n“In Section 4, Section 5, and Section 6, we demonstrated that symbol tuning can significantly improve a language model’s in-context learning and algorithmic-reasoning abilities. At the same time, however, there are also some settings for which symbol tuning did not significantly improve performance. For example, in Section 4, we found that performance gains from symbol tuning are most significant when relevant natural language labels or task instructions are not available, and that symbol tuning results in smaller improvements when relevant labels are available. We also found that after symbol tuning, the smallest model (IT-LLM-8B) actually saw a decrease in performance when relevant labels are available, which may suggest that a small model overrides its prior knowledge and overfits to the symbol-tuning data. Moreover, we found that symbol tuning did not necessarily improve model performance on benchmarks such as MMLU and BIG-Bench-Hard (Appendix C.1), in chain-of-thought (Wei et al., 2022b) settings (Appendix C.2), and in zero shot settings (Appendix C.3).”", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Reasons to Accept:\nStrong motivation: The sensitiveness of prompts in the LLMs performance has been a concern in IT. This paper provided a strong motivation behind introducing symbol tuning.\n\nUsefulness to the NLP community: The symbol tuning approach is useful to NLP community for training LLMs using in-context examples.\n\nStrong experiment results: The authors performed comprehensive experiments on a broad range of NLP tasks and with a varied sizes of LLMs.", "reasons_to_reject": "None.", "questions_to_authors": "1. Will the authors make their code once accepted? \n2. Why does symbol tuning performs worse for smaller LMs like Flan-PaLM-8B?\n3. In Table-1 we see for the IT-LLM-62B-cont that without instructions performed better than with instructions? So, in this case even without instructions the model was able to learn from in-context examples. Does it mean we don't need symbol tuning in all the cases?", "paper_topic_and_main_contributions": "What is this paper about:\nThis paper proposes a new method called symbol tuning to overcome the disadvantages of instruction tuning. In symbol tuning, the instructions are removed and natural language outputs are replaced with a random symbol. The LLMs are the fine-tuned on the input-masked outputs. This forces the models to reason only from examples not from the instructions. Experimental results on several NLP tasks shows the effectiveness of the proposed approach.\n\nMain Contributions:\n1. New method symbol tuning to effectively learn from in-context examples.\n2. Proposed approach helps model to override knowledge if provided with contradictory information aka. editing knowledge in LLMs.\n3. Strong results on all benchmarks compared to IT.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6Hwj61BTA4", "reviewer": "Reviewer_MTeF", "summary": "", "questions": "1) What is the performance of symbol tuning an LLM without instruction tuning? Will it still increase its ability on algorithmic reasoning and following flipped labels?  \n2) What is the relationship betweem instruction tuning and symbol tuning (only for classification problems)? \n3) Does symbol tuning increase algorithmic reasoning in CoT settings, like 8-shot CoT on GSM8K, which is also open-ended generation like list function tasks?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the encouraging comments and feedback. Our paper explored a new method of finetuning which can improve algorithmic reasoning and in-context learning, and we are glad that you found our approach to be simple, effective, and detailed.\n\nWe revised the manuscript based on your feedback and believe it is more clear (we cannot revise the submission during the review process but will do so as soon as possible). Please let us know if you have any further comments or suggestions.\n\n> What is the performance of symbol tuning an LLM without instruction tuning? Will it still increase its ability on algorithmic reasoning and following flipped labels?\n\nThanks for this question about applying symbol tuning to non-instruction-tuned language models. In our paper, we applied symbol tuning to the instruction-tuned variants of language models since symbol tuning an instruction-tuned model means that the model does not need to relearn the information learned during the instruction-tuning process. We agree, however, that an open question is whether symbol tuning would be affected by whether the language model has undergone instruction tuning, as we did not experiment with symbol tuning base models. As mentioned in our limitations section, a limitation of our work is that we only experimented using a family of instruction-tuned internal language models, so it is unclear if symbol tuning would be affected by various changes in the language model.\n\nTo make this limitation more clear, we modified Section 8 as follows:\n\n“Furthermore, we experimented with symbol tuning using a family of internal language models and their instruction-tuned variants. It is unknown, however, whether the effects of symbol tuning that we showed may be affected by changes to the pretraining objective, model architecture, or training process. For this reason, future work may investigate how these factors impact the effectiveness of symbol tuning for improving in-context learning abilities in language models.” → “Furthermore, we applied symbol tuning to a family of instruction-tuned internal language models. It is unknown, however, whether the effects of symbol tuning that we showed may be affected by changes to the pretraining objective, model architecture, or training process. Similarly, symbol tuning may have different effects on language models that are not instruction tuned, as we did not specifically experiment on this factor. For this reason, future work may investigate how these factors impact the effectiveness of symbol tuning for improving in-context learning abilities in language models.”\n\n> What is the relationship between instruction tuning and symbol tuning (only for classification problems)?\n\nThanks for this comment about how symbol tuning is related to instruction tuning. In Figure 1, we provided a comparison of instruction tuning and symbol tuning. These two finetuning procedures are related in the sense that they both show the model some in-context exemplars for a given NLP task and then provide an evaluation example. \n\nThe key difference between them, however, is that instruction tuning specifies the task with instructions and relevant natural language labels. Thus, in instruction-tuning prompts, models do not necessarily need to look at the in-context exemplars to learn the task, as they could instead just read the instruction, for example. In symbol tuning, on the other hand, we underspecify the task by removing instructions and providing irrelevant labels. This means that the model can only learn the task by reasoning over in-context exemplars to learn the relationship between inputs and labels.\n\nPlease let us know if this helps clarify the relationship between instruction tuning and symbol tuning.\n\n> Does symbol tuning increase algorithmic reasoning in CoT settings, like 8-shot CoT on GSM8K, which is also open-ended generation like list function tasks?\n\nThanks for allowing us to elaborate on how symbol tuning might improve algorithm chain-of-thought (CoT) reasoning. We also were interested in how symbol tuning might translate to CoT settings, and in Appendix B.2, we experimented on whether symbol-tuned models were better at MMLU and BIG-Bench-Hard, two well-known benchmarks. \n\nWe found that there was no significant improvement on these benchmarks in a CoT setting (see Figure 14), despite the fact that the intermediate steps in CoT could be viewed as a symbolic mapping problem. This result may actually be expected, however, as our symbol-tuning data did not include any data with CoT reasoning (instead only containing input–label mappings without any intermediate reasoning steps). Although we did not explore how to apply symbol tuning to CoT data in this work because it is unclear how to best replace intermediate steps with symbols, we agree that the idea is an exciting direction that future work can explore.\n\nTo make this section more noticeable for readers, we added a footnote in Section 5 following “while these tasks are more open-ended generation problems”: \n\n“Although chain-of-thought (Wei et al., 2022b, CoT) can be viewed as an open-ended generation problem, in Appendix C.2, we found that symbol-tuning did not significantly affect a model’s CoT reasoning abilities, possibly because our symbol-tuning data did not incorporate any CoT prompts.”\n\n> The authors only experiment with one class of closed-source models [...] the effectiveness of symbol tuning on the open-source LLMs such as Llama 1,2, is still unclear.\n\nThank you for bringing up this important comment about the generalizability of symbol tuning towards other families of language models. We agree that this is a limitation of our work, as successfully applying symbol tuning to other language models would likely require some modifications that are currently unknown. For this reason, we added a paragraph discussion this limitation in Section 8:\n\n“Because we only experimented with one family of language models, it is still unclear whether symbol tuning is effective for other models. Applying symbol tuning to other language models would likely require adjustments to the finetuning procedure to be successful (e.g., number of finetuning steps, mixing with previous data, number datasets), but a definitive conclusion about these factors cannot be drawn without further experimentation. We thus note that a crucial direction for future work is to explore how well symbol tuning translates to other language models.“\n\n> The font does not seem to follow the EMNLP2023 template.\n\nThank you for pointing out this detail, we have fixed this in the revised manuscript. With this corrected font, our manuscript is still within the page limits after moving the limitations section to be in the extra space allowed after the 8th page (as allowed by the EMNLP style, 2023.emnlp.org/calls/style-and-formatting/). Our revised manuscript with the font change conforms to the 9-page limit allowed for accounting for reviewer comments (2023.emnlp.org/calls/main_conference_papers/#long-papers).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The symbol tuning approach is simple and effective. It seems to be complementary to instruction tuning in terms of the algorithmic reasoning ability and strict instruction-following ability (e.g. flipped labels).\n2. The extensive experiments across model scales make the results convincing.\n3. The details of the experiments are available.", "reasons_to_reject": "1. The authors only experiment with one class of closed-source models which are inaccessible to most researchers. The effectiveness of symbol tuning on the open-source LLMs such as Llama 1,2, is till unclear.\n2. The template does not look like EMNLP 2023 template.", "questions_to_authors": "1) What is the performance of symbol tuning an LLM without instruction tuning? Will it still increase its ability on algorithmic reasoning and following flipped labels?  \n2) What is the relationship betweem instruction tuning and symbol tuning (only for classification problems)? \n3) Does symbol tuning increase algorithmic reasoning in CoT settings, like 8-shot CoT on GSM8K, which is also open-ended generation like list function tasks?", "paper_topic_and_main_contributions": "This paper proposes symbolic tuning for in-context learning.\nIn symbol tuning, we replace the semantic labels of the few-shot exemplars with random irrelevant labels and remvoe the instructions in the exemplars. Then the instruction-tuned models are finetuned with the new data.\nAccording to the authors, symbolic tuning not only improves performance on few-shot classification problems, but also enhances algorithmic reasoning ability and can restore the flipped-label following ability.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "The font does not seem to follow EMNLP2023 template."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes symbol tuning, a novel fine-tuning method for LLMs that uses random symbols instead of natural language labels and instructions. The paper shows that symbol tuning boosts LLMs’ performance on in-context learning and algorithmic reasoning tasks, and enables them to override prior knowledge with contradictory information. The paper tests symbol tuning on various benchmarks.
The reviewers like the paper for its novelty, effectiveness, motivation, analysis  and comprehensive experiments. The reviewers  only mention some minor issues, as well as reproducibility problem due to the closed-source models. The reviewers recommend accepting the paper, as it offers a simple yet effective  technique for improving LLMs’ in-context learning and algorithmic reasoning abilities. The paper also opens new possibilities for creative uses of input-label pairs during in-context learning.

Pros:
The symbol tuning approach is simple, effective, and seems to complement instruction tuning.
Novelty: The authors provide a novel and effective approach to fine-tuning models.
Robustness: The methodology is supported with clear and logical analysis on the impact of symbol tuning in various contexts.
Strong Motivation: The paper addresses the concern of prompt sensitivity in LLMs performance, providing a strong motivation for introducing symbol tuning.
Usefulness to the NLP Community: The symbol tuning approach is beneficial for training LLMs using in-context examples.
Strong Experiment Results: Comprehensive experiments were performed on a broad range of NLP tasks and with varied sizes of LLMs. 
Detailed Experiments: The details of the experiments are available, making the results convincing.

Cons:

Reproducibility Issue: Due to the models tested being closed-source, reproducibility could be an issue, although the process of symbol-tuning is described in detail.
Limited Experimentation: The authors only experiment with one class of closed-source models which are inaccessible to most researchers. The effectiveness of symbol tuning on open-source LLMs such as Llama 1,2, is still unclear.
Lack of Evidence: While the authors hypothesize why symbol tuning allows models to perform better, there’s a lack of clear evidence backing these model’s performance.

Please note that a reviewer  mentioned template Issue: The template does not look like EMNLP 2023 template.
Decision: Accept-Main