Compositional Task Representations for Large Language Models
Authors: NAN SHAO
Zefan Cai
Hanwei xu
Chonghua Liao
Yanan Zheng
Zhilin Yang
Venue: ICLR 2023 poster (2023)

Abstract: Large language models have shown a remarkable cross-task generalization ability. Most prior work assumed that prompts effectively extract knowledge from language models to facilitate generalization to new tasks. This perspective led to numerous studies on improving prompts. In contrast, we introduce a new perspective, compositional generalization, that views each task as a composition of latent codes and generalizes to test tasks by a new composition of seen codes. To this end, we propose a novel prompt-free approach, Compositional Task Representations (CTR), that employs multi-task training to learn a discrete, compositional codebook. Empirically, our CTR substantially outperforms prompt-based methods in zero-label learning on average. According to our analysis, some of the learned CTR codes are interpretable to human and demonstrate a certain degree of controllability.


Published as a conference paper at ICLR 2023
COMPOSITIONAL TASK REPRESENTATIONS FOR LARGE LANGUAGE MODELS
Nan Shao"!, Zefan Cai*, Hanwei Xu', Chonghua Liao", Yanan Zheng", Zhilin Yang 34451 Recurrent AI, "Beijing Jiaotong University. TTsinghua University *Shangaai Artificial Intelligence Laboratory, "Shanghai Qi Zhi Institute (windoker, zefncai 0 @mgall com Izyanan, zhiliny) stsnnguu .edu.
ABSTRACT
Large language models have shown remarkable cross-task generalization abil- ity. Most prior works assumed that prompts effectively extract knowledge from language models to facilitate generalization to new tasks. This perspective led to numerous studies on improving prompts. contrast, we introduce new perspec- tive, compositional generalization, that views each task as composition of latent codes and generalizes to test tasks by a new composition of seen codes. To this end, propose novel prompt-free frrept-free approach, Compositional Task Representa- tions (CTR), that employs multi-task training to learn a discrete, compositional codebook. Empirically, our CTR substantially outperforms prompt-based meth- ods in zero-label learning on average. According to our analysis, some of the learned CTR codes are interpretable to humans and demonstrate a certain degree of controllability.
INTRODUCTION
Large language models (LLMs) have shown remarkable performance in cross-task generalization. Without using any labeled data for the target task, GPT-3 (Brown et al., 2020) obtains reasonable per- formance on wide range of tasks. Later extensions such FLAN (Wei et al., 2022) and TO (Sanh et al., 2022) continue training the LLMs on a large number of supervised tasks, which further im- proves cross-tasi taakk generalization performance. The aforementioned studies have used an important assumption that natural language prompts extract knowledge from LLMs to facilitate generalization new tasks. In this direction, numerous studies have focused on different aspects of improving prompt-based learning, such as designing better prompts (Xu et al., 2022), increasing the number of prompts (Wang et al., 2022; Aribandi al., 2022), and improving the training efficiency of prompts (Lester al., 2021).
In contrast, we explore an alternative perspective for cross-task generalization, i.e., compositional generalization. Specifically, we explore whether it is possible represent tasks using discrete com- positions of latent codes. This perspective enjoys several potential benefits. First, since the latent codes have been trained for seen tasks, expect the LLMs to have strong cross-task generaliza- tion abilities because new tasks can also be represented as a composition of these trained codes. Second, it provides way analyze and understand cross-task generalization by investigating the association between tasks and the learned representations. Third, has the potential of being more controllable than prompts for task generalization due its built-in compositionality.
Motivated by the aforementioned potentials, we propose a new method, Compositional Task Rep- resentations (CTR), that employs multi-task training learn a discrete. compositional codebook. Specifically, given large number of training tasks, we use an encoder to map each randomly- initialized task embedding fixed-length sequence of query vectors. Each query vector is used to retrieve code from a codebook, which is formulated as an embedding lookup table. This produces
"qqual contribution.
"Corresponding author.
"The code will be available wwtpt:tt/////...... http:s:///ittt http://tttttttte/o com/ shaonan 993/CTR.

---

Published as a conference paper at ICLR 2023
Task Inputs
Compositional Task Code
Generated Outputs
!oochee the MrDiin Ctema dehote but night was fall of aomments had to pusse the DVR bosunac fedi dovane whet they wame caying my Iushand: llaaree about
CoamesQA (Gode 59 is shared tasks that multiple- choice/ hetpe//ressnnnngeee. based QA):
Moderate
points sawtthm both
[23. 90, 69, 15, 2, 79, 25, 103, 120, 44]
The unemppyymmnt este exteorupped 8.2% lastmmnnt, butthe sousunt unly salãed 201000 jobs when 203 new jobs had been pendiatesl, report the Will lournal's Marionllee ilkagoon worts "Woat!!! Sod numberr 'hhe unamsploymmnt rate, hanwever, bettor nrwee it had bost sopaected hold swady at 9.% Bhe nobe that the dip montty disetto more Amseiion kosss alring soeking emphgggmmlmmeeeet
biundlrs Hose would thi persson be dasefled i- Moderatt- Liberal- Consengtive- Nane the shove chassos.
MultiNNwss (Cade exists long song-eentesceegeee geseration tasks):
Document: National Archavon Yes, agatt,
time Seiintareuss
Fritay
WailiShivee, Wosshingttn Main Street arvall silgged one thing, jobs. A frish updat: the U.S. emplyymmnt sit cation for jintory hite the wire 8.00att
[54, 14, 58, 28, 117,33, 16, 53, 109, 111]
CTR+LLM
AGNews (Code (152,79.911119119 accurs in tasks that select from muitiple options]:
Rand on chack rste ond intened ratecare milnnoed to by South African world politios spors, business, science tcchsology??
[23, 90, 76, 15, 79, 119.19]
Training Plusse
Zero-Label inforence Phose
COPA femtennce
persion,
She received her pursion.
paod mortiags
[111,7, 69, 15. 2. 39 119,19]
The girl Sound a bug her coreal so... She poured milk the bosel She lust her appetite
She lest her appetiie.
Figure 1: An illustration of how CTR generalizes to zero-label tasks. In this real example produced by our model, CTR combines the abilities of reasoning-basee baaed QA, sentence generation, and multi- choice selection from training tasks to perform a new task COPA.
sequence of codes, compositional representation of the current task. These compositional codes are fed as the input to an LLM in place of prompts to make predictions. At test time, given new task, we unlabeled data to search for high-performing composition of codes, which enables zero-label cross-task generalization. CTR is also applicable to the few-shot setting where the few labeled examples used for code search.
Empirically, we demonstrate improved performance under both the settings of zero-label learning and few-shot learning, outperforming strong baselines including prompt tuning, model tuning, and genetic prompt search (Xu et al., 2022). Importantly, we analyze the learned task representations and show that they demonstrate certain degree of interpretability and controllability. For example, as shown in Figure 1, CTR learns to generalize to new task by a new composition of existing codes.
2 RELATED WORK
Language Model Prompting. Brown et al. (2020) showed that GPT-3 performs well in the few- shot setting if properly handcrafted prompts are provided. Other works (Shoeybi et al., 2019; Rae et al., 2021: Schick & Schütze. 2021) also report promising zero-shot or few-shot performances of LLMs. Besides, Wei et al. (2022) and Sanh al. (2022) collect a set of labeled datasets and use manual templates to transform them into a sequence-to-sequees possible to continue training LLMs on these labeled datasets and improves cross- generalization. Wang et al. (2022); Mishra et al. (2022) introduced a benchmark of over 1600 tasks and their expert- written instructions Gao et al. (2021); Shin et al. (2020) studied automating the search process of discrete prompts. Li & Liang (2021); Liu et al. (2021) propose continuous soft prompts with gradient- based optimization. Compared to these approaches. we study a different approach that learns compositional task representations, which benefits cross-task generalization.
style. Such a formulation makes it
Compositional Architecture for LLMs. Previous work has explored designing compositional architectures. Sparsely Gated Mixture of Expert (MoE) (Lepikhin et al., 2021) 1) activates a subset of network given the input data. Artetxe et al. (2021) trained an MoE model with 1.IT parameters, which is shown to outperform a dense model with similar computational cost. SkillNet-NLU (Tang et al., 2022) and SkillNet-NLGG (Liao et al., 2022b) employed similar sparsely activated mechanism to handle different NLU or NLG tasks. Different from these approaches, our approach focuses on learning compositional task representations using discrete codebook.
2

---

Published as a conference paper at ICLR 2023
G. go
Oaddbook Ambeddnng
(ookap
Compesilocal
[Gll(a] [5allGal t
Codebook Enbookinng
(Lolegg)
Gorpodtizcal
(. Real Bal Rale
Dicnete
Conposttionl
Decoder
Uzery Embhakding
Embeiding loss Connilimnnt Less
Speclyy code cortain Statage code endemble or (3) hitwlee
Task Embodlinng for No.k: C
para aro talk:
(Laokup)
Task Iepute: TTh tollet filldd with water" "Waser flowed fron the-t"ou" The ran turned on the Sucet. so
LLM
Randomly Initislize Tunable Task
Embolding (85) (fa) gooo
(Passic Labelall Takk Outpate The tolke filed with volee
TD
Scornng code by evalssttng on the poeeds- cats. spectty
TiskID:k
Iopots: ternralshest years.
Task toputs and Targets
LM Loss
fteding
thrratively oods.
Trak Outpptts poetttve
Training Phase CTR
Inference Phase CTR
Figure 2:
An overview of the architecture of our proposed CTR. The left part illustrates the training phase, while the right part shows how CTR works during the inference phrase.
3 COMPOSITIONAL TASK REPRESENTATIONS
The motivation of CTR to explore the cross-task generalization ability of LLMs from a brand- new
generalization, and to further improve the performance of cross- task generalization. Specifically, our main hypothesis is that being trained on a variety of natural language tasks, LLMs will be able learn represent each task as a composition of discrete latent codes, where each latent code is associated with certain aspects of task. As result, CTR poten- tially enjoys the advantages of better cross-task generalization since could represent new tasks by forming new compositions.
This section introduces the overall architecture of CTR and how it trained to overcome optimiza- tion challenges. As Figure 2 shows, CTR consists of the CTR learning module and an LLM,, where the CTR learning module contains an encoder, a decoder, task embeddings, and a codebook.
3.1 DISCRETE LATENT TASK CODEBOOK
We define latent task codebook embedding space C E RSxD where S is the size of the latent codebook embedding space (i.e., each task code can take either of the S categorical values), and D the dimension of each latent code embedding C. RD, i E [1,2.... .S. This is analogous to the idea of VQ-VAE (van den Oord et al., 2017) where discrete latent codebook is also employed.
As Figure 2 shows, given a training task the input (assuming the task id is k), CTR first obtains its task embedding Eg E R" by retrieving from a randomly indttllliittializeed task embedding lookup table NND where N is the number of training tasks. The task embedding Es is then passed to the encoder module and is mapped into fixed-length sequence of query vectors Q E R where I, the length of the sequence. For each query vector Qt. E [1, 2, T.). it is used retrieve a task code embedding from the codebook embedding space. Specifically, it calculates the l2 distance with each the latent code embedding C₁ RP, 21,2,.-. ... .S], and find the nearest neighbor,
,S].
CTR: Czyy where žı arg min ||Q Cill2
In this way, all query vectors together produce a L-length sequence of code embeddings, denoted as the compositional task representations (CTR). They are then passed through a decoder, followed by being used as the input to an LLM in place of prompts to make predictions.
Let be vector of L latent codes 21, 22, .... ZL. We consider each latent code describing an attribute or necessary skill of certain task, such as task type, output space, etc. Intuitively, we need multiple codes to fully describe task, and each task is formulated as a composition of codes.
3.2 TRAINING
Training be challenging due the existence of discrete latent variables. Moreover, during the initial training phase, the CTR learning module is randomly initialized. As a result, the codebook embeddings C will have very different distribution to the query vectors Q, which increases the
3

---

Published as a conference paper at ICLR 2023
difficulty of optimization. We decouple training into two phases. In the first phase, freeze the LLM and only update the CTR learning module. This is followed by tuning all parameters.
In terms of the loss function, following den Oord al. (2017), employ two separate losses, an embedding loss and a commitment loss, to match the query vectors with the compositional task representations. These losses used in combination with standard language modeling loss,
I. - LIM +
Σ [gs [Q:] 2 BIQ - sg. (CTTR[]]
(2)
where LLM is standard language modeling loss for solving the target task, sg denotes stopping the gradients, and / is a hyperparameter.
3.3 INFERENCE
We consider two settings: zero-label learning and few-shot learning, and describe how we apply CTR in these settings.
Code Ensemble for Zero-Label Learning. In the zero-label setting, we are given a new task along with set of unlabeled data. The question is how to decide the code for this new task without labeled data. Our main idea to select one of the training tasks (319 in total in our experiments) and use its learned code for this new task. We first obtain set of candidate codes by examining how much a code gives predictions that deviate from a uniform distribution on the unlabeled data (Zhao et al., 2021). The candidate set is formed by N (set to 60 in our experiments) codes with the lowest deviations. We then ensemble the candidate set of codes to predict pseudo labels on unlabeled data, and select the code with the highest pseudo-label accuracy.
Bitwise Search for Few-Shot Learning. In the few-shot setting, we are given a new task along with set of labeled data. We this of labeled data nigh-performing code. Our preliminary study shows that one can control the output of CTR by changing one bit in the code vector z. Inspired by this, we first examine the validation-set accuracy for codes of training tasks, and select the code with the best accuracy an initialization. Then we iteratively change a single bit of the selected code and evaluate the validation-set performance. At each iteration, we keep the updated code with higher performance. Finally, the code vector that obtains the best result on the validation set taken as the test-task code. Our preliminary study shows that a certain code value usually occurs in a specific position of the code vector, and each position usually only has a small set of code values. Motivated by this, we only changes a small set of valid code candidates for each position.
validation set to search for
In both the zero-label and feo- settings, after we obtain a code vector z for the new task, we use the vector to obtain composition task representation by indexing in the codebook C. The task representations are then passed through the decoder and the LLM to perform the new task, similar to training time. This is also illustrated in the right part of Figure 2.
4 EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
We conduct extensive quantitative experiments to validate the performance cross-task general- ization of CTR. We mainly consider two settings, the zero-label setting and the few-shot setting. Aside from quantitative experiments, we perform qualitative analysis to understand the cross-task generalization ability by investigating the association between the discrete latent codes and the tasks.
4.1.1 DATASETS
CTR requires a large multi-task set for training, and held-out set of tasks whose types are never seen during training for evaluation. We follow the TO benchmark. The training part consists of
"To clarify, for the zero-label setting, the unlabeled data equals the inputs of the test task. Note that the inputs naturally exist long as we perform the evaluation. So it requires no extra effort to acquire data.
4

---

Published as a conference paper at ICLR 2023
39 tasks of 8 task types, including closed-book question answering (QA), multiple- QA, tractive QA, sentiment analysis, paraphrase identification, topic classification, summarization, and structure to text. The test part consists of 11 tasks of 4 task types, including natural language in- ference (RTE (Dagan et al., 2006), CB (De Marneffe et al., 2019), ANLI/R1-R3 (Nie et al., 2020)), coreference resolution (WSC (Levesque et al., 2012). Winogrande (Sakaguchi al., 2020)), sen- lence completion (COPA (Roemmele et al., 2011), StoryCloze (Mostafazadch et al., 2017), Hel- laswag (Zellers et al., 2019)), and word sense disambiguation (WiC (Pilehvar & Camacho-Collados. 2019)). Both the training and test parts are disjoint in task types, ensuring the zero-label setting. We follow TO to use the accuracy on the validation split of test tasks as the metric.
4.1.2 BASELINES
For the zero-label setting, we primarily compare CTR with the following baselines. It is noteworthy that all baselines share a similar model size as CTR (i.e., about 770M), thus are comparable. We provide implementation details of baselines in Appendix A.7.
TO (Sanh et al., 2022) shares similar goals as CTR, which uses prompted multi-task training to improve the generalization performance. TO reports the average results multiple prompts.
Self-Training Since unlabeled data are accessed, we consider the self- self-training (Schick & Schütze, 2020) method as one of the baselines. Starting from TO, it uses TO to label the unlabeled data, and further finetunes TO with the pscudo-labeled data. It also reports average performance
prompts. Manual-Code uses artificial feature vectors in place of CTR vector. Specifically, we manually label set of artificially-designed features for each task, including the number of input fields of the task, whether it requires reasoning, and whether it is a classification task, etc. Each task can be represented as an discrete feature vector, each dimension associated with one of the aspects.
ZPS (Liao et al., 2022a) is method for zero-label prompt selection. It first labels set of unlabeled data through prompt ensemble, and use the pscudo-labeled data to select the best natural language prompt for the test task. We apply ZPS to multi-task TO baseline.
For the few-shot setting where there are 32 test-set labeled examples, based on the multi-task TO, experimented the following five baseline methods.
Model Tuning directly finetunes the pretrained model using the test-set labeled data. Specifically, follow the few-shot setting Zheng al. (2021) use 16 labeled data for finetuning and another 16 labeled samples for model selection.
Prompt Tuning (Lester et al., 2021) introduces additional continuous prompts to the backbone model (i.e., TO) and trains the continuous prompts using the few labeled data.
GPS (Xu et al., 2022) is a genetic prompt search method. Based on TO, GPS gradually mutates the prompts with a generative model and uses the few labeled data to selects prompt candidates.
GRIPS (Prasad et al., 2022) is a gradient-free edit-based method for optimal prompt search.
Black-Box Tuning (BBT) (Sun et al., 2022) is a gradient-free few-shot prompt selection method. Unlike GPS and GRIPS, it searches for the best soft prompt embedding in continuous space.
4.1.3 TRAINING DETAILS
We instantiate our CTR with T5-Large (Raffel et al., 2019) being the LLM. We implement both the encoder and the decoder in our CTR model as two linear networks. We have experimented with various architectures (see Appendix A.4 for more details). For the first stage of training where the LLM is frozen, we use the Adam optimizer with learning rate of le-2, a decay rate 0.1, and batch size of 2048, We use codebook embedding dimension of 1024, which is the same as the hidden dimension. The CTR length is set at 10 and each position can be assigned values ranging from 0 to 127; i.e., the codebook size is 128. The hyperparameter is set at 0.1. We experimented with different codebook sizes and CTR lengths, and detailed results are provided in Appendix A.5. For the second training phase where all parameters are updated, we use the Adam optimizer with a learning rate of lc-4 and a batch size of 1024. We follow the training recipe as TO (Sanh et al., 2022) for the rest of the hyperparameters.
TO uses natural language prompts from PromptSource (Bach et al., 2022).
5

---

Published as a conference paper at ICLR 2023
Natural Language Inference
Sentence Completion COPA
Co-reference WSC
WSD
Method
Avg.
RTE
CB
ANLII
ANLI2
ANLI
Hella.
Story.
Wino.
WIC
Zero-Label Setting (unlabeled data of each test task)
TO-Large
72.67
56.55
32.77
32.15
34.38
85.36
27.18
93.04
63.94
54.35
50.33
54.79
Sclf-Training
73.57
76.14
34.42
32.90
37.44
87.45
30.33
94.54
57.08
56.56
50.75
57.38
Manual-Code Code
75.19
56.89
33.12
32.49
33.48
75.76
30.84
93.10
61.16
54.10
51.45
54.33
ZPS
79.06
67.86
31.20
31.10
34,25
88.00
29. 16
93.43
65,38
53,43
49.84
56.61
Our CTR
80.51
87.50
33.40
34.40
33.80
92.00
27.50
90.10
56.58
49.40
62.50
58.88
Few-Shot Setting (32 labeled data of each test task)
Model Tuning
75.31
80,95
35.73
31.31
35.93
82.05
41.86
92.04
55,96
56.74
52.15
58.18
Prompt Tuning
77.08
76.90
31.89
31.86
35.53
81.70
31.18
94.10
62.88
55.42
51.22
57.25
GPS
77.68
79.64
32.71
31,49
37.56
81.08
28.11
93.40
64.23
52.72
52.52
57.38
GRIPS
71.56
70.89
32.14
32.26
34.77
77.56
26.44
93.40
62.12
52.96
52.12
55.11
BBT
71.19
57.26
33.79
32.00
35.30
76.49
28.95
93.11
62.
53.40
52.93
54.23
Our CTR
80.51
83,93
34.40
34.20
36.60
89.00
35.07
91.70
68.18
55.00
58.62
60.66
Table 1: Main results of CTR and baselines on 11 test tasks under the zero-label setting and the few- shot setting. The zero-label setting allows using unlabeled data of the test task while the few-shot setting uses 32 labeled data of the test task. All methods share a similar model size (i.e., 770M).
4.2 MAIN RESULTS AND ANALYSIS
The performance of cross-task task generalization, respectively under the zero-label setting and the few- shot setting, are shown in Table 1. Our CTR outperforms all baseline methods on average under both settings. Comparing CTR with TO-Large and its variant (i.e., self-training), CTR outperforms them respectively by more than points and by almost 1.5 points on average. The improvements potentially originate from aspects- (a) the learned compositional task representations benefit from better generalization abilities than the discrete manual prompts used by TO/self-training: (b) our CTR can select gegh-performing compositional task representation for the unseen task. Compared with Manual-Code, our CTR demonstrate significant advantages of more than 4.5 points, proving that tttficialll-designee features of tasks are unreliable, and CTR provides an effective way of automatically training such data-driven compositional task representations using multi-tasks.
Compared with Model Tuning, Prompt Tuning, and BBT, where they require parameter updates over the test-task data, CTR shows better performance of cross-task generalization on average. Compared with baselines that do not tune parameters (i.e., GPS, GRIPS), CTR shows even larger and more con- sistent advantages on most of the test tasks; i.e., on 9/11 tasks CTR shows dominating performance.
On co-reference tasks, CTR performs worse under the zero-label setting and better under the few- shot setting. The zero-label setting uses pseudo-labeled data select test codes while the few-shot setting uses real-labeled data to select codes. The reason for the decreased performance lies in that the pseudo data of the co-reference task were of low quality and did not select effecctive task codes.
Please refer to Appendix A.3 for generated zero-label cases from CTR.
4.3
ABLATION STUDY
4.3.1 ARE MANUAL PROMPTS NECESSARY?
We are interested in whether CTR will be further improved when combined with discrete man- ual prompts. We conduct comparative experiments of CTR respectively with and without manual prompts. Specifically, for CTR without manual prompts, the inputs are direct concatenation of multiple text fields, with compositional task representations appended in front of it. For CTR with manual prompts, the inputs are constructed by leveraging TO prompts, with CTR placed in front.
The results are presented in Table 2. We report both the CTR results as well as its results. It is noteworthy that the "upper-bound" results are post-hoc, which are obtained by the best code/prompt with observing the test task performance, and are merely given for estimating the potential of the two methods. We shall observe that adding additional discrete manual prompts does not improve the CTR performance well as the tupper-bounts From the results, we shall conclude that (1) our CTR does not rely on manual prompts to obtain optimal performance. CTR can work as an alternative to the prompt-based methods. (2) By comparing the CTR performance and the CTR

---

Published as a conference paper at ICLR 2023
upper-bound, there is still room to optimize the code-searching algorithm, and as result to further improve the generalization performance.
Natural Language Inference
Sentence Completion
Co-reference
WSD WIC
Avg.
RTE
CB
ANLII
ANLI2
ANLI3
COPA
Hella.
Story.
WSC
Wino.
Our CTR
80.51
87.50
33.40
34,40
33.80
92.00
27.50
90.10
56.58
49,40
62.50
58.88
upper-bound
81.95
89.29
38.77
39,00
40.60
94.23
42.81
94.20
73.86
60.00
61.76
65.13
w/ Manual Prompt
72.56
82.14
35.00
35,20
38.90
90.00
30.30
92.70
55.68
52.00
55.33
58.17
upper-bound 80.87
85.71
41.50
39.80
44.60
93.00
38.76
95.60
71.59
59.90
60.97
64.75
Table 2: Ablation study on manual prompts. It shows the results of our CTR respectively without and with manual prompts. The experiments are under the zero-label setting. The "upper-bound" results of each method are obtained by using the best code/prompt after observing test task performance and are merely given for estimating the potential of the method. Results show that additionally adding manual prompts does not necessarily improve performance, and the codebook learned by CTR can act as an alternative to previous manual prompts.
4.3.2 TRAINING OBJECTIVES
Our CTR consists of three training loss items. Intuitively, the LM loss is optimized to predict the correct answer of the tasks. The embedding loss and the commitment loss are optimized to minimize the distance between the query embeddings and the CTR that are lookup from the codebook embed- dings. To study the effectiveness of each loss item, conduct an ablation study on them, and the results are shown in Table 3. Results show that removing either of the loss items will drastically hurt the zero-label performance, proving that either loss item indispensable for the training of CTR.
Loss Function
Natural Language Inference
Sentence Completion
Co-reference WSC
WSD
Avg.
RTE
CB
ANLII
ANLI2
ANLI3
COPA
Hella.
Story.
Wino.
WIC
Our CTR Loss
80.51
87,50
33.40
34.40
33.80
92,00
27.50
90.10
56.58
49.40
62,50
58.88
wa/ Commitment Loss
76.53
76.79
33.30
32.60
38.00
85.00
33.50
74.70
64.77
49.90
54.39
56.32
wa/ Embedding Loss
74.01
75.00
33.70
33.20
34.30
75.00
25.40
86.00
61.36
50.00
54.75
wo/ Ceamitmeen--Embedding
79.06
76.79
30.90
33,70
35.70
84.00
30.50
85,90
57.95
55.60
50.00
56.37
Table 3: Ablation study on CTR loss function. Results show that by removing either of the loss items, i.e., commitment loss, embedding loss or both, the performance decreases varying degrees.
4.4 IN-DEPTH ANALYSIS
4.4.1 INTERPRETARILITY
Since CTR is trained represent tasks with compositional codes, each code associated with one of the key aspects of tasks, CTR demonstrates a certain degree of interpretability. Table 4 presents examples of CTR that shows how each compositional task code is possibly interpreted.
Interestingly, it frequently occurred that tasks that share similar features indeed have the same com- positional code. For example, the code 52 occurs in tasks that require extracting information from the given contexts, including samsum_* * (summarization task), wiki_bio_* (structured data to text task), and labeled wwwpsslaeeeddfdallppppeeeee paraphrase task (paraphrase generation) etc. Another example that code 111 exists in most of the tasks that require generating long setences, including multi.news.* and samsum (both are summarization tasks).
We also validate these explanations unseen test tasks. Results show that these possible explana- tions still hold. For example, the COPA task generates long answers given two candidate choices, and its compositional code has the code 111 (indicating sentence generation).
4.4.2 CONTROLLABIILITY
CTR also demonstrates certain degree of controllability. Specifically, by modifying one bit of the compositional task code, our CTR will exhibit a different task behavior. Table 5 shows several examples of controlling CTR. We shall observe that, given the same inputs, by simply changing one
7

---

Published as a conference paper at ICLR 2023
Code: 41
Explain: generate key words based passages
TASKS:
2 wiki wiki_hop.orge hhwp.original.coooe Lchoose www.ooooooooooooooooio.oeee.e interrogative _2 wiki.hop.oris htt.hhogoiiigll.....o por.oirnal_cchoseeeeee choose_hest best_ohject.affrmmalle common, gen topics. from the sentence
wiki_hop.ool www.ooopppoolll..oomo www.arinalleeeeeattteeeeeae. generate.suujecl wuujesst.anddd......... .object
conmon.genn,eeteneeennoo wewteence_nnoooeceeeeee
Code: 52
Explain: extract the main information of the article
TASKS:
samsum Samsum_Sum_uu Sp_the following_dialogee
samsum,
samsum_Summart Summarize sattmmrrzze_thss...oo dhilooi
samsum_ samsum_To_sum _u_sum_ htt.muupttisssillll..
samsum. samsum_Generate Generate...summmattne summary _fr www.this..dllguu. wiki bio.kooooeeooooooo boo.key_conenl
samsum. CannsumGGiven... the wtthe_aboeeddee .olooou.eelouee.... summary
wiki_bio_cee sihoeemprreennionn paws laheled final paraphrase task
wiki_bio_whail wha__ccttttttttt
Code: 69
Explain: generate the given candidate choices
TASKS:
qusc gusc.qa_with seraratedd facts_1
quse.qa_witt sere-paattttsssaaaeeeddd facts
qasc_qa_with seepppp_withheeprateed
qase_qa_with htt.:p__wit_.ommbeeeeeeee frmbined_factk...
cosmos qa context description question answer text cosmos qa context questiin description answer text social qaqa.iqaaua......... Show
cosmos qa description context question answer text
context_question.
cosmos, qu no pwwmpttotsss text
Shw..Shw............oo.......... wwww.choices.......eeeeeeeeeoee. choices and generate answer
tree fine grained WOC.L.OCC.oomc contextttttttoooc first super glue copa hest option
tree fine grained L.CC
Code: III
Explain: Generate long sentence
TASKS:
multi_new, wwwil_nwssssme hwwps_summrryscccoa
multi_news, htr__nnssssmmmme summarize multi news expand (reverse task) super glue copa best option
multi news distill
samsum Write dialogue that match this summary
Table 4: Examples of how compositional task codes can be possibly interpreted. We use codebook size of 128. It shows the co-occurrence of tasks and codes. By analyzing common features shared by multiple tasks, we shall find that CTR embraces the advantage of interpretability.
bit of the compositional task code, the task behavior of CTR turns from DIALOGUE GENER ATION to TOPIC CLASSIFICATION, from REVIEW RATING to SUMMARIZATION, from SENTIMENT ANALYSIS to TOPIC CLASSIFICATION, etc.
We shall conclude- (1) Since the inputs are randomly sampled from all tasks, some of which are quite different in data distribution, CTR performs well on them, proving that CTR indeed learns the ability to perform different tasks, instead of simply memorizing/overfitnng (2) CTR is capable of switching between different tasks by simply changing one bit the compo- sitional code, proving that CTR effectively encodes the "task behavior" factor into the bits of the compositional code while disentangling other factors.
certain dataset.
4.4.3 How CTR GENERALIZES TO NEW TASKS
To reveal how CTR essentially works,
Figure explain how CTR generalizes from training tasks to unseen tasks such as COPA. During the training phase, CTR learns the compositional task code for each task, each code associated with certain features of the task. For example, code 69 is associated with many of the pallippe-coiiccrraaoonnn---- with most of the long-sentence-genn generation
QA tasks; code 111 is associated sentence--enerraiion tasks, etc. For new task, its task code is a re-arranged composition of the code existing in training tasks. If the new task shares the same code snippet as the training tasks, it inherits corresponding task features as well. the example of Figure 1, real case produced by our model, the new task COPA is performed by combining the abilities of reasoning-baat based QA, sentence generation, and culti- selection from training tasks. These observations consistent with motivation of learning CTR for cross-task generalization.
4.4.4 GENERALIZING TO NEW UNSEEN DOMAINS/CLASSER
Another interesting finding is that, for tasks with limited categorical classes, e.g., topic classifica- tion tasks, CTR is capable of generalizing to new unseen classes/domains, Table 6 shows several examples. Given different inputs that are randomly sampled from a large set of tasks and the com- positional task code of AGNews, CTR not only predicts topic classes that are originally defined
8

---

Published as a conference paper at ICLR 2023
Jim moves some particles of matter farther apart, what will happen to the which they can pass vibrations on to nearby particles decrease or increase? Assuming that: When particles matter are closer logether, they can more quickly pass the energy of vibrations to nearby particles.
INPUT:
TASK CODE: [61, 96, 127, 113, 55, 107, 5, 87, 5. 5]
FILII 52 > DIALOGUE: W: Hi, Jim. I'm Jim. I'm a physicist at the University of California, Berkeley.
FILL 37
TOPIC CLASSIFICATION: Science and Technology
INPUT: A historic victory for the separatist Parti Quehecois was marred by an attack on a victory rally for premier-elect Pauline Marois. A man opened fire during her victory speech, killing one person and critically wounding another, reports CTV. ... "I have convictions and I am going defend them." Marois said during her victory speech. "There will be referendum when the Quebec population wants referendum."
TASK CODE: [31, 4, 113, 55, 107, 18, 87, 12, 63]
FILL 68 REVIEW RATING: star
FIL.I 93 + SUMMARIZATIONS : One person was killed and another critically injured a shooting at a victory rally for the separatist Parti Quebecois in Quebec City on Sunday.
INPUT: Paragraph: v've been here few times and I like that it's right up the street. On Tuesday and Wednesday they have specials on manicures and pedicures which is $5 cheaper then the original price.
TASK CODE: 123, 90, 76, 15, 2, 79, 1, 119,
FILL 13 -> SENTIMENT ANALYSIS: Positive
FILII 5 + TOPIC CLASSIFICATION Business.
Table Examples of controlling the compositional codes. Given the input and a compositional task code, each time we modify one bit of the code, CTR begins to perform a different task.
Predict with the code for AGNews (topic classification): [23, 90, 76, 15, 2. 79, 39, 1, 119, 19] Original News classes: (World, Sports, Business, Science and technology]
INPUT: Fears for T N pension after talks Unions representing workers at Turner Newall say they are 'disap- pointed' after talks with stricken parent firm Federal Mogul.
OUTPUT: Business
INPUT: There are 10 apples on an apple tree. Three fall off. Now there are X apples.
OUTPUT: Math problem
INPUT: Stuning even for the non- non-gamer This sound track was heautiful! It paints the senery in your mind so well I would recomend it even to people who hate game music! It would impress anyone who cares to listen!
OUTPUT: Music
INPUT: Slack (2003) compares three groups that conducted biological research at Yale during overlapping periods between 1910 and 1970. Yale proved important as a site for this research. Hutchinson's 's example shows that new models for research groups are needed, especially for those that include extensive field research. OUTPUT: Ecology
Table 6: Examples of CTR generalizing to new unseen classes/domains. The inputs are randomly selected from all tasks other than AGNews. The first case predict the same classes as AGNews defines, while the latter three cases predict new classes that are never seen within AGNews. It shows the codebook of CTR can generalize to new unseen classes/domains.
by AGNews, but also predicts new topic classes that never occur in AGNews. Please refer to Ap- pendix A.I and Appendix A.2 for more cases.
CONCLUSIONS
In this paper, we explore cross-task generalization from new perspective perspective-cooppsiiionn ization. We propose the Compositional Task Representations (CTR) method that learns a discrete compositional codebook for tasks and generalizes to new unseen tasks by forming new compositions of the task codes. For the inference of CTR, we propose two algorithms Bitwise Search, respectively for zero-label and few-shot settings. Experiments demonstrate that our CTR significantly outperforms existing coompedd-based methods on both zero-label and feo-shot settings. Analysis of the learned compositional task codes proves that some of the CTR codes show certain degrees of interpretability and controllability.
compositional general-
Code Ensemble and

---

Published as a conference paper at ICLR 2023
REFERENCES
Vamsi Aribandi, Yi Tay. Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mchta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. In International Conference on Learning Representations, 2022. URL https: / opwpreveeeeeeeeeeeeeeeee. net forum? SL=Vzh1BFUCiIX,
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victo- ria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv: 12. 10684, 2021.
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Ab- heesht Sharma, Taewoon Kim, Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, An- drea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. PromptSource: An integrated de- velopment environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp- 93-104, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ URL https: /aclaathhlloyyyyoooo org.org/////o./o..oo /2022 acl-demo.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL https: htttttss: http:://coceeeeeegssseee neurips cc http//pape/2222////1/// 1457c0d6bfcb4Ber 111.441fffac1112211 -ateeff64a--aaaeere pdf.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment chal- lenge. In Joaquin Quiitonero-Candela Ido Dagan, Bernardo Magnini, and Florence d' Alché Buc (eds.), Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classi- fication, and Recognising Tectual Entailment, pp. 177-190, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg, ISBN 978- 778-3-540-33428-6
Marie-Catherine De Mamneffe, Mandy Simons, and Judith Tonhauser. vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pp. 107-124, 2019.
The commitmenthank: In-
Tianyu Gao, Adam Fisch, and Dangi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 011633816633 3830, Online, August 2021. Association for Computational Linguis- tics. doi: 10.18653/w1/202 18065:/1/002...ddooogg2229 URL https: http::///allaahhhhaaeae acl-long. 295.
org/2021.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. {GS] hhard: Scaling giant models with condi- tional computation and automatic sharding. In International Conference on Learning Represen- tations, 2021. URL https: /ttp///ppweeeew.eeeeeeeee net/ forum forum?id=qree htt.:eeeeweeeee.oouuoeeeeeeeeeee XHTmYb.
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10111165/11/222/1 lemnlp- main.243. URL https: / //cl//aaaaaaaa antho loyy. www.///g111 emnlp-ma i n. 244.
10

---

Published as a conference paper at ICLR 2023
Hector Levesque, Emnest Davis, and Leora Morgenstern. The winograd schema challenge. In Thir- teenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer, 2012.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/ v1/2021 11/2021.acl-long.10 URL https: http:///aaaaaatooolooooeo
www..aclloong...... 353.
Chonghua Liao, Yanan Zheng, and Zhilin Yang. Zero-label prompt selection. arXiv preprint arXiv:2211.04668, 2022a.
Junwei Liao, Duyu Tang, Fan Zhang, and Shuming Shi. Skillnet-nlg: General-purpose natural language generation with sparsely activated approach. arXiv preprint arXiv:2204. 12184, 2022b.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103. 10385, 2021.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470-3487, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 18118/8//200222acllloo/ 244. URL https: /tti://allaa /aaaaaaannotoooooooooooao org/ / 20222 aal-laallnnno 244.
Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. LS- DSem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Link- ing Models of Lexical, Sentential and Discourse-level Semantics, pp. 46-51, Valencia, Spain, April 2017, Association for Computational Linguisties. doi: 10. 10.18663/11/W17-00006 URL org /w////1709900
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901, Online, July 2020. Association for Computational Linguistics. doi: 111166/3//222222ccmmmmiii https:
URL
.rg wttp://.ooo.//.....mmmaaaaaaaaa. acl-main. 441.
Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 1267-1273, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/11/11-1111 URL https: /ate:///aalaaa www.cannhhologo...
Marg/N19-11221 9-1128
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-fres edit-based in- struction search for prompting large language models. arXiv preprint arXiv:2203.07228, 2022.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 22Xiv:2122.1144 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.1068s 2019.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible altemna tives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, pp. 95, 2011.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. In AAAI, pp. 8732-8740. AAAI Press, 2020.
11

---

Published as a conference paper at ICLR 2023
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Repre- sentations, 2022. URL https: /epp/pppeeeieeeeeeeeeeeeeeeeeeee net./ forum?
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. CoRR, abs/2009.077 18, 2020.
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255-269, Online, April 2021. Association for Computational Linguistics. doi: 10. 11118653/112021..aa0 mail-main.20. URL https: / //aclanthology org/ saacllmaannnna 20.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Auto- Prompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pp. 4222-4235, Online, November 2020. Association for Computational Linguis- tics. doi: 10.18653/w1/2020.com main..46. URL https: / /aaaaaaaaanttoooooooooo org/202C. emn.p-mmain. 346.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism. arXiv preprint arXiv:190900805. 2019.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-modell a- arXiv preprint arXiv:2201.035// 2022.
Duyu Tang, Fan Zhang, Yong Dai, Cong Zhou, Shuangzhi Wu, and Shuming Shi. One model, multiple tasks: Pathways for natural language understanding. arXiv preprint arXiv:2203.03312, 2022.
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learn- ing. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL https: http:///cccceececeeeeeess/ccssss neurips cctppaper/211//EE... pdf.
7898af17 73 wwwee oar.e//e00000000eeee000000000000
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv.2204.07705, 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In Interna- tional Conference Learning Representations, 2022. URL Srrumiid=qE22rGozddg
n
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang. Ze- roprompt: Scaling prompt-based pretraining 1,000 tasks improves zero-shot generalization. arXiv preprint arXiv:2201.069) 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can ma- chine really finish your sentence? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics. pp. 4791-4800, Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10.18653/w1/P19. 1472. URL https: / /aacaanthooogyy .rg// P1 1472.
12

---

Published as a conference paper at ICLR 2023
Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. ArXiv, abs/2102.09690, 2021.
Yanan Zheng, Jing Zhou, Yujie Qian, Ming Ding, Jian Li, Ruslan Salakhutdinov, Jie Tang, Sebastian Ruder, and Zhilin Yang. Fewnlu: Benchmarking state-of-the-art methods for few-shot natural language understanding. CoRR, abs/2109. 12742, 2021.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 EXAMPLES OF AGNEWS
QUESTION: Fears for N pension after talks Unions representing workers Turner Newall say they disappointed" after talks with stricken parent firm Federal Mogul.
MODEL ANSWER (WITHIN THE ORIGINAL CLASSES); Business
QUESTION: Prediction Unit Helps Forecast Wildfires (AP) AP- It's barely dawn when Mike Fitzpatrick starts his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring. Lightning will strike in places he expects. Winds will pick up. moist places will dry and flames will roar. MODEL ANSWER (WITHIN THE ORIGINAL CLASSES): Science and technology
INPUT: How did serfdom develop in and then leave Russia?
MODEL OUTPUT (OUT OF ORIGINAL CLASSES): History
INPUT: There 10 apples
apple tree. Three fall off. Now there are X apples. MODEL OUTPUT (OUT OF ORIGINAL CLASSES); Math problem
INPUT: the rock is destined to be the 21st century's new "conan" and that he's going make splash even
greater than aroold schwarzenegger, jean-claud van damme or steven segal.
MODEL OUTPUT (OUT OF ORIGINAL. CLASSES): Film
INPUT: D. Abbott Ltd Abbott of Farnham
Abbott Limited was British coachbuilding business based in Farnham Surrey trading under that name from 1929. A major part of their output was under sub-contract to motor vehicle manufacturers. Their business closed in 1972.
MODEL OUTPUT (OUT OF ORIGINAL CLASSES): Company
Stuning even for the con-gamer This sound track was beautiful! It paints the senery in your mind so well would recomend it even people who hate vid. game music! I have played the game Chrono Cross but out of all of the games I have ever played it has the best music! backs away from crude keyboarding and takes a fresher step with grate guitars and soulful orchestras. It would impress anyone who cares to listen!
INPUT:
MODEL OUTPUT (OUT OF ORIGINAL CLASSES): Music
INPUT: E. D. Abbott Ltd Abbott of Farnham E D Abbott Limited was a British coachbuilding business based in Farnham Surrey trading under that name from 1929. A major part of their output was under sub-contrace to motor vehicle manufacturers. Their business closed in 1972.
MODEL OUTPUT (OUT OF ORIGINAL CLASSES): Company
QUESTION: Slack (2003) compares three groups that conducted biological research at Yale during overlapping periods between 1910 and 1970. Yale proved important as site for this research. The leaders of these groups were Ross Granville Harrison, Grace F. Pickford, and G. Evelyn Hutchinson, and their members included both graduate students and more experienced scientists. All produced innovative research, including the opening of new subfields in embryology, endocrinology, and ecology. respectively. over long period of time. Harrison's 's group is shown to have been classic research school; Pickford's and Hutchinson's 's were not. Pickford's group was successful in spite of her lack departmental or institutional position or power. Hutchinson and his graduate and postgraduate students were extremely productive, but in diverse areas of ecology rather than one focused area of research or the use of one set of research tools. Hutchinson's 's example shows that new models for research groups needed, especially for those that include extensive field research. MODEL ANSWER (OUT ORIGINAL CLASSES): Ecology
Table 7: Examples of CTR by the compositional code of AG News. Given different inputs that randomly sampled from large of tasks, and the compositional task code of AGNews, CTR not only predicts topic classes that are originally defined by AGNews, but also predicts new topic classes that never occur in AGNews.
14

---

Published as a conference paper at ICLR 2023
A.2 EXAMPLES OF DBPEDIA.14
Federation of International Trade Associations - The Federation of International Trade Associations (FITA) based in Reston Virginia and New York New York USA was founded in 1984. It fosters international trade by seeking to strengthen the role of associations in the United States Mexico and Canada. FITA is the strategic partner of the United States Commercial Service for e- commerce. company, educational institution, artist, athlete, office holder, mean of transportation, building, natural place, village, animal, plant, album, film or written work MODEL ANSWER (WITHIN THE ORIGINAL CLASSES): Company
QUESTION:
Furian knife In this movie sequel, Vin Diesel returns Riddick, escaped convict with a price on his head. Riddick has been hiding on a snow planet for the last five years, when a group of mercenaries try capture him. Riddick returns the planet Helion, and finds out that his friend Jack, is in prison on Crematoria, a very hot planet. While on Helion, the planet is invaded by the Necromongers, led by Lord Marshal (Colm Feore), who wants to rule the universe. Riddick is captured by the mercs and flown to the same unsavoury and possibly illegal prison Kyra is in. Turns out Jack is now known as Kyra (Alexa Davalos) and tough as nails.
QUESTION:
MODEL ANSWER (WITHIN THE ORIGINAL CLASSES); Film
lila abu- lughod' s: name alma mater- website known for birth date employer nationality occupation Bio: lila abu-lughod -Irb- born 1952 -rrb- is an american with palestinian and jewish ancestry who is professor of anthropology and women's and gender studies columbia university in new york city. specialist of the arab world, her seven books, most based on long term ethnographic research, cover topies from sentiment and poetry to nationalism and media, from gender politics to the politics of memory.
QUESTION:
MODEL ANSWER (WITHIN THE ORIGINAL CLASSES): Educational Institution
when he dug his massive body into the batter's box tuesday night at legends field during an exhibition game against the new york yankees, adam dunn not only wasn't 't hitting his weight, he wasn't hitting the weight of an UNK bag of baseballs.
INPUT:
MODEL OUTPUT (OUT OF ORIGINAL CLASSES): Sports
We know that, thanks our DNA, each of us is little bit different. Some of those differences are obvious, like eye and hair color. Others are not so obvious, like how our bodies react medication. Researchers are beginning to look at how tailor medical treatments to our genetic profiles, in relatively new field called pharmacogenomies. Some of the biggest breakthroughs have been cancer treatment. MODEL OUTPUT (OUT OF ORIGINAL CLASSES): Science
INPUT:
INPUT: how did athenians make money? Other Greek cities set up democracies, and even though most followed an Athenian model, none were as powerful, stable, nor well-documented as that of Athens.
MODEL OUTPUT (OUT OF ORIGINAL CLASSES): City
Great food, portions could be smaller. little pricey for Middleton. Would try somewhere else next time but would also not hesitate to return
INPUT:
MODEL. OUTPUT (OUT OF ORIGINAL CLASSES): Restaurant
us - caption: mr. october 2007 marcus at a porn star karaoke event, height: 5 8 alias: mister marcus, marquis image: mr marcus, 2007. birth date: september 1970 - birth name: jesse spencer weight: 200 lb kg on ethnicity: black birth place: pomona, california, usa hair color: black number of films: 1,782 -lrb- per iafd -rrb- performer& & 36 as director eye color: brown mr. marcus.
INPUT: Facts: name: mr. marcus - spelling:
MODEL OUTPUT (OUT OF ORIGINAL CLASSES): Actor
Table 8: Examples of CTR by the compositional code of DBpedia_14 Given different inputs that are randomly sampled from large set of tasks, and the compositional task code of DBpedia_14 14, CTR not only predicts topic classes that are originally defined by DBpedia_14, 14, but also predicts new topic classes that never occur in DBpedia.
15

---

Published as a conference paper at ICLR 2023
A.3 EXAMPLES OF ZERO- TASKS
COРA Task Code:
INPUT: The woman retired. so... She received her pension. She paid off her mortgage.
OUTPUT: She received her pension.
INPUT: The girl found bug in her cereal. so... She poured milk in the bowl. She lost her appetite.
OUTPUT: She lost her appetite.
Winogrande Task Code: 1,,27,761,.22799,,,,,,11911
INPUT: Sarah was much better surgeon than Maria so always got the easier cases. Sarah or Maria?
OUTPUT: Sarah
INPUT: Terry tried bake the eggplant in the toaster oven but the was too big. eggplant or toaster?
OUTPUT: eggplant
3lozeCloz/22016 Task Code: [23.68.5..0.2272225510....
Sam loved his old belt. He matched it with everything. Unfortunately he gained too much weight. It became too small. Sam went on a diet. Sam was happy.
INPUT:
OUTPUT: Sam went on diet.
Larry bought a new motorcycle. He was excited to look cool. The first time he tried riding it he dropped it. He hurt his leg and had to go to the hospital. Larry loved going to the hospital. Larry became careful.
INPUT:
OUTPUT: Larry hecame careful.
Winograd Schema Challenge Task Code: [5. 68. 4. 10, 8, 22, 18, 124, 12, 97]
INPUT: Jane gave Joan candy because she was hungry. Jane was hungry. Yes or no? OUTPUT: No
INPUT: Joe Joe's uncle can still beat him tennis, even though 30 years older. Joe is 30 years older. Yes no?
OUTPUT: No
Table 9; Examples of generalizing to tasks of unseen types. It shows cases of the CTR code for zero-label test tasks, and corresponding generated examples.
A.4 COMPONENT ARCHITECTURE
Natural Language Inference
Sentence Completion
Co-reference WSC
WSD
Encoder-Decce Decoder
Avg.
RTE
CB
ANLII
ANIJ2
ANLJ3
COPA
Hella.
Story.
Wino.
WIC
CTR: Linear - Lineur
80.51
87,50
33.40
34.40
33.80
92.00
27.50
90.10
56.58
49.40
62.50
58.88
Linear- None
77.98
82.14
32.40
33.30
33.60
86.00
29.43
89.30
55.68
57.70
50.00
$7.05
Lineur MLP
77.26
80.36
32.40
34,00
35.60
84.00
26.86
84.10
54.55
55.70
50.31
55.92
Linear Transformer
77.98
80.36
32.70
31.80
35.70
88.00
27.72
89.10
54.55
58.00
49.84
56.89
Lineur- RNN
78.70
80.36
33.80
32.50
36.30
87.00
30.70
87.30
55.68
57.20
56.11 11
57.79
MLP- - Lincar
79.06
71.43
32.40
33.40
35.20
88.00
30.25
89.70
60.23
55.70
49.80 84
56.84
MLP -one
78.34
76.79
32.90
33.30
33.80
87,00
30.08
90.60
52.27
56.60
50.31
56.54
MLP- MLP
76.53
76.79
33.20
33.30
34.30
84,00
28.14
90.90
52.27
55.00
50.47
55.90
ML.P Transformer
78.70
75.00
32.40
33.00
33.90
82.00
28.70
91.00
56.82
55.70
50.16
56.12
MI.P- RNN
79.78
73.21
33.20
33.70
34.50
28.64
90.90
57.95
50.00
56.71
Table 10: Ablation study on different architectures of the CTR encoder and decoder. All experiments are conducted under the zero-label setting. We experiment two of the encoder, respectively the linear net and the Multi-layer perceptron (MLP), and five alternatives of the decoder, including the linear net, bidirectional RNN, Transformer, MLP and removing the decoder (None). Results show that the simple "Linear- Linear" combination achieves the best performance.
Table 10 shows the zero-label results when using different architecture of the encoder and the de- coder. We experiment two of the encoder, respectively the linear net and the multi-layer perceptron (MLP), and five alternatives former, MLP and removing the decoder (None). Results show that simply using a linear network for both the encoder and the decoder performs best, while complicated architectures. e.g.. Transformer and MI.P, yield poor performance. This could because that the learning of codebook embeddings
the decoder, including the linear net, bidirectional RNN, Trans-
16

---

Published as a conference paper at ICLR 2023
well the code mapping does not necessarily need complicated transformation. When using complicated architectures, it instead increases the difficulties of the learning process.
A.5 CODEBOOK SIZE AND CTR LENGTH
There are critical hyper-parane prper-parameters for CTR -he codebook size and the CTR length. We conduct experiments of different selection of the two performance of zero-label generalization. Table 11 presents the results. We shall observe that when the codebook size decreases below 64, the performance decreases to large degree. We conjecture that it is due the capacity of codebook size below 64 is not sufficient for representing the multiple aspects of the tasks. In addition, experimenting with CTR length of 10 generally outperforms those with larger CTR length.
and see how it influence the
Codebook Size
CTR Length
Natural Language Inference
Sentence Completion
Co-reference
WSD WiC
Avg.
RTE
CB
ANLII
ANLI2
ANLI3
COPA
Hella.
Story.
WSC
Wino.
128
10
80.51
87.50
33.40
34.40
33.80
92.00
27.50
90.10
62.50
49.40
56.58
58.88
128
20
77.62
80.36
33.10
34.40
38.50
87.00
31.60
90.90
54.55
50.90
$5.80
57.70
64
10
75.81
73.21
31.90
31.70
36.70
89.00
26.90
65.40
48.86
$2.50
54.70
53.34
20
79.06
82.14
36.80
31.70
85.00
30.86
92.00
53.20
48
10
77.26
78.57
35.30
33.30
38.80
88.00
33.54
66.10
60.23
50.00
53.45
55.87
48
20
79.42
60.71
33.80
32.50
33.20
81.00
29.00
92.50
50.00
48.00
53,92
54.01
Table 11: Ablation study different codebook size and CTR length. All experiments con-
ducted under the zero-label setting. We experiment three of the codebook size, respectively 128, 64 and 48, and two of the CTR length, including 10 and 20. Results show that "128 10" combination achieves the best performance.
A.6 PERFORMANCE SENSITIVITY TO DIFFERENT SELECTION OF CODES
We want to verify whether different selections of codes will affect the performance of the model. Intuitively, a code is merely noise that gets ignored by the model eventually, the model will not sensitive to different selections of codes and vice versa. Therefore, we randomly sampled several codes of training tasks and evaluate their performance on different test tasks. The results are shown in Table 12. As shown in the table, the model is highly sensitive different selection of codes. For example, the code of task amazon polarity user satisfied can perform well on RTE and significantly better than the performance of code of task enn dailymail 3.0.0. nww...www..mmmmmmmm.mmmm.o...l.. card view.
Test Task
Code of Train Task
Results
Min./Max.
usum_rcad.n bwwow..eooowwwwooooooo.o.ooowww DOC.write_ate abbstrat
35,83
quoref_Guest BeeeGGGusTsttee Titec_ For_Context
41.79
CB
35.83/52.74
writer Expressed Sentiment
52.74
gigaword_wrinn wwwtt_twiittttteeeeeeeeeeeee for_this, sentence
42.50
amazon polarity user satisfied kilt tasks hotpotqa final exam can dailymail 3.0.0 news card view
75.13
72,74
RTE
49.36/72.74
49.60
wiki wwwpppriiiiiii.leiinnaa.am
49.36
52.74
yelp_review _fo.-fffolllfattttcommmooo _score
63.51
WSC
52.74/63.51
cos_c_vl.1 w.q.stit..eetonnn...mmmmmm L_l_uesttoo_ddssciipiiiii www.dddppttoooopolecc citiooo_opttoottxtt
57.90
xsum DOC. write. summary of above
53.19
cos c vv.11 rationale quarel_logee sesstttettt...
55.95
81.92
COPA
59.05/81.92
wiki htophooopggaeoo original choose best wtw.:oobbeeeettenneeeee adversarial qa www.er.bbsseeee.
59.05
81.40
Table 12: Performance sensitivity to different selections of codes.
17

---

Published as a conference paper at ICLR 2023
A.7 EXPERIMENTAL DETAILS
For the data preprocessing of all experiments, to balance the number of data for different tasks, we restrict the maximum data examples for each training task to be 50,000, which empirically yields better results.
Training details of each baseline method under the zero-label setting are illustrated as follows.
To-Large Based on T5-Large-LMs Ma-ge-LM-Adapted, it performs multi-task training for 10000 steps. We set the maximum length of input and target sequences to 384 and 32 respectively. We use the Adam optimizer with learning rate of le-4, dropout rate of 0.1, and a batch size of 1024. Following TO Sanh et al. (2022), we use the same task prompts from PromptSource(Bach et al., 2022). We report the average accuracy of multiple prompts for each test task. Note that our reproduced TO- Large results are much better than those reported in the original paper Sanh et al. (2022), which sets much stronger baseline for comparison. We report the average accuracy of prompts for each test task. We believe our bascline is well-optimized. Because the performance on test tasks is comparable the results of TO-3B reported in Sanh et al. (2022), even our baseline only contains 770M parameters.
Self- Seff-Trannng For a fair comparison, we randomly sample 32 unlabeled data for self-training, which the same as CTR. The Self-Training method trains from the TO-Large with these pseudo-labeled examples for 5 epochs and reports average performance over prompts. For the training, we use batch size of 32 and the Adam optimizer with a learning rate of le-4.
manually label set of artificially-designee features for each task, including the number of input fields of the task, whether it requires reasoning, whether it includes options into inputs, and whether it is classification task, etc. Each task can be represented as an artificially-deffned discrete feature vector, each dimension associated with one of the aspects. Manual-Cou follows exact the same training recipe as the second training phase of our CTR. Train- ing details are presented in Section 4.1.3.
Manual-Code In practice,
Zero-Labee Prompt Selection (ZPS) (Liao et al., 2022a) For each task, we use 32 unlabeled data for producing pseudo-labeled data. Finally, we report the accuracy of the selected prompt.
For the few-shot setting, we consider the following five baseline methods. All few-shot baselines are based on our reproduced TO-Large.
Model Tuning We use the Adam optimizer with a batch size of 256 and a learning rate of lc-4. We combine all training data of the test tasks for training. The maximum training step is use validation set for model selection. Finally, we report the average accuracy of the selected best checkpoint.
100. We
Prompt Tuning (Lester et al., 2021) We use the Adam optimizer with a batch size of 128 and learning rate of 0.05. We combine all training data of the test tasks for training. The maximum training step is 100. The length of continuous prompts for each task is 20. We use validation set for model selection. Finally, we report the average accuracy of the selected best checkpoint.
GPS (Xu et al., 2022) We follow hyper- paremeerrssetrss reported in the original paper. Specifically, run the GPS for steps. At each step, new prompts are generated by a 1m-xxi--m-adaapeed model.
GRIPS In our experiments, we set max patience P 2, candidate 5, and step yn 5. For the rest of the paramee,r,,,,rs,, we follow the original GRIPS paper(Prasad et al., 2022).
Black-Box Tuning (BBT) (Sun et al., 2022) In practice, we use the Adam optimizer with learning rate of 0.05. For each test task, we train the soft prompt for 200 steps. We set the prompt length I = 10, subspace dimension d = 500, and cma_budged 1000. For the rest of the pareeeparameters, we follow the original BBT(Sun et al., 2022).
18


Reviews:
[{"review_id": "jysieU16lU", "reviewer": "Reviewer_FsjZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for multitask learning of NLP tasks with compositional codes. Given a task id, a sequence of discrete codes are obtained. Code vectors obtained from these codes using an embedding table are fed as input to the language model as a task representation. The codebook and the language model are jointly trained with a language modeling loss and additional losses to enforce consistency between task ids and corresponding codes. Results show that the proposed approach performs better than prior methods in zero-label and few-shot settings. In addition, the trained control codes offer some degree of interpretability and controllability.", "summary_of_the_review": "Updating score to 6 post author response. \n\nThe authors propose an interesting approach to learn task representations as compositional codes. However, due to missing details about the proposed approach, experimental setup and baselines it is difficult to gauge the impact of the approach. In addition, the claims about interpretability and controllability are not entirely convincing. See more detailed comments below.", "strength_and_weaknesses": "Pros\n* Viewing task representations as compositional codes is an interesting perspective\n* Proposed method offers some degree of interpretability and controllability in task learning \n\nCons\n* The problem settings and proposed approach  are vaguely described. \n* Several aspects of the experimental setting were unclear which makes it hard to interpret the results. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PeoxyGWLMsV", "reviewer": "Reviewer_BP2L", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies learning composable task codes for soft-prompting of language models. The authors propose an approach similar to VQ-VAE where each task is associated with a set of discrete codes and embedding of these codes from a codebook lookup table is stacked as a soft-prompt for a language model. First, a 2D embedding for each training task is learned via another lookup table. Next, each row of this embedding matrix is used to search for the nearest code embedding in codebook lookup table. The model is trained with typical language model loss in addition to commitment and embedding losses for learning codebook embeddings. By sharing codes across tasks, this approach presents a natural task compositionality. The authors propose code ensembling for zero-shot learning and bitwise searching for few-shot learning. On a set of benchmark tasks, the model compares favorably to previous models; achieving on-par or better on average. The authors also experiment with interpretability -- similar tasks share codes -- and controllability -- bitwise perturbation exhibits different behaviors.", "summary_of_the_review": "I think composable task codes, VQ-VAE style training for LLM fine-tuning, and generalization to novel tasks during inference are strong points of the paper. But, there are still pieces that are unclear and needs improvement.", "strength_and_weaknesses": "**Strengths** The paper studies a very important and interesting problem -- learning composable task codes for generalization to unseen tasks. I find the formulation of the problem using VQ-VAE natural and inference methods convincing. It is interesting to see that empirical results, especially zero-shot learning, improves compared to self-training or T0 models.\n\n**Weaknesses** There are a few points that require clarification and improvement.\n1. Does sharing a code necessarily mean that two tasks are similar in some sense? I found no evidence as to how important a code is for a task; hence, I can't say if a code is merely noise that gets ignored by the model eventually. For example, if you increase the number of codes, *L*, the model might be given more codes but not all of them would be used. There is not loss that encourages the LLM to utilize all codes as well.\n\n2. What is the effective number of codes that discrete compositional task code, *z*, has? Are all codes in z unique or are they frequently reused?\n\n3. Comparison to previous work needs clarification.\n- Are baselines multi-task learners or are they initialized with some pretrained embedding? Like, prompts can be initialized or they can be trained in mutli-task fashion.\n- What are the architectural details of other models, including model size, are they comparable?\n- What is the performance of model tuning if you use all 32 examples for training? I think it is a bit unfair to use all examples for CTR while using only half of them for others.\n- You mention that other methods update extra parameters as a weakness but this is not true. Prompt tuning only updates an additional prompt while the LLM is fixed. On the other hand, CTR updates a codebook and LLM is also fine-tuned. I think this is less scalable overall compared to parameter efficient updates.\n\n4. How do you choose hyperparameters? Like, \"CTR length\", \"codebook size\", or *N=60*. I don't see a clear pattern on final performance and no training/dev results are given.\n\n5. Could you clarify:\n- How do you combine with manual prompts? Are they appended as additional soft-prompt vectors?\n- What is the performance if you haven't updated the LLM? This would be a fairer comparison to parameter efficient models.\n- What is the performance w.r.t. increasing data size for few-shot learning?\n- Why does the model perform worse on co-reference?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IXO2hmClGH", "reviewer": "Reviewer_xT8s", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a prompt-based architecture to improve the performance of large language models on zero-label learning and few-shot learning. The authors employ multi-task training to learn a discrete, compositional codebook, which is for mapping the given task to task types and then generate a task-related prompt for LLM inference.\n\nLearning a discrete latent codebook to enhance the interpretability of prompts is impressive, and the experiments are comprehensive and convincing. \n", "summary_of_the_review": "The idea is useful and interesting. I generally like this idea. I recommend acceptance.", "strength_and_weaknesses": "# Strength\n- The idea is useful and interesting\n- the models provide some interpretability, controllability, and generilization\n\n# Weaknesses\n- have not found any significant one\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2IB-TxqCiW", "reviewer": "Reviewer_zu2t", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper introduces Compositional Task Representations, an approach to train language models to achieve strong zero-shot and few-shot performance on unseen tasks without the need to rely on prompts at test time. \n\nGiven a pre-trained LM, CTR is trained on a variety of tasks. For each task, a vector is initialized and trained. An encoder then produces a sequence of query vectors from this vector, These query vectors are used to retrieve a sequence of codes from a global set of trainable codes. Those codes are given as input to the LM, serving as a task representation meant to replace the prompt. At inference time on a new task, the authors search for a good composition of codes for the new tasks using unlabelled examples.\n\nThe authors compare CTR to a comprehensive set of recent baselines on both the zero-shot and few-shot settings on the T0 benchmark. They achieve strong results overall on both these settings, outperforming other methods. They present ablations on the use of prompts at test time for CTR and on the importance of sub-components of their loss. Finally, there is a rich analysis section showing the interpretability, controllability and generalization ability of CTR. \n", "summary_of_the_review": "This is a well-written paper that introduces an interesting method to train LM to have strong zero and few-shot capabilities. In addition to good results, the method offers nice advantages in terms of controllability and interpretability. The main issue with this paper is in the lack of clarity on hyper-parameter selection and details of the baselines. Given the results seem to be quite sensitive to these minutes details, this casts some doubt into the claimed performance. ", "strength_and_weaknesses": "Pros:\n* The paper is well-written and well-structured. The approach is well explained, related work is presented clearly and the experimental setup is clear. \n* The set of baselines is comprehensive and very helpful in assessing the value of the method presented. Having both the zero-shot and few-shot settings is also helpful.\n* The method achieves good results overall, although there may be some issues with measurement (see Cons section).\n* Both ablations are interesting and helpful.\n* The analysis section is rich and sheds light in some of the properties and benefits of CTR.\n\nCons:\n* My main concern is on the experimental setup, especially hyper-parameter tuning. For this setup, the proper way to do hyper-parameter tuning is not always obvious. Table 10 and 11 shows the authors have tried several configurations/options (at least 15) and this paper presents the best one. Given the wide variation on results and limited data of some tasks, the authors need to clarify how they performed hyper-parameter tuning for their method as well as the baselines. This will ensure the comparison in Table 1 is fair.\n* There needs to be more information on the experimental setup for the baselines. For instance, the authors introduce the \"Manual-code\" baseline but very little detail is given. For the self-training baseline, we do not know the experimental details at all. In general this section is lacking and makes the comparison of Table 1 harder.\n* There are details missing that harm reproducibility. For instance, the value of the beta parameter selected for the loss is not given, let alone how selection was done. \n* I would comment on the text that there is wide variability across tasks for Table 1. For instance, Discarding the CB outperformance the method is very similar to prior ones on the zero-shot setting. Given the aforementioned sparsity of details on hyper-parameter selection and baselines this is an even bigger issue.\n* Some further ablations would be interesting:\n  * How does the method perform if fewer tasks are provided during training (currently 319). \n  * Does the order of the latent codes matter for performance? At inference time, how does performance deteriorate if you shuffle the order of the codes selected for a task. \n  * Visualizing the task embeddings / the codes may be interesting.  \n* Minor: Typo p8 combing -> combining ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster