Tree Prompting: Efficient Task Adaptation without Fine-Tuning
Authors: Chandan Singh
John Xavier Morris
Alexander M Rush
Jianfeng Gao
Yuntian Deng
Venue: EMNLP 2023 Main (2023)
Keywords: Decision tree
large language model
chain prompting
prompt engineering
TL;DR: A decision tree of prompts can be competitive with fine-tuning

Abstract: Prompting language models (LMs) is the main interface for applying them to new tasks. However, for smaller LMs, prompting provides low accuracy compared to gradient-based fine-tuning. Tree Prompting is an approach to prompting which builds a decision tree of prompts, linking multiple prompt-LM calls together to solve a task. At inference time, each call to the LM is determined by efficiently routing the outcome of the previous call using the tree. Experiments on classification datasets show that Tree Prompting improves accuracy over competing methods and is competitive with fine-tuning. We also show that variants of Tree Prompting allow inspection of a model's decision-making process.

Tree Prompting: Efficient Task Adaptation without Tunnng
John X. M Chandan Singh
Alexander M. Rush Jianfeng Gao Yuntian Deng*
Cornell University & Microsoft Research
Harvard University
(jxm3, arush)@coooeell..dd. (chonsiiehh.feeeeeecccsoof....
dengyuntian@seas,
hww.:aanesssssa.aaad....o harvard.
Abstract
Prompting language models (LMs) is the main interface for applying them to new tasks. How- ever, for smaller LMs, prompting provides accuracy compared to gradient-base based fine- tuning. Tree Prompting is an approach to prompting which builds decision tree of prompts, linking multiple LM calls together to solve a task. At inference time, each call to the LM determined by efficiently routing the out- of the previous call using the tree. Exper- iments on classification datasets show that Tree Prompting improves accuracy over competing methods and is competitive with fine-tuning. We also show that variants of Tree Prompting allow inspection of model's decision-making process."
A
6 A
Figure Illustration Tree Prompting. each node of the decision tree, subset of training data is used to prompt the LM to partition the input space into sub- regions. This process is repeated until classification decision is made at a leaf node.
1 Introduction
In this work, we propose Tree Prompting as an alternative method for incorporating task supervi- sion. The key idea is to use training data to form decision based on simple prompt-LM calls, with each prompt determined by the outcomes of previous calls. The method does not change the parameters of the language model, but instead uses its outputs to determine an effective tree structure. To determine the prompts used at each node of the decision tree, we propose simple bagging- inspired approach that samples few-shot examples to find the most informative prompt (see Fig. I). To convert LM outputs into split features for deci- sion path determination, we consider both using a pre-defined verbalizer (Hu et al., 2022) and more expressive &NN Prompting approach (Xu et al., 2023). To learn the tree structure, we employ a classic decision tree learning algorithm (Breiman et al., 1984). The constructed tree is a sparse rep- resentation of the fine-tuning data, incorporating large number of few-shot examples, but only requir- ing constant number of LM calls for inference.
Pretrained language models (LMs) have made re- markable progress in recent years (Vaswani al., 2017; Brown et al., 2020; OpenAI, 2023), but their large size makes them difficult to fine-tune with gradients for specific downstream tasks. As such, prompting has become the main interface for ap- plying pretrained language models (LMs), where task-specific instructions are provided to guide an LM's behavior. The most common way to adapt LMs is few-shot in-context examples, where input-output pairs are shown to the model.
Yet, few-shot prompting has a clear downside. Prompt expressiveness limited by the context length of the language model. This constraint prevents using more than handful of examples for few-shot in-context learning, particularly in memory censtry-conttraieed environments. If there is ad- ditional supervised data available for task, users need either ensemble together many prompts or back off to alternative LM fine-tuning approaches.
Eqqual contribution. Seikit-eamm-compatbbee API for using Tree-Prompt is available at ? githuh.com/csinva/tre.. prompt.
Tree Prompting offers several advantages over existing prompting approaches. It allows users to

---

easily incorporate large supervised training datasets without requiring larger contexts. It also allows experts to examine the decision-making process underlying a prediction in detail, which can be improved by combining with prompt generation methods. Finally, Tree Prompting can be adapted to be compatible with many existing LMs that are only accessible to the public via API calls. We demonstrate these advantages in experiments on multi-class classification benchmarks.
3 Tree Prompting
Tree Prompting utilizes decision trees as method of adapting LMs to specific tasks without fine- tuning the model. Assuming access to a set of text- label pairs (z, y), the goal is to determine a tree to best classify this data. The algorithm then proceeds in a top-down manner, where at each node, it se- lects the best prompt based on the chosen method for finding prompt candidates and constructing split features.
However, unlike standard decision trees, we do not have access to predetermined set of features Φ(x). Tree Prompting instead constructs this fea- ture function dynamically by constructing prompts. The value of a feature (x is determined by run- ning a prompt through the LM and mapping its response to a binary value.
2 Background: Decision Trees
Decision trees are a classic model for classification and regression. They provide a graphical, intuitive model of decision-making, based on a cascading series of binary decisions". At each node in the tree, a decision is made based on single feature of the input, which leads to the next node, and ultimately leaf node representing a prediction.
A major benefit of utilizing decision trees in this setting is their efficiency at inference time. Con- structing the tree lets compactly represent a large amount of task-specific training examples. If each o; requires running one prompt, there are 2D fea- tures. At inference time, we only need D prompt calls to classify a single datapoint.
Learning Decision trees are constructed greed- ily in a top-down manner, starting from the root node, where all training data (I, y) and features ((x) {0, 1)d are available. At each node, fea- ture that best splits the dataset into two subsets is chosen. The "best split" is determined by a crite- rion that measures the quality of split. A com- monly used criterion is the Gini impurity from the CART algorithm (Breiman et al., 1984). The se- lected feature creates two child nodes, each contain- ing the subset of data that satisfies the respective split condition. This process is repeated recursively for each child node with the corresponding subset of the data until a stopping condition is met?. Each leaf node in the final decision tree represents a deci- sion (such as a class label prediction), determined by the majority class of the instances in the leaf.
Our primary approach to find prompts for fea- tures (x(x) is to select random few-shot exam- ples drawn from the training data, as shown in Figure 1. We take inspiration from bagging ap- proaches (Breiman, 1996) that combine random training samples produce complementary paral- lel models. By sampling random x, y pairs from the task training data and passing them to an LM, we are effectively bagging small training splits. Each prompt is constructed by alternating classes with their corresponding labels in a templated form.
Once prompt is constructed, the feature value is set using pre-defined verbalizer to transform the LM's output (Hu et al., 2022). A verbalizer is a function that maps the LM's output probabilities into a discrete decision. A simple implementation of verbalizer is to determine whether the pre- dicted probability for the token Yes/No is higher. In this work, we experiment with two different verbal- izers: the first maps the logits to class labels (such as Positive/Negative for binary sentiment classifica- tion), and the second more generic verbalizer maps the logits to Yes/No.
Inference A decision tree makes predictions on unseen data by traversing the tree from the root node to leaf node. Starting from the root, the feature value of the example corresponding to the split feature at the current node is used to determine whether the left child or the right child node is visited next. This process is repeated until a leaf node is reached. The class label associated with this leaf node is then used as the prediction.
When the output logits of the LM are inacces- S we can discretize the LM's outputs into
We exclusively focus on binary trees in this paper. 'Stopping conditions include reaching a maximum number of leaf nodes or when no feature improves the split quality.
"Some recent 1.Ss such as OpenAl's GPT-3.5 and GPT-4 do not provide output logits.

---

813
had Benind
Seaild Love.
"no"
"yes"
When considering the tene and used in this tweet
would clasery emotion as Fear.
made Happy
The mood conveyed this tweet is Sad.
mued convessci thi twell miaed this Happy
The emotion of sentence
When considering toneland language used in tweet classity emotion
This tweet cont theree tiseellyeggg this tweet Anger.
Figure 2: Extension to the Tree Prompting approach using instruction prompts the Emotion dataset. Each path represents a decision sequence, and colors correspond to different emotions (classes). The line width indicates the number of instances within particular class. As the decision process advances down the tree, the classes get separated.
categories defined by the verbalizer using word matching. With few-shot prompts, large LMs have empirically been found to respect the template for- mat and output only labels that they have seen in the demonstrations most of the time (Brown et al., 2020).
to identify a list of prompt candidates that best ex- plain the subset of data at this node. The prompt that best splits this subset into two further sub- sets is then selected. The prompt search algorithm used in this paper is iPrompt (Singh et al., 2023b), which employs an LM to generate potential prompt candidates, ranking them based how well they explain the data. Dynamic prompts offer enhanced flexibility and adaptability, the cost of additional computation.
3.1 Extensions
Instruction Prompts To leverage expert insights on specific tasks, we can instead use human-curated prompt candidates (Bach et al., 2022) as shown in Fig. 2. To diversify and enrich the pool of prompt candidates, we leverage the capabilities of GPT- 3.5 to generate paraphrases of the original prompts. This method provides the ability incorporate domain knowledge, and the prompts are more interpretable compared to random few-shot examples. However, might be less adaptable to novel or unique task specifications compared to the other automatic prompt candidate generation methods.
ANN Prompting Features As more expressive alternative to predefined verbalizers, consider the kNN Prompting approach (Xu et al., 2023). ANN Prompting employs the label of the nearest neighbor an anchor set as the split feature, with the distance measured terms of the KI. diver- gence between output probabilities. This approach allows for the use of a large number of examples at each node, extending beyond the limitations im- posed by the restricted context window size, mak- ing it more expressive. A downside of this ap- proach is its dependence on access the LM's output logits. Moreover, as multiple prompts are utilized at each node of the decision tree, this can compromise the interpretability of the model.
Dynamic Prompts Instead of pre-constructing random prompt-based features, we can generate dynamic prompts while building the decision tree. At each node, we conduct discrete prompt search

---

Approach
SST2
SUBJ
MPQA
AGNews
CR
DBPedia
MR
RTE
TREC
Emotion
FPB
IMDB
AVG
#LM Calls
Classes
14
Finetuning
BERT"
88.3
90.7
74.5
88.0
78.6
88.0
95.1
83.0
78.8
82.3
1.0
GPT12 Large!
90.7
86.1
87.6
88.3
70.0
86.7
96.3
86.2
55.4
71.2
81.9
1.0
ICT. GPT-2 Small
ESPromppingg
Greedy
48.4
49.3
73.0
73.2
51.2
50.4
71.7
64.7
54.6
51.4
37.4
61.8
63.9
57.8
40.0
Boosting
TreePrompt
72.3
50.4
83.5
80.6
55.4
68.0
82.9
66.7
56.0
63.4
43.0
67.9
77.5
66.7
TredPrompt Ens
72.7
30.4
83.3
81.2
68.2
$7.6
66.5
55.9
71.7
43.3
68.7
78.7
68.3
32.6
ICL GPT-2 Medium
ESPrompting
$2.0
52.1
66.7
35,9
48.2
59.1
42.6
49.5
50.9
53.6
28.6
23.5
22.1
45.0
1.0
Greedy
83.1
74.3
63.8
70.8
64.3
50.4
78.9
77.9
54.8
63.0
46.8
61.8
74.3
66.5
40.0
Boosting
15.4
K3,9
77.9
79.0
50.3
75.9
80.6
69.5
TreePrompt
83.6
76.2
83,9
78.3
66.7
S0.6
90.5
78.8
54.9
72.8
53.3
78.4
81.1
75.3
6.2
TreePrompt
ICL GPT-2 Large
PSPrompting
81.6
92.2
50.0
49.5
40.5
64.6
33.6
59.4
51.0
54.7
27.5
52.1
50.8
51.4
1.0
Greedy
81.5
Boosting
91.9
87.4
84.0
79.9
65.5
85.4
25.0
86.5
54.0
59.8
45.0
78.6
90.6
71.8
40.0
TreePrompt
91.5
87.8
85,2
58.9
81.9
91.0
85.0
53.9
76.8
46.4
79.7
90.1
77.6
TreePrompt Eas
91.7
87.4
85.5
83.2
63.7
84.0
94.7
86.7
55.6
79.2
47.4
81.7
90.6
79.3
35.7
ICL GPT-2 XL.
ESPrompting
83.3
62.1
65.2
55.2
47.6
69.4
74.5
67.4
53.9
57.2
24.9
48.5
96.9
Greedy
90.5
77.9
76.3
83.7
62.5
73.6
68.8
84.2
57.7
63.5
35.2
65.8
86.3
72.8
40.0
Bousting
91.7
82.7
84.5
84.1
67.9
83.7
41.5
88.5
56.2
57.9
54.9
78.5
90.7
74.1
40.0
TreePrompt
76.2
78.3
90.5
78.8
72.8
78.4
81.1
75.3
TreePrompt Ens
85.5
75.9
85.2
79.8
64.9
80.6
94.5
79.0
55.6
77.9
54.9
79.4
81.8
76.5
36.2
ICL GPT-J
ESPrompting
79.2
76.6
89.7
78.4
77.3
Greedy
91.3
89.8
85.4
83.9
72.0
87.2
93.8
89.5
58.9
$1.0
49.4
69.4
93.7
80.4
40.0
Boosting
87.4
TreePrompt
91.5
92.2
87.4
85.5
73.2
87.8
97.5
90.5
59.8
83.2
59.6
76.5
93.1
82.9
8.9
TreePrompt Ens
93.1
92.2
88.5
85.8
75.0
87.5
98.2
90.5
87.2
61.9
79.2
93.7
84.0
32.3
ICL LLAMA-27HH
PSPrompting
92.7
55.1
78.6
84.9
55.4
90.9
94.0
90.8
59.2
80.2
37.5
70.2
60.1
73.1
1.0
Greedy
95.1
84.1
85.3
86.6
64.9
90.4
98.2
94.7
71.7
83.9
47.6
70.2
86.0
81.4
40.0
Boostig
94.4
89.2
86.1
87.2
76.2
90.9
21.5
92.7
74.0
39.3
55.1
81.6
91.9
75.4
40.0
TreePrompt
93.6
89.2
85.2
88.5
75.0
88.5
98.6
93.9
69.7
88.0
54.8
84.0
93.8
84,8
6.6
TreePrompt
Table 1: Main results. ICI: In Context Learning. ICL Prompting and ICL. Prompting Ensemble use 128 examples per class construct the prompt. 1; results taken from Xu al. (2023).
Tree Ensembles Trees generated via Tree Prompting can be used to construct typical tree en- sembles such as random forests (Breiman et al., 1984) or gradient-boosted trees (Freund et al.. 1996) by using Tree Prompting tree as the base estimator. This incurs very little overhead when using a fixed of prompts, as the split features can be shared across all trees after being computed once.
et al., 2015), CB (De Marneffe et al., 2019), DB- Pedia (Zhang et al., 2015; Lehmann et al., 2015), TREC (Li and Roth, 2002; Hovy et al., 2001), FPB (Malo et al., 2014), and Emotion (Saravia et al., 2018). Appendix A.I provides dataset statis- ties in Table 6. and examples in Table
Model Settings For the LM, we run experi- ments using five pretrained models: GPT-2 Small (117M parameters), GPT-2 Medium (355M param- eters), GPT-2 Large (774M parameters), GPT-2 XL (I.5B parameters) (Radford et al., 2019), and GPT- (Wang and Komatsuzaki, 2021) (6B parameters).
4 Experimental Setup
Datasets We evaluate Tree Prompting on 13 text classification datasets. Among them are binary classification datasets SST2 (Socher et al., 2013), SUBJ (Pang and Lee, 2004; Wiebe al., 2005),
Baselines We compare our approach Trec Prompting, TreePrompi, standard fine-tuning. In addition, we also compare against conven- tional prompting baseline FSPrompting, which di- rectly uses few-shot example demonstrations as the prompt. We also compare the performance
MPQA (Deng and Wiebe, 2015), CR (Hu and Liu, 2004), MR (Pang and Lee, 2005), RTE (Da- gan et al., 2006), IMDB (Maas et al., 2011), and multi-class classification datasets AGNews (Zhang

---

Emotion
FPB
IMDB
MR
SST2
0.8
ROC AUC
0.95
0.96-
0.8
0.90
Teeehomp
0.7
0.90
Twehhomp: (GBDT)
0.94
Froemo (Greed))
Trowmo (fcesting)
0.6
0.7
0.85
0.85
# LM calls
# LM calls
LM calls
# LM calls
LM calls
Figure 3: Performance as function of the number of LM evaluations per example (#LM calls). We GPT-J the base LM and class names as verbalizers. GBDT, gradient-boosting tree using Tree Prompting the base classifier, fitted to a maximum 40 LM calls provides an upper bound of accuracy we can get on individual datasets.
our ensembling approach, TreePrompt Ens², with two baseline ensembling strategies: Greedy, which adds prompts an ensemble in order of their cross- validation accuracy), and Boosting, which adds prompts an ensemble using AdaBoost (Freund and Schapire, 1997; Hou et al., 2022; Pitis al., 2023).
Relative to the baselines, Tree Prompting is typi- cally able outperform them all while making fewer queries than Greedy and Boosting. Tree Prompting makes large improvements over few- show prompting in most cases, even when the model size large. We observe a failure of the boosting strategy when the number of classes is large (specifically for 14-class DBPedia). Tree Prompting with gradient boosting generally gives an increase at performance at the cost of a 5.7-times increase in queries.
5 Results
5.1
Classification Accuracy
Our main results are summarized in Table 1. The table compares Tree Prompting across multiple lan- guage model sizes to other few-shot prompting and ensembling approaches, as well as to gradient- based fine-tuning Approaches are allotted maxi- mum of 40 LM inference calls per example.
5.2 Inference Efficiency
As computing outputs from language models can be costly, particularly with large LMs, the effi- ciency of Tree Prompting at inference time is cru- cial. In Fig. we plot test accuracy against the number of language model evaluations per exam- ple (#LM Calls) to gauge this efficiency Tree Prompting frequently surpasses competing ensem- ble strategies in performance under the same num- ber of LM calls, indicating an improvement in effi- ciency. This gain is more significant for multiclass datasets, such Emotion and Financial phrasebank (FPB).
Results show that Tree Prompting outperforms basic few-shot prompting and also ensembling- based approaches across model sizes and almost all datasets. The performance difference is particularly large across smaller model classes. For instance, while FSPrompting averages an accuracy of 44.3% with GPT-2 Small, Tree Prompting elevates this 6..5% Tree Prompting can also be ensembled, which produces accuracy improvements the cost of more LM calls.
To establish an upper bound of accuracy, we con- sider Tree Prompting ensembling. This approach generally achieves the best test performance across all methods. although it also demands more LM calls than single tree (up to 40 calls).
We also compare Tree Prompting to gradient based fine-tuning, particularly on GPT-2 Large. Re- sults show that Tree Prompting is less stable than fine-tuning, performing poorly some tasks, but outperforms it on 5 of 10 tasks. This result shows that Tree Prompting can learn well from task su- pervision the cost of additional runtime queries. (Tree Prompting could likely perform better com- pared to fine-tuning if we increased the maximum number of prompts beyond 40.)
5.3 Interpretability and Dynamic Prompts
A practical benefit of decision trees is increased interpretability. Each node of the decision tree can be inspected, offering insights into the decision- making process when predicting the label of given input. Our few-shot approach for Tree Prompting
"This ensemble uses scikit-learn's 's GradientBoostingClas sifier with its default parameters: 100 trees, each with max depth 3, with a learning rate of 0).1.
"The mean number of LM calls may differ from the max depth of a tree, as reaching a leaf node can require fewer calls in an unbalanced trec.

---

CB
SUBJ
MPQA
TREC
DBPedia
SST2
AGNews
AVG
Train Size (k)
0.25
1.77
2.49
5.45
8.00
8.60
8.66
50.00
67.35
120.00
KNN Prompting'
62.1
87.5
54.1
86.7
87.6
84.8
83.6
96.7
85.5
87.6
81.6
Tree Prompting
58.2
86.6
54.4
85,9
88.4
84.8
86.2
97.2
90.3
88.8
82.1
Table Comparison between Tree Prompting and ANN Prompting. Both approaches use GPT-2 Large as the base LM. Tree Prompting predictions from ANN Prompting construct split features. Tree Prompting results are averaged over random seeds. i; results taken from Xu et al. (2023).
the MR dataset.
comtent storytellin?
N
5.4 Comparison with &NN Prompting
mentoon the siggiiliant
Nonparametric methods like ANN Prompting (Xu et al., 2023) be employed to improve model expressivity, which allows using multiple prompts per node and avoids the reliance on pre-defined verbalizers. Table 2 provides a comparison be- tween Tree Prompting and ANN Prompting. In this comparison, Tree Prompting uses ANN Prompting predictions as split features". The results show that Tree Prompting outperforms vanilla kNN Prompt- ing most datasets, potentially due its added flexibility of partitioning the input space using the decision tree, although underperforms on three of the smaller datasets CB (250 training examples). CR (1.77k training examples), and TREC (5.44k training examples).
N
ssprnss
primertt
Uses the nevtew teghtight a poodiive sepect or pralse the subject malker?
N
Dows resieur mentann positto expect movie of stalas contant??
Done the mevnnw preie (&mmskking sutject diccseaed in
Figure 4: Tree Prompting tree learned using dynamic prompts on the MR dataset, We use GPTT4 for prompt generation in AutoPrompting and GPT-J as the base LM in Tree Prompting.
5.5 Comparison to Larger LMS
is challenging to interpret, but we can instead use interpretable prompts that are human-curated or dynamically constructed.
Tree Prompting allows enhancing the performance of small LMs to match the performance of large LMs, as shown in Table 3. For these experiments instead of few-shot prompting we use instruc- tion prompts curated from PromptSource (Bach et al., 2022)8. In this setting, even GPT-2 Small paired with Tree Prompting Ensemble competi- tive against GPT-3 (rext-davinci-0033, outperform- ing it on two datasets (FPB and MR), albeit being slightly worse on the other two datasets (IMDB and SST2). With the larger LM GPT-J, Tree Prompting outperforms GPT-3 with conventional prompting across all datasets, demonstrating the potential of using a smaller model in decision tree repeat- edly outperform larger model, which might be useful in resource-constrring scenarios.
Fig. 2 demonstrates instance of decision tree learned from human-curater prompts the Emotion dataset, where different colors represent the true labels of the data. At the root node, high-level query is posed regarding whether the tweet's underlying emotion is love. Deeper in the tree, more granular questions are presented, e.g. whether the sentiment of the sentence is anger.
Dynamic prompts offer an additional advantage over human-curated prompts; they are capable of better reflecting the specific subset of data at each node, making the decision process more aligned with the data distribution at each tree node. Fig. shows tree learned using iPrompt to create dy- namic prompts each node of the tree. Prompts are suggested by GPT-4 and reranked according to the iPrompt algorithm with the verbalizer Yes/No corresponding to the positive/negative classes of
"We binarize Prompting predictions, which multi- class labels, into multiple split features (each evaluating whether output matches a certain class).
" "Initial experiments showed that instruction prompts out- perform few-shot prompts for GPT-3.

---

Approach
FPB
MR
IMDB
AVG
Zero-Shot Instruction Prompting
39.6
82.7
75.6
90.5
72.1
GPT-3
AutoPrompting (Singh et al., 2023a)
57.2
77.4
86.6
82.4
75.9
GPT-2 Small
TreePrompt Ensemble
71.4
77.5
85.8
80.8
78.9
GPT-J
TreePrompt Ensemble
80.2
91.3
94.5
93.7
89.9
Table 3: Tree Prompting with supervision achieves comparable accuracy GPT-3 zero-shot and supervised auto-
prompting, Tree Prompting uses instruction prompts, class names as the verbalizer, and fits gradient bootted trees with up to 40 prompts. Averaged over random seeds.
Verbalizer
FPB
MR
IMDB
SST2
AVG
and source of prompts.
Yes/No
6.2
Prompt Source Sensitivity
Greedy
58.9
72.7
58.8
75.7
66.5
Boosting
57.8
71.6
58.7
75.8
66.0
Table 5 examines the sensitivity of various ap- proaches to the source of prompt candidates. The comparison between using instruction prompts and few-shot prompts demonstrates that Tree Prompt- ing consistently outperforms baselines regardless of the source of prompt candidates. It's worth not- ing that instruction prompts generally result in bet- ter performance than few-shot prompts, corroborat- ing previous findings that in-context learning with single prompt can work as well as multiple data demonstrations (Le Scao and Rush, 2021), How- ever, curating instruction prompts requires extra human effort, since new prompts must be written for each new dataset.
TreePrompt
64.4
79.4
68.6
77.2
72.4
Class Names
Greedy 59.6
64.5
62.3
78.2
66.1
Boosting 61.9
65.2
62.4
77.1
66.6
TreePrompt 74.2
73.4
65.7
80.4
73.4
Table 4: Accuracy with different verbalizers. We em- ploy GPT-2 Small as the LM, limiting to maximum of 5 average calls during inference.
Prompt Source
FPB
MR
IMDB
SST2
AVG
Few-shot
Greedy
64.5
62.3
78.2
Boosting
61.9
65.2
62.4
77.1
TreePrompt
74.2
73.4
65.7
80.4
73.4
Instructions
Greedy 71.8
83.3
88.2
86.5
82.4
6.3
Sample Complexity
Boosting 77.4
84.2
91.3
87.6
85.1
Fig. visualizes the performance of Tree Prompt- ing in relation to the fraction of training sam- ples used for training. When compared to base- line ensembling techniques, Tree Prompting some- times underperforms in low-data regimes (on FPB, IMDB, and MR), but it eventually outperforms baselines as more training data is available.
TrecPrompt
80.9
85.1
92.0
88.5
86.6
Table 5: Comparative results using different prompt sources. We use GPT-2 Small with class names the verbalizer, limiting the LM to maximum of 5 average calls during inference.
6 Analysis
7 Related Work
6.1
Verbalizer Sensitivity
Prompting Language Models The rise of large language models (LMs) has led to surge in the development of effective prompting methods (Stro- belt et al., 2022; Lu et al., 2022: Bach et al., 2022; Logan IV et al., 2022; Zhong et al., 2022: Singh et al., 2023b). Building on top of these methods, emsembling techniques for averaging multiple LM calls have shown that they often improve perfor- mance (Jiang et al., 2020; Zhang et al., 2023a), e.g. boosting (Hou et al., 2022; Pitis et al., 2023). Chain prompting (Wang et al., 2022; Press et al., 2022; Chase, 2023; Rush, 2023) is a widely used method that divides complex tasks into manage- able subtasks, linking these via prompt-LM calls.
Table 4 shows the robustness of different ap- proaches when employing a generic Yes/No ver- balizer versus a class-name verbalizer. The results show that Tree Prompting consistently outperforms the baseline regardless of the verbalizer used, de- livering decent performance even when using the generic Yes/No verbalizer. This feature could be useful in applications where class names not meaningful words, such in distinguishing be- tween texts generated by different decoding set- tings (Naseh al., 2023). Table 8 in Appendix A.3 shows full performance sensitivity results across different settings for the underlying LM, verbalizer,

---

Emotion
FPB
IMDB
MR
SST2
0.80
0.85
0.93
ROC AUC
0.895
0.80
0.75
0.92
0.85
0.890
Teethomp: (GBD))
(Grendy)
0.75
Froemo (Roosting)
0.70
0.91
0.84
0.885
0.5
1.0
0.5
1.0
0.5
1.0
0.5 1.0
Fraction samples
Fraction samples
Fraction samples
Fraction samples
Fraction samples
Figure Accuracy plotted against the fraction of samples used for training. The performance improvement facilitated by Tree Prompting (shown here with wradient-boosting becomes more noticeable as the number of training samples escalates. We use GPT-2 with Instruction prompts and a set of 10 prompts for this visualization.
This approach has proven effective across various applications, aligning with our intuition underlying this work: LM can handle individual steps of task more accurately than exccuting the task in full (Ma al., 2023; Madaan et al., 2023; Zhang et al., 2023b). However, while chain prompting links prompt-LM calls, our approach organizes them within decision tree, learning the tree struc- ture and selecting appropriate prompts for each node.
Guestrin, 2016), and BART (Chipman et al., 2010) have found use in areas like part-of-speech tag- ging (Magerman, 1995), syntactic parsing (Collins, 1997), and text classification (Sebastiani, 2002; Singh al., 2023a). However, these studies predominantly utilize pre-defined textual features within their decision tree frameworks, contrasting our approach where the decision tree is used to direct the language model's ' behavior.
Decision Trees for Interpretability Decision trees have also been applied to increase the inter- pretability of neural models. For example, Wan et al. (2021) used a decision tree structure where each node neural classifier for image classifica- tion. Zhang and Zhu (2019) learned a decision tree to explain the decisions made by an image classifier post hoc. While these works primarily target vision- based applications, we adopt a similar strategy for natural language processing, where each node in our decision tree embodies distinct prompt-LM call. Furthermore, our dynamic prompt setting ables the concurrent learning of prompts and the decision tree structure, distinguishing our method from conventional decision tree applications that function within pre-defined feature space.
Frugal GPT (Chen et al., 2023) also bears rele- vance to our work, proposing a cascade of LMs that stops when an intermediate output is considered re- liable, resulting in better computational efficiency. Viewed from the perspective of decision trees, this approach resembles right-branching decision tree.
Concurrent to our work, Tree of Thoughts (Yao et al., 2023; Long, 2023) organizes LM-generated "thoughts" within tree structure for solution search. While we also use tree structure, our aim is partition the input space to simplify the LM's tasks lower tree levels. We search the tree's struc- ture and the prompt at each node during training, while keeping these elements static during infer- ence. In contrast, Tree of Thoughts adjusts node prompts dynamically based on upper-level results. This sets it apart from our approach, where prompts remain constant post-training. Collectively, these works demonstrate the growing interest merging tree structures with LMs for task decomposition, albeit with varied focuses and methodologics.
Conclusions and Future Work
We introduce the Tree Prompting approach, a use of decision trees for task adaptation. Experiments demonstrate that Tree Prompting can offer im- proved performance across various text classifi- cation tasks while still remaining efficient during inference. On many tasks, the model is compet- itive with gradient fine-tuning. Additionally, the approach can be used with dynamic prompt cre- ation yield interpretable models.
Decision Tree Applications Dating back decades, decision trees have been prevalent choice in the realms of classification and re- gression problems (Costa and Pedreira, 2022). In the field of natural language processing, decision trees and their ensemble variants such as Random Forest (Breiman, 2001), Gradient-booster Trees (Freund et al., 1996), XGBoost (Chen and
Our results suggest a future direction of explor- ing a flexible and modularized assembly of models. One exciting direction to extend Tree Prompting

---

generalize to tasks beyond text classification, us- ing previous oulputs guide subsequent prompts
10 Acknowledgements
JXM supported by an NSF GRFP. YD is sup- ported by an Nvidia Fellowship and NSF 2242302. AMR is supported by NSF 2242302, NSF CA- REER 2037519, and Sloan Fellowship. We would also like to thank Harvard University FAS Research Computing for providing computational resources.
and LMs. Further exploration could involve ex- tending Tree Prompting to jump across nodes in
the tree (similar to Long (2023)) or introduce cy- cles in the tree (similar to Besta et al. (2023)), and ultimately developing program of prompts by navigating various nodes in decision tree as though calling different functions. Another direc- tion could explore incorporating different criteria into the tree-building algorithm, e.g. fairness (Jo et al., 2022), sparsity (Hu et al., 2019; Tan et al., 2022), or smoothness (Agarwal et al., 2022).
References
Abhineet Agarwal, Yan Shuo Tan, Omer Ronen, Chan- dan Singh, and Bin Yu. 2022. Hierarchical shrinkage: improving the accuracy and interpretability of tree- based methods. arXhv:2202.00858 (cs, stat), ArXiv: 2202.00858.
9 Limitations
Stephen Bach, Victor Sanh. Zheng Xin Yong. Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, 7aid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gun- jan Chhablani, Han Wang. Jason Fries, Maged Al- shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang. Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. 2022. Prompt- Source: An integrated development environment and repository for natural language prompts. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguisticx: System Demonstra- tions, pages 93-104, Dublin, Ireland. Association for Computational Linguistics.
Sample Complexity While Tree Prompting's adaptability and flexibility are its strengths, they also contribute to its higher sample complexity. As shown in Sec. 6.3, Tree Prompting lags behind few- shot prompting in low-data environments. Decision trees inherently risk overfitting, particularly when dealing with numerous features. This shortcom- ing can be partially offset through the use of larger training sets, and by restricting the tree's size in relation to the training set size.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Ger- stenherger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. 2023. Graph of thoughts: Solv- ing elaborate problems with large language models. arXiv preprint arXiv:2308.084 096877
Training Cost Although Tree Prompting de- mands fewer LM calls during inference compared analogous techniques, its training process, which involves learning the decision tree, requires com- puting prompt features for every example in the associated data subset at each node. This can be resource-intensive for large LMs. Additionally, when paired with dynamic prompts that leverage automatic prompting methods (which are typically computation-heavy: the training process can be substantially expensive as each node necessitates running the autoprompting method once.
L. Breiman, H. Friedman, R. A. Olshen, and C. Stone. 1984. Classification and Regression Trees. Wadsworth and Brooks, Monterey, CA.
Leo Breiman. 1996. Bagging predictors. Machine learning, 24:123-140.
Leo Breiman. 2001. Random forests. Machine Learn- ing, 45(1):5-32.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates. Inc.
Interpretability While decision trees are typi- cally celebrated for their interpretability, the inter- pretability of Tree Prompting is bounded by the nature of the prompts and the verbalizer. Specif- ically, when employing pre-defined prompt, its interpretability may not be as intuitive as that of dynamic prompts. If the prompt itself (such as when using few-shot demonstrations) lacks inter- pretability, the entire decision trec's s interpretability is likely to be compromised.

---

Harrison Chase. 2023. Langchain: Building appli- cations with lims through composability. https: //github. comm//wchaaee1/1aagchhe
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin- Yew Lin, and Deepak Ravichandran. 2001. Toward semantics-based answer pinpointing. In Proceedings of the First International Conference on Human Lan- guage Technology Research.
Lingjaa Chen, Matei Zaharia, and James Zou, 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.015 05330.05776...
Minqing Hu and Bing Liu. 2004, Mining and summa- rizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 168-177.
Tianqi Chen and Carlos Guestrin. 2016. Xghoost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785- 794.
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. 2022. Knowledgeable prompt-tuning: Incor- porating knowledge into prompt verbalizer for text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2225-2240, Dublin, Ireland. Association for Computational Lin- guistics.
Hugh Chipman, Edward George, and Robert E Mc- Culloch. 2010. Bart: Bayesian additive regression trees. The Annals of Applied Staristics, 4(1):266-298.
Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In 35th Annual Meet- ing of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguisties, pages 16- 23, Madrid, Spain. Association for Computational Linguistics.
Xiyang Hu, Cynthia Rudin, and Margo Seltzer. 2019. Optimal sparse decision trees. Advances in Neural Information Processing Systems (NeurIPS).
Zhengbao Jiang, Frank F Xu, Jun Araki, and Grabam Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Vinícius G Costa and Carlos E Pedreira. 2022. Recent advances in decision trees: An updated survey, Arti- ficial Intelligence Review, pages 1-36.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment chal- lenge. In Machine Learning Challenges. Evaluat- ing Predictive Uncertainty, Visual Object Classifi- cation, and Recognising Tectual Entailment: First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK. April 1 11-13, 2005, Revised Selected Papers. pages 177-190. Springer.
Nathanael Jo, Sina Aghaei, Jack Benson, Andrés Gómez, and Phebe Vayamos. 2022. Learning optimal fair classification trees. arXiv preprint arXiv:2201.09933
Teven Le Scao and Alexander Rush. 2021. How many data points is prompt worth? In Proceedings of the 2027 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2627-2636. Online, Association for Computational Linguistics.
De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef. Sören Auer, et al. 2015. Dbpedia-a large-scale, mul- tilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167-195.
Lingjia Deng and Janyce Wiebe. 2015. MPQA 3.0: An entiy/evvet-lleell sentiment corpus. In Proceedings of the 2015 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1323-1328, Denver, Colorado. Association for Computational Linguistics.
Xin Li and Dan Roth. 2002. Learning question clas- sifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
Yoav Freund and Robert Schapire. 1997. A decision-
theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 5591):119-139
Robert Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2022. Cutting down prompts and parameters: Simple few-shot learning with language models. Find- ings of the Association for Computational Linguis- tics: ACL 2022, pages Association for Computational Linguistics.
Yoav Freund, Robert E Schapire, et al. 1996. Experi- ments with a new boosting algorithm. In ieml, vol- ume 96, pages 148-156. Citeseer.
Dublin, Ireland.
Bairu Hou, Joe O' O'Conno,, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2022. Promptboosting: Black-box text classification with ten forward passes.
Jieyi Long. 2023. Large language model guided tree-of- thought.

---

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguisticy (Volume 1: Long Pupers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Silviu Pitis, Michael Zhang, Andrew Wang. and Jimmy Ba. 2023. Boosted prompt ensem- bles for large language models. arXiv preprint arXiv:2304.0597
Ofrr Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models.
Yubo Ma. Yixin Cao, YongChing Hong, and Aixin Sun. 2023. Large language model is not good few-shot information extractor, but a good reranker for hard samples!
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Πly Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. 201 I. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Humun Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Alexander M. Rush. 2023. Mini chain: A tiny library for coding with large language models. https: gi github. wtt.:/oo//ssu// con/sruush/iinChai..
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. CARER: Con- textualized affect representations for emotion recog- nition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687-3697, Brussels, Belgium. Association for Computational Linguistics.
Aman Madaan. Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdan- bakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback.
Fabrizio Sebastiani. 2002. Machine learning in auto- mated text categorization. ACM computing surveys (CSUR), 34(1):1-47.
Chandan Singh, Armin Askari, Rich Caruana, and Jian- feng Gao. 2023a. Augmenting interpretable models with Ilms during training.
David M. Magerman. 1995. Statistical decision-tred models for parsing. In 33rd Annual Meeting of the As- sociation for Computational Linguistics, pages 276- 283, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
Chandan Singh, John X. Morris, Jyoti Aneja, Alexan- der M. Rush, and Jianfeng Gao. 2023b. Explain- ing patterns in data with language models via inter- pretable autoprompting.
P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and Takala. 2014. Good debt or bad debt: Detecting se- mantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference Empiri- cal Methods in Natural Language Processing, pages 1631 1631-16422 Seattle, Washington, USA. Association for Computational Linguistics.
Ali Naseh, Kalpesh Krishna, Mohit lyyer, and Amir Houmansadr. 2023. On the risks of stealing the de- coding algorithms of language models.
OpenAl. 2023. Gpt-4 technical report.
Hendrik Strobelt, Albert Webson, Victor Sanh, Ben- jamin Hoover, Johanna Beyer, Hanspeter Pfister, and Alexander M. Rush. 2022. Interactive and visual prompt engineering for ad-hoc task adaptation with large language models.
Bo Pang and Lillian Lee. 2004. A sentimental educa- tion: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 271-278, Barcelona, Spain.
Yan Shuo Tan, Chandan Singh, Keyan Nasseri, Abhi- neet Agarwal, and Bin Yu. 2022. Fast interpretable greedy-tree sums (figs). arXiv:2201.1 11931 [cs, stat]. ArXiv: 2201.11931..
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Compu- tational Linguistics (ACL '05), pages l 115-124, Ann Arbor, Michigan. Association for Computational Lin- guistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F, ukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, Curran Associates, Inc.

---

Alvin Wan, Lisa Dunlap, Daniel Ho, Jihan Yin. Scott Lee. Suzanne Petryk, Sarah Adel Bargal, and Joseph E. Gonzalez. 2021. (NBDT): Neural-backed decision tree. In International Conference Learn- ing Representations.
A
Appendix
A.1 Data
Table 6 presents dataset statistics, and Table 7 shows example input, output pairs from each dataset.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J- 6B: A 6 Billion Parameter Autoregressive Lan- guage Model. /ttps://githi com/ mesh- tett--ttraffrrmr---a00 jax.
A.2 Tree Visualizations
Appendix A.2 shows another example tree learned on the MR dataset.
Boshi Wang, Xiang Deng, and Huan Sun. 2022. Itera- tively prompt pre-trained language models for chain of thought. Proceedings the 2022 Conference on Empirical Methods in Natural Language Processing, pages 27 2730, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
A.3 Full Ablation Results
Full ablation results for different choices prompts and verbalizers can be found Table 8.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39:165 210.
Benfeng Xu, Quan Wang, Zhendong Mao, Yajuan Lyu, Qiaoqiao She, and Yongdong Zhang. 2023. $kSNN prompting: Beyond-context learning with calibration- free nearest neighbor inference. The Eleventh In- ternational Conference on Learning Representations.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao. and Karthik Narasimhan. 2023, Tree of thoughts: Deliberate problem solving with large language models.
Chenrui Zhang, Lin Liu, Jinpeng Wang, Chuyuan Wang. Xiao Sun, Hongyu Wang, and Mingchen Cai. 2023a. Prefer: Prompt ensemble learning via feedback- reflect-refine. arXiv preprint arXiv:2308.1200
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023b. Summit: Iterative text summarization via chatgpt.
Tianyuan Zhang and Zhanxing Zhu. 2019. Interpreting adversarially trained convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7502-7511. PMLR.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems, volume 28. Curran Associates, Inc.
Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Stein- hardt. 2022. Describing differences between text distributions with natural language. In Proceedings of the 39th International Conference Learning, volume 162 of Proceedings of Machine Learning Research, pages 27099-27116 27116. PMLR.
Machine

---

CB
CR
RTE
TREC Emotion
FPB SST2
SUBJ
MPQA
MR
AGNews DBPedia IMDB
Classes
3
2
Train Size (k) 0.25
1.77
2.49
5.45
2.31 67.35
8.00
8.60
8.66
120.00
50.00
25.00
Table 6: Dataset statistics
Dataset
Text
Label
SST2
that loves its characters and communicates something rather beautiful about human nature
positive
SUBJ
the script isn't very good; not even someone as gifted as hoffman the actor can make it work.
subjective
MPQA
victory of democracy
positive
AGNews
Wall St. Bears Claw Back Into the Black (Reuters). "Reuters Short-ssllers, Wall Street's 's dwindling band of ultra-cynics, are seeing green again."
business
CB
Premise: "Do you mind if use your phone?" Ronni could see that Guido's 's hrain whirring. Hypothesis: Guido's brain was whirring
entailment
CR
didn have major prohlems installing this software..
positive artist
DBPedia
Geoffrey D. Falksen (born July 31 1982) American steampunk writer. the film flat
MR
negative
RTE
Sentence I: No Weapons of Mass Destruction Found in Iraq Yet. Sentence 2: "Weapons Mass Destruction Found in Iraq.
nol_ennailment
TREC FPB
What 's known The queen of Drinks
entity
According Gran the company has plans
all production Russia
neutral
although that where the company is growing.
IMDB Emotion
would put this the top my list of films in category unwatchable trash! I...l go from feeling hopeless someone who cares awake
negative
damned hopeful just from being around
sadness
Table 7: Data examples
The soundraccs the
"negative"
"positive"
The action scones n the mevie were intonde
The emotional pepact of the movie was
The pacins bf movie was
The action scenes movie intense
The makeup and in the movie were
makeup movie were
The diracting the movie was
The movie had to ups and downs but ultim ollyy it was
The slot te movee
Figure 6: Example tree for the MR dataset. We use GPT-J and search for 10 instruction prompts.Class names (positive/negative) are used as the verbalizer.

---

Single prompt
Tree Prompting
Tree Prompting (GBDT)
LLM for prompt generation
GPT-2 (1.58)
0.74
0.66
0.65
0.74
0.68
0.87
0.72
0.74
0.78
0.72
0.90
0.77
0.74
0.82
0.76
GPT-2 (1177)
0.69
0.68
0.63
0.69
0.64
0.77
0.83
0.71
0.77
0.72
0.79
0.85
0.73
0.80
0.76
GPT-2 (345M)
0.73
0.65
0.69
0.71
0.68
0.78
0.75
0.84
0.79
0.72
0.77
0.76
0.86
0.50
0.74
GPT-2 (762M)
0.73
0.64
0.64
0.75
0.66
0.77
0.73
0.72
0.87
0.72
0.81
0.75
0.73
0.90
0.78
GPT-) (68)
0.72
0.66
0.66
0.71
0.70
0.74
0.72
0.69
0.80
0.85
0.80
0.76
0.74
0.82
0.89
GPT22 (1.38)
GPT-2 (117M)
GPT-2 (345M)
GPT-2 (762M)
GPT: GPT:)/88 (68)
GPT2 (1.38)
GPT-22 CPT-2 (117M)
GPT-2 (345M)
2 (7620)
GPT: GPT:) (68)
GPT-2 (1.58)
GPT-2 (117M)
GPT-2 (345M)
(P:2200 (7622))
GPT: (68))
LLM for evaluation
LLM for evaluation
LLM for evaluation
Figure 7: Performance when varying the LLM used to select prompts versus evaluate performance. Each heatmap shows the test ROC AUC achieved averaged across 4 datasets (SST2, Emotion, Financial Phrasebank, and Rotten tomatoes) using either single prompt, Tree Prompting, or Tree sarpriiggrradeet-bbotiiiii ensemble. Unsur- prisingly. performance is when building a tree prompts on same dataset as is used for testing. Trees from Tree Prompting still induce better performance when transferring new LLMs than individual prompts. All prompting strategies select from 40 "Instructions" prompts as Table 5).

---

Ensemble (Greedy)
Ensemble (Bonsting)
Tree Prompting
Dataset
Model
Prompts
Verbalizer
1-Shot
Class names
0.56
0.56
0.59
GFT-2
5-Shot
Class names
0.59
0.54
0.63
Instruction
Class names
0.76
0.63
0.77
Emotion
1-Shot
Class names
0.71
0.68
0.72
GPT-J
5-Shot
Class names
0.72
0.70
0.76
Instruction
Class names
0.71
0.57
0.73
Class names
0.62
0.60)
0.74
1-Shot
Yes/no
0.58
0.59
0.64
GPT-2
Class names
0.73
0.74
5-Shot
Yes/no
0.72
0.73
0.76
Instruction
Class names
0.77
0.72
0.81
FPB
Class names
0.72
0.72
0.76
1-Shot
Yes/no
0.57
0.57
0.72
GPT-J
Class names
0.72
0.72
0.84
5-Shot
Yes/no
0.58
0.58
0.65
Instruction
Class names
0.80
0.77
0.86
Class names
0.62
0.62
1-Shot
GPT-2
Yes/no
0.59
0.59
0.69
Instruction
Class names
0.91
0.88
0.92
IMDB
Class names
0.96
0.96
0.97
1-Shot
GPT-J
Yes/no
0.95
0.96
0.97
Instruction
Class names
0.93
0,92
0.95
Class names
0.65
0.64
0.73
1-Shot
Yes/no
0.72
0.73
0.79
GPT-2
Class names
0.50
0.50
S-Shot
Yes/no
0.58
0.58
0.64
Human-1
Class names
0.84
0.83
MR
Class names
0.92
0.92
0.93
1-Shot
Yes/ho
0.88
0.87
0.93
GPT-J
Class names
0.93
0.95
0.96
5-Shot
Yes/no
0.80
0.80
0.93
Human-1
Class names
0.86
0.85
0.88
Class names
0.77
0.78
1-Shot
Yes/no
0.76
0.76
0.77
GPT-2
5-Shot
names
0.61
0.61
Yes/no
0.75
0.75
Instruction
Class names
0.88
0.87
0.88
SST2
Class names
0.93
0.94
0.97
1-Shot
Yes/no
0.90
0.90
0.94
GPT-J
Class names
0.93
0.94
0.97
5-Shot
Yes/no
0.58
0.58
0.58
Instruction
Class names
0.87
0.87
0.90
Table 8: Performance (ROC AUC) for different ensembling strategies when using at most 5 LM calls across different datasets, models, prompts, and verbalizers. Emotion not compatible with a Yes/No verbalizer, so it has fewer rows. Some rows missing for datasets for which 5 demonstrations too long fit in context.


Reviews:
[{"review_id": "gib5vGYgYF", "reviewer": "Reviewer_kxdC", "summary": "", "questions": "* Question A: Do \"dynamic prompts\" apply during inference time?\n* Question B: As for $k$NN-prompting, is my following understanding correct? At each node, there are a bunch of prompts. The LM does inference for each prompt and the final feature is the NN label.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\n\n### Re: It would be better if the authors could provide more insights into the failures of baselines.\n\nWe've conducted further analysis on the training and testing performances, and these analyses will be included in the updated appendix. Below are the key takeaways:\n- FSPrompting: Our data shows that FSPrompting generally underfits, indicated by its low overall performance and minimal difference between training and testing accuracies.\n- Boosting vs. Greedy: The relative performance of Boosting and Greedy becomes more nuanced when considering variations in model size. Larger models serve as more expressive base learners; however, they are also more susceptible to overfitting. The nuances are further complicated by the fact that we use pretrained models, which themselves have varying amounts of pretraining data depending on the model size.\n\n### Re: The authors did not show the effectiveness of their method using the most recent LLMs, such as LLama.\n\nThanks for the suggestion. We have now conducted experiments using the Llama-2 model with 7 billion parameters. The updated performance comparisons will be added to Table 1. Our preliminary results indicate that TreePrompt continues to show strong performance when compared with baselines.\n\nHere's a snippet of the additional rows in Table 1 for Llama-2:\n\n\n| Method |   SST2 |   SUBJ |   MPQA |   AGNews |   CB |    CR |   DBPedia |    MR |   RTE |   TREC |   Emotion |   FPB |   IMDB |  Mean | LLM Calls | \n|-----------------------------------------------------|-------|-------|-------|---------|-----|------|----------|------|------|-------|----------|------|-------|-------|----|\n| FSPrompting |   92.7 |   55.1 |   78.6 |     84.9 | 55.4 |  90.9 |      94.0 |  90.8 |  59.2 |   80.2 | 37.5 |  70.2 |   60.1 |   73.1 | 1.0 |\n| Ensemble |   95.1 |   84.1 | 85.3 | 86.6 | 64.9 |  90.4 | 98.2 |  94.7 |  71.7 |   83.9 | 47.6 |  70.2 |   86.0 |   81.4 | 40.0 | \n| TreePrompt Ensemble |   94.5 | 90.0 | 86.6 | 88.2 | 76.8 |  89.2 | 98.6 |  93.8 |  73.6 |   89.7 | 57.2 |  84.9 | 93.5 |   85.9 | 6.6 | \n| TreePrompt | 93.6 |   89.2 |   85.2 | 88.5 | 75.0 |  88.5 | 98.6 |  93.9 |  69.7 |   88.0 | 54.8 |  84.0 |   93.8 | 84.8 | 35.4 |\n\n\n### Re: Do \"dynamic prompts\" apply during inference time?\n\nThank you for your question; we apologize for any confusion. To clarify: the \"dynamic prompts\" are generated during the training phase, where they are optimized to suit each specific node in the decision tree. In other words, the prompts are \"dynamic\" in that they are adapted to the specific characteristics of the subset of data that flows to each node during training.\n\nHowever, once the model is trained, the decision tree—including its structure and associated prompts—is fixed. So while the prompts are dynamically generated during training, they are static during inference. We'll make sure to include this clarification in the updated paper.\n \n### Re: As for kNN-prompting, is my following understanding correct? At each node, there are a bunch of prompts. The LM does inference for each prompt and the final feature is the NN label.\n\nYour understanding is correct. At each node in the decision tree, we use the language model to generate logits for both the test example and a set of training examples. We then find the nearest neighbor in the set of training examples based on the generated logits. The label of this nearest neighbor is used as the final feature for that node.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed method is novel and interesting. It is a nice combination of traditional simple methods and current LMs. \n* Experimental results on multiple datasets indicate the effectiveness of the proposed method.", "reasons_to_reject": "* It would be better if the authors could provide more insights into the failures of baselines. For example, as shown in Table 1, on TREC, Greedy and Boosting show similar performances with GPT-2 XL, but Boosting significantly underperforms Greedy with GPT-2 Large and Medium.\n* The authors did not show the effectiveness of their method using the most recent LLMs, such as LLama. The computational budget should be affordable since the authors have adopted GPT-2 XL and GPT-J. Recent LLMs should be included for comprehensive evaluation.", "questions_to_authors": "* Question A: Do \"dynamic prompts\" apply during inference time?\n* Question B: As for $k$NN-prompting, is my following understanding correct? At each node, there are a bunch of prompts. The LM does inference for each prompt and the final feature is the NN label.", "paper_topic_and_main_contributions": "This paper presents a method for better prompting LMs without fine-tuning. The proposed method, Tree Prompting, works similarly to a decision tree. For each node of the decision tree, generated labels of individual prompts are regarded as the split features. The authors also present several extensions of the proposed method, including human-written instruction prompts, dynamic prompts, $k$NN prompting as split features, and tree ensemble.\n\nExperiments are conducted on 13 text classification datasets. Results show that Tree Prompting outperforms various baselines across multiple LM sizes.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "U0dfMWoK2K", "reviewer": "Reviewer_8DXL", "summary": "", "questions": "A. Can you provide further clarity on the method used to identify prompts for features mentioned in lines 133-143 and $\\Theta_1$ - $\\Theta_4$ in Figure 1?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\n\n### Re: While Section 3.1 outlines several ways to construct prompts for features, this paper does not provide performance results for these methods, \n\nIndeed, the primary focus of Table 1 is on \"few-shot prompts,\" which is the main prompt-construction setting discussed at the outset of Section 3. However, results for alternative settings that are covered under Section 3.1 “Extensions” can be found in other tables:\n- \"Instruction Prompts\" results are presented in Tables 5 and 8.\n- \"kNN Prompting Features\" results are shown in Table 2.\n\nRegarding \"Dynamic Prompts,\" due to the computational demands involved—each decision tree node's construction requires running the autoprompting method—we opted for a qualitative presentation in Figure 4 to showcase its interpretability.\n\nWe acknowledge the need for greater clarity in distinguishing between the main contribution and these extensions in the paper, and will revise the manuscript accordingly.\n\n### Re: The main results in Table 1 lack a comprehensive comparison to SOTA methods. \n\nWe will incorporate SOTA results from a recent preprint [1]. Additionally, we have expanded our evaluations to include the latest 7-billion parameter Llama-2 model, as suggested by reviewer kxdC. The results are presented below:\n| Method |   SST2 |   SUBJ |   MPQA |   AGNews |   CB |    CR |   DBPedia |    MR |   RTE |   TREC |   Emotion |   FPB |   IMDB |   mean | LLM Calls | \n|-----------------------------------------------------|-------|-------|-------|---------|-----|------|----------|------|------|-------|----------|------|-------|-------|------|\n| FSPrompting |   92.7 |   55.1 |   78.6 |     84.9 | 55.4 |  90.9 |      94.0 |  90.8 |  59.2 |   80.2 | 37.5 |  70.2 |   60.1 |   73.1 | 1.0 |\n| Ensemble |   95.1 |   84.1 | 85.3 | 86.6 | 64.9 |  90.4 | 98.2 |  94.7 |  71.7 |   83.9 | 47.6 |  70.2 |   86.0 |   81.4 | 40.0 |  \n| TreePrompt | 93.6 |   89.2 |   85.2 | 88.5 | 75.0 |  88.5 | 98.6 |  93.9 |  69.7 |   88.0 | 54.8 |  84.0 |   93.8 | 84.8 | 6.6  |\n| TreePrompt Ensemble |   94.5 | 90.0 | 86.6 | 88.2 | 76.8 |  89.2 | 98.6 |  93.8 |  73.6 |   89.7 | 57.2 |  84.9 | 93.5 |   85.9 | 35.4  | \n| SOTA [1] | 97.4 | - | - | 96.4 | - | - | - | 92.4 | - | - | - | - | - | - | -\n\nFor the datasets that overlap with the SOTA, our results are on average 3.2 accuracy points below. However, it's crucial to note that the SOTA model is significantly larger (GPT-3 text-davinci-003, with 175 billion parameters) compared to our Llama-2 model, which has 7 billion parameters. We also observe that better pre-trained models yield better results; for instance, Llama-2's average score of 84.8% significantly outperforms GPT-2 XL’s average of 77.2%. Furthermore, we'd like to emphasize that the prompting techniques used in [1] are complementary to TreePrompt.\n\n### Re: the ensembling method used in TreePrompt Ens is not illustrated, and the stopping condition of constructing the decision tree in TreePrompt is not elucidated.\n\nWe apologize for not clearly specifying the ensembling method and stopping conditions used in TreePrompt. We will add the below information to the manuscript:\n- Ensembling Hyperparameters: We create ensembles of 100 trees, each with max depth 3, and we use a learning rate of 0.1.\n- Stopping Condition: In TreePrompt, the construction of a decision tree stops after reaching 40 splits. This number was chosen to match the number of splits used in our greedy and boosting ensemble baselines.\n\n### Re: Can you provide further clarity on the method used to identify prompts for features mentioned in lines 133-143 and  -  in Figure 1?\nThank you for your question. For the segment between lines 133-143 and in Figure 1, which describe our approach to using \"few-shot prompts\" with k shots, the method works as follows:\n\nWe construct a prompt by concatenating k text-label pairs, formatted in the manner of:\n```\nInput: example_text_1\nOutput: example_label_1\n...\nInput: example_text_k\nOutput: example_label_k\nInput: test_sentence\nOutput: [logits]\n```\nThe prompt is then processed through the language model to produce logits for the test sentence. A verbalizer is then applied to these logits, transforming them into discrete class names. These class names serve as the features utilized in the decision tree nodes.\n\nWe will make sure to elaborate on this method in the revised manuscript for added clarity.\n\n### References\n[1] Text Classification via Large Language Models (https://arxiv.org/abs/2305.08377)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents a novel Tree prompting method wherein LMs are prompted using a structured decision tree of prompts. A potential advantage of this method is the enhanced interpretability of models.\n- Several ways to find prompts for features in the decision tree are proposed.\n- Empirical results show that Tree prompting method outperforms the baseline methods presented in the paper.", "reasons_to_reject": "- While Section 3.1 outlines several ways to construct prompts for features, this paper does not provide performance results for these methods, which leaves readers in the dark regarding which method might yield superior results.\n- The main results in Table 1 lack a comprehensive comparison to SOTA methods. I mean, the improvements from the baselines are good but I cannot assess how this pushes the current SOTA. The paper limits its comparison with KNN prompting to GPT2-Large. However, given that the improvements derived from KNN prompting aren't markedly significant and the fact that multiple LMs are evaluated in both Tree prompting and KNN prompting, a broader comparison across different LMs would be more informative.\n- The paper is unclear in some technical details. For instance, the ensembling method used in _TreePrompt Ens_ is not illustrated, and the stopping condition of constructing the decision tree in _TreePrompt_ is not elucidated.", "questions_to_authors": "A. Can you provide further clarity on the method used to identify prompts for features mentioned in lines 133-143 and $\\Theta_1$ - $\\Theta_4$ in Figure 1?", "paper_topic_and_main_contributions": "The paper explores prompting language models (LMs) on classification tasks by constructing a decision tree composed of prompts. The proposed Tree prompting method outperforms baseline methods showcased in the paper and is competitive with fine-tuning methods. Additionally, a notable advantage of this approach is the enhanced interpretability of models it offers", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- It appears that the results for the SUBJ and RTE datasets in Table 2 are swapped."}, "extra_text": {}}, {"review_id": "Gi2SI9hgT2", "reviewer": "Reviewer_VnSE", "summary": "", "questions": "1. Has the author explored training Tree Prompt on smaller language models, such as GPT-2 small, and subsequently evaluating its performance on larger language models? Additionally, did the author manually inspect the trained Tree Prompt and subsequently test it on large language models?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\n\n### Re: The experiments primarily focus on small language models, and as a result, the paper does not effectively demonstrate the impact of the proposed method on large language models like ChatGPT and GPT-4.\n\nOur focus on smaller models has been dictated by computational constraints as well as a desire to create solutions that are practical for applications with limited computational resources. That said, we have expanded our experiments to include results on a slightly larger model—Llama-2, with 7 billion parameters. Here are the updated results added to Table 1:\n\n| Method |   SST2 |   SUBJ |   MPQA |   AGNews |   CB |    CR |   DBPedia |    MR |   RTE |   TREC |   Emotion |   FPB |   IMDB |   mean | LLM Calls | \n|-----------------------------------------------------|-------|-------|-------|---------|-----|------|----------|------|------|-------|----------|------|-------|-------|----|\n| FSPrompting |   92.7 |   55.1 |   78.6 |     84.9 | 55.4 |  90.9 |      94.0 |  90.8 |  59.2 |   80.2 | 37.5 |  70.2 |   60.1 |   73.1 | 1.0 |\n| Ensemble |   95.1 |   84.1 | 85.3 | 86.6 | 64.9 |  90.4 | 98.2 |  94.7 |  71.7 |   83.9 | 47.6 |  70.2 |   86.0 |   81.4 | 40.0 |\n| TreePrompt Ensemble |   94.5 | 90.0 | 86.6 | 88.2 | 76.8 |  89.2 | 98.6 |  93.8 |  73.6 |   89.7 | 57.2 |  84.9 | 93.5 |   85.9 | 6.6 | \n| TreePrompt | 93.6 |   89.2 |   85.2 | 88.5 | 75.0 |  88.5 | 98.6 |  93.9 |  69.7 |   88.0 | 54.8 |  84.0 |   93.8 | 84.8 | 35.4 |\n\nWe find it encouraging that the performance improves with the quality of the base language model. Specifically, Llama-2 exhibits an average accuracy of 84.8%, as compared to GPT-2 XL's 77.2%. We hope this additional result offers a clearer picture of how TreePrompt scales with the performance of the base language model.\n\n### Re: Has the author explored training Tree Prompt on smaller language models, such as GPT-2 small, and subsequently evaluating its performance on larger language models? Additionally, did the author manually inspect the trained Tree Prompt and subsequently test it on large language models?\n\nThanks for this interesting question. To investigate this issue, we conducted an experiment where we trained TreePrompt on smaller models (e.g., GPT-2 small) and subsequently evaluated its performance on larger language models (e.g., GPT-2 medium, large, and even GPT-J). We present the results in the tables below. These tables show the test ROC AUC achieved on an average of four datasets (SST2, Emotion, Financial Phrasebank, and Rotten Tomatoes) when applying various strategies, such as using a single prompt, tree prompting, and gradient-boosted tree prompting.\n\n**Single prompt**\n\n|     model         | GPT-2 (117M) | GPT-2 (345M) | GPT-2 (762M) | GPT-2 (1.5B) | GPT-J (6B) |\n| ----------- | ----------- | ----------- | ----------- | ----------- | --------- |\n| GPT-2 (117M) |         0.68 |         0.63 |         0.69 |     **0.69** |       0.64 |\n| GPT-2 (345M) |         0.65 |         0.69 |         0.71 |     **0.73** |       0.68 |\n| GPT-2 (762M) |         0.64 |         0.64 |     **0.75** |         0.73 |       0.66 |\n| GPT-2 (1.5B) |         0.66 |         0.65 |         0.74 |     **0.74** |       0.68 |\n| GPT-J (6B)   |         0.66 |         0.66 |         0.71 |     **0.72** |        0.7 |\n\n**TreePrompt**\n\n| model   | GPT-2 (117M) | GPT-2 (345M) | GPT-2 (762M) | GPT-2 (1.5B) | GPT-J (6B) |\n| :----------- | -----------: | -----------: | -----------: | -----------: | ---------: |\n| GPT-2 (117M) |     **0.83** |         0.71 |         0.77 |         0.77 |       0.72 |\n| GPT-2 (345M) |         0.75 |     **0.84** |         0.79 |         0.78 |       0.72 |\n| GPT-2 (762M) |         0.73 |         0.72 |     **0.87** |         0.77 |       0.72 |\n| GPT-2 (1.5B) |         0.72 |         0.74 |         0.78 |     **0.87** |       0.72 |\n| GPT-J (6B)   |         0.72 |         0.69 |          0.8 |         0.74 |   **0.85** |\n\n**TreePrompt (GBDT)**\n\n| model   | GPT-2 (117M) | GPT-2 (345M) | GPT-2 (762M) | GPT-2 (1.5B) | GPT-J (6B) |\n| ----------- | ----------- | ----------- | ----------- | ----------- | --------- |\n| GPT-2 (117M) |     **0.85** |         0.73 |          0.8 |         0.79 |       0.76 |\n| GPT-2 (345M) |         0.76 |     **0.86** |          0.8 |         0.77 |       0.74 |\n| GPT-2 (762M) |         0.75 |         0.73 |      **0.9** |         0.81 |       0.78 |\n| GPT-2 (1.5B) |         0.77 |         0.74 |         0.82 |      **0.9** |       0.76 |\n| GPT-J (6B)   |         0.76 |         0.74 |         0.82 |          0.8 |   **0.89** |\n\nIt appears that TreePrompt models don't transfer as seamlessly across different sizes of language models as we had hoped. In general, we find that performance is the best when the decision tree is learned on the same language model that is used for inference. This suggests that while TreePrompt boosts performance within a particular model, its knowledge does not easily transfer between models of different sizes (at least for the small scale language models that we tested).\n\n### Re: In Table 2, does Tree Prompting achieve superior performance on the DBPedia dataset?\n\nUpon reviewing our data, we found an oversight: the performance numbers for MR and DBPedia were inadvertently swapped when we rearranged columns based on dataset sizes. In the corrected table, Tree Prompting achieves a performance of 97.2% on the DBPedia dataset, while the performance on MR should be 86.2%, and both numbers should be bolded. We appreciate your keen attention to detail and have corrected this error in our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed Tree Prompt demonstrates enhanced performance on diverse classification datasets, while maintaining efficiency during inference.\n2. The paper conducts comprehensive experiments to showcase the effectiveness and robustness of the proposed method.\n3. The paper presents a clear and promising direction for future prompting design.\n4. The code provided in the paper ensures reproducibility.", "reasons_to_reject": "1. The experiments primarily focus on small language models, and as a result, the paper does not effectively demonstrate the impact of the proposed method on large language models like ChatGPT and GPT-4.\n\n2. The proposed strategies for Tree Prompt, including learning, verbalizer, and extensions, lack proper organization in the paper. As a reader, one needs to constantly refer back to previous pages to find relevant information. It would be beneficial to have an overall figure that depicts the training process and the different extensions for Tree Prompting and baselines in a more coherent manner. This would enhance the clarity and understanding of the proposed approach.", "questions_to_authors": "1. Has the author explored training Tree Prompt on smaller language models, such as GPT-2 small, and subsequently evaluating its performance on larger language models? Additionally, did the author manually inspect the trained Tree Prompt and subsequently test it on large language models?", "paper_topic_and_main_contributions": "The paper presents an interesting approach to address the limitations of few-shot in-context examples in language models. These in-context examples have shown effectiveness, but their expressiveness is limited by the context length. To tackle this problem, the authors propose a solution called Tree Promptings, which involves using training data to create a decision tree based on prompts. This decision tree is constructed using a bagging-inspired approach. Additionally, the paper utilizes pre-defined verbalizers and a kNN Prompting approach to split features.\n\nThe experimental results demonstrate that the proposed method outperforms various prompting baselines on different text classification tasks. Notably, the approach remains efficient during inference, making it a promising solution for improving performance in a practical and scalable manner.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In Table 2, does Tree Prompting achieve superior performance on the DBPedia dataset?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an innovative approach called Tree Prompting, which aims to enhance the performance of Language Models (LMs) without the need for fine-tuning. In this method, Tree Prompting operates in a manner akin to a decision tree. Each node of the decision tree is associated with generated labels derived from individual prompts, serving as the split features. Furthermore, the authors present various extensions of this novel method, including the incorporation of human-written instruction prompts, dynamic prompts, the use of Neural Network (NN) prompting as split features, and the implementation of a tree ensemble technique. However, it would be better if the authors could provide more insights into the failures of baselines.
Decision: Accept-Main