Learning to Plan for Language Modeling from Unlabeled Data
Authors: Nathan Cornille
Marie-Francine Moens
Florian Mai
Venue: COLM (2024)
Keywords: planning, representation learning, hierarchical models
TL;DR: We propose to condition LMs on writing actions predicted via an external planner trained in a self-supervised way.

Abstract: By training to predict the next token in an unlabeled corpus, large language models learn to perform many tasks without any labeled data. However, their next-token-prediction objective arguably limits their performance in scenarios that require planning, such as writing a coherent article. In this paper, we train a module for planning the future writing process via a self-supervised learning objective. Given the textual context, this planning module learns to predict future abstract writing actions, which correspond to centroids in a clustered text embedding space. By conditioning on these actions, our model extends the successful language model formula to more abstract planning in an unsupervised way. Empirically, we demonstrate that our method improves language modeling performance in general, particularly with respect to the text structure. Because our framework uses a planner module that is unsupervised and external to the language model, new planner modules can be trained at large scale and easily be shared with the community.

Published as a conference paper at COLM 2024
Learning to Plan for Language Modeling from Unlabeled Data
Nathan Cornille, Marie-Francine Moens Florian Mai
Department of Computer Science
KÚ Leuven
Leuven, Belgium
{nathan. cornille, sien.moens, florian. mai @ewllijeuleeeeccc
Abstract
By training to predict the next token in an unlabeled corpus, large language models learn to perform many tasks without any labeled data. However, their mext-token-predictton
objective arguably limits their performance in scenarios that require planning, such as writing a coherent article. In this paper, we train module for planning the future writing process via a self-supervised learning objective. Given the textual context, this planning module learns predict future abstract writing actions, which correspond to centroids in clustered text embedding space. By conditioning on these actions, our model extends the successful language model formula to more abstract planning in unsupervised way. Empirically, that our method improves language modeling performance in general, particularly with respect to the text structure. Because our framework uses a planner module that is unsupervised and external to the language model, new planner modules be trained at large scale and easily be shared with the community.
demonstrate
1 Introduction
Despite their simple learning objective, language models (LMs) can solve surprising range of challenging generation tasks such as summarization (Zhang et al. 2024b), style transfer (Reif et al. [022], and machine translation (Zhu et al. 2023). Ihis is widely attributed to the that the self-supervised next-token- peeeecc-reeiitioon objective allows for training large models on unprecedented amounts of unlabeled data (Radford et al. 2019), unlocking new capabilities at sufficient scale (Wei et al.) 2022a), e.g. in-context learning (Brown et al. 2020) and chain-of-thought reasoning (Wei et al. 2022b). The dual-process theory posits that human intelligence comprises intuitive (type I) and deliberate (type 2) reasoning systems (Evans 1984 Kahneman, 2011). [Bengio et al. (2021) describe current self-supervised deep learning systems as most successful at system tasks such as perception, reading, and talking. However, it is currently unclear how to transfer the success of self-supervision to system 2 tasks that require planning, such as writing a coherent article.
Most popular approaches for eliciting deliberate reasoning capabilities in LMs are based on either advanced prompting techniques such as Mhain-of-thoughh (Wei al. 2002b), plan-and- solve (Wang et al., ) or of-h-uhtthought (Long] 2023), on finetuning on task-specific reasoning data (Havrilla et al. (024)). While these techniques already lead to substantial improvements, they ignore the key ingredient that made LMs so successful in the first place: the ability to obtain ever-improving performance through scalable self-supervised pretraining. Hence, it is widely recognized that combining planning with self-supervised foundation models is promising research direction (Yang et al. 2023).
In the context sequential decision making, planning is the process of determining decision policy from model of the environment (Sutton & Barto 2018). In this paper, we propose a method for learning to plan future writing from unlabeled data. Concretely, as illustrated in Figure I we first transform unlabeled text into a sequence of abstract writing actions which we obtain from a clustered text embedding space. We then train an external planner network which predicts the next writing action from the current context. Finally,

---

Published as a conference paper at COLM 2024
we integrate the new learned planning capability into regular LM by finetuning it on the self-supervised next-token-preeictinn task in a way that allows it to extract relevant information from predicted writing actions
In summary, we report the following findings:
1. Our model makes effective use of abstract writing actions predicted by the planner.
2. An external planner outperforms an LM-internal planner.
3. The LM improvements are specifically due to improvements in anticipating and matching the text structure due to planning.
2 Related Work
2.1 Deliberative Reasoning and Planning in LMs
Existing work on deliberative reasoning in LMs can be roughly categorized into three approaches: 1) Prompting-basee based approaches such as chain-of-thought (Wei et al. 2022b Kojima et al.) 2022) and plan-and-solve (Wang et al. 2023a) elicit reasoning and planning
capabilities from pretrained LMs via advanced prompting methods but without any addi-
tional finetuning (Wei et al. 2022b Kojima et al. 2022 Wang et al. 2022b a: 2023a Zheng ee al. 2023b Kim et al. 2023 Xue et al. 2023] Zhao et al. 2023; Sun et al. 2024 Yao et al.
Hao et al. 2023). 2) Through combination with external tools, LMs can learn to use e.g. calculators when appropriate (Schick et al. 2023 Chen et al. 2023 ( Paranjape et al. 2023
2023
or receive feedback from domain specitic solvers (Nye et al. 2021 Valmeekam al. 2023)
3) Finally, finetuning on task-specific reasoning data can result in substantially improved
in-domain performance for e.g. mathematical reasoning (Hendrycks et al., 2021 Cobbe et al.,
2021 Havrilla et al. 20024,, code generation (Shen et al., 2023), and navigation tasks (Carta al. 2023). .. While the above three approaches lead to substantial performance gains on
individual reasoning tasks, they deviate from the paradigm of large-scale selffsupprvised
training on unlabeled data that gave rise to the dominance of LMs, which our method addresses. The Planning Tokens (Wang et al., 2023b) method is perhaps most relevant to our work: Here, the dataset is augmented with planning tokens that represent high-level information similar to our writing actions. However, planning tokens are generated by the LM itself rather than by an external planner, and are hence limited in their flexibility and ultimately performance. Most importantly, again their method is applied to narrow mathematical reasoning dataset rather than general-purpose corpus. Nonetheless, in our experiments, we compare our external planner the LM-internal planning approach by Wang et al. (2023b).
2.2 Hierarchical Text Processing in Deep Learning
By planning in the space of abstract writing actions derived from text units (e.g., sentences), our method explicitly addresses the hierarchical nature of language. This is common theme in NLP models; for example, hierarchical attention mechanisms have been employed for summarization (Nallapati et al. 2016), text classification (Yang et al., 2016) and machine translation (Miculicich et al.) 2018). 10....2. Several works have decomposed text generation tasks
into sentence-level and word- level prediction tasks (Ian al. 2017 Perez-Beltrachini et al.,
Marfurt & Henderson 2021 [hamtani & Berg-Kirkpatrick 2020), but in contrast to
our work do not condition on abstract writing actions. [ppolito et af. (2020) propose to do language modeling at the sentence-level, while we ultimately still perform language modeling at the token level.
2.3 Language Modeling
Language models have been an important subject of study for decades. While initial models were based on count-based n-gram LMs, the models that are ubiquitous today are based on
The code is available on Github.
2

---

Published as a conference paper at COLM 2024
neural LMs (Bengio et al. 2000). Algorithmic advances one major source of progress in neural language modeling (Ho et al. 2024), through e.g. recurrent LMs (Mikolov et al.
2010: 2011), specialized attention mechanisms such pointer-networks (Vinyals et al.
2015: Merity et al. 2017), and eventually transformer-based
(Vaswani et al. 2017) language models (Radford] 2018). While the contribution of our paper is on the algorithmic side, it is important to remember that, as Ho et al. (2024) identify, in recent years progress has been driven mostly by scaling up similar architectures (Radford et al. 2019: Brown et al. 2020
Hoffmann et al. 2022), which gives rise to the key motivation of our paper: New algorthmss for planning and reasoning in LMs need to scale unlabeled data to be relevant long-term.
Related methods Among the important algorithmic improvements, Retrieval Augmented
Generation (RAG) (Guu et al., 2020 Lewis al.) 2020) informs next-token generation by querving a large database of text. A special case of RAG, nearest neighbor language models (Khnneelwal et al. 2020), retrieve the closest matches to the past based on similarity in a latent space, transtorm the distances into probability distribution, and interpolate it with the LM's distribution without any additional finetuning. Orthogonal to the above, we propose method that predicts sentence into the future in order to inform the generation of the next tokens via finetuning from unlabeled data. In Controllable Text Generation (Zhang
et al. 2024a: Mudgal et al. 2024 Guo et al., 2022 Deng & Raffel, 2023 Yang & Klein 202T), generation is guided by user-provider attributes such as topic, style, or keywords. Our method can be seen as a form of controlled text generation, where guiding (latent) attributes are predicted by an external model rather than provided by the user. Aiming at separating the global context from the local context, topic-guided LMs (Lau et al., 2017] Wang et al., 2019) condition the generation on topic models inferred from the context. Recent results indicate that this does not improve language modeling because well-trained LMs already possess good awareness of the current topic (Zheng et al. 2023a In contrast, our planner predicts future writing actions, which often carry information that is structural rather than topical.
3 Methodology
We consider the general task of language modeling, i.e., estimating the probability pxxx22 ... Xx) for any text sequence X X1 Xy E X. As standard practice, we factorize
p(z x,) II p(x](x)...xxx 11 However, in contrast to standard LM, we condition our i=1
LM on additional writing actions a E A, which are predicted by an external planner module P: X + A. To this end, we split the input X into text units X f....ta Im consisting of one or more tokens 1 x -ny A text unit could for example be paragraph or chapter, even be determined dynamically. However, in this paper we choose sentences for simplicity. We
associate one writing action @j with each text unit fj, which we write as X' = ait am fm
m "
Hence, we frame language modeling as estimating
II j=1 i=1
II
P(x/1a
1-1 14,2
i.e., the probability of the next token in the current text unit given the sequence of previous text units with their corresponding writing actions and the previous tokens in the current text unit, via our language model pa.
Our method consists of three core contributions as sketched in Figure 1 to derive an abstract writing action for every text unit in our unlabeled data, providing us with training data. Second, we propose a simple planner for predicting the next writing action. Finally, we finetune the language model by conditioning on predicted actions.
First, we show how
3.1 Generating Abstract Writing Actions from Unlabeled Data
Given a training corpus X with articles X = !1... In E X, we first embed every text unit into a low-dimensional vector z; E(t;) E Rd via text encoder E (e.g. Sentence-
We choose this notation for convenience. The actions aj are not in textual form.
3

---

Published as a conference paper at COLM 2024
E
titi
bm
E
label text unit closest action
Ovey ampym
CLUSTER
a
[
4
E
[2c aitı
tmx
E
X A
am mmpn
1) Unlabeled corpus of X articles
2) Text unit encoding
3) Clustering into abstract writing actions
4) Unlabeled corpus with actions
(a) Generating abstract plans from unlabeled data.
à classifler
Action
Adapter
Transformer Encoder
add pos
pos emb
Planner
Language Model
emb
P
E
4-11
Zt
(b) Predicting writing actions from context via the planner
(c) Finetuning the language model con- dition on predicted writing actions.
Figure 1: The three core phases of our proposed method to learn a planner from unlabeled data. Blue indicates frozen parameters, yellow indicates trainable parameters, and grey indicates no learnable parameters
BERT (Reimers & Gurevych, 2019)). principle, we could now associate each text unit with its corresponding embedding, i.e, choose the action embedding aj = zj. However, while this would retain most information, in this case is very difficult to predict due to the large number of options. Instead, we posit that necessary to compute abstractions that correspond to common writing patterns. Similar to Wang et al. (2023b), a simple k-means clustering step in the embedding space to narrow down the possible actions. We reuse the resulting cluster centroids as the actions embeddings 7 E Rd. Since the embeddings carry semantic information, we expect the resulting centroids correspond to general writing patterns. We set k = 1024 by default. In Section 5.3 we investigate the effect of and what our clusterings have learned.
employ
3.2 Predicting Future Writing Actions with Planner Module
During inference, we do not have access the true writing actions, so they have be predicted. We could simply treat writing actions as tokens and try to predict them along with every other token by finetuning the language model on the new data, as is done in Wang et al. (2023b). However, we argue that it is beneficial to treat the prediction of writing actions differently from tokens, namely through an external prediction module, for three reasons: (1) Importance: Writing actions weigh higher in importance than individual tokens, as they carry information regarding entire text units (e.g., sentences, paragraphs), rather than subword units. (2) Planning: The predicted writing action has significant influence over the future generated text, demanding elaborate planning to ensure the right choice is made, which next-token prediction in LLMss cannot provide. (3) Modularity: An external

---

Published as a conference paper at COLM 2024
planner is not specific to any language model. Hence, it can be trained once and henceforth be used for conditioning various language models downstream.
In this paper, we propose a simple planner that only predicts the next writing action. However, our framework can encompass planners of arbitrary complexity, allowing for long-term prediction multiple steps ahead and the integration of look-ahead search, similar to successful reinforcement learning approaches to playing games (Schrittwieser et al. 2020).
We model action prediction as classification task, by training it via cross-entropy to predict the next action obtained from unlabeled data in the first step (Section 3.1). The default choice for such text classification setup is to embed the whole context at once. However, in preliminary experiments, we found it helpful to embed the input in the same space as the output. As illustrated in Figure lb our planner first embeds each text unit t independently into single vector 2j using the same (frozen) text encoder in Section 3.1
Z,-1 {E(h) + P1. P1...., E(f--) + Pi-1)-
p is an absolute position embedding to integrate sequence information (Vaswani et al. 2017).
Given the embedding matrix Z,-1 E Σ = R with (Z:-1); representing vector 2j. we predict the subsequent writing action a; Given that the embedding is a set of vectors, we employ a simple Transformer encoder with mean pooling and linear classifier head:
- = Transformer (Z,-1); 0,
I (ZLi)); -- -
PFΛ = softmax (W,0;-1 + b).
Here, rr is the predicted probability distribution over actions. Since the semantic relation- ships between actions are encoded in their embeddings d E Rd which we obtained from clustering, we initialize the output matrix W. E R|AAxd with action embeddings.
3.3 Fine-tuning the Language Model Conditioned on Predicted Writing Actions
State-of-the-art LMs are already powerful next-token- next-token-rredictors that possess various capa- bilities. When training these models make use of the information in provided writing actions, we want to finetune them in minimally invasive way in order to avoid catas- trophic forgetting and overspecialization to a particular training domain. To achieve this, we propose adapter-style conditioning module.
For an input sequence of length P, for each position É P, we select an action to be merged into the model. Specifically, we select the action whose centroid embedding is closest to the sentence embedding of the text unit to which the token at the nex! position p+ corresponds. This is done because the final-layer prediction at that position p predicts the probabílity of the token at position p + 1. Call this action ap+1- Following Zhang et al. (2023), we insert actions only in the last layers of the language model. In each layer 1, we first embed the action using an embedding table E We will show in Section 5.3 that it is beneficial to initialize E with the action embeddings 7 p+1 IR that correspond the centroids of the clustering phase. We project this embedding once more with linear layer W Rd xd to get the action representation r, at position p and layer I:
p w (apt)(
The action representation is then added element-wise to the d' o'-dimensional feature rep- resentation inside the transformer at layer and position p, just after the multiplication with attention weights, and just before the output projection. This is similar to the Llama- Adapter ZZhang et al., 2023) with adaptation prompt sequence length 1, except that we do not use gating, which we found not to be helpful. With action-information inserted into the model as explained above, we finetune the LM by minimizing cross-entropy for next-token-prediction as
L - IB x 11 1x/ P(X) log Po (x/1ant) ........... 1 ...x x(--),
5

---

Published as a conference paper at COLM 2024
where P(X)) is the distribution of sequences and denotes planner-predicted writing actions.
An alternative approach is to finetune the language model with oracle writing actions, which we also evaluate in Section 5.1 One the one hand we expect that tuning with oracle actions will increase the mismatch between training (oracle actions) and testing (predicted actions), but the other hand the more reliable oracle actions might induce the LM to rely more strongly on them.
4 Experimental Setup
4.1 Hypotheses
The goal of our paper is to demonstrate the viability of our method for learning an external planning algorithm from unlabeled data whose predictions inform language modeling. Hence, our experiments are designed to test the following hypotheses: (H1) Our method successfully integrates the information from predicted writing actions into language model- ing. (H2) The external planner is more successful than an internal planner. (H3) Our method particularly improves performance in terms of text structure.
4.2 Experimental Details
In all our experiments, we train and evaluate our models on a subset of English Wikipedia articles. To this end, we utilize cleaned Wikipedia dump from March of 2022 that is publicly available via Huggingface") We train on subset of 285310 articles (corresponding to roughly 300M tokens), we split off 1000 articles for early-soopping validation, and 1000 articles for testing. Due to the high computational cost of training model, most of our experiments are performed on the small GP12 model (Radford et al. 2019). However, in order to show that our method is viable for larger, more powerful models, we additionally perform an experiment with the recent OLMo 1B (Groeneveld et al. 2024). The full list of hyperparameters is provided in Appendix A
In Section 5.2 we compare our external planner against the LM-internal planner by Wang et al. (002bb). Since this model uses finetuning to adapt the LM itself to predict writing actions, fair comparison of internal and external planner requires adaptation of our method to this scenario, which we describe in detail in Appendix Briefly, the adaptation consists of inserting externally predicted writing actions as tokens, which allows every token that follows condition on via attention. Moreover, the adaptation allows finetuning of the original LM parameters.
4.3 Evaluation
Primary evaluation The primary objective of our work is to improve language modeling whose standard metric is perplexity PPL = exp(
Surface-level generation evaluation Because perplexity does not evaluate actual gener- ations of the model, we also report ROUGE-2 (F1) (Lin 2004) and MAUVE (Pillutla et al., 2021) scores. ROUGE-2 measures the overlap between generated text and real text in terms of bigrams. Concretely, given a real text X = X1 Xy = t....tm fm, we first use the trained models to generate text continuations X = f 1.....f411 ... im given the prefix .,.. some We then compare true and generated continuations at different lengths, we report the average of the different lengths in Section and the individual results in Appendix B
f for
...
MAUVE a metric designed to evaluate open-ended text generation. It aims to measure the divergence between the probability distribution of the model and the true data distribution. To do so, it takes a set of true articles and generated articles, and deduces a tractable proba- bility distribution from each by embedding and clustering the articles to form histograms. then calculates a divergence frontier (Djolonga et al. 2020) between the two distributions,
https///ggga.eee/ttattaaae///aaa We use the "20220301.ee version.

---

Published as a conference paper at COLM 2024
which is analogous to precision-recall curve applied to generative models. The area under the curve is the MAUVE score. Because MAUVE doesn't require pairwise comparisons, generate texts unconditionally, ie. without context. We generate 1024 tokens, which corresponds roughly to the average article length.
Abstract generation evaluation Additionally, we hypothesize that our planning improves the language modeling performance in particular with respect to text structure. To this end, we measure how well model-generated texts corresponds to ground-truth texts in terms of writing action sequences. We use the procedure described Section 3.1 obtain the sequence of writing actions that corresponds to a real or generated text.
We provide two metrics for this: Levenshtein (Levenshtein et al., 1966) distance and latent perplexity (Deng et al. 2022). The Levenshtein distance (also referred to as edit distance) measures the number of insertions, deletions, and substitutions necessary to make the sequences equal. Similar to ROUGE-2, it compares generated text to ground-truth text in pairwise manner. Hence, we report results for the same text continuations as for ROUGE-2.
For latent perplexity, we train simple HMM-based critic to model the distribution of sequences of abstract writing actions of ground- Iruth text. Latent perplexity results as the perplexity of the sequence of abstract writing actions in the generated text. Latent perplexity evaluates open-ended text generation, and we thus report on the same unconditionally generated texts MAUVE.
Planner evaluation While the end-goal of our work is improving language modeling, the way in which we aim to do that hinges on two components. First, how accurately the planner module can predict oracle writing actions, and second how successfully we can finetune the language model to make use of the information provided by the planner. To measure the former in isolation, we report two metrics: accuracy (i.e., percentage of writing actions predicted correctly), and average rank. The planner predicts a score for each possible action. To calculate the average rank, we order the predicted scores of each action from high to low and observe which rank the oracle action has on average.
4.4 Baselines
Since method involves finetuning language model via the action adapter (cmp. Fig- ure Ic), improved performance over the language model with no finetuning (henceforth called None) is expected due to adaptation to the finetuning dataset (Wikipedia). However, since the goal of this paper is to identify how well the language model is able to make use of the information in the predicted writing actions (called Predicted), we would like to isolate the benefit introduced from predicted writing actions alone for our main experiments. To this end, employ the Fixed baseline, which finetuned receives the same (uninformative) writing action regardless of the context. Finally, we em- ploy the Oracle baseline, which always receives the true next writing action, serve as an upper bound for planners trained on our extracted writing actions. The different baselines correspond to different writing actions being provided to the language model, but each baseline always uses the same type during training and testing. For Predicted, however, it is possible to train using either oracle actions (Predicted-OA) or planner-predicter actions (Predicted-PA).
the same dataset, but always
Wang et al. 2023bb train an LM-internal planner, to which we compare our external planner. To enable fair comparison that isolates the effect of internality externality of the planner, we re-implement their approach in our framework. The implementation details can be found in Appendix
7

---

Published as a conference paper at COLM 2024
Base-LM
Model
PPL
1AUVE
Latent PPL
ROUGE-2
Edit
Oracle
23.92
Baselines
GPT-2 Small
None Fixed
32.89
0.20
438.10 10 352.8
0.013
4.26
26.69
0.38
0.015
3.88
Proposed
Predicted-OA Predicted-PA
26.94 25.55
0.45
91.6 205.9
0.019
3.69
0.44
0.017
3.78
Oracle/ Oracle/Oracle /Orace
10.99
Baselines
OLMo 1B
None Fixed
13.08
0.14
374.22 250.90
0.019
4.83
11.81
0.45
0.022
3.42
Proposed
Predicted-OA 11.99
0.43
74.2
0.026
3.39
Predicted-PA
11.46
0.56
178.2
0.025
3.31
Table 1: Model evaluation under different training and conditioning scenarios. Predicted- OA was trained with oracle actions, while Predicted-PA was trained with planner-predicted actions. The improvement of Predicted-OS to which the model learns to make use of the information in the action (H1), and Oracle shows the best possible performance under perfect prediction of writing actions.
over Fixed over all metrics indicates the extent
Results
5.1 Main Result
Table the results. The main result that the LM trained with planner-predicted actions (Predicted-PA) outperforms the Fixed baseline by 1.14 PPL for GPT-2 and 0.35 PPL for OLMo. It also outperforms Fixed on both surface-level (ROUGE-2, MAUVE) and abstract (Latent PPL, Edit) generation metrics. The results also show trade-off between finetuning actions (Predicted-PA) and oracle actions (Predicted-OAA)
the LM on planner-predicee Finetuning oracle actions damages perplexity,
it introduces mismatch between training and testing data. Despite this, it is sometimes better in terms of generation metrics. An explanation
be found in the different extent to which the resulting LMs actually follow the plans proposed by their planner, i.e., whether they produce sentences that belong to the cluster corresponding to the action used in generating the sentence. We find that Predicted-OA follows the plan about 40% of the time, compared to only 20% for Predicted- PA. This makes sense, oracle actions are more informative than predicted actions. Hence, there is a trade-off between reducing the train/ /test mismatch to achieve low perplexity and following the plan achieve high generation quality. Finding a balance between these two is promising direction for future work.
We also observe that there remain differences of 1.63 PPL and 0.47 PPL. with respect to Oracle. While it is not possible to train planner that is exactly equivalent to oracle access, this does suggest that a planner that better at matching oracle actions can further improve language modeling performance. Moreover, in Appendix D we find that, while the oracle action is much better than a random choice, it is not optimal. Hence, further perplexity gains could be made by learning to consistently predict those actions that lead to an even lower perplexity than the oracle actions.
5.2 External vs. Internal Planner
Conducting a fair comparison between internal and external planner as described in Ap- pendix C we find that the external planner performs 1.22 PPL better than the internal planner by [Wang et al. (2023b) if we restrict finetuning to parameter-efficient finetuning with LoRA, and PPL. better in full finetuning.
8

---

Published as a conference paper at COLM 2024
5.3 Ablations
Model Architecture In Table we perform an ablation showing the contribution of several architecture components. Having multi-vector representation improves the performance of the planner, which in turn is slightly beneficial when conditioning the language model on predicted actions, As expected, initializing the action embeddings with the respective cluster centroids is also crucial to the performance of the model. While initializing the planner with action embeddings helped for preliminary experiments at small scale, the effect disappears with more training data.
Clustering Figure shows the model behavior as we increase the number of possible actions. The perplexity improves up 8192 actions, after which it decreases again. This is explained by the tradeoff between informativeness and predictability: More clusters correspond to less compression of information contained in the sentence embeddings. On the other hand, more available actions make the prediction task harder, as indicated by an increasing average rank.
In Tables ind the Appendix, we analyze the ten largest clusters qualitatively. We find that they often correspond well to structures typically found in Wikipedia article. For example, an article about person typically contains information about person's origin, education, and professional life in certain order, which a planner would be able to predict in terms of writing actions. Some actions, however, are rather topical than structural. For example, cluster #7 contains mentions of Italian people and places. This indicates that our procedure of generating abstract writing actions (see Section 3.1 can be improved further, e.g. through an encoder whose representations contain more structural rather than topical information.
Model
Acc. f
Rank
PPL 1
full model
24.7
36.4
25.55
25.80
Perplexity
600
no sent rep
21.9
38.3
25.59
Average
no P init
24.3
37.3
25.51
25.75
500
no
24.7
36.4
27.29
25.70
Perplexity
400
Table 2: Ablations. no sent rep: In- stead of encoding sentences sepa- rately, encode whole context into one vector via mean-pooling. Use
25.65
Average
300
25.60
25.55
200
an MLP on top instead of a Trans- former. no P init: Randomly initial- ize action output layer of the plan- ner. no AD init: Randomly initial- ize action input layer of adapter.
25.50
25.45
256
1024
4096
8192
16384
Number of Clusters
Figure 2: Performance by number of clusters.
Content
Example sentence
Demographics
The rural population 693,566 (89.03%) and urban 85,496 (10.97%).
Plot
During private argument about whether not they should move, Elaine
reveals she is pregnant.
3
Business
2010: Snecma and GE formed CFM Materials 50/50 joint vefnture.
Geography
Khuzama is situated in Jakhama circle of Kohima District in Nagaland.
Schools
The district consists of an elementary school, middle school, and high school.
Education
He has a bachelor's 's degree from the University of Florida.
Italy
Maffco Olivicri (1484, Brescia 1543 or 1544) was an Italian sculptor and
wood carver.
Acronyms
Comput.
Origin
Shokler was born in Cincinnati on April 27, 1896.
10
Career
In 1954 he played impressionable navigator opposite Gregory Peck in The Purple Plain.
Table 3: Analysis of the ten largest clusters.

---

Published as a conference paper at COLM 2024
6 Discussion
The experimental results (Section 5.1
confirm that our method is able to enhance language model performance by integrating externally predicted writing actions (H1) inferred from unlabeled data. This finding holds both for small LMs and more powerful, larger LMs. Naturally, the effect smaller for larger models because their general language modeling ability is already stronger.
As evidenced by experimental results in a fair comparison (Section 5.2) externality of the planner (H2) is beneficial. Besides performance, an external planner enables greater modularity such that the language model and the planner can be developed and reused independently.
The enhanced LM ability of our method can be attributed to a stronger ability to plan their writing structurally (H3), since conditioning on predicted writing actions causes the model to follow the ground truth text structure, which is indicated by a lower Levenshtein distance and latent perplexity in comparison to finetuning without a planner (Section 5.1] Our qualitative analysis of clusters (Section 5.3) suggests that writing actions indeed correspond to common structures in the dataset.
Finally, our method's performance depends on the technical contributions that we have made, such as better neural architecture for predicting writing actions (Section (.3) and conditioning the LM on predicted actions rather than ground truth actions during training (Section 5.1
6.1 Limitations
Our study is limited in some ways. First, while we try to show the generality of our method through experiments with a more powerful, medium-sized OLMo 1B model, due to computational constraints, we cannot train models of even larger scale and strength. Moreover, while we trained on the Wikipedia corpus, which is often considered general- purpose, its diversity is still considerably more limited than the datasets modern LLMs are trained with, which typically subsume a snapshot of the web. Finally, since LMs only achieve nontrivial performance on zero- and few-shot tasks after large amounts of pretraining compute, we do not evaluate the benefit of our method on such downstream tasks. In the future, we want to apply our method to an LLM of medium size (e.g. OLMo 7B) finetuned on diverse dataset (e.g. Dolma (Soldaini et al.) 2024
Second, while we improve over standard architecture with a model better suited for predicting future text, there are several obvious avenues for improvement that remain unexplored. Since we obtain abstract writing actions via clustering in a text embedding space, they do not necessarily encode information that is best suited to help language modeling. We view learning actions tailored to this use case as a promising research direction. Moreover, we want to explore planning multiple text units in the future and using a search method such as beam search or Monte Carlo Tree Search at inference time to improve planning and find the action that optimizes downstream LM performance.
7 Conclusion
Large Language Models have impressive capabilities, but they lack the ability to plan their writing well. Our proposed method utilizes an independently trained, external planner module, whose outputs can be used to inform and improve the generation process of the language model. Crucially, the planner is trained from unlabeled data in a self-supervised fashion. These qualities enable researchers to develop and train new planners at large scale and publicly share them, allowing anyone to integrate new planning capabilities with the language model of their choice. Hence, we envision future where progress on planning algorithms advances at a similar speed as language models today.
10

---

Published as a conference paper at COLM 2024
Acknowledgments
This research was financed by the CALCULUS project- and Anticipation enriched Learning of Continuous represntations--uuueen Research Council Advanced Grant H2020-ERC-2017-ADD 788506, http:://calculussppjjeetttt
References
David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In SODA, pp. 1027-1035. SIAM, 2007.
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. pp. 932-938, 2000.
Yoshua Bengio, Yann LeCun, and Geoffrey E. Hinton. Deep learning for AL. Commun. ACM, 64(7):58-65, 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Cameron Browne, Edward Jack Powley, Daniel Whitehouse, Simon M. Lucas, Peter Cowl- ing, Philipp Rohlfshagen, Stephen Tavener, Diego Perez Liebana, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Trans. Comput. Intell. Al Games, 4(1):1-43, 2012.
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. ICML, volume 202 of Proceedings of Machine Learning Research, pp. 3676-3713. PMLR, 2023.
Zhipeng Chen, Kun Zhou, Beichen Zhang, Zheng Gong, Xin Zhao, and Ji-Rong Wen. Chat- cot: Tool-augmented chain-of-thought reasoning on chat-based large language models. In EMNLP (Findings), pp. 14777-14790. Association for Computational Linguistics, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, November 2021. URL
14168 arXiv:2110.14168 [cs].
Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 11781-11791, Singapore, December 2023, Association for Computational Linguistics. doi: 2023. 1223p-mnlp-mall manellppmmin.777 721
URL httphh::/acaaathhllllaa org/
Yuntian Deng, Volodymyr Kuleshov, and Alexander Rush. Model criticism for long-form text generation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, Pp. 11887- 11912, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 16653/v1/222.emm/// 2022. 1022.emllp-mann m
URL httphtttt//////attoooooo org/
Josip Djolonga, Mario Lucic, Marco Cuturi, Olivier Bachem, Olivier Bousquet, and Sylvain Gelly. Precision-recall curves using information divergence frontiers. In International Conference Artificial Intelligence and Statistics, pp. 2550-2559, PMLR, 2020.
11

---

Published as a conference paper at COLM 2024
Jonathan St BT Evans. Heuristic and analytic processes in reasoning. British Journal of Psychology, 75(4) 1984.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Ólmo: Acceler- ating the science of language models. arXiv preprint arXiv:2402.00836 2024.
Han Guo, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. Efficient (soft) Q- learning for text generation with limited good data. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6969-6999 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi:
L0.18653/vv 10.18653/v1/20222 1e2022..indinng-emmitt
URL https
//acllanthllogyorrg//8// wrg.:e....rg///00 wttp://.orr.........nno..oeee.e. findings--mm.naa em.lp..-em...... 518
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang, Retrieval augmented language model pre-training. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 5929-3938. PMLR, 2020.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. EMNLP, pp. 8154-8173. Association for Computational Linguistics, 2023.
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. Xiv preprint ar Xix::2403. 04642, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR. OpenReview.net, 2021.
Anson Ho, Tamay Besiroglu, Ege Erdil, David Owen, Robi Rahman, Zifan Carl Guo, David Atkinson, Neil Thompson, and Jaime Sevilla. Algorithmic progress in language models. Xrv preprint arXio:2403.05812 2024.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In NeurIPS, 2022.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, Adriane Boyd, et al. spacy: Industrial-strength natural language processing in python. 2020.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In TCLR. Revnnevie 2022.
Daphne Ippolito, David Grangier, Douglas Eck, and Chris Caltison-Burch. Toward better storylines with sentence-level language models. In ACL, pp. 7472-7478. Association for Computational Linguistics, 2020.
Harsh Jhamtani and Taylor Kirkpaaikk. Narrative text generation with latent discrete plan. In EMNLP (Findings), volume EMNLP 2020 of Findings of ACL, pp. 3637-3650. Association for Computational Linguistics, 2020.
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Gener- alization through memorization: Nearest neighbor language models. In ICLR. OpenRe- view.net, 2020.
12

---

Published as a conference paper at COLM 2024
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. In NeurIPS, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurlPS, 2022.
Jey Han Lau, Timothy Baldwin, and Trevor Cohn. Topically driven neural language model. In ACL (1), pp. 355-365. Association for Computational Linguistics, 2017.
Vladimir Levenshtein et al. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pp. 707-710. Soviet Union, 1966.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurlPS, 2020.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summariza- tion branches out, pp. 74-81, 2004.
Jieyi Long. Large language model guided tree-of-thought, 2023, URL https://xxxxi///rr/ abs/2305. 08291
Andreas Marfurt and James Henderson. Sentence-level planning for especially abstractive summarization. In Proceedings of the Third Workshop on New Frontiers in Summarization, pp- 1-14, 2021.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR (Poster). OpenReview.net, 2017.
Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. Document- level neural machine translation with hierarchical attention networks. In EMNLP, pp. 2947-2954. Association for Computational Linguistics, 2018.
Tomás Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In INTERSPEECH, pp. 1045-1048. ISCA, 2010.
Tomás Mikolov, Stefan Kombrink, Lukás Burget, Jan Cernocky, and Sanjeev Khudanpur. Extensions of recurrent neural network language model. In ICASSP, pp. 5528-5531. IEBE, 2011.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models. In Forty-first International Conference on Machine Learning, 2024. URL /ttphtts///opeeeeiee.... wwwpwviwwwwieeefrruunnee
id=bvIc2b7Qa0
Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. Abstractive text summarization using -oueeeeee-----eeeccee rnns and beyond. In CoNLL, pp. 280-290. ACL, 2016.
Maxwell I. Nye, Michael Henry Tessler, Joshua B. Tenenbaum, and Brenden M. Lake. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. In NewrlPS, pp. 25192-25204, 2021.
Bhargavi Paranjape, PaaagjaiParaajjpee, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models, 2023. URL httptp::///aaxi.oor////////... 09014
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high--peformmnce deep learning library. Advances in neural information processing systems, 32, 2019.
13

---

Published as a conference paper at COLM 2024
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Coumapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-leam: Machine learning in Python. journal of Machine Learning Research, 12:2825-28331 2011.
Laura PerezzBeltrachini, Yang Liu, and Mirella Lapata. Generating summaries with topic templates and structured convolutional decoders. In ACL (1), pp. 5107-5116. Association for Computational Linguistics, 2019.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Adoances in Neural Information Processing Systems, 34: 4816-4828, 2021.
Alec Radford. tive Pre-Training.
Improving 2018.
Language URL
Understanding by Genera-
https://www. wwt.://stmtiicchoo..ooooooo org/
hygp:r/annuuuu--eeeerrrnnnnnnnnr cd1 ccl8800ffe66aaacc113ee4550006060
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al.
Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Kallison-Burchh and Jason Wei. A recipe for arbitrary text style transfer with large language models. In ACL. (2), pp- 837-848. Association for Computational Linguistics, 2022.
Nils Reimers and Iryna Gurevych. Sentence-bert, Sentence embeddings using siamese bert-networks. In EMNLP/IJCNLPP (1), pp. 3980-3990. Association for Computational Linguistics, 2019.
Timo Schick, Jane Dwivedi-Yo Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In NeurlPS, 2023.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timo- thy P. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Nat., 584,588878399):04-666 2020.
Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, et al. Pangu-coder2: Boosting large language models for code with ranking feedback. arXiv preprint ar 2ttt:23307.449366, 2023.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, lan Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Pe- ters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint, 2024. httphttts///arrt http://axxvvooorraab/ / /rgg///////2..0/0 22tp:/2422.00111 00159
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. In NeurlPS, 2020.
Simeng Sun, Yang Liu, Shuohang Wang, Dan Iter, Chenguang Zhu, and Mohit Iyyer. PEARL: prompting large language models to plan and execute actions over long documents. In EACL (1), pp. 469-486. Association for Computational Linguistics, 2024.
Richard Sutton and Andrew Barto. Reinforcement learning: An introduction. MIT press, 2018.
14

---

Published as a conference paper at COLM 2024
Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. Abstractive document summarization with a graph-based attentional neural model. In ACL (1), pp. 1171-1181. Association for Computational Linguistics, 2017.
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models critical investigation. In NeurlPS, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, LlionJones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 5998-6008, 2017.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NIPS, pp. 2692- 2700, 2015.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In ACL (1), pp. 2609-2634. Association for Computational Linguistics, 2023a.
Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. Topic-guided variational auto-encoder for text generation. In NAACL-HLT (1), pp. 166-177. Association for Computational Linguistics, 2019.
Xinyi Wang, Lucas Caccia, Oleksiy Ostapenko, Xingdi Yuan, and Alessandro Sordoni. Guiding Language Model Reasoning with Planning Tokens, December 2023b. URL arXiv:2310.05707 [cs].
http:///axxivorrg/bbap 23100.5707
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Rationale- augmented ensembles in language models, 2022a. URL httpss:///aaxvv.og/aabbss// 00747
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought rea- soning in language models, 2022b. URLLhhttps:/oopeneeviiee wwtp://eeooooomoeoppoomm
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In EMNLP (Demos), pp. 38-45, Association for Computational Linguistics, 2020.
Tianci Xue, Ziqi Wang, Zhenhailong Wang, Chi Han, Pengfei Yu, and Heng Ji. Rcot: Detecting and rectifying factual inconsistency in reasoning by reversing thain-of-thought, 2023. URL //arxiv. http:/awwvv.or.rr//////./ org/abs/2305. 11499
Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3511-3535, Online, June 2021. Association for Computational Linguistics. doi: 10.18653 v1/2021.11. 1022l.aacl-mainn 276. URL https!
//ac.lanthology.rrr/28888 naacl-main 276
15

---

Published as a conference paper at COLM 2024
Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foundation models for decision making: Problems, methods, and opportunities. arXio preprint ar Xio:2303.04129, 2023.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy, Hierarchical attention networks for document classification. In HLT-NAACL, PP. 1480-1489. The Association for Computational Linguistics, 2016.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS, 2023.
Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. A survey of controllable text generation using transformer-basee pre-trained language models. ACM Comput. Suro., 56(3):64:1-64337 2024a.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adane adaater Efficient fine-tuning of language models with zero-init attention. arviv preprint ar Xiv:2303.161999 2023.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tat- sunori B. Hashimoto. Benchmarking Large Language Models for News Summarization. Transactions of the Association for Computational Linguistics, 12:39-57, 01 2024b. ISSN 2307-
387X. doi: 10.1162 tacllaaallaa 00632. URL https://doi.orr//e/ 1162/tac1111aatccc a 00632
Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. Verify-and- edit: knowledge-enhancee chain-of-thought framework. ACL (1), pp. 5823-5840. Association for Computational Linguistics, 2023.
Carolina Zheng, Keyon Vafa, and David Blei. Revisiting topic-guided language models. Transactions Machine Learning Research, 2023a.
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hint prompting improves reasoning in large language models, 2023b. URL https://arxiv org/abs/2304.095
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang. Multilingual machine translation with large language models: Empirical results and analysis. arXiv preprint arXio:2304.0467 046675, 2023.
A Hyperparameters and Experimental Details
Implementation
The code is available Github.
Data processing All of experiments are based cleaned Wikipedia dump from March of 2022 that is publicly available via Huggingface preprocessing to the dataset.
We do not apply any further
For generating abstract writing actions (Section 3.1 2020) to split every article into sentences. Then, we use MPNet-base-v2 (Song et al.) 2020 via the SentenceeTransorme embeddings. For clustering, we use the existing k-means implementation from Scikit- Learn (Pedregosa et al.) 2011) with k-means+ initialization (Arthur & Vassilvitskii 2007 and run for the default maximum 300 iterations.
first use spaCy (Honnibal al.
library (Reimers & Gurevych, 2019 to encode sentences into
hnttps://huggingfacc htt.:/ggg.ggggtee.oo//aaaeee//aa https://huggingffce http:///wwoacocoeooeoeeeeeeeeeoa
16

---

Published as a conference paper at COLM 2024
Neural network implementation All methods compared in paper implemented within the same PyTorch-based (Paszke et al. 2019) framework. Language models are loaded via the Huggingface Transtormers library (Wolf et al. 2020). Experiments with the GPT2 mode@@mode were 1F
single 12GB GPU, whereas experiments including OLMo single 24GB GPU.
were
Evaluation To compute the Levenshtein distance, we use the python-Levenshtein package with default settings. For ROUGE and MAUVE, we used the python packages 'rouge' and 'mauve-text', respectively. For latent perplexity, we made minor changes to the implementation byDDegg et al. (2022), available at https://github. httpa//w........eeaaaaaaaaeeiiee wtt.rrttiicccee.e..eeeeettiiio
(nttps://pypi. wwwoorrg/ooooooooo http//oojec/ppyteeeeeeeee roojjeettpyyho--Lennstteeinn
Hyperparameters
Table shows hyperparameters used for our experiments.
Hyperparameter
Value
Context window size
128
Train test val split sizes
285310 1000
1000
K-means initialization
k-means++
Number of tokens generated for edit distance
128
Default action count
1024
Action embedding dimension
768
Language Model Finetuning
Batch size
32
Learning rate (Ir)
1e-4
Planner Training
Batch size
32
Learning rate (Ir)
1e-4
Table 4: Hyperparameter Settings
B Detailed ROUGE and Edit Distance Results
Table shows the Rouge-2 and edit distances between true and generated continuations, when cut off different generation lengths. Because the edit distance scales linearly with the number of tokens, we normalize results: for 128 tokens we report the actual edit distance, for 256 the edit distance divided by 2, etc.
C Internal vs External Planner Setup
We want to compare our approach that uses an external planner to using the language model itself planner, as is done in Wang et al. (2023b).
the vocabulary size, W the number of writing actions, and h the hidden dimension, us- ing the language model planner entails expanding the V x h-dimensional final prediction layer to a (V + W) x 1--dimensional layer. Just training those extra W ht parameters does not suffice to let the language model learn to predict planning tokens, which is why Wang et al. (2023b) also finetune the existing language model parameters (either full-finetuning, or parameter-efficient finetuning with LoRA (Hu et al. 2022
co/
17

---

Published as a conference paper at COLM 2024
Base-LM
Model
128
256
512
1024
2048
ROUGE-2 f
None
0.0125
0.0134
0.0136
0.013
0.0116
Fixed
0.0148
0.0167
0.0165
0.0154
0.01
Predicted-OA
0.0185
0.0205
0.0211
0.0196
0.02
GPT-2
Predicted-PP
0.0166
0.0178
0.0184
0.0169
0.01
Edit
None
4.82
4.42
4.12
4.01
3,94
Fixed
4.63
4.09
3.76
3.50
3.39
Predicted-O
4.39
3.88
3.55
3.35
3.25
Predicted-22 P
4.45
3.92
3.67
3.50
3.37
ROUGE-2 1
None
0.0212
0.021
0.0194
0.0168
0.0141
Fixed
0.0242
0.0247
0.023
0.0203
0.0165
Predicted-OA
0.0275
0.0296
0.0286
0.0255
0.0209
OLMo
Predicted-PAA
0.0277
0.0284
0.0273
0.024
0.0192
Edit
None
5.01
4.68
4.71
4.84
4.92
Fixed
4.21
3.69
3.38
3.07
2.77
Predicted-O
4.15
3.64
3.29
3.06
2.84
Predicted-PA
4.2
3.64
3.25
2.89
2.55
Table 5: Details of ROUGE-2 and Edit distance results for different numbers of tokens.
Besides having an external planner, our model also differs from Wang et al. (023b) the way predicted actions presented to the model. We present the predicted writing actions to the language model with the adapter-style conditioning module detailed in section 3.3 whereas Wang et al. (2023b) insert the actions like tokens in between sentences. The insert- style requires extending the imput embedding matrix the language model in the same way as the final prediction layer above.
Consider sequence of tokens X = x x7... x....mm .1....11 ... mgg where x/ is the ith token of the jth text unit in the sequence. Let A = am be the corresponding sequence of writing actions where each a; E {0... W - 11. Call idx-x; the index the vocabulary of x/, we then pass the following indices the language model:
(a1 + V) i)xxxx-f...idxx
(am + V) idx-x"" idx-x
To isolate the effect of using an external planner vs using the language model as planner, we evaluate our model in an adapted setting where we also fine-tune original parameters, and condition on planner-predicter actions in insert-style rather than adapter-style.
Because learning to plan and learning to condition on the plan are intertwined for the internal planner, does not allow training just a planner separately beforehand in order to condition on caanner-predicted actions. Hence we condition the internal planner on oracle actions during training.
When training the model to condition on oracle actions, there is slight mismatch with how will be used at generation time (when it uses self- self-predicted actions): During training with oracle actions, the action embedding always corresponded to the centroid closest to the embedding of the text unit following it, while that does not necessary hold during generation. One option to reduce this mismatch during generation would be to retroactively change past predicted actions in the context of the language model to match the generated sentence that follows them. We leave evaluation of this option for future work.
18

---

Published as a conference paper at COLM 2024
D Are Oracle Actions Optimal?
An oracle action is the action whose clustered embedding is closest to the concrete em- showed bedding of the true next sentence. Showeed that conditioning on an oracle code consistently leads to lower perplexity than conditioning on fixed or predicted code. A natural question whether providing a language model with the oracle action leads to lowest perplexity among all actions.
To test this, we first evaluate every action at every step, resulting in a ranked list of actions per step. Then, we compute the perplexity that one would get when always choosing the k-th best action. We plot the the results for every in Figure
Average Perplexity at each Rank
10a
Perplexity
6 10
x 10
Oracle rank among actions: 18.
10
Oracle: 24.01
Best action: 21.54 2
200
400
600
800
1000
Rank
Figure The blue dots show what the average perplexity is when conditioning on the k'th best action (in terms of what perplexity it leads to), with the rank k on the horizontal axis. The red horizontal line displays the average perplexity of selecting the oracle code, the blue vertical line shows the rank with the nearest average perplexity to the oracle perplexity
The figure shows that the average perplexity using oracle codes equivalent to using the 18th best action (out of 1024) in each step, meaning that average there are 17 actions that would lead to lower perplexity than the oracle action.
The fact that better-than-oracle-attee planner that is able to consistently find those saater-hha--rrcce-cctiono alternative explanation is the better-than-oracl actions are better purely due luck: If one would make 1024 noisy variations of the oracle action embedding, some of those would also be better than the oracle embedding by chance. We investigate this alternative hypothesis by adding Gaussian noise (μr 0, o set to empirical standard deviation across the action embeddings over all layers) to the oracle action embeddings.
exist does not guarantee that it is feasible to train One possible
Figure 4sowws shows that indeed about half of the noise variations outperform the oracle embed- ding. However, also shows that the lowest perplexity achievable by selecting the best random noise is significantly higher than that of selecting the best alternative action. This suggest random luck is not the full explanation for the success of the better-than-oracle actions, and it might be feasible to predict them consistently.
indeed feasible, this suggest the planner can be made more effective by training it to predict the value of an action directly (in terms of the perplexity it leads to), rather than only training to match oracle actions. A planner that can successfully estimate the values of its actions could then be combined with an algorithm like MCTS (Browne et al. 2012) at
19

---

Published as a conference paper at COLM 2024
Average Perplexity at each Rank
Learnt action embeddings
Noisy variants of oracle embedding
10²
Perplexity
10
Oracle rank among noise variations: 4.06
4 10
Oracle rank among actions: 18
3 10
Best noise variant: 22.58
Best action: 21.54
2 10
Rank
Figure 4: The green curve shows what the average perplexity when conditioning on the k'th best noisy variation of the oracle action embedding (in terms of what perplexity it leads to), with the rank k on the horizontal axis. The green dotted horizontal line indicates the perplexity of the best noise variant, and the green dotted vertical line the rank with the nearest average perplexity to the oracle perplexity among noise variations. The rest is the same as in Figure
inference time for even greater performance gains. We consider this promising avenue for future work.
E Examples of Clusters
Table shows an extension of Table i.e, multiple examples per action cluster.
F
Complexity Analysis
In the following, we discuss the number of parameters that method requires well as the computational complexity.
Parameter count The base language model has approximately 130M (GPT-2) or approx- imately 1B (OLMo-1B) parameters. The adapter needed for finetuning add about 14M parameters. The planner consists of sentence transformer with 110M frozen parameters plus a small Transformer encoder with 6M additional parameters. Having a separate plan- ner with its own sentence representations means we can reuse the same planner for different language models. As memory-efficient alternative we can use the sentence representations of the LM (which have slightly worse overall performance), losing the portability advantage.
Model complexity The complexity (number of computations) of our proposed sentence- based planner (Section 3.2), is O N2
M for the representation function (computation of Z'; 1) plus ((M3) for the prediction function, where N is the total number of tokens and M the number sentences. Since the representation function (110M parameters vs 6M for the prediction function) represents the vast majority of the complexity, invoking the planner is
Because evaluating perplexity conditioned on all codes is slow, the average is taken only over 58 articles for this graph.
20

---

Published as a conference paper at COLM 2024
Content
Example sentences
1
Demographics
1: According to the 2011 census, it had 841 inhabitants, all of whom were Albanian. 2: The population was 23 as of 2002. 3: Based on the 2007 national census conducted by the Central Statistical Agency of Ethiopia (CSA), this woreda has total population of 112,396 of whom 56,245 are men and 56,151 Women, no urban inhabitants were reported. 4: 1790 Population: 16,014. 5: Its population as of 1990 was 344; its population as of 2000 was 680.
2 Plot
1: Feeling angry and betrayed, Nadine separates from Chloe, but they make amends. 2: The pair ultimately decide to move in together after Britney kicks out her previous roommate. 3: After Coré's death, Shane becomes strange and distant. 4: When Nico finds out that he never showed the police Antonia's s diary she questions their relationship and Philipp's ' love for her again. 5: Louise reveals that she has been helping both Bob and Gene with their pranks without the other knowing and announces that she is quitting.
3 Business
1: Then in May 1998, all of PolyGram and its associated labels were purchased by Seagram which announced its plan to integrate Poly- Gram with UMG to produce an estimated cost savings, within a couple of years, of between US$275 million and $300 million an- nually. 2: As of April 2020, the ownership of the pipeline was as represented in the table below. 3: Wendy's International is owned by The Wendy's Company. 4: October 2007: Nakheel announced the sale of Ireland, and Shanghai in October 2007. In 2016, Endeavor entertainment group acquired a reported 70%-contro stake in Frieze, which includes its publishing, fair and music interests.
a Hollywood-basee
Geography
1: Districts of Khyber Pakhtunkhwa 2: and in Kyrenia District. 3: Birhors are found mainly in the area covered by the old Hazaribagh, Ranchi and Singhbhum districts before these were broken down into numerous smaller units, in Jharkhand. 4: Perumpanachy falls under Madapally Panchayat and Chanaganacherry Thaluk. 5: It is about from Kuknur and from Lakkundi.
5 Schools
1: The old high school building is now being reused as the middle school. 2: High School is one of the four-star high schools in Jiangsu Province. 3: The school has complete grade school levels, and high school levels. 4: The sole entrance and exit to the school is along Texas State Highway Loop 1604. 5: Students that attend the school come from the central parts of Prince County.
Table 6: Analysis of the ten largest clusters. (Part 1)
roughly M times less expensive than the language model (GPT-2: 128M parameters, OLMo: 1B parameters), which have complexity ON22).
At training time, due to its externality, the planner can be trained once and henceforth be integrated with variety of language models. Therefore we focus on inference-time cost here. The planner consists of a representation function (sentence-transformee and prediction function (Transformer encoder on top of sentence embeddings).
Representation function: The bulk of the compute of the planner is in the representation function (110M parameters). Since our architecture encodes sentences in the previous context independently, sentence embeddings can be efficiently cached to avoid re-encoding when moving from one sentence to another. Computing the sentence embedding with Transformer-basee architecture has quadratic complexity in the length of the sentence. Assuming a simplified view where each sentence has length N/M M, where N is the total number of tokens in an article and M is the number of sentences, encoding the necessary representations for all calls to the planner in article has complexity O(M (N/M)2)
21

---

Published as a conference paper at COLM 2024
Content
Example sentences
6 Education
1: In 1957, he graduated with thesis on Henry James. 2: In 1996, he graduated with a BA in Political Science and Government, Philoso- phy, and Business Administration. 3: He studied at Dalton School in New York and The Putney School in Vermont, later on graduating from Harvard University in 1965, in Far Eastern history. 4: Guinle meanwhile earned a degree in Accountancy from the University of Morón in 2000. 5: In 1986, Darren Barrett attended Berklee College of Music on a full scholarship, receiving a BA in Professional Music in 1990.
7 Italy
1: Non qui, Barbara, nessuno ci sta guardando (1989) 2: Marchesa
Olga Benucci Granieri Solaro - Monaldo wife, mother of Vittoria and Costanza and mother-in-law of Martino, Olga is very car- ing woman to the image of his own family. 3: Ugo Gregoretti (28 September 1930 - July 2019) was an Italian film, television and stage director, actor, screenwriter, author and television host. 4: This was the second Garibaldi group that succeeded the first group of the same name. 5: Gaetano Marco "Guy" Nardulli (born May 31, 1974 in Norridge, Illinois) is an American actor and producer who is most associated for his character role as a street fighter turned MMA professional fighter in the 2006 movie -And Then It Breaks with actress Anne Dudek.
8
Acronyms
1: Soc. 2: Dir. 3: Ber. 4: Ed. 5: 2nd ed.
Origin
1: Ashley Sessa was born and raised Schwenksville, Pennsylvania.
2: Born on August 9, 1883, in Reading, Pennsylvania, Daisy Elizabeth
Adams was educated in Reading, Pennsylvania. 3: He was born in Keokuk, Iowa in 1877. 4: Ada E. Schnitzer was born November
22, 1887, in Hannibal, Missouri, the daughter of O. C. Schnitzer. Morse was born in Abington, Massachusetts on August 19, 1911.
10 Career
1: He appeared in the 1995 Seinfeld episode "The Beard" playing Robert's boss.
2: Tisch also made an appearance on the reality show Shark Tank in season 5. 3: He starred as the surgeon Theodor Hristea, who, after some of his belongings are stolen, involves himself in the inquiry and directs the interrogation of a seemingly innocent man. 4: He also appeared, uncredited, in the movie It's a Wonderful Life (1946), playing piano in the scene where George Bailey gets thrown out of Nick's Bar. 5: Clint Eastwood was among the first of many actors to adopt this wandering ronin with no name persona for foreign films, which he used to great effect in his Western roles, especially in Spaghetti Westerns directed by Sergio Leone where he played the Man with No Name, a character similar to Mifune's seemingly-nameles ronin in Yojimbo.
Table 7: Analysis of the ten largest clusters. (Part 2)
O(N) ). In contrast, non-sentence--asedd representation (see ablation 'no sent rep' in Table 2 has complexity (N2). Since there are many sentences in an article, the sentence-based planner is considerably more efficient.
Prediction function: The additional Transformer encoder is lightweight (6M parameters) and has complexity (M?2, where M' denotes the number of sentences in the current context. Overall, this brings the complexity to O (M²) across the article. This could potentially be brought down to (M2) by using a Transformer decoder instead of a Transformer encoder for the prediction function, which allows for caching due to causal masking.
22

---

Published as a conference paper at COLM 2024
G Color-coded Example Article
In order to give the reader an impression of how abstract writing actions are distributed within an article, below we print an article that is color-coded with different actions. Differ- ent text colors correspond to different actions.
G.1 Article
In 1917, Weeks defeated incumbent John J. Mullen by 230 votes to become Mayor of Everett. The Boston Daily Globe described the race between Mullen and Weeks as "one of the bitterest campaigns in years" and in his inaugural address, Weeks referred to his predecessor as "caterwauling demagogue" and vowed to overturn many of his acts, including firing of Police Chief William E. Hill and the closure of the Everett Tuberculous Hospital. In 1918, Christopher Harrison defeated Weeks by 390 votes, with Mullen, who supported Harrison after being eliminated in the preliminary election, taking credit for "putting [him] over".
In 1922, Weeks was the Progressive Party candidate for United States Senate. He finished sixth with less than 1% of the vote. In 1923 Weeks moved to Reading, Massachusetts. However, in 1933, he returned to Everett to run for Mayor. He made the runoff election, but was defeated by another former Mayor, James A. Roche. During the 1934 gubenatoral election Weeks supported Democrat James Michael Curley. In 1935 Curley appointed Weeks to the State Alcoholic Beverage Control Commission. In 1941, Weeks again ran for Mayor of Everett. He finished last in a four candidate primary.
Legal career In 1922, Weeks defended George H. Mansfield, former Everett resident who was charged with murdering his lover, Alice Jones. The medical examiner later ruled that Jones committed suicide and District Attorney Thomas C. O'Brien asked the grand jury to return no bill against Mansfield. In 1924, Weeks served as a special counsel for defendants accused of being part of a extortion ring led by former Middlesex County district attorney William.. J. Corcoran. They were found guilty and Corcoran was sentenced to 7 to 10 years in prison. Weeks also defended Corcoran when he and Daniel H. Coakley were also charged with conspiracy to extort later that year. They were found not guilty on all counts. In 1927 Weeks represented Jerry Gedzium, a convicted murder who was appealing his death sentence. The conviction was upheld by the Massachusetts Supreme Judicial Court.
See also 1916 Massachusetts legislature
References
1880 births 1972 deaths Mayors of Everett, Massachusetts Massachusetts lawyers Mas- sachusetts Progressives (1912) Massachusetts Republicans Members of the Massachusetts House of Representatives People from Reading, Massachusetts
23


Reviews:
[{"review_id": "4pmOBq1GbW", "reviewer": "Reviewer_BMkB", "summary": "This paper presents a method for explicit planning for LLMs for text generation: \n1. It breaks text generation into “chunks” (or sentences”); and for generating the next chunk, it explicitly injects an “action” variable, obtained from the planner module,  to condition the next chunk generation.\n2. It trains a separate transformer model to predict a sequence of action variables, as the “plan” of the generation.\n3. The action variable is obtained by clustering the sentence level embeddings for sentences in the training corpus, which means no explicit supervision. \n\nEmpirically, the authors find that the planning can somewhat improve generation quality (in terms of lower perplexity).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "*Methodology*:\n1. We do not agree that the proposed actions are called **“next sentence embeddings”** as they represent a cluster of sentences at a more abstract level. Consequently, the planning model  does not perform  “next sentence language modeling” for the same reason.  \n2. As discussed in Section 3.1, we posit that it is necessary to compute abstractions that correspond to common writing patterns. By nature, planning in natural language involves a form of abstraction to deal with the vast action space. As Figure 2 demonstrates and we discuss in Section 5.3, there is a **tradeoff between the granularity of actions (and hence informativeness for generation) and the difficulty of predicting the correct action**. In the case that two consecutive sentences are guided by the same action, the language model provides fine-grained control conditioned on previous words and sentences through attention.\n3. We have tested **your suggested alternative way of using a signal from the language model**, namely in our ablation that encodes the entire context using GPT-2 at once rather than encoding each context sentence with a sentence transformer and then fused into one representation with a trained transformer. We observed that this yielded no improvement (see Table 2 second row and third paragraph of Section 3.2). We will state explicitly that this ablation uses GPT-2 in the final paper.\n\n*Experiments*: \n1. It is not true that **None** is our only baseline. The main baseline to our proposed method (“Predicted/Predicted” in Table 1) is “Fixed/Fixed”, which finetunes the same architecture on the same data using the regular causal LM objective.\nOur improvement is consistent across 2 models, and more than 1 perplexity point is a moderately sizable difference commonly reported in the LM literature. \n2. & 3. We now evaluate generation performance via **ROUGE-2** and **edit distance on long articles** up to 2048 tokens. The table below shows the *relative* improvements (in %) of our method over the Fixed baseline, cutting off generation at different points.\n\n| |128|256|512|1024|2048|\n|---|---|---|---|---|---|\n|GPT-2 Edit|6.0|3.1|2.7|1.0|-0.2|\n|GPT-2 R2|12.8|11.0|12.7|12.0|9.8|\n|OLMo Edit|-0.1|1.7|2.6|2.3|2.2|\n|OLMo R2|10.2|15.5|19.2|20.3|15.2|\n\n*Literature*: Thanks for the suggested controlled generation literature. We will include it in our related work section on controlled text generation, where, in contrast to our method, the control codes are provided by the user.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents an interesting idea that explicitly models the “planning” of the writing then uses it to guide/improve the generation. It automatically derives the planning in an unsupervised fashion using sentence-level embeddings of the training data. It separates the planning module and the language model, and the needed training is relatively lightweight. This idea is relevant to some of the previous work like sentence-level language modeling (Ippolito et al) and FUDGE (Yang and Klein).", "reasons_to_reject": "This paper can be improved as follows: \n\nMethodology: \n1. The reference of “plan” / “action” is based on the sentence embeddings rather than explicit writing plans like “write a sentence about concept X”, “detail the characteristics of the city”. In essence, the planning module is doing sentence level language modeling. I would recommend the authors clarify that in the paper and I would not call the $a$ variables as “actions” but just next sentence embeddings. \n2. Because the author uses clusters of semantic embeddings, the granularity for expressing the fine-grained actions for the next sentence might not be fine-grained enough to guide the next generation. For example, in table 3 and table 6, the authors show examples of the major clusters of the action vectors – it only reveals major semantic/topic differences like business/geograph. That fails to capture the fine-grained control of the details during the writing process – e.g., two consecutive sentences might talk about the same general topic but focus on different aspects/details. Hence, the usefulness of the planning module is actually unclear (which is also reflected in the empirical results).\n3. While it’s good to decouple the planning module from the language models, currently the training of the planning module does not use any signal from the language models, which might make the plans less helpful. Alternatively, one can use sentence representation from the same language model to train the planning module (rather than sentence bert). \n\nExperimental design and results:\n1. Missing baselines: In table 1, the author only compares with a baseline that does not train the language model (the None column). At least the baseline where simply fine-tuning the language model on the same training data using a regular causal LM objective should be compared. \n2. Perplexity might not be the best measurement, and one should compare the actual generation quality (In fact, improving perplexity from 27.89 to 26.69 might not necessarily indicate better generation quality). It would be helpful to directly compare generation quality, e.g., comparing rouge-2 of the generations, with/without using the plan. \n3. Current table 1 evaluation (edit distance) is done at the sentence/chunk level, which might not be a realistic setup. It would be interesting to compare the quality of generating the complete article.", "questions_to_authors": "There are some missing references:\nFUDGE: Controlled Text Generation With Future Discriminators\nFor an in depth analysis of controlled generations, table 1 in Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo could be a helpful pointer.", "extra_scores": {}}, {"review_id": "OipNCfNuoy", "reviewer": "Reviewer_Dmpy", "summary": "This work proposed a hierarchical way of generating text, where a high-level planning action is generated before generating the tokens in a sentence. These actions are induced by clustering sentence embeddings into K clusters, and are used to train two models: an action prediction model which predicts the action of the next sentence from the current sentence using a transformer; and a pretrained language model which is finetuned by injection actions into it. Experiments are conducted on language modeling, and the main evaluation happens at two levels: at a surface level, this method achieves a lower PPL compared to baselines; and a high-level evaluation of actions in a conditional setting, where the actions corresponding to the predicted continuation are closer to the ground truth actions compared to baselines.\n\n===Post Rebuttal Update===\nAfter reading authors' rebuttal, my concerns about surface-level and high-level generation evaluation have been resolved (Weakness 2 and 3). Besides, authors have clarified how the model knows sentence boundaries at generation time as well as the number of introduced parameters (which seems small). Overall, I like the idea of separating high-level planning with low-level realization, and would love to see followup works on pretraining a generally useful high-level writing planner. Therefore, I have bumped up my rating from 6 to 7.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "*Reasons to reject:*\n1. We use a simple unigram classifier for **detecting sentence boundaries**, which classifies a token to be at the end of a sentence if it appears more often at the end of a sentence in our training dataset than not, which reaches 99.3% accuracy and 88% f1-score, which is invoked during generation.\n2. & 3. We now implemented **latent PPL** (lower=better) with abstract writing actions as latent variables. We trained the same HSMM critic as used for the discourse coherence experiment in the paper you provided, which achieves a test latent PPL of 100.8. Furthermore, we provide **MAUVE** scores (higher=better).\n\n| |GPT-2 Ours|GPT-2 Fixed|OLMo Ours|OLMo Fixed|\n|---|---|---|---|---|\n|Latent PPL|204.9|352.3|168.95|249.74|\n|MAUVE|.429|.379|.555|.364|\n\n4. **Parameter count**: The base language model has ~130M (GPT-2) or ~1B (OLMo) parameters. The adapter parameters are about 14M parameters. The planner consists of a sentence transformer with 110M frozen parameters plus a small Transformer encoder with 6M additional parameters.\nHaving a separate planner with its own sentence representations means we can reuse the same planner for different language models. As a memory-efficient alternative one can reuse the sentence representations of the LM, losing the portability. \n\nWe will add all new results and extra information to the final paper.\n\n*Questions to Author:*\n1. We address the **impact of P** init in Section 5.3 “Model Architecture”, where we state that while it was beneficial in our preliminary experiments on little data, it doesn’t help with more training data.\n2. Indeed, since the previous actions are derivable from the previous text, **adding the actions explicitly does not add any new information**. We suspect adding it might help when training on little data, but would not be beneficial with more data.\n3.  Exposure bias is indeed **the reason for training the LM on predicted actions**, as we discuss in Section 5.1 in the paper (“it decreases the train/test mismatch”).\n4. **Color-coding actions** is a nice visualization. We will add it to the final paper.\n5. You are correct that **Oracle/Oracle performance is likely not achievable** since there is no such thing as a perfect planner. Note, however, that the level of leakage of concrete text is small due to the level of abstraction of the actions (millions of sentence embeddings clustered into 1024 actions). We will modify this statement accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is an interesting idea to separate high-level planning with low-level surface realization.\n2. Experiments showed improved results both at low-level realization and high-level planning.", "reasons_to_reject": "1. Some details are not clear. For example, the actions are at a sentence level, but it is not clear from the paper how the model knows when to transition to the next action at inference time: is it by introducing a special end-of-sentence token? Or by using some punctuations like \".\" or \"?\" to mark sentence boundaries?\n2. The high-level action evaluation using the edit distance between the actions of ground truth continuation and generated continuation does not consider that there might be multiple possible ways to continue writing an article (even at the high level). It would be nice to see results using better evaluations, such as comparing the distribution of generated action sequences versus the original action sequences, as in \"Model Criticism for Long-Form Text Generation\" (https://aclanthology.org/2022.emnlp-main.815/).\n3. For surface-level evaluation, this work mainly relies on PPL, but not on metrics that directly evaluate the generations themselves. It would be nice to see evaluation results under open-ended generation metrics such as MAUVE (https://openreview.net/pdf?id=Tqx7nJp7PR).\n4. The method introduces additional parameters (the sentence BERT, and the transformer used in planner) compared to the baseline language model. I think it would be nice if the authors show a parameter count comparison.", "questions_to_authors": "1. According to table 2, isn't no P init better in terms of PPL and edit distance?\n2.  It seems that the action prediction doesn't take into account previous actions but instead only previous text, is the rationale that previous text contains information about previous actions?\n3. What's the intuition that finetuning using oracle actions is worse than using predicted actions? Is it some sort of exposure bias argument?\n4. Can you visualize an example article by color-coding actions in a full article? I think that would be more useful than table 6.\n5. I think this statement \"However, there remain differences of 1.63 PPL and 0.47 PPL with respect to Oracle ( Oracle/Oracle ), indicating that less than half of the potential of integrating the information in the writing action is actuated due to a suboptimal planner\" doesn't consider the fact that the oracle actions might leak information about the text, so the PPL it achieves might not be achievable. For example, if you use a very large action space, then each action might already be able to recover every word in the sentence, and the PPL (at least for training) would be very low.", "extra_scores": {}}, {"review_id": "fnvMwB0ceh", "reviewer": "Reviewer_Btei", "summary": "The paper proposes to improve language models by including a planner that predicts the next writing action, in addition to the tokens. The approach is to first encode all the sentences of the model, then cluster them into writing actions, use a planner to predict the next action and fine-tune the LM to condition on the predicted planning actions.\n\n===\n\nUpdate after rebuttal: \n\nAfter reading all of the other reviews and the authors' responses, I am updating my score from 6 to 7. \nOne thing that could be improved, is phrasing in the beginning of the paper what exactly are actions and what planning them means. At first, it is not very clear and I had the impression that you were talking about edit actions (later it was specified, but it would be good to clarify this in the beginning).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "*Reasons to Reject*:\n1. **Definition of writing actions**: Abstract writing actions are equal to the indices of the clusters obtained in the clustering step, which we already clarify in the third paragraph of the introduction and in Figure 1. Intuitively, they correspond to common writing patterns in the dataset, e.g. origin or profession in a biography. We will add further clarification to the abstract.\n2. **Evaluation**:  We will add a sentence to Section 4.4 and the caption of Table 1 clarifying that Predicted/Fixed/Oracle actions can be evaluated both at training and test time. This evaluation setup allows for valuable insights such as that training the LM with oracle actions rather than predicted actions leads to worse performance due to the exposure bias.\n\n*Questions to Authors*:\n1. **Internal planner**: As we refer in Section 4.4. the details of the internal planner (which corresponds to [1] which we reimplement as a baseline) are described in Appendix B. Intuitively, the language model acts as an internal planner that treats writing actions just like every other token. Those tokens are added to the vocabulary and the LM is finetuned on action-inserted text.\n2. **Model complexity**: The complexity (number of computations) of our proposed sentence-based planner (Section 3.2) is $\\mathcal{O}( \\frac{N^2}{M} )$ for the representation function (computation of $ Z_{i-1} $, assuming caching of sentence embeddings during inference) plus $\\mathcal{O}(M^3)$ for the prediction function (computation of $\\bf{Z’}_{i-1}$), where $N$ is the total number of tokens and $M$ the number of sentences, where we consider a simplified analysis where each sentence has length N / M. Since the representation function (110M parameters vs 6M for the prediction function) represents the vast majority of the complexity, invoking the planner is roughly $M$ times less expensive than the language model (GPT-2: 128M parameters, OLMo: 1B parameters), which has complexity $\\mathcal{O}(N^2)$.\nWe will add a more elaborate discussion to the appendix of the paper. \n\n[1]  Xinyi Wang, Lucas Caccia, Oleksiy Ostapenko, Xingdi Yuan, and Alessandro Sordoni. Guiding Language Model Reasoning with Planning Tokens, December 2023b. URL http://arxiv.org/abs/2310.05707. arXiv:2310.05707 [cs].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed approach of using an action planner is meaningful, and seems to improve the model performance.", "reasons_to_reject": "From the beginning of the paper, it is not very clear what the writing actions are, which makes it a bit hard to understand what exactly is the idea and what the model is trying to learn.\nThe evaluation is a bit hard to follow - particularly the separation of Predicted/Fixed/Oracle.", "questions_to_authors": "- What does \"internal planner\" mean in this context - is this fine-tuning of the model?\n- It could be interesting to discuss the complexity of the model with the added planner - does the model perform much slower, and what is the trade off between memory requirements and the improvement of the language model?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a technique for generating a high level plan of sentence latents, then followed by the text.

The reviewers generally agree on the quality of the modeling and execution, and find the overall idea interesting. The main issues are around the premise that clusters of sentence embeddings represent writing actions, and the evaluation. 

I think both these limitations are important. 
- There isnt convincing evidence that the clusters represent actions around writing. Rather it is likely content modeling previously explored in generation and summarization literature. In addition, using only data from wikipedia is likely to accentuate this behavior as wikipedia has a very regular topic structure.   
- So the evaluation needs to be strengthened with experiments on texts with different types and purpose. Eg. summarization outputs or compare/contrast where writing actions need to be captured in a general way beyond topic. Currently, the perplexity type evaluations only on wikipedia are less convincing. The paper can be strengthened with evaluations on actual tasks/benchmarks with downstream metrics.

[comments from the PCs] Please make sure to follow up on the AC comments and revise as needed.
Decision: Accept