Language modeling via stochastic processes
Authors: Rose E Wang
Esin Durmus
Noah Goodman
Tatsunori Hashimoto
Venue: ICLR 2022 Oral (2022)
Keywords: contrastive learning
language modelling
stochastic processes

Abstract: Modern language models can generate high-quality short texts. However, they often meander or are incoherent when generating longer texts. These issues arise from the next-token-only language modeling objective. To address these issues, we introduce Time Control (TC), a language model that implicitly plans via a latent stochastic process. TC does this by learning a representation which maps the dynamics of how text changes in a document to the dynamics of a stochastic process of interest. Using this representation, the language model can generate text by first implicitly generating a document plan via a stochastic process, and then generating text that is consistent with this latent plan. Compared to domain-specific methods and fine-tuning GPT2 across a variety of text domains, TC improves performance on text infilling and discourse coherence. On long text generation settings, TC preserves the text structure both in terms of ordering (up to +40% better) and text length consistency (up to +17% better).  Human evaluators also prefer TC's output 28.6% more than the baselines.

Published as a conference paper at ICLR 2022
LANGUAGE MODELING VIA STOCHASTIC PROCESSES
Rose E. Wang, Esin Durmus, Noah Goodman, Tatsunori B. Hashimoto Stanford University frewang, edurmus, ngoodman,
ABSTRACT
Modern language models can generate high-quality short texts. However, they often meander or are incoherent when generating longer texts. These issues arise from the next-token-only language modeling objective. To address these issues, we introduce Time Control (TC), a language model that implicitly plans via a la- tent stochastic process. TC does this by learning a representation which maps the dynamics of how text changes in a document to the dynamics of stochastic pro- cess of interest. Using this representation, the language model can generate text by first implicitly generating document plan via a stochastic process, and then gen- erating text that consistent with this latent plan. Compared to domain-specinn methods and fine-tuning GPT2 across variety of text domains, TC improves performance on text infilling and discourse coherence. On long text generation settings, TC preserves the text structure both in terms of ordering (up to +40% better) and text length consistency (up to +17% better). Human evaluators also prefer TC's output 28.6% more than the baselines
INTRODUCTION
Large language models (LLM) such as GPT-2 have been extremely successful text generation [Radford et al. 2019 Brown et al. 2020). However, LLMs are known to generate incoherent long texts. One reason is that they are unable to plan ahead or represent long-range text dynamics (Kiddon)
et al. 2016 Fan et al. 2019 Hua & Wang 2020 Duboue & McKeown, 2001 Steent et al. 2004
Tamkin et al. 2020). As a result, they oftentimes produce wandering content with poor discourse structure and low relevance (Hua & Wang 2020 /hao et al. 2017, Xu et al. (020);; the text reads as if the model has no anchored goal when generating. These problems with coherence are further exacerbated when forcing autoregressive models generate longer texts as the model struggles to extrapolate beyond its expected text end point. These problems suggest that LLMs currently fail to properly capture how documents evolve from beginning to end. Doing so is critical for succeeding in goal-oriented tasks such as story, dialog or recipe generation.
Prior work has explored the use of planning based methods for generating globally coherent text (Kiddon et al. [0166] Fan et al. 2019 Hua & Wang. 2020 Duboue & McKeown, 2001 [Stent et al.) 2004). However, these methods rely on manually defining text dynamics for specific domains. Other work has attempted to use sentence representations for modeling text, such as with variational auto- encoders (Bowman et al. 2016) or contrastive learning (Gao et al. 2021 Devlin et al. 2019). Their shortcoming in text generation settings is that the latent representations are static: they capture semantic similarity between sentence neighbors, but don't 't capture how sentence embeddings evolve document. Methods including/van van den Oord et al.) (2099) have tried to remedy this by learning a model of local latent dynamics. However, it is difficult to use learned local dynamics for generating accurate coal-conditioned trajectories, especially long-horizon ones. We explore an alternative that explicitly assumes a simple, fixed dynamics model with goal-conditioned generation.
In this work, we propose Time Control as way to learn latent space with known, coal-conditioned dynamics. We begin by assuming that meandering text generated without goal can be represented as Brownian motion in latent space; this motion enforces the embeddings of distant sentences to be dissimilar. Goal-directed be- havior can be incorporated into this model by conditioning on a fixed start and end point. In this
neighboring sentences
to be similar to each other, whereas those
'The accompanying code can be found here: http::////gthuu...co//eeeeeeeee
model i ing_viii a agg_via_tttchssttt_____p_ processes

---

Published as a conference paper at ICLR 2022
case, the Brownian motion becomes Brownian bridge and the resulting latent trajectories abide by simple, closed-ford forsed-ormooo dynamics.
In Time Control, we derive novel contrastive objective for learning latent space with Brownian bridge dynamics. We can then use this latent space to generate text that retains local coherence
and has improved global coherence. To perform text generation, Time Control first plans latent trajectory via the Brownian bridge process pinned at a start and end point. It then conditionally generates sentences using this latent plan. In our work, we decode latent plans by fine-tuning GPT2 to generate text conditioned on Time Control's 's latent trajectory. Trajectories from Time Control act as abstract semantic positions in document that guide generation of fine-tuned language models.
In summary, our work's contributions are the following:
We derive Time Control, language model which explicitly models latent structure with Brownian bridge dynamics learned using a novel contrastive objective.
Across a range of text domains, we show that Time Control generates more equally coherent text on tasks including text infilling and forced long text generation, compared to task-specific methods.
We validate that our latent representations capture text dynamics competitively by evaluat- ing discourse coherence with human experiments.
We ablate our method understand the importance of the contrastive objective, enforcing Brownian bridge dynamics, and explicitly modeling latent dynamics.
2 RELATED WORKS
Generating long, coherent text is conceptually difficult for autoregressive models because they lack the ability to model text structure and dynamics (Lin et al. 2021). This means that they struggle to plan and look ahead which leads to generating globally incoherent text. Forcing autoregressive mod- els to generate longer texts exacerbates this incoherence because the models struggle to extrapolate beyond their expected text end point. Prior work has tried address the problem of generating glob- ally coherent text with planning based approaches (Puduppully et al. 2019:; Moryossef et al. [219]] Fan et al.] 2019 Kiddon al. 2016). However, planning- based approaches rely on Nomain-speciine heuristics for capturing text structure and dynamics.
Our work uses a contrastive objective to learn latent dynamics text without domain-specifi heuris- tics. Contrastive objectives have been applied to several domains, including language (Devlin et al.)
2019 [ter al.] 20200] Liu & Liu 2021), vision (Chen et al., 2020)), and general time series data (Hyvainnnn & Morioka 2016 Hyvarinen et al., 2019). In particular for language, contrastive ob-
jectives have been applied to the nexi-sentence prediction task for improving BERT embeddings
(Devlin al. 2019) and to the discourse coherence setting (Nie et al. 2019 Chen et al. 2019b) for evaluating how coherent pairs of sentences are. However, these methods have two shortcomings which address with our work. One is that the resulting sentence embeddings are often static: they capture semantic similarity between sentence neighbors, but don't capture how sentence em- beddings evolve over a document. Two is that they are not used for generation and are limited to classification tasks like discourse coherence. Prior work has also tried fitting latent variable models (Bowman et al., 2016), however these generally result in poor language generation (He et al. 2018) or are domain- specific (Weber et al. 2020 Arora et al. 2016).
Our work closely related to Contrastive Predictive Coding (CPC) from van den Oord al. (0119) The key difference is CPC implicitly learns unconditioned latent dynamics, whereas we impose known goal-conditioned dynamics on our latent space. Doing so, we can extrapolate successfully further in time. Additionally, our method builds off of recent findings that contrastive objectives can be used to approximate local transition kernels stochastic processes (Liu et al. 2021). The main difference between Liu et al. (2021) and our work is that they focus on provable conditions for latent recovery; we focus on empirically effective methods that leverage similar insights for recovering latent representations from language. Finally, use of stochastic processes draws similarities to diffusion models (Song et al. 2020; SShh-DDickttinn et al.) (015)) which apply a chain of diffusion steps onto the data and leam reverse the diffusion process. However, our application
2

---

Published as a conference paper at ICLR 2022
x_ [UBER] Hello, I'd like to buy tickets for tomorrow. [ASSISTANT] what movic theater you prefer? [USER] Could you confirm tickets just in case?
[USER] Bi, I'm looking to purchase tickets for family.
Mt
exp(d Hb))
log exp (d(2, (4) exp(d 1b)
Figure 1: Latent space for a positive triplet sentences (No, 2T) that part of the same conversation. Time Control maps positive triplets to a smooth Brownian bridge trajectory. It embeds 2, close to the expected embedding
pinned by 20, 27. The green oval area illustrates the uncertainty over 2: as a function of how close t is to 0 and T. In contrast, negative random sentence x' from different conversation is not coherent with zo and 27; thus, it is embedded far from pe- This captured by contrastive loss, L.
is conceptually different: diffusion processes characterize properties of our latent space and are not a fixed inference method in our work.
TIME CONTROL
The intuition behind Time Control to learn a latent space with smooth temporal dynamics for modeling and generating coherent text. We detail Time Control in three sections. The first section discusses training the encoder via contrastive learning to map sentences to a Brownian bridge (Revuz & Yor 2013) latent space. The second section discusses training decoder to reconstruct sentences from this latent space. The third section discusses generating text from Time Control.
3.1 TRAINING AN ENCODER WITH BROWNIAN BRIDGE DYNAMICS
Our encoder is a nonlinear mapping from raw input space to latent space, fo X + objective for the encoder is map high-dimensional sequential data into now-dimensional latents which follow a stochastic process of interest in this paper, is the Brownian bridge process. The density of Brownian bridge process between arbitrary start point at and end point ør at /. 7 is,
The
p(= 2 (1-
t(T t) T
2T,
(1)
This density is intuitive to understand: It acts like a noisy linear interpolation between the start and end point of the trajectory, where 2, should be more like zn the start and more like 27 at the end of the trajectory. Uncertainty is highest in the middle region, and low near the end points (rf. Figure
Consider
of triplet observations, (2,, x33. The goal of our work is ensure that fe(x1), f0(22), fe( follow the Brownian bridge transition density in Equation ] We ensure this
using a contrastive objective. Formally, given multiple sequences of data points, X {01 [Zi....,NN.] NN). we draw batches consisting of randomly sampled positive triplets To, Tz, 2r where 0 < t T: Our encoder optimized by,
((r0, Ft, TT)
exp(d(ro, I4, 27; fo))
LN Ex
log
where
Σ
exp(d(x0, we, zr1 fo))
d d(zo. Zt, NT; fe)
1
fo(
(1-4)
+ fo(xx) - L - fo(or)
(3)
202
mean Equation 1
We use indices 0, t, to denote the start, middle and end point of Brownian bridge, but these do not correspond to strictly sampling the first, middle and last sentence of document. No, Te, JT can be any sentence document long z0 comes before 2, and xe before di in the document.
3

---

Published as a conference paper at ICLR 2022
of is the variance in Equation| <(T-)) Note that Equation sums negative middle contrasts, Ze. This objective can be viewed as maximizing the extent to which true triplets from the data follow the Brownian bridge process while minimizing the extent which an alternative mid-point sampled from another sequence does so.
Figure I illustrates how the objective translates into the language setting for training the encoder. The objective samples triplet sentences from a document. Sentences drawn from the same document make up smooth latent trajectory; they should be close to each other and follow the conditional density in latent space. Sentences drawn from different documents should not make up a smooth trajectory and should less likely follow bridge dynamics.
Connection mutual information estimation and triplet classification We draw connections between our contrastive loss and the mutual information estimation setup from van den Oord et al.
(2019) Poole et al. (2119) (as [ + 00) and the classification setup from Liu et al. (2021) ) (B) 1).
Following van den Oord et al. 2019); Poole et al. (2019), this objective can be seen as a lower bound on the mutual information between the two end points and the middle point: J(X,, (Xo, XT)) 2 log(N)) Lx. Hence, by minimizing the contrastive loss, are maximizing the amount of infor- mation between the trajectory and the linear interpolation its end points.
Assuming [8] = 1, we can draw connection to the classification setup studied in Liu et al. (202I). They train a classifier to distinguish in- vs. out-of-order input pairs and show that the Bayes optimal logits for pair-wise classification can be written a function of the stochas- tic process transition kernel. This is equivalent to the our loss on a single triplet i: 4 =
- log expid(=n.21.27 fo)) texp()) a...s...................... fo))" Liu et al. 2022 consider pairs whereas our work con- siders triplets; we show in Appendix A the parrwise and triplet setups equivalent.
3.2 TRAINING A DECODER WITH LATENT PLANS
Here we discuss how to train language model to decode latent plans for generation. We first map all the sentences in the training dataset to our learned latent space using the pretrained encoder fo- This gives us Brownian bridge trajectory of sentence-level latent codes (zu. ... 2t...... 2r) for a document in the dataset. Then, rather than learning decoder from scratch, we fine-tune GPT2 (Rafford et al. 2019) to generate text conditioned on past context and the latent plan.
We fine-tune in the following manner. .... J'W be a document with W tokens and T sentences used to train the decoder. Using the encoder Se. we can obtain embeddings 21 .....2 2T for each sentence. The decoder standard auto-regressive language model that is modified in the following way: at time f, the decoder must predict 24 using all tokens in the past Tet, as well as the sentence embedding žs,, where the index St [T] is map which takes each token to its corresponding sentence. This is a form a reconstruction objective, as the identity of xe is encoded in 2s,.
3.3 GENERATING TEXT WITH LATENT PLANS INFERENCE TIME
Figure illustrates how the trained decoder generates text at inference time. Given two end points 20: 2T, we sample trajectory from latent Brownian bridge, and then generate from the decoder conditioned on this bridge. In many situations, we may not know the endpoints of the Brownian bridge explicitly. In this case, we encode set of sentences corresponding to start and end points (eg. the first and last sentences of our training set), and fit Gaussian to these points to form density estimate. Generating in this case involves first sampling from the Gaussian, and then generating as before from the bridge. More details on training and generation can be found in Appendix B
EXPERIMENTS
We now evaluate the ability of Time Control to capture text dynamics. Specifically, we aim to answer the following research questions (RQ):
Empirically, we found Brownian bridge dynamics easier to recover with triplets rather than pairs of con- trasts. Appendix 1.2 discusses some of the piir- contrast results.
4

---

Published as a conference paper at ICLR 2022
Generate latent plan pinned at in and
TIME CONTROL
TIME CONTROL
Conditionally generate each sentence using the latent plan.
CONTROL
[OBER] HL, I'd 1ike to order tickets for [208]
[ASSTSTANT] Sure, what movie aze these 1057 [MOS]
[UBER] thinking to vatch Iron Han 2
[EOS]
Figure 2: Time Control generates text conditioned on a latent plan. A latent plan is first generated by running Brownian bridge dynamics pinned between sampled start 2n and goal latent variable ar forward. A decoder then conditionally generates from this latent plan sentence-level.
RQ1: Can Time Control model local text dynamics? Section 4.1 investigates this question using a sentence ordering prediction task: given two sentences from the same document, we evaluate whether different models can predict their original order.
RQ2: Can Time Control generate locally coherent text? Section 4.2 the text-infilling task: given prefix and suffix, we evaluate how well different models can fill in between.
investigates this question using
RQ3: Can Time Control model global text dynamics? Section 4.3 investigates this question on text generation for Wikipedia city articles by examining the length of generated sections.
RQ4: Can Time Control generate long coherent documents? Section 4.4 investigates this question on forced long text generation: we evaluate how well models preserve global text statistics (such as typical section orders and lengths) when forced extrapolate during generation.
We run Time Control with different latent dimensions (d 8, 16, 32). Our encoder architecture is a frozen, pretrained GPT2 model from Huggingface (Radford et al., 2019 Wolf et al.) 2020) and a trainable MLP network. We extract GPT2's last layer hidden state that corresponds the end-of- sentence (EOS) token and train the 4-layer MLP on top of the hidden state. The MLP network has intermediate ReLU activations and is trained with stochastic gradient descent with learning rate of le-4 and with momentum 0.9.
Ablations We perform three ablations on our encoder model. Recall that Time Control (A) explic- itly models latent structure with (B) Brownian bridge dynamics using (C) contrastive loss. (A) replaces explicit dynamics with Implicit Dynamics (ID) where future latents are directly predicted with autoregressive model (van den Oord et al., 2019). (B) replaces Brownian bridge dynamics with Brownian motion (BM): latents follow the transition density 24 & ~ [2,,, t-s) s) in Equation Note 2, is centered 2, and is not conditioned with a Variational Auto-Encoder (VAE) and centers the priors over 2g to and 27 to 1, as done in setup. We use GPT2 as the decoder. Appendix D includes more detail on the ablations.
goal end-point. (C) replaces the contrastive loss
Datasets We use language datasets that elicit different kinds of structure, from section structure to discourse structure to narrative structure. Time Control does not take any information about the structure, treating each domain the same under its encoding objective. More information and dataset examples are provided in Appendix E Wikisection (Arnold et al. 2019) includes Wikipedia articles on cities split by sections. We adapt this dataset such that each article contains four ordered sections (abstract, history, geography, demographics) marked with section id tokens: Each article repre- sented as, "[ABSTRACT] text [HISTORY] text [GEOGRAPHY] text [DEMOGRAPHICS) text". Wikihow (WH) (Koupaee Wang 2018) contains how-to articles organized by a title, method, and steps. We mark each with its section id tokens: Each article is represented "[TITLE] text [METHOD] text [STEP] 1 text [STEP] 2 text " Recipe NLG (Bieñ et al., 2020 contains recipes, each with title, ingredients and set of directions. recipe is constructed as "[TTTLE] text
5

---

Published as a conference paper at ICLR 2022
Wikisection
TM-2
TicketTalk
Method
k: = 5
k
==
k 10
1 5
h 10
GPT2
50.3 5.8
50.2 + 6.3
55.7 5.3
63.6
54.7
8.1
BERT
50.9 =
47.8
6..8
8.87
68.4
80.4
ALBERT
49.9 + 12.1
49.6 18.0
8.06
8.3
78.4
89.4 3.1
S-BERT
50.8 = 6.0
4.1
3..
83.3 + 4.3
72.1 5.3
8..2+
Sim-CSE
49.1 6.4
8..
7..4
86.2
75.1 5.9
85.2
VAE (8)
5.5
5.1
5.4
1.0
49.9 1 1.0
51.2011.0
VAE (16)
10.1=588
4.7
4..8
5.9
50.1 1.0
49.5
VAE (32)
501
6.0
4..0
4.9
50.0
49.3
ID (8)
19.8 5.9
5.0
6.2
65.2 1 6.8
59.2 1.9
66.5 1.1
ID (16)
5
55.8
67.7 6.0
60.3
68.4 6.4
ID (32)
50.0 = 5.0
5..1
6.3
67.6 7.1
61.0 1.0
6.5
BM (8)
49.8
50.0
4.48
49.9 5.2
49.7 5.0
50.6 5.8
BM (16)
5.5
50.5 5.2
49.9
51.1 11.16600 6.0
50.3 4.6
50.8 + 5.5
BM (32)
5.6
48.8 : 5.8
49.5 4.7
49.6 1.2 5.2
4
49.1
TC (8)
49.23 - 5.72
6.8 48.3 6.8
77.6 7.8
87.7 1: 6.9
71.6
82.9 1 4.1
TC (16)
57.25 1 5.30
65.8 5.4
78.2 + 8.1
88.0 1 7.1
71.3 + 3.3
82.9 4.1
TC (32)
50.1 - 4.8
49.8 1 5.8
77.9 1 7.9
87.9 1 7.4
72.0 1 3.9
84.4 1 3.9
Table 1: Discourse coherence accuracy measured by the test accuracy of the trained linear classifier, reporting μ&t standard error over runs. Random accuracy is 50%. Values are bolded they are within range the highest mean score and its corresponding standard error. The highest mean score are marked in gray cells. When applicable, the methods are run with varying latent dimensions marked in parentheses (dim).
UNGREDIENTS] text [DIRECTIONS] text". Taskmaster-2 (TM-2) (Byrne et al. 2019) contains conversations on finding restaurants between an assistant and user. The assistant's turn is marked with an "[ASSISTANT]" tag, and the user's 's turn is marked with a "[USER]" tag. Ticket (Byme et al. 2021) contains conversations on booking movie tickets between an assistant and user. The assistant's s and user's turns are similarly marked as in TM-2. ROC Stories (Mostafazadeh et al.) 2016 is short 5-sentence stories dataset. No additional tokens are added in this dataset.
4.1 MODELING LOCAL TEXT DYNAMICS
We evaluate how well Time Control models local text dynamics (RQ1) on the discourse coherence setting (Jurafsky] 2000). Discourse coherence often measured by how well representations capture discourse structure by testing for whether linear classifier can detect in-order and vs. out-of-order sentence pairs (Chen et al., 2019a). We compare Time Control's 's encoder against GPT2's last layer's hidden state corresponding to the EOS token (Radford et al. [2119) BERT (Devlin et al. 2019)
ALBERT (Lan et al. 2019). Sentence BERT (Reimers et al. 2019). and SimCSE (Gao et al. 2021 The latter 4 methods are designed sentence embedding models. We also compare to ablations.
The setup the following: The encoder takes in two sentences It. T+++ to produce their latents 21, Zt+& Rdd At random, the latents are fed either in- or out-of-order. A linear classifier trained on 100 epochs with stochastic gradient descent with a learning rate of le-4 and with momentum 0.9. We varied the sentence distance [1, 5, 10] and found that on some domains and for 1, all methods scored near random accuracy; we have omitted those results in the main paper. Otherwise, the results are summarized Table where we report the mean and standard eтror accuracy on held-out discourse dataset of 3000 examples on 3 runs. Our method is able to outperform compete with sentence embedding specific methods, like ALBERT and SimCSE. Additionally, though GPT2 is used as a base encoder for Time Control, observe that Time Control greatly improves upon GPT2 with significant gains on TM-2, TicketTalk, and Wikisection (around 7 25%).
Neither VAE nor BM perform better than random accuracy any of domains. This suggests that the variational lower bound does not recover the latent embeddings as well as contrastive objectives and the choice of stochastic processes matter for learning informative structural embeddings. ID does best on the conversation datasets, which indicates that implicitly learning text dynamics yields latent representations that be used for inferring discourse though not effectively Time Control.

---

Published as a conference paper at ICLR 2022
Method
MM % (1)
GPT2
17.5 1 0.1
Method
WH
TM-2
TT
SD
10.0 0.1
SS
10.6 1.1
GPT2
10.7
86.8
22.0
Method
BLEU (1)
VAE (8)
10.8 0.1
VAE (8)
10.6
83.4
16.2
LM ILM
1.54 1.00
VAE (16)
9.6
VAE (16)
73.5
35.1
3.03 0.11
VAE (32)
8.7
VAB (32)
15.5
90.2
54.5
VAE (8)
0.17
ID (8)
101
ID (8)
23.1
119.1
111.1
VAE (16)
0.62
ID (16)
154.8
ID (16)
38.1
87.9
55.4
VAE (32)
0.03 -
ID (32)
138.6
ID (32)
30.1
113.3
78.5
(8)
0.3
BM (8)
9.1
BM (8)
18.1 1
52.0
34.9
ID (16)
0.9
BM (16)
17.8
BM (16)
12.7
44.9
75.8
ID(2)
0.1
BM (32)
1.1
BM (32)
15.5
47.9
78.5
TC
3.80 1 0.06
TC
16.8
(8)
9.6
31,1
8.0
TC (16)
4.30 0.02
TC (16)
7.9 0.1
TC (16)
15.0
9.3
5.5
TC (32)
5.4 0.11
TC (32)
B.1
TC (32)
15.8
5.2
12.0
Table 2: BLEU on ground Table 3: Percentage of length Table 4: Section lengths deviating from ex- truth infill and generated mismatch (MM) during genera- pected length in forced long text generation reported (L).
tence.
Noticeably, discourse is more easily inferred on task-oriented conversations like TM-2 and Tick- etTalk. We hypothesize that the ordering of responses matters more conversations than enumera- tive, factual settings like Wikisection. Nonetheless, our model performs above chance on both types of domains. This answers RQI the positive: Time Control can model local text dynamics, like in conversations and articles.
4.2 GENERATING LOCALLY COHERENT TEXT
We evaluate how well Time Control generates locally coherent text (RQ2) on the text-infilling set- ting. Text-infilling requires model to take an incomplete text, with missing sentences, and complete it. An example input could be, "Patty was excited about having her friends over. [blank] Patty had great time with her friends." The challenge in performing text infilling to generate sentence that is locally coherent with the left and right neighboring sentences.
We follow the task setup in Donahue et al. (2020) where they use the ROCStories dataset. Each story in the dataset contains sentences, one which is randomly masked. Time Control fills the masked sentence by rumning the bridge process pinned at the latent for the prefix, 20, and the latent for the suffix, zr. We compare our method against ILM (Donahue et al.] 2020). state-of-the-ar method for text-infilling that only sees the prefix. The VAE ablation takes the average of the prefix and suffix embeddings for generation. When using the average, BM equivalent Time Control we omit the BM results. ID's autoregressive model predicts the next latent given the prefix for generation.
well Donahue al. (2020)'s L.M model, an autoregressive baseline
We evaluate the text coherence with the BLEU score (Papineni al. 2002), ROUGE (Lin 2004). BLEURT (Sellam et al., 2020) and BERTScore (Zhang et al. 2019) between the generated and ground truth infill sentence. Due space constraints, the last three metrics are in the Appendix, Table 17 We also report human evaluations on how coherent the generated sentence is fill-in sentence. Participants were asked to rank the generated fill-in sentence from ILM, LM, and Time Control scale of 1-5 (not reasonable to very reasonable). Appendix H includes more details on the human experiment setup.
The BLEU scores are summarized in Table 2 Time Control generates fill-in sentences that much more closely overlap with the ground truth than ILM and LM. The ablated methods perform worse to varying degrees. On average, VAE performs worse than LM and ID. This suggests that the interpolating embeddings learned via the variational objective yields embeddings which hurt the autoregressive model's ability to generate locally coherent text.

---

Published as a conference paper at ICLR 2022
Method
Human
LM ILM
2.4 0.06
1.077
TC (8)
3.64 0.07
Method
Wikisection
Wikihow
TicketTalk
Recipe
Table 6: Human evaluations
GPT2
50.4 + 1.0
61.5 1 3.5
75.8 + 1.6
36.8 3.7
text infilling. Scores were ranked between and 5. Higher better.
VAE (8)
57.1 + 3.5
66.3 + 1.7
66.1 = 4.1
71.4 0.6
VAE (16)
47.3 = 3.3
60.5 + 2.9
0.8 = 0.0
45.6 + 0.5
VAE (32)
58.3 1: 0.3
60.9 + 5.6
38.9 + 1.7
87.5 1 0.2
ID (8)
34.6
2.9
30.8 11x+10 1.6
68.7 + 2.3
Method
Human
ID (16)
35.7
31.7 0.9
63.9
78.1 18.13.11
ID (32)
2.0
16.9 0.6
45.6 0.6
85.6 + 2.3
GPT2
2.8 +: 0.06
BM (8)
54.7 54.7
56.0 3.5
77.0 4.7
82.7
TC (8)
3.6 + 0.07
BM (16)
61.4
63.8
43.3 2.1
31.9
TC (16)
3.4 0.07
BM (32)
44.3 1.4 1.4
50.2
14.8 = 1.0
8.1
TC (32)
3.3 + 0.07
TC (8)
76.7
81.8 1.0
41.7 1.0
TC (16)
57.9 + 1.0
63.1 + 6.2
88.0+1.
64.1 1.0
Table 7: Human evaluations on tail quality in forced long generation. Scores were ranked between and 5. Higher is better.
TC (32)
36.5
1
2.8
59.1 1
83.6
76.4
Table Ordering in forced long text generation. ROC Stories and TM- 2 omitted because they are not applicable.
Human evaluations in Table 6 indicate that Time Control performs competitively with ILM. Upon inspection of the model output, we find that the ablated models struggle to meaningfully decode from the interpolated latent embeddings (Appendix G.T). Oftentimes, the model would start sentence coherently, but end the sentence by repeating a word over and over again; examples can be found in Appendix G This suggests the importance of learning latent space that supports interpolation by construction. All together, the results provide positive evidence for RQ2: Time Control can generate locally coherent text due to its well-defined latent dynamics.
4.3 MODELING GLOBAL TEXT DYNAMICS
We evaluate how well Time Control models global text dynamics (RQ3) by assessing whether the methods mimic document structure on Wikisection. We check whether the generated section lengths match with the average lengths in the dataset. We focus Wikisection because it is the only dataset with long, well-defined sections (rf. Appendix E on dataset statistics). Each document contains an abstract, history, geography, and demographics section city.
Time Control plans a latent trajectory by running the bridge process hetween the start and end latent, 20 p(z0) and zт p(2T). We compare Time Control to fine-tuned GPT2. We also include two oracle methods that are fine-tuned GPT2 models with additional section-embeddiig supervision. One "sec-dense GPT2" (SD) where each token's 's embedding contains the current section identity; section i's embedding is added onto the token's 's positional token embedding. The other is "sec- sparse GPT2" (SS) where the token embedding contains an indicator that is if the token is the start of a new section, and 0 otherwise. For VAE and ID, we calculate the density estimate p(20) and p(2r) and run a linear interpolation between the start and end latents. For BM, we calculate the density estimate p(2o) and run Brownian motion.
Table reports percentage how much the generated section lengths deviate from the average sec- tion lengths. Time Control best matches the section lengths out of all the methods. One surprising observation was that tne-ttueed GPT2 mismatches the section lengths significantly by almost 20%. Upon further investigation, tions; this causes the section length deviations. Adding domain-specific supervision like with SD and SS slightly alleviates this issue. Surprisingly, we find that our method beats the oracle certain settings, eg. d = 16. ID performs extremely poorly in contrast to VAE and B.:: this highlights the challenge of interpolating on learned dynamics models which is exacerbated in the long text genera-
noticed GPT2 overshoots short sections and undershoots long sec-
'Note that ILM is in effect upper bound model performance as it is trained specifically for text- infilling; TC matching the performance of ILM is strong result for using latent Brownian bridges for locally coherent generation.
8

---

Published as a conference paper at ICLR 2022
Time Control: : USER I'm looking for movie tickets pleasc. ASSISTANT OK, where would you like to see the movie? [ USER ] I'd like to see it Creek's 's End, Oregon please. [...] [ [ASSISTANT Is it OK to go ahead and purchase these tickets? USER Yeah, that would be great. [ ASSISTANT OK. I understand that you are interested in tickets for Emma at AMC Mercado 24 tonight at 7: 7:10pm. Is
that OK? | USER Yes, please do that. ASSISTANT 1 OK
GPT2: [ USER Hi! Tonight, I'd like to go to the movies. ASSISTANT) ] Okay. What theater would you like to go to? [ USER ] Center City. [...] [ ASSISTANT ] That will be all for now. Thank you for all your help. N/A [ USER Bye Bye. I ASSISTANT J N/A ASSISTANT NIA N/A N/A N/A N/A N/À N/A N/A [ USER N/A ASSISTANT I N/A N/A N/A | USER N/A [ ASSISTANTI J N/A N/A
N/A [ USER N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A USER J N/A N/A N/A N/A [...]
Example of forced long text generation on TicketTalk with Time Control vs. fine-tuned GPT2. Both models are forced to extrapolate when generating long texts. They start coherently, but only Time Control extrapolates coherently. For space reasons, some of the text has been removed, marked with [...].
Table 8:
tion settings. The results affirm the importance of Time Control for modeling global text dynamics, such as in matching document structure, thus answering RQ3 positively.
4.4 GENERATING GLOBALLY COHERENT TEXT
We evaluate how well Time Control generates globally coherent text (RQ4) where the EOS token is omitted. We refer to this as the forced long text generation setup because the model must extrapolate beyond its natural end point in generation. Appendix C.2 details how the latent plan is generated. For reference, 1000 tokens is about 50% longer than the average Wikisection document (the longest text domain). This setting is intrinsically difficult for auto-regressive models which do not have the ability to "look ahead" during generation (Lin et al. 2021).
We evaluate model extrapolation on three metrics. First is ordering: how well do the models main- tain the structure of the document (eg. turn-ordering in TicketTalk)? Second is length consistency. Length consistency captures common failure modes such as a model which stops modeling conver- sation between two agents, and instead outputs a long monologue from one agent. Third is human evaluations, which measures the long-tail text quality. We examine the long-tail behavior as the complete 1000-token generation is long.
The results are summarized Table for length consistency, human evaluations. Overall, our method maintains text flow the best according to these metrics. common failure mode of GPT2 that it produces nonsensical text where it would naturally end generation; this is particularly noticeable on TM-2 and TickerTalk, Figure X shows an example of our model's behavior versus GPT2's s behavior on generating long texts in TicketTalk. The example illustrates how Time Control continues the dialog whereas GPT2 utters nonsensical text. These sults are additionally confirmed by our human evaluation experiments: human evaluators rank Time Control's 's extrapolation ability better all latent dimension settings than that of GPT2 (Table
for ordering and Table 7 for
ID performs poorly in forced long text generation, similar to the normal text generation setting: It significantly misorders sections and overshoots section lengths. Although VAE and BM do poorly on length consistency, they perform slightly better than ID on ordering; in Appendix[] we investigate why the VAE baseline performs much better than TC on ordering. This suggests the importance of good latent dynamics for long text generation. These results answer RQ4 positively: Time Control generates globally coherent text thanks to its ability to plan ahead and correctly generate future latent variables following its latent dynamics.
5 CONCLUSION
We propose Time Control, a language model that implicitly plans via a latent stochastic process. Specifically. Time Control looks Brownian bridge processes as desirable latent space. Time Con- trol learns to map coherent text to smooth Brownian bridge trajectories. Empirically, we demonstrate this leads to several benefits such as generating more locally and globally coherent text. Although work focuses on benefits of learning stochastic process latents for language, it can be extended to other domains with sequential data like videos or audio, or extended to handle arbitrary bridge processes without known fixed start and end points.

---

Published as a conference paper at ICLR 2022
6 REPRODUCIBILITY STATEMENT
In the supplemental, we include zip file containing our code and processed datasets. We've also in- cluded in the appendix how we processed the datasets. Our results on showing equivalence between the triplet classification setting and pairwise classification setting are also included in the appendix.
ACKNOWLEDGMENTS
REW is supported by the National Science Foundation Graduate Research Fellowship. The authors would give special thanks CoColab members Mike Wu, Gabriel Poesia, Ali Malik and Alex Tamkin for their support and incredibly helpful discussions. The authors would like to thank the rest of Co and Rishi Bommasani for reviewing the paper draft. The authors also thank Chris Donahue for helpful pointers on using II.M.
REFERENCES
Sebastian Arnold, Rudolf Schneider, Philippe Cudré-Mauroux, Felix A. Gers, and Alexander Löser. Sector: A neural model for coherent topic segmentation and classification. Transactions of the Association for Computational Linguistics, 7:169-184, 2019, doi: 10. 10.1162hacll,ay -00611.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385-399, 2016.
Michal Bicń, Michat Gilski, Martyna Maciejewska, Wojeiech Taisner, Dawid Wisniewski, and Ag- nieszka Lawrynowicz. RecipeNLG: A cooking recipes dataset for semi-structured text gener- ation. Proceedings of the 13th International Conference on Natural Language Generation, pp. 22-28, Dublin, Ireland, December 2020. Association for Computational Linguistics. URL https: /wwww//wwww//.////////////.../// ac wwwwlwebb.o 2 in1.00nnngg11..
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference Computational Natural Language Learning, pp. 10-21, 2016
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arxiv:2005 14 465,, 2020.
Bill Byrne Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik. Taskmaster- 1: Toward realistic and diverse dialog dataset. In Proceedings of the 2019 Conference Em- pirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4516 2019.
Bill Byrne, Karthik Krishnamoorthi, Saravanan Ganesh, and Mihir Kale. TicketTalk: Toward human-level performance with end-to-end, transaction-basel dialog systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In- ternational Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 671 671-680, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.acl-long.55. URL
acl ac1.
Mingda Chen, Zewei Chu, and Kevin Gimpel. Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations. In Proceedings of the 2019 Conference Empirical
Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 149-662, Hong Kong, China, November 2019a.
Association for Computational Linguistics. doi: 10. 10.18553/v1//19. 10600.9--10.....00 TtRLhttps::///
www. aeabbergg/nnin htrgg/antho.oo hottchologyyDDo ogy 1060
Mingda Chen, Zewei Chu, and Kevin Gimpel. Evaluation benchmarks and learning criteria for discourse-aware sentence representations. In Proc. of EMNLP, 2019b.
10

---

Published as a conference paper at ICLR 2022
Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs], May 2019. URL htRth/tp://axixi
abs 1810. 04805 arXiv: 1810.04805.
Chris Donahue, Mina Lee, and Percy Liang. Enabling language models to fill in the blanks, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Pablo A. Duboue and Kathleen R. McKeown. Empirically estimating order constraints for content planning in generation. In Proceedings of the 39th Annual Meeting Association for Com- putational Linguistics. ACL '01, pp. 172-179, USA, 2001. Association for Computational Lin- guistics. doi:
URL https:
1073035
Angela Fan, Mike Lewis, and Yann Dauphin. Strategies for Structuring Story Generation. 10Xiv:1902.01109 [cs], June 2019. URL Ottp:pp./aara...ccrr///////22/.2 arXiv: 1902.01109.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2 104.081,,,, 2021.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. Lagging inference net- works and posterior collapse in variational autoencoders. In International Conference on Learning Representations, 2018
Xinyu Hua and Lu Wang. Pair: Planning and iterative refinement in pre-trained transformers for long text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 781-793, 2020.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised Feature Extraction by Cime-Contrattive Learning and Nonlinear ICA. arXiv:1605. 105.03660 [cs, stat], May 2016. URL (Rttpp////aax
org/abs/166l 605. 06336
arXiv: 1605.06336.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turne. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky. Pretraining with Contrastive Sentence Objectives Improves Discourse Performance Language Models. arXiv:2005. 10389 [cs], May 2020. /tRL:t/pp////:......
abs /2005.20000mm 10389
arXiv: 2005. 10389.
Dan Jurafsky. Speech & language processing. Pearson Education India, 2000.
Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi. Globally coherent text generation with neural checklist models. In Proceedings of the 2016 conference empirical methods in natural lan- guage processing, pp. 329-339, 2016.
Mahnaz. Koupaee and William Yang Wang. Wikihow: A large seale text summarization dataset. arXiv preprint arXiv:1810.0955 2018.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1942, 2019.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona. Spain. July 2004, Association for Computational Linguis- tics. URL http::ttt//.... //acl::::/clcccc ant
hoto www.hoooy..oo...........oo -

---

Published as a conference paper at ICLR 2022
Chu-Cheng Cheng Lin, Aaron Jacch, Xin Li, Matthew R. Gormley, and Jason Eisner. Limitations of autore- gressive models and their alternatives. In Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5147-1475 Online, June 2021. Association for Computational I Linguistics. doi: 10. 18 URL ht.tttttt://eet. htt.tppss/acclaattooooooooo 0rgg/020000//2211 naacl 405
Bingbin Liu, Pradeep Ravikumar, and Andrej Risteski. Contrastive learning of strong-mixing continuous-time stochastic processes. arXiv:2103.0222 [cs, stat], March 2021. |RLhttp:
/
2210/ 02740
arXiv: 2103.02740.
Yixin Liu and Pengfei Liu. SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization. Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing (Volume
2: Short Papers), pp. 1065-1072, Online, August 2021. Association for Computational Linguis- tics. doi: 10. 1/1/6665//112222.aaaahhhoooa sel-short.111
URL https: htttp:://aclanttlooooo.or////2//
Amit Moryossef, Yoav Goldberg, and Ido Dagan. Step-by-step: Separating planning from real- ization in neural data-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pp. 2267 2267-2277, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 11863//1//119-2233 URL https:
7/ac Sttpchhoooooo.rg/M133
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper under- standing of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 839-88 439-849, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/N166 1098. URL
aachhannhhhooooooo caanthology.cooo wgg.ooog.ol................. 6018
Allen Nie, Erin Bennett, and Noah Goodman. DisSent: Learning Sentence Representations from Ex- plicit Discourse Relations. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. 1497-4510, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P199 1442. URL https: www acwweb. org. wwt.:/thooogaaggoo.oooooooooooo.
P19- P19-1442)
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171- 5180. PMLR, 2019.
Ratish Puduppully, Li Dong, and Mirella Lapata. Data-to-text generation with content selection and planning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 6908 6915, 2019.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Nils Reimers, Iryna Gurevych, Nils Reimers, Iryna Gurevych, Nandan Thakur, Nils Reimers, Jo- hannes Daxenherger, Iryna Gurevych, Nils Reimers, Iryna Gurevych, et al. Sentence-bert: Sen- tence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empir- ical Methods Natural Language Processing. Association for Computational Linguistics, 2019.
Daniel Revuz and Mare Yor. Continuous martingales and Brownian motion, volume 293, Springer Science & Business Media, 2013.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. generation, 2020.
Bleurt: Learning robust metrics for text
12

---

Published as a conference paper at ICLR 2022
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2265, PMLR, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2020.
Amanda Stent, Rashmi Prasad, and Marilyn Walker. Trainable sentence planning for complex in- formation presentation in spoken dialog systems. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguisties, ACL '04, pp. 79- es, USA, 2004, Association for Computational Linguistics. doi: 10.3115/11189551211888 URL
3115/1218955. 1211199955.1129966 1218966
Alex Tamkin, Dan Jurafsky, and Noah Goodman. Language through prism: A spectral approach for multiscale language representations. Advances in Neural Information Processing Systems, 33: 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019.
Noah Weber, Leena Shekhar, Heeyoung Kwon, Niranjan Balasubramanian, and Nathanael Cham- bers. Generating narrative text in switching dynamical system. In Proceedings of the 24th Conference on Computational Natural Language Learning, pp. 520-530, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL https: wwww. aclweb. org/ org/anthology. 2020. emn 1e-deos.....
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. Discourse-aware neural extractive text summarization. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 5021-5031, Online, July 2020. Association for Computational Linguis- URL /ttttt/////ccaaaooooooo.../
tics. doi: 10.186534 0t//3//1/2220.ee a/2000.acl-main.444. acl 1ammmain.444 451
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. In International Conference Learning Representations, 2019.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 654-664, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/ P17-1061. URL httttp///aclaathoooyy..c///////
A SHOWING THE EQUIVALENCE OF THE PAIRWISE AND TRIPLET CLASSIFICATION TASK
In this section, we focus on the classification setup where the goal is to train a classifier to detect in- want to show that the classification setup from
out-of-order groups of inputs. Specifically, Liu et al. (202T) where the inputs come in pairs is equivalent to the setup where the inputs come in triplets. The triplet inputs are what we assume for Time Control.
13

---

Published as a conference paper at ICLR 2022
Notation We denote latent space 2; typically do not observe this space directly. We denote the observation space as X:; this is the space we observe directly.
We define the Brownian bridge for E [0, I]
B, B(t) = ((1) _ -
(4)
where (t) is a standard Wiener process,N
We state the assumptions:
We assume latents (2:)20 IRd are drawn from the Brownian bridge process defined by the stochastic differential equation,
2T 21 dt + dB.
da
(5)
T
The intervals at which latents are sampled are every At step: z' 21+
We denote the transition probabilities as
pa (= 2T) = V (
T - (1- T 2T, UT (T-11)
We denote the proposal distribution over possible intermediate triplets q( = 2/7).
Liu et al. (2021) characterize properties of an optimal classifier 1"(zz, z") which observes pairs of latents (z, 2") and outputs in [0, 1], probability indicating whether the pair comes in order (ie.
= + At- d2ạ) or not in order (ie. a randomly sampled latent). They train h" using an L2 loss,
C(h, (2,22)).
Lemma of their work states the following : The optimum of the contrastive learning objective arg min, ((()))))))) [Ch,, (C(h, {(z, z'), satisfies
(2,2)) h" (2,22)) g(21) + (2,
Manipulating this equality, we observe that the transition kernel has the following relation to the classifier which takes in pairs of observations,
q(='))
(2,
(8)
1 (2,222))
log (2, log q(>') log hggh"(a (2, log(1 (00(11122,,2))))) 22)
Our setting however assumes that the algorithm receives triplets of data points, 2t, Ptg Stg. We want to show below that minimizing C with triplet contrasts in the classification setting still approximates the transition kernel. In particular, we we're interested in transitions 20, Zr: At (F0s 24, 2r) Pr(z.] 2o. 2r).
Brownian bridge pinned at
Let's say have positive triplet samples, (2) =6,, =f, and (=/, =/s 2 where t < <t2 < la- Following Liu et al. (202), minimizing L yields the following on each triplet:
logp (2) 2g 24,) logq(z') log hi (2f, 2t,))t) log(1 h" (2), 2 242))
(10)
log p (2) bby 24,) = log q(=') ) + log (2), 2 z1-)) -log(1 h" (2/, 2 2422))
(11)
Taking the difference in log probabilities between Equations 10 results in
lo (=t 2/ 211) - loog Al 24 z() = [log h (=t, = z(0) log(1 (= 2/, =1)))]
h (2/, 2/,, zf,)) - log(1 - h² (2/, 2 2
(12)
Similar to the wiir-wise classification setting, we' ve shown that minimizing L in the triplet classifi- cation setting results in approximating the transition kernel of the Brownian bridge process.
14

---

Published as a conference paper at ICLR 2022
B TRAINING DETAILS
Here we describe the training procedure for the encoder and the fine-tuning procedure for decoding.
frozen, pretrained GPT2 model from Huggingface (Radford al. 2019) Wolf et al. 2020) and trainable MLP network. We extract the GPT2's last layer hidden state that corresponds to the end-of-sentence (EOS) token and train the 4-layer MLP top of the hidden state. The MLP network has intermediate ReLU activations and is trained with stochastic gradient descent with learning rate of le-4 and with momentum 0.9. We train the encoder for 100 epochs on each of the datasets.
Brownian bridge encoder The encoder architecture
The text fed into GPT2 are fed in on a sentence level. This means that the input 2e refers to the t'th sentence of document. The sentences are separated from each other in the main text as " which is added to the tokenizer as a separate token for indexing convenience.
Fine-tuning GPT2 with latent embeddings After training the encoder, we run it on the training dataset collect an accompanying latent trajectory for each text. The encoder run on the dataset at sentence level: we separate the text by sentences and pass the sentences through the encoder.
The sentence latent embeddings are aligned with the tokens of that sentence and offset by one token before the start of that sentence token. Let's illustrate by an example. We denote [SOS] as the start of the document token, [sl] as sentence I tokens and [s2] as sentence 2 tokens. [. ] is the period token which we've added into the tokenizer. denote the latent variable corresponding to the i'th sentence.
Let's say that the sequence fed into GPT2 is "[SOS] [s]] [s1] [s1] [. [s2] [s2] [s2]". Then the corresponding latent trajectory is "21, 21, ø1, 21, 22, 22, 82, 22". The latent variables are added onto the positional embeddings. We then fine-tune GPT2 normal.
C GENERATION WITH EMBEDDINGS
C.1 NORMAL TEXT GENERATION
We first sample start and end latent, 20 p(za), 29 p(2r) where p(zo). as the density estimates over the training dataset. We pin our trajectory to the start and end latent, and run the Brownian bridge using Equation ??. For normal long text generation, set T be the average number of sentences in each of the dataset. For forced long text generation, we set 7 to be proportional the number of sentences needed in order to generate 1000 tokens. By the end, have a trajectory Ø1 ....
calculated
Generation starts with feeding the SOS token and the first latent zg. Once GPT2 emits [. token and terminates sentence t, we transition to the next latent 2++1- This process continues until GPT2 is finished with generation. If GPT2 generates more sentences than there are latents the trajectory, the last latent 27 is used until the end of generation.
C.2 FORCED LONG TEXT GENERATION
Let Sagg(D) denote the average number of sentences in a document and T Tavg(D) denote the average number of tokens document. Rather than planning trajectory of length Savg(D) (the average number of sentences in document) which is what is done in Section 4.3 for normal text generation, we scale the trajectory length to c Savg(D)- e determined by how many more tokens we need in order to fill up to GPT-2 maximum context length of 1024: c 1021-Tap(D)
Tag(2)
D ABLATIONS
The following methods ablate the encoder model in Time Control; in other words, the ablations dissect the assumptions we make in the first step of Time Control described in Section [3.] Recall that Time Control (A) explicitly models latent structure with (B) Brownian bridge dynamics using
15

---

Published as a conference paper at ICLR 2022
(C) contrastive loss. (A) replaces explicit dynamics with Implicit Dynamics (ID) where future latents are directly predicted with an autoregressive model (van den Oord et al. 2019). (B) replaces Brownian bridge dynamics with Brownian motion (BM) which doesn' rely pinning trajectories. Latents follow the transition density z 2 ~ (2,,, s). (C) replaces the contrastive loss with the
Variational Auto-Encoder (VAE). Below detail how these ablations are implemented.
D.1 IMPLICIT DYNAMICS (ID)
The Implicit Dynamics ablation is where we compare our explicit dynamics objective to van den Oord et al. (201)) which suggests an implicit latent dynamics objective. This ablation thus changes the learned latent dynamics. In den Oord al. 2019), they train two models. One is non-linear encoder Gene (x)) = 2ς which takes observation Ta (eg. a sentence) and maps it to a latent represen- C₁ which summarizes sequence of latent variables 2ge into a context latent representation сt. Rather than directly predicting a future observation xz++& (k steps from the current timestep they model the density ratio that preserves the mutual information between I++ and the context variable ez:
tation 24- Two is an autoregressive context model 9ar (2ge)
fa(x44)) x p(ax+kl())
(13)
p(xe+k)
They model this with log-bilinear model f*(#++*) exp(= Wecz) which applies a linear transformation W₁ for modelling latents k-steps away from timestep This way, they avoid directly learning the generative model p(xr+*/()).
They train both models jointly via a contrastive InfoNCE loss. Given a set X = { EN } of N random samples (eg. sentences from different documents) containing one positive sample from p(z++a/c)) and N negative samples from proposal distribution p(I++*), they optimize:
L -Ex log falx)tk, Exjex Sk((
(14)
The encoder Jan for Implicit Dynamics has the same architecture as Time Control. The context encoder for using a 2400-hidden-uml GRU, as done in van den Oord et al. (2019). We then train both encoders using the InfoNCE loss, done in prior work. Since Jene and far are trained to align latents up to linear rotation, we use Jene for extracting the sentence embeddings.
D.2 BROWNIAN MOTION (BM)
The Brownian motion ablation is where remove goal-conditioning in the dynamics. The main change is in distance function in Equation 3, BM instead optimizes the following equation:
d(It, Le' fo)
fo(xe)
fo (x
(15)
20
where 2² is the variance in of the Wiener process, 0² = t' t.
D.3 VARIATIONAL AUTO-ENCODER (VAE)
The Variational Auto-Encoder ablation is where we compare our contrastive objective with a vari- ational one. This ablation changes the encoder objective from Section 3.1 from Equation 2 to the ELBO objective. Below we derive the ELBO objective. Similar to the contrastive objective, we' 'r deriving the BLBO over the triplet dynamics in the Brownian bridge.
16

---

Published as a conference paper at ICLR 2022
logp(x 2 R (oo(( Dlo p(x, z)
(zz) x)
plro, 2i, 2T; 20, 24, 2T
(x) flog
g 20, Rt; 2T [00 Tt, VT
E (x/xx) llog p 20 }p(1 ( (p 2T) 27
(T) p( (((2))((TT)
(2 2o: 2r. 34 ) Z0 (2) (T)
p p(21
E (x) lo (2) [T) Eg (m/xx) [log
(z 20 24)
E (2)x) log p(20)
p(2T)
Egg (m/x) (x/x)
llog
q(20|
(22)(r)))
= (xtb)) [log pl 2o 2 2T)]
DeL 21 20, 27, xz p (2 2T)) - DR. (r( -
We assume the priors over 2g O-centered and 2r is I I-centered, which is similar to our Brownian Bridge setup. The encoder (2|2z) )) is parameterized with the same architecture encoder. The decoder p(z|2) is fine-tuned GPT2 model.
E DATASET INFORMATION
For each dataset, text examples were filtered out if they did not fit within GPT2's context length of 1024 tokens. We also added token " each setting to mark the end of sentence. This was done for indexing purposes, cg. when aligning the latent embeddings.
Wikisection (Arnold et al. 2019) includes Wikipedia articles on cities split by sections. We adapt this dataset such that each article contains four ordered sections (abstract, history, geography, demo- graphics) marked with section id tokens: Each article is represented as, "[ABSTRACT] text [HIS- TORY] text [GEOGRAPHY] text [DEMOGRAPHICS] text". These section id tokens are added to the tokenizer.
The training dataset contains 1420 articles. The section lengths have the following breakdown mea- sured in the number of BPE tokens (GPT2 tokenizer):
Abstract: 75.8 + 1.4
History: 191.5 + 3.7
Geography: 83.9 + 1.5
Demographics: 342.6 + 4.6
The test dataset contains 431 articles. The section lengths have a similar breakdown:
Abstract: 73.5
History: 180.2 + 6.2
Geography: 85.2 2.7
Demographics: 332.5 + 8.6
The ordering metric used Table if all four section ids occur exactly once and in the order as they are listed above.
The length mismatch in % used in Table calculated with respect to the training lengths.
Wikihow KKupacee & Wang 2018) contains how-to articles organized by title, method, and steps. Each article includes multiple methods for completing a multi-step procedural task such as "How Register Vote". We scraped all the available English articles covering wide range of
17

---

Published as a conference paper at ICLR 2022
topics following Koupace & Wang 20188. We mark each with its section id tokens: Each article is represented as "[TITLE] text [METHOD] text [STEP] text [STEP] 2 text
The training dataset contains 1566 articles. The section lengths are,
Title: 10.4 + 2.2
Method: 8.7 2.2
Steps (total step length): 480.2 + 231.5
The test dataset contains 243 articles. The section lengths are,
Title: 10.7 + 2.2
Method: 8.6 +: 2.1
Steps (total step length): 480.1 + 224.0
The ordering metric used in Table 5 is 1 if all the section ids appear in order, the TITLE and METHOD section ids not repeated, and the step numbers come in order. It's 0 otherwise.
The deviation in section length measured in Table is calculated with respect to the training set lengths. We check for whether the models are able to maintain the section lengths when has to extrapolate. The most common failure mode is that the model generates incoherent text and allocates this text the last section of what it's generated thus far, resulting in the deviation section lengths.
Recipe NLG (Ben et al. 2020) contains recipes, each with title, ingredients and of instruc- tions. A recipe is constructed as "[TITLE] text [INGREDIENTS] text [DIRECTIONS] text".
The training dataset contains 4000 recipes. The section lengths are,
Title: 9.7
Ingredients: 23.8 + 4.5
Directions (total step length): 62.0 14.5
The dataset contains 1000 recipes. The section lengths are,
Title: 9.4
Ingredients: 24.1 + 4.5
Directions (total step length): 63.3 13.7
The ordering metric used in Table is 1 if all the section ids appear exactly once and in order. It's 0 otherwise.
Taskmaster-2 (TM-2) (Byrne et al., 2019) which contains conversations on finding restaurants between assistant and user. The assistant's turn is marked with "[ASSISTANT]" tag. and the user's turn is marked with a "[USER]" tag.
The training dataset contains 2000 conversations. The section lengths are,
User: 11.8 + 5.6
Assistant: 18.0 + 3.7
The test dataset contains 1276 conversations. The section lengths are,
User: 11.7 + 5.6
Assistant: 19.5 + 3.0
The ordering metric used Table 5 does not apply here because the user and assistant don't take turns in the dialog.
18

---

Published as a conference paper at ICLR 2022
Method
Wikisection
Wikihow
TM-2
TicketTalk
Recipe
GPT2
5.9
15.3
4.5
4.4
sec-dense
5.9
sec-sparse
5.9
VAE (8)
5.5
15.3
4.5
4.1
7-3
VAE (16)
5.5
15.3
4.5
4.1
7.3
VAE (32)
5.5
15.3
4.5
4.1
7.3
ID (8)
5.5
14.9
4.5
4.0
7.3
ID (16)
5.4
14.9
4.3
3.9
ID (32)
5.3
14.9
4.2
3.9
6.7
BM (8)
5.5
15.2
4.5
4.1
7.3
BM (16)
5.5
15.3
4.5
4.1
7.3
BM (32)
5.5
15.2
4.5
4.1
7.3
TC (8)
5.5
15.2
4.3
4.0
6.9
TC (16)
5.5
15.3
4.3
4.0
6.9
TC (32)
5.5
15.2
4.3
4.0
Table 9: Perplexity after fine-tuning.
The deviation in section length measured in Table calculated with respect the training set lengths. We check for whether the models are able to maintain the utterance lengths between the user and assistant when it has extrapolate. The most common failure mode is that the model generates incoherent text and allocates this text to the last section of what it's generated thus far, resulting in the deviation in section lengths.
TacketTalk (yyne al. 2021) which contains conversations booking movie tickets between an assistant and a user. The assistant's 's and user's turned are similarly marked as in TM-2.
The training dataset contains 2000 conversations. The section lengths are,
User: 11.8 5.6
Assistant: 18.0 + 3.7
The test dataset contains 1276 conversations. The section lengths are,
User: 11.7 = 5.6
Assistant: 19.5 3.0
The deviation in section length measured Table 4 calculated similarly to TM-2.
The ordering metric used in Table 5 applics because the user and assistant take turns in the dialog. The ordering metric is the user and assistant take turns in the conversation. It's otherwise.
2016 is short stories dataset. Each story contains 5 sen- tences. No additional tokens are added in this dataset. The training dataset contains 2000 stories, and the test dataset contains 1000 stories.
ROC Stories (Mostafazadch
F PERPLEXITY AFTER FINE-TUNING
Table reports the final perplexity scores after fine-tuning GPT2 on the different domains with the methods. We fine-tune for 10 epochs and checkpoint the models every 1000 steps; we keep the model checkpoint that scores the lowest PPL. held-out validation set.
19

---

Published as a conference paper at ICLR 2022
G GENERATION EXAMPLES
We provide example outputs from the different models on text infilling. We denote the prefix sen- tences "LEFT CONTEXT", the suffix sentences as "RIGHT SENTENCES", the generated output from the models as "GENERATED" and the ground truth infill sentence as "GT".
G.1 TEXT INFILLING
LM
See Table 10
Table 10: Example of text infilling with Donahue et 2020 LM model on ROCStories.
LEFT CONTEXT:
John couldn't wash his clothes because his washing machine broken. GENERATED: had idea.
GT: John didn't know how to fix washing machines.
RIGHT CONTEXT: He bought a book to tell him how to repair it. Hc changed the fan bolt. When he plugged it back in, worked.
LEFT CONTEXT: found a stray cat yesterday. gave some milk. It came back again today. was going to give it some milk but didn't. GENERATED: guess need get something.
GT: had run milk.
RIGHT CONTEXT:
Toni took trip to the local fair with her family. She immediately laid eyes preuty balloon. The balloon was red star shaped. Unfortunately slipped out of her hands. GENERATED: Her swollen fingers had horrible rashes. GT: She was sad to see it go.
LEFT CONTEXT:
RIGHT CONTEXT:
LEFT CONTEXT:
Miles was a roofer. He loved his joh and was very good at it. GENERATED: He dreamed of finding good title.
GT: However, day he was paying attention and fell off the roof.
RIGHT CONTEXT: Luckily, only broke his leg. Miles decided that needed be more careful.
LEFT CONTEXT: Lindsey feeling especially antsy school. GENERATED: The school
paying much attention her.
GT: She not want to go to Lacrosse practice after school.
RIGHT CONTEXT: She tried to fake sickness, but nobody believed her. Lindsey ended up skipping practice without an excuse. She hoped she wouldn't I get in trouble.
LEFT CONTEXT:
Last night Jack went on blind date. When he arrived at the restaurant a he heard someone yell his name. He spotted blonde waving from across the room. GENERATED: She
waiter.
GT: She was nothing like the photo she sent.
RIGHT CONTEXT: But he figured would try to make the most
LEFT CONTEXT:
Rico working really hard his essay for English. After working it all night, decided take short nap. slept in and was late for school. When he got school he noticed that his essay was gone.
GENERATED: teacher found the essay in his bag and he passed it. GT: Rico's teacher let him turn in the essay the next day.
RIGHT CONTEXT:
ILM
See Table
20

---

Published as a conference paper at ICLR 2022
Table 11: Example text infilling with Donahue (2020) 's ILM model on ROCStories.
LEFT CONTEXT:
GENERATED: My 98 year old friend and played blackjack yesterday. GT: Last week's family game night was intense.
RIGHT CONTEXT: We were playing Monopoly and nobody made any headway for hours. Everyone was trying their hardest win and the game kept going. It wasn't until we finally decided to check the rules that knew why. There
many different pieces missing.
LEFT CONTEXT:
Tom was jealous of his brother. His brother was older and stronger. His brother went wherever he wanted.
steroids. GT: One day his brother was grounded for staying out too late.
GENERATED: Tom decided
RIGHT CONTEXT: This made Tom really happy.
His first time the opera. only went after girlfriend begged. He sat for three hours in sheer boredom. Finally it was over. GENERATED: He turned on the tv and heard her thanking him. GT: He learned he didn't t like the opera.
LEFT CONTEXT:
RIGHT CONTEXT:
My best friend played video game online. One day. she met nice boy there. They talked every day and fell in love. They finally met in person and hit it off. GENERATED: Now, they are best friends.
LEFT CONTEXT:
GT: The two became a very loving couple.
RIGHT CONTEXT:
LEFT CONTEXT:
The family decided to go to park Veteran's 's Day. GENERATED: They asked permission from the park manager.
GT: We had a barbecue outside.
RIGHT CONTEXT: In addition, everyone decided to take swim in the water. Before left, we had a huge cake. then swam for an hour at lake.
LEFT CONTEXT:
Jennifer always wanted to have the birthday of her dreams. Her parents surprised her with hig birthday party the park. All of her friends and family came celebrate her special day. was so happy as she blew out the candles on cake. GENERATED: Her parents had made a big surprise to her by turning the cake into a carnival. GT: It was the best day of birthday she could have imagined.
RIGHT CONTEXT:
LEFT CONTEXT:
Terry's daughter was in extreme pain in her mouth. GENERATED: Terry and her husband decided stop the baby from teething. GT: After evaluation. Terry realized that her daughter
teething.
RIGHT CONTEXT: Once Terry realized was teething. Terry administered medication. After the medication, Terry placed teething gel on the baby's 's teeth. Terry got the baby to calm down.
Time Control
See Table 12
21

---

Published as a conference paper at ICLR 2022
Table 12: Example of text infilling with Time Control (d=8) on ROCStories.
LEFT CONTEXT:My day on Sunday this week went as expected. My family and attended Church from to 11 in the morning.
GENERATED: We went to local church in the afternoon
GT: Afterwards, went eat dinner restaurant.
RIGHT CONTEXT: The rest my day was spent doing homework assignments. I then went sleep feeling happy about my life.
LEFT CONTEXT:
His first time at the opera. He only went after his girlfriend beggod. Hc sat for three hours in sheer boredom. Finally it was over.
GENERATED: He was happy he didn't I want to leave.
GT: He learned he didn't like the opera.
RIGHT CONTEXT:
My best friend played video game online. One day, she met nice boy there. They talked every day and fell in love. They finally met in person and hit it off. GENERATED: Until that day, they were married.
LEFT CONTEXT:
GT: The two became a very loving couple.
RIGHT CONTEXT:
The other day at the clinic had to help put a dog down. He seemed really sad and lonely. Like he knew what was going to happen. GENERATED: He was going die soon GT: As laid down and it took its final breaths stared RIGHT CONTEXT: stayed calm, cried after we were finished.
LEFT CONTEXT:
LEFT CONTEXT:
Tom was jealous his brother. His brother was older and stronger. His brother went wherever wanted.
GENERATED: Tom was jealous of his brother. GT: One day his brother grounded for staying out too late.
RIGHT CONTEXT:
LEFT CONTEXT:
Jays habit of buying expensive beer was catching up to him. He was spending more money beer than food. He needed to find another source of income support this habit. A friend recommended he try out Amazon MTurk. GENERATED: found the site and bought a heers.
GT: Jay become slightly less poor from Amazon Mturk.
RIGHT CONTEXT:
LEFT CONTEXT: John couldn't it wash his clothes because his washing machine was broken. GENERATED: John went the
buy GT: John didn't 't know how fix washing machines. RIGHT CONTEXT: He bought book to tell him how to repair it. He changed the fan belt. When he plugged it back in, worked.
22

---

Published as a conference paper at ICLR 2022
Table 13: Example of text infilling with Time Control (d=16) on ROCStories.
Tom was jealous of his brother. His brother was older and stronger. His brother went wherever he wanted.
LEFT CONTEXT:
GENERATED: Tom's brother was very jealous his brother. GT: One day his brother was grounded for staying out too late.
RIGHT CONTEXT: This made Tom really happy.
Jackie was 11 and had to get braces. She was worried about what her friends would think. She tried to hide them when she first got them.
LEFT CONTEXT:
GENERATED: But she was too embarrassed tell them
GT: Eventually her friends saw them and she was embarrassed.
RIGHT CONTEXT:
Her friends noticed she was embarrassed and decided to comfort her.
LEFT CONTEXT: Sally was going to surprise the office with a cake. Sally felt that a cake would be good way to make them smile. She went to the supermarket to pick up the cake. GENERATED: She bought the cake and was very happy with it. GT: At the office she gathered the employees around the conference table. She then brought out a cake and they all felt better.
RIGHT CONTEXT:
LEFT CONTEXT:
Lars was playing XBOX. His controller stopped working during a game. he had walk all the way the store. The store was being remod- eled when he got there so he went to another.
Lars didn't t have
GENERATED: Video games were all over the floor.
GT: Lars wasn't 't able to find a controller.
RIGHT CONTEXT:
LEFT CONTEXT: Emma had been working as dishwasher. Her hands cracked and bled from the hot soapy water. Then her mom noticed and concocted special salve for her. Emma used the salve every night before bed.
GENERATED: Soon she was able to keep her hands clean and happy.
GT: Her hands got better. RIGHT CONTEXT:
LEFT CONTEXT:
Jerome dribbled the ball quickly. Sam tried to grab it from him, but wasn't fast enough. He chased Jerome down the court.
GENERATED: He scored point with long shot..
GT: Jerome pushed Sam backwards and threw the ball.
RIGHT CONTEXT: Jerome scored points for his team.
LEFT CONTEXT: Class started in 10 minutes and had math assignment due. My older friend volunteered to do it for me. Unfortunately, my teacher found out about the copying. GENERATED: She was very upset with me for doing the assignment.
GT: She crumbled my paper and threw it in the trash.
RIGHT CONTEXT: My teacher gave me an F for the assignment.
23

---

Published as a conference paper at ICLR 2022
Table 14: Example of text infilling with Time Control (d=32) on ROCStories.
LEFT CONTEXT: I went the park to play frishee with my dog. GENERATED: We played all day
GT: I tossed the frisbee my dog and he would catch in his mouth.
RIGHT CONTEXT: accidentally threw the frisbee too far. The frisbee landed into the pond. But my dog went and got it.
LEFT CONTEXT: I was tired of working at Walmart. The hours were bad. The store music was awful.
GENERATED: was very bored..
GT: I handed my two weeks in to the manager.
RIGHT CONTEXT: then found another job and was happy.
LEFT CONTEXT: Sam bought new SUV. was all wheel drive. He figured he would take it off road. He hit few hard bumps and broke his suspension. GENERATED: Unfortunately he had to pay a lot of money for it. GT: Sheepishly, he brought it to the dealership for repair. RIGHT CONTEXT:
LEFT CONTEXT:
Missy got drunk and went to get a tattoo. She decided to get a tattoo on her forehead. The next day, Missy was horrified at what she had done.
GENERATED: Her tattoo was on her forehead!
GT: Missy scraped up her money to pay for a tattoo removal procedure. After much wasted money, the tattoo was gone.
RIGHT CONTEXT:
LEFT CONTEXT: Jake was going on a road trip to see his family. He Got in the car and drove.
GENERATED: Jake's family was driving down the road
GT: The car's tires exploded due too much air.
RIGHT CONTEXT: Jake hitchhiked for 30 miles. When Jake got to his family he was happy
his trip was over.
LEFT CONTEXT:
Bob decided to start a business. GENERATED: However, he did not know the market at all GT. He opened up grocery store and was doing very well. RIGHT CONTEXT: After a year, his profits dropped and he had to declare bankruptcy. Bob was sad to see his business fail. Bob worked hard and reopened his business.
LEFT CONTEXT: hadn't seen my girlfriend in a while. She got new job so it's hard to talk. The job takes up all of her time. GENERATED: had to ask her to out with me.
GT: Finally she called me hang out.
RIGHT CONTEXT: I was really happy to see her and we made plans.
Variational Auto-Emnn Euto-Enncoder See Table 15
24

---

Published as a conference paper at ICLR 2022
Table 15: Example of text infilling with Variational Auto-Encoder on ROCStories.
LEFT CONTEXT:
went to the park to play frisbec with my dog. GENERATED: served served served [...]
GT: tossed the frisbee to my dog and he would catch in his mouth.
RIGHT CONTEXT: accidentally threw the frishee too far. The frisbee landed into the pond. But my
dog went and got it.
LEFT CONTEXT: Tim had a dentist appointment today. He was going to get his wisdom teeth pulled. GENERATED:
GT: His dentist numbed his gums.
RIGHT CONTEXT: His wisdom teeth popped out instantly. less hassle than he thought it would
be.
LEFT CONTEXT:
Tom studying for the big test. He then fell asleep do boredom. GENERATED: The The The The The The The The The [...]
GT: He slept for five hours.
RIGHT CONTEXT: He woke up shocked. Tom hurried to study as much as possible before the test.
Implicit Dynamics See Table 16
Table 16: Example of text infilling with Implicit Dynamies on ROCStories.
LEFT CONTEXT:
Tom was jealous of his brother. His brother was older and stronger. His brother went wherever wanted.
GENERATED:
wasn't like him. wasn't like him.............................
isn't t...
doesn't.. t... doesn't.. don't. don't. don't. don't. don't. don't. don't t. don't t. don't. don't. don't t. don't. don't. don't t. don't. don't. t. don't. tt. don't. don don't. 't. don don't 't' don't. don don't 't don't don't 't don't don't don't 't don't don't don don don don [...]
GT: One day his brother was grounded for staying out too late.
RIGHT CONTEXT: This made Tom really happy.
LEFT CONTEXT:
The family decided go the park for Veteran's Day. GENERATED: didn't knew why why why why [...]
GT: We had a barbecue outside.
RIGHT CONTEXT: In addition, everyone decided to take swim in the water. Before we left, we had huge cake. We then swam for hour at the lake.
G.2 FORCED LONG TEXT GENERATION
We include examples of GPT2 long text generation in Figures and
H HUMAN EXPERIMENTAL SETUP
We use Mechanical Turk to collect human evaluation for long text generation and text infilling models. Examples from all the systems are selected randomly and each example is evaluated by 10 annotators.
H.1 LONG TEXT GENERATION EVALUATION
The annotators are provided with a dialogue between a movie assistant and user generated by GPT and Time Control, where the goal of the assistant is to help the user to book movie ticket. We show the last few generated interactions between the assistant and the user to the annotators, and ask them to assess whether the provided snippet could be reasonable ending for such conversation. We use a 5-point Likert scale where 1 corresponds to a "Very Unreasonable Ending" and 5 corresponds to a "Very Reasonable Ending".
25

---

Published as a conference paper at ICLR 2022
USER ] I'm interested buying some tickets for movie tonight. [ ASSISTANT I CK. Can you tell me the city and state where the theater is located? USER It's been long since we've been the movies that I want to get out. It can't get any better. [ ASSISTANT ] Great. Can you tell me the city and state where you plan to see the movie? USER I'll be in Santa Barbara, California at the moment. ASSISTANT Got it. Are you intereated in town.
particular movie? USER There's couple of nice theaters ASSISTANT ] CK. Is there a movie you have in mind? USER I'm thinking action. [ ASSISTANT ] Ok. Let's see what action films are out in that area. [ ASSISTANT ] I found one action movie at AMC Mountain 16: No Time To Die. Are you interested in that one? USER My best friend sally has been wanting to see that one, and I quess ww'll with that one. [ ASSISTANT ] OK, Are you interested this movie though? USER Definitely interested in seeing No Time to Die. [ ASSISTANT ] oK. Remaining showtimes for No Time To Die at AMC Mountain are 4:30pm, 6:40pm
9:10pm. USER 9:10 pm good.
ASSISTANT ] Gct it. And how many tickets? [ USER ] Just me and couple other guys. ASSISTANT All right. Le me confirm that you'd like three tickets for No Time тс Dic at MMC Mountain 16 tonight at 9:10pm. Is that all correct? [ USER ] Yes that's right. ASSISTANT Is these ticketa? [ USER Yes go right ahead. I ASSISTANT ] OK. Your tickets are purchased and details for how to proceed have been sent to your email address. Can I help with anything else? I USER I No. [ ASSISTANT ] OK. Enjoy your novie! As always, have a good day. javascript ive been! havn bout ive heard it. dont think to much. in there. lmao. id appreciate it. reccommended. The email address for your tickets theater. There's no problem at all with anything else. thanks! sign off. Your tickets are confirmed and they've ve been sent to your email address. thanks a lot. your welcome. enjoy your movie. -1bye. I'm very, very sorry for leaving you behind. Thanks for calling. Is there anything else I can help with? dont remember the last time we went out. I'm very sorzy. Thank you very much. arn. Thanks for all of your help. Enjoy the movie. wwww, bye. -en Chinese girlfriend, bye. can you enjoy your movie. Bye. remaining showtimes for No Time To Die at AMC Mountain 16 are 4:30pm, 4:40pm and 9:10pm. This is all for right now. N/À Thanks for all your assistance. @ é b/r/r/r/r abys mam. @ согnуииитииииуу BULLY BULLY MOVIE FOR YOU! Thank you for calling. Enjoy your movie! Your movie. Aww, bye. happy Thanks for using services.
to go ahead and purchase
movie. thanks for calling.
Figure 3: Example of GPT2 forced long text generation.
H.2 TEXT INFILLING EVALUATION
Given the preceding and following context, we ask annotators to evaluate whether the generated fill-in sentence a reasonable completion. We evaluate generated fill-in sentence by ILM, LM and Time Control on scale of 1-5.
I EXAMPLE TRAJECTORIES OF THE LEARNED LATENT SPACE
See Figures 7 10 for example latent trajectories over coherent vs. incoherent (randomly scrambled) held-out Wikisection documents. These trajectories show the recovered latent structure by Time Control and the three ablations from our work. Figure 7 are latent trajectories from Time Con- trol. Figure are latent trajectories from Implicit Dynamics. Figure 9 are latent trajectories from Brownian motion. Figure latent trajectories from Variational
J WHY DOES VARIATIONAL AUTO-ENCODER PERFORM WELL ON THE RECIPE DOMAIN?
In Table 5 VAE does much better than TC. Here we investigate potential reasons why.
26

---

Published as a conference paper at ICLR 2022
I USER I'm interested in buying some tickets for a movie tonight. | ASSISTANT Ok. Can you tell the city and state where the theater is located? USER ] It's been so long since we 've been to the movies that I want to get out. It can't get any better. [ ASSISTANT Great. Can you tell me the city and state where you plan to see the movie? USER 1 I'll be in Santa Barbara, California at the moment. ASSISTANT Cot it. Are you interested in a particular movie? [ OSER There's couple of nice theaters in town.
ASSISTANT OK. Is there movie you have in mind? [ USER ] I'm thinking what action filma
action. ASSISTANT Ok. Lat's ASSISTANT I found one action movie at ANC Mountain 16: No Time To Die. Are you interested in that one? USER My beat friend Sally has been wanting to see that one, and guess we'll go with that one. ASSISTANT OK. Are you interested in this movie though? USER Definitely interested in seeing No Time to Die. [ ASSISTANT DK. Remaining showtimes for No Time To Die at AMC Mountain 16 are 4:30pm, 6:40pm and 9:10pm. USER 9:10 pm is good. ASSISTANT Got it. And many tickets? | USER Just and a couple other guys. ASSISTANT All right. Let me conti:m that you'd like three tickets at Mountain tonight at 9:10pm. Is that all correct? [ USER Yes that's s right. I ASSISTANT ] IS it OK to go ahead and purchase go right ahead. ASSISTANT оk. Your tickets purchased and details for how to proceed have been sent to your email address. Can help with anything else? USER No. I ASSISTANT ] оK. Enjoy your movie! As always, have good day. javaecript ive been! havn bout ive heard it. dont think to much. in there. lmao. id appreciate it. reccommended, The email address for your tickets is theater. There's s no problem at all with anything else. thanks! sign off. Your tickets are confirmed and they've ve been your email address. thanks lot. your welcome. enjoy your movie. -lbye. I'm very, very sorry for leaving you behind. Thanks for calling. Is there anything else
in that area. [
for No Time
these tickets? [ USER
help with? dont remember the last I I'm very sorry. Thank you very much. arn. Thanks for all of your help. Enjoy the movie. www,, bye. Bye. remaining showtimes for No Time To Die at AMC Mountain 16 are 4:30pm, 6: 4:00pm and 9:10pm. This is all for right now. N/A Thanks for all of your assistance. 8 ê b/x/r/z/r abys mam. I соτлууиииииииуу BULLY BULLY MOVIE FOR YOU! Thank you for calling. Enjoy your movie! Your movie. Awn, bye. happy movie. thanks for calling. -en Thanks for using our services. >
we
out.
Chinese girlfriend, bye. you enjoy your movie.
Figure 4: Example of GPT2 forced long text generation.
We looked into comparing the latent structure recovered by the VAE baseline and TC; see Figure 11 What found is the Time Control is best extracting time over the course of the document (ie. its latents recover a Bridge process correlated with time), whereas the VAE doesn't 't elicit strong temporal structure. We hypothesize that temporal structure is not all you need to succeed in the Recipe domain.
K ADDITIONAL EXPERIMENTS
On discourse coherence, we ran ALBERT on = 1 since ALBERT is trained on this setting. We found that it got slightly above random chance, unlike the most of the baselines we compare against. The discourse accuracy is 50.1 + 9.0 on Wikisection, 55.8 + 4.0 on TicketTalk and 60.8 + 2.8 on TM-2.
27

---

Published as a conference paper at ICLR 2022
[ USER 1 Hey! I wanna go tonight to watch movie! [ ASSISTANT ] Hi, what movie would you like to watch? USER I want to know about Birds of Prey! ASSISTANT Sure, is there any particular theater you would like to watch Birds of Prey at? USER Can you tell what theaters are showing the movie? [ ASSISTANT I What are you looking at? I USER Can you tell me about what theaters are showing Birds Prey at? [ ASSISTANT ] Are you interested in one theater? [ USER ] What theaters are showing Birds of Prey at? [ ASSISTANT ] Okay, what movie theatre are you going to? [ USER ] I want to go to AMC. ASSISTANT Very well, what time would you like to watch the movie? [ USER ] What is the movie playing tonight at 1:00pm and 3:00 pm? ASSISTANT Can you look up times for Birds of Prey at AMC theaters? USER ] Can you tell about the time and the theater where Birds of Prey is playing? [ ASSISTANT ] Sure, would you like to watch an IMAX,
Standard showing? USER 1 want the IMAX show especially there an IMAX showing. [ ASSISTANT 1 Sure, how many tickets would you like to purchase? [ USER I I need five tickets. ASSISTANT Okay, I found the IMAX showing at 1:00pm, does that work for you? [ USER ] Sure, let's do that! ASSISTANT ] How many tickets would you like to purchase? [ USER ] I want two tickets, for me, my daughter and my doctor. [ ASSISTANT ] Okay, have purchased two tickets for Birds of Prey at AMC theater for 1:00pm. for three people. Is that all correct? [ USER ] Yes, that's correct. ASSISTANT Okay, would you like confirm your purchase and book your tickets? USER 1 Yes, want to purchase tickets ASSISTANT Please provide me your email address, I'll see what I can find. USER ] I'd like to use google, is there an information available for emailing tickets to birds of prey? [ ASSISTANT ] Sure, would you like me to check where my tickets are located? [ USER ] Can you tell me the closest theaters to me for tickets showing tonight? [ ASSISTANT Sure, how would you like to the movie playing in theaters near you? USER Can you look for any similar theater, get food, and tickets? ASSISTANT Okay, showing theaters in Davis, California, is there way you can search? USER Look up some movie showtimes for me, see if they have anything that is showing in IMAX or 3D. [ ASSISTANT J Would you like me to search all theaters for tickets showing any available IMAX shows? [ USER ] Can you look for showing times of Dolittle at AMC? [ ASSISTANT ] Suze, could I look at the showtimes for the movies showing at 3D? [ USER ] I want the earliest showing times, maybe at 10:20 pm. [ ASSISTANT ] Not a problem, not an issue, let me know if can change that. Thanks, I'll get it my phone. C'mon, have you decided on time? ASSISTANT Yes, saw your email for confirmation and booked three tickets for Birds of Prey at AMC theater for 1:00 pm, does that work for you? USER ] Oh ok, forget about it now. [ ASSISTANT Would you like to checkout the booking details and book your tickets online, please. I USER ] Actually, can you help me find tickets for Dolittle showing at 10:20 pm in 3D instead of 3D? [ ASSISTANT ] Not a problem, can you look up tickets to that showing instead instead? [ USER J Can you think of anything else? ASSISTANT Sure, I'll check out the tickets at AMC showing time showing in 3D well, if there is anything else? [ USER ] Yea, cool, I just need to get those booked and I'll get more information I ASSISTANT 1 Awesome,
AMC
help buy those tickets! have just booked three tickets for Dolittle showing in 3D AMC showing time in 3D at 10:20 pm tonight, does that work for you today? [ USER ] I want to upgrade to 3D, and then use my Apple Pay. I ASSISSAATT ] Okay, I'm turning this over and getting those tickets online now, Back to you, I want to purchase your movie tickets to purchase the movie tickets tonight at 10:20 pm at AMC showing in 3D.
Correct? [ USER ] Sure, I want to purchase movie ticket for the movie showing in 3D instead of 3D instead, and then use my Apple Pay. ASSISTANT Okay, will now see if you can purchase the movie tickets to purchase your movie tickets to purchase the movie tickets tonight at 10:20 pm at AMC showing in 3D. Is there anything else you need help with? [ USER ] Not really, not really. Thanks!
Figure 5: Example I Time Control forced long text generation.
L PREVIOUS EXPLORATORY THREADS
L.1 SUCCESS AND FAILURE MODES FINE-TUNING GPT2
We found that fine-tuned GPT2 was able to replicate certain aspects of a document corpus, such as section header ordering ( 92% accurate). However, when going from document-level to section- level statistics, we noticed that GPT-2 seemed to be either undershooting or overshooting the section lengths. These results are reported in Table 3
28

---

Published as a conference paper at ICLR 2022
USER ] Hello. am thinking of going to see a movie today. ASSISTANT 1 OK, And where will you be seeing the movie? [ USER ] Maybe in Creek's End, Oregon? [ ASSISTANT I Creek's s End, Oregon. Got it. Is there a particular movie you have in mind? [ USER ] Just any James Bond movie. ASSISTANT Ok. think you are talking about No Time To Die. Does that sound right? [ USER Yes it is, No Time To Die is the name.
ASSISTANT
Creek's End, Oregon, see that No Time To Die is showing in two theaters: AMC Mountain 16 and AMC Mercado 24. [ USER ] I want the AMC Mercado 24 its what I used to call it. ASSISTANT It looks like tonight's remaining showtimes for No Time To Die at AMC Mercado 24 are 5:00pm, 7:10pm, and 9:45pm. Which is best for you? [ USER ] Can I change the theater from the one I picked because the another one has other showings around it?
think action movies are always great choice.
ASSISTANT OK, No Time To Die it is. USER Can I have showtimes for two instead of one? ASSISTANT Sure, you chose Mercado 24. USER I Can I change want the earliest showtime instead? ASSISTANT Let me check showtimes at AMC Mercado 24. I USER Can actually change the showtime to the showtime you said? [ ASSISTANT Sure, you chose the showtime at 3:00pm. [ USER Okay that works better, can I change the time again. ASSISTANT Remaining showtimes for No Time To Die at AMC Mercado 24 are 4:30pm, 6:40pm and 9:45pm. [ USER ] Can purchase two tickets to Mercado 24? ASSISTANT Sure, you would like
time,
the showing of Time To Die two tickets for No Time Die at AMC Mercado at 3:00pm. Correct? USER ] Yes, please. [ ASSISTANT ] Okay. So to confirm you would like two tickets for No Time To Die at AMC Mercado 24 at 3:00pm tonight at the AMC Mercado 24 theater? [ USER ] That's right. ASSISTANT Is it OK to go ahead and purchase these tickets? [ USER Sure, let's do it. [ ASSISTANT ] OK. Your tickets are purchased and details for how to proceed have been sent your email address. Can help with anything else? [ USER ] Actually, let me think before purchase the tickets since it will be rough night work. [ ASSISTANT 1 understand. Could you please detail how I got these tickets? USER ] Can I just send them to my email? [ ASSISTANT ] Sure. Let me see what I can find. [ ASSISTANT ] Unfortunately description. [ USER ] Can I get the dealer's list of movie names from the AMC Mercado 24 website? [ ASSISTANT ] Showtimes for No Time To Die at AMC Mercado 24 are 4:30pm, 7:10pm, and 4: USER Ok, let Unfortunately cannot find any movie title that matches your description. [ USER 1 think have to choose between No Time To Die and King Johnny please. ASSISTANT ] Got it. Let me see what I can find. ASSISTANT Unfortunately I cannot find any movie title that matches your description. [ USBR Ok, well, I don't want to buy any tickets. [ ASSISTANT Showtimes for No Time To Die at AMC Mercado 24 are 3:00pm, 5:10pm, 7:30pm and 9:45pm. USER Can you fix something else? [ ASSISTANT ] That's it for now. USER Thanks. Could you actually order the tickets? ASSISTANT No need, I think we are done with this. [ USER ] Can you just order these? ASSISTANT Sure. I will be buying these tickets for you at AMC Mercado 24 at 3:00pm today. Is there anything else you would like to know? [ USER ] Can I make sure my debit card got charged again? [ ASSISTANT ] Sure. Let me see what I can find. [ ASSISTANT ] Unfortunately I cannot find any movie title that matches your description. I USER ] ok! No problem. ASSISTANT Showtimes for No Time To Die at AMC Mercado 24 are 4:30pm, 7:10pm, and 9:45pm. [ USER Ok great, would like to make this change. ASSISTANT Is there anything can help with now? USER ] Please, could you send the tickets to my email? ASSISTANT I Sure. am glad to help. Let me know what you need help with.
cannot find any movie title that matches your
what I
find. ASSISTANT ]
Figure 6: Example 2 of Time Control forced long text generation.
L.2 INVESTIGATING THE IMPORTANCE OF DISTANCES BETWEEN SAMPLED SENTENCES
In earlier iterations of the work, we did explore learning with pairwise contrasts with fixed t- distances, e.g. distances of length 1, 5, and 10 sentences. We observed that the resulting latent trajectories elicited different fits to Brownian bridge dynamics, and the quality in fit varied in t across domains; we've we've include some examples in Figure 12 on Wikisection.
29

---

Published as a conference paper at ICLR 2022
coherent abslz)
inceherent abx
0.70
0.50
0.45
0.50
(a) Coherent, Time Control d=8.
(b) Incoherent, Time Control d-8.
5.6
0.25
50
200
120
150
182
20
60
100
120
140
150
(c) Coherent, Time Control d=16.
(d) Incoherent, Time Control d=16.
coherent abstzl
Incoherent abstz!
0.45
0.40
5.6
53
0.20
300
120
140
120
(c) Coherent, Time Control d=32.
(f) Incoherent, Time Control d=32.
Figure 7: Time Control's latent trajectories over coherent vs. incoherent (randomly scrambled) held-out Wikisection documents. The encoder learns Brownian bridge-like latent trajectories the coherent documents. The incoherent documents map to noisy trajectories that don't 't evolve over time.
30

---

Published as a conference paper at ICLR 2022
coherent
incehernnt abeld
9.2/5
3.250
3.225
9.20
5.200
2.72
0.55
2.350
200
1.00
1.0
340
180
(a) Coherent, Implicit Dynamics d=8.
(b) Incoherent, Implicit Dynamics d=8.
coherent absizi
Incohorent abstrl
3.0
2.
355
1.5
1.0
(c) Coherent, Implicit Dynamics d=16.
(d) Incoherent, Implicit Dynamics d=16.
coherent
inceherent
255
3.0
15
1.0
0.5
(c) Coherent, Implicit Dynamics d=32.
(f) Incoherent, Implicit Dynamics d=32.
Figure 8: Implicit Dynamics's 's latent trajectories coherent incoherent (randomly scrambled) held-out Wikisection documents.
31

---

Published as a conference paper at ICLR 2022
coherent
inceherent abslrl
2.532
0.20
0.08
3.000
0.026
0.024
2.004
0.032
200
1.00
1.0
340
160
(a) Coherent, Brownian motion d=8.
(b) Incoherent, Brownian motion d=8.
coherent absizi
Incohorent abstll
0.07
0.20
0.008
5.008
0.006
3.006
5.000
(c) Coherent, Brownian motion d=16.
(d) Incoherent, Brownian motion d=16.
coherent
inceherent
0.000
0.28
5.000
0.024
0.0
(c) Coherent, Brownian motion d=32.
(f) Incoherent, Brownian motion d=32.
Figure 9: Brownian motion's 's latent trajectories over coherent incoherent (randomly scrambled) held-out Wikisection documents.
32

---

Published as a conference paper at ICLR 2022
coherent
inceheennt abeld
0.11
0.00
0.59
200
1.00
1.0
340
150
(a) Coherent, Variational Auto-Encoder d=8.
(b) Incoherent, Variational Auto-Encoder d=8.
coherent absizi
Incohorent abstel
0.00
9.08
0.28
(c) Coherent, Variational auto-Encoder d=16.
(d) Incoherent, Variational Auto- Auto-Encoder d=16.
coherent
inceherent
0.09
0.10
0.18
0.27
0.00
0.00
3.55
0.00
(c) Coherent, Variational Auto-Eneoder d=32.
(f) Incoherent, Variational Auto-Encoder d=32.
Figure 10: Variational Auto-Encodert latent trajectories over coherent incoherent (randomly scrambled) held-out Wikisection documents.
33

---

Published as a conference paper at ICLR 2022
abstat
15
5.2
10.0
125
26.3
13.5 20.0
(a) Latent trajectories across dimensions for TC (b) μ + in latent trajectories across dimensions TC (d=8).
(d=8).
coherentaasii
0.50
3.250
0.25
9.20
0.25
0.36
7.
160
135
15.0
26.
60
255
5.0
3/5
150
18x
16.
17.5
26.0
(c) Latent trajectories across dimensions for VAE (d) μ u ar in latent trajectories across dimensions (d=8).
for VAE (d=8).
Figure 11: Comparing the latent structure recovered by the VAEbaseline and TC on held-out Recipe documents. Noticeably, TC learns temporally relevant activations whereas VAE latent does not correlate with time.
34

---

Published as a conference paper at ICLR 2022
test72
12
1.0
14
12-
10-
08-
66-
0.4
04
00
120
(b)
10-
84
(c) k=10
Figure 12: Recovered latent trajectories held-out Wikisection documents where used pair- wise contrasts with varying time distances; this was on TC, d= 8. Notice how the recovered latent structure varies depending on k the distance between sampled sentences.
35

---

Published as a conference paper at ICLR 2022
BERTScore
ROUGE
BLEURT
Method
Latent dim d
Precision
Recall
F1
1-F1
2-F1
L-F1
LM
0.45
0,50
0.47
22.6
1.8
14.0
0.31
ILM
0.50
0.51
0.50
21.6
1.8
20.7
0.33
VAE
8
0.21
0.26
0.21
1.1
0.9
1.1
0.27
VAE
16
0.17
0.25
0.18
1.4
0.1
1.5
0.26
VAE
32
0.10
0.10
0.12
0.2
0.6
0.2
0.26
InfoNCE
8
0.22
0.29
0.23
7.7
1.3
7.4
0.18
InfoNCE
16
0.18
0.28
0.20
1.9
0.0
1.9
0.18
InfoNCE
32
0.20
0.28
0.21
3.0
1.0
5.8
0.18
TC (Ours)
8
0.51
0.51
0.51
17.5
1.6
16.1
0.30
TC (Ours)
16
0.47
0.49
0.49
15.9
1.5
14.3
0.34
TC (Ours)
32
0.50
0.50
0.50
18.4
1.4
17.1
0.32
ROUGE, and BLEURT truth infilled sentence and the generated sentence.
Table BERTScore (Zhang et al. 2019),
(Sellam et
2020) on ground
36


Reviews:
[{"review_id": "mILODfUxog", "reviewer": "Reviewer_mT85", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a generation from a language model not only from an initial\nstate, but also using a goal state. Instead of Brownian motion, the authors\nemploy a draw from Brownian bridge by designating initial and end states,\ncalled Time Control.\nExperimental results show the proposed generation from Brownian bridge is more\nnatural and coherent for text-infilling task, and also preserves text \nstructures both by automatic evaluation and human evaluation.\n\nUsing Brownian bridge is a very simple and effective idea for text generation.\nMy only concern is the range of its applicability: while it is far more natural\nthan a simple random walk, Time Control only allows designating the first and\nlast states for generation. However, in the actual situation, it is not always\nthe case for the first (and sometimes, last) sentence should have a designated\nstate. First few sentences might constitute just an ice-break, and the actual \ncontent might start after that. \nMore generally, it is more desirable that we can condition the generation at\narbitrary time. In fact, I think that this can be done by a conditional draw\nfrom a Gaussian process. Since Brownian motion corresponds to using an \nexponential kernel of GP, sentence generation from conditional GP would be \nthe way for the future extention of this work.\nAnyway, this work will surely pave the way for such principled generations.\n\nMinor\n\n- Some tables are located within the main text. Tables and Figures should be\nplaced top or bottom of the paper for readability: please use \\begin{figure}[t]\nfor something like that.\n- Numerical results in Tables can be rendered in a smaller font (i.e. \\small).\nAlso I recommend to condense line spacing for Tables for readability, using \n\\usepackage{setspace} and begin{spacing}{0.9} ... end{spacing}, for example.\n", "summary_of_the_paper": "This paper proposes a generation from a language model not only from an initial\nstate, but also using a goal state. Instead of Brownian motion, the authors\nemploy a draw from Brownian bridge by designating initial and end states,\ncalled Time Control.\nExperimental results show the proposed generation from Brownian bridge is more\nnatural and coherent for text-infilling task, and also preserves text \nstructures both by automatic evaluation and human evaluation.", "summary_of_the_review": "Nice attempt for random generation from neural language models using the idea of Brownian bridge. This work will pave the way for more princpled random generation from language models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Ct0RJ731kF3", "reviewer": "Reviewer_JgDa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has an interesting approach and tackles an important problem of streamlining sequence generation from autoregressive models. \n\nThe experiments show the value of learning a manifold over the latents that have a linear relationship with some stochastic perturbation. They provide evidence that learning in such a manner is promising in order to maintain coherence over long text generation.\n\nHowever, the setting is fairly limited because this approach requires two contextual endpoints, the start and finish. This is especially underwhelming given that the introduction states that this approach aims to perform \\emph{controllable goal-oriented} generation. In my view, the setting described and experimented with doesn't reflect this goal. For example, there are limited experiments with regard to controllable generation, or goal-oriented generation tasks.\n\nSecondly, the assumption that autoregressive generation follows a Brownian motion is strong and I would like to see some empirical evidence or theoretical argument supporting this. One simple experiment could be to actually try to fit a Brownian motion model to a bunch of sequences generated from GPT-2, and show that this fitted model is not suitable for naturally occurring text. \n\nExperiment wise, my biggest concern is the VAE baseline. The point of this baseline is to show that for the same setup of Brownian bridge process, contrastive learning is better than the VAE objective, but I feel that the VAE implementation as described in the appendix does not make the comparison fair. Due to lack of details in the paper, I am assuming that the priors p(z0) and p(zT) are standard gaussians. If this is not true, then a clarification would ease this concern of mine. But assuming this is true, the loss basically tries to match the encoder distributions q(z0) and q(zT) obtained by f_{\\theta}(x0) and f_{\\theta}(xT) to the standard Gaussian. What this means is that there is a pressure to make the 0 and T embeddings similar which is not at all what we want from this bridge process model. A more careful instantiation of prior for VAE, or even learning a time sensitive prior would be a better implementation of the VAE baseline.\n\nTable 1 is another concern. This experiment basically trains a linear classifier over the encodings to identify if they are in-or-out of order. The proposed approach is naturally suited for this metric/classifier because the encodings at different times are more or less linearly related with some stochasticity. However, this is not true for the other baselines, so I am not sure what is the takeaway message from this experiment.\n\nAlso, more exposition on the Brownian motion baseline would be helpful. The current description is not enough to get an idea about what exactly was done for generation and other experiments with this baseline. On a related point, I don't get why BM for Table 2 would be the same as the brownian bridge. Isn't it the case that Brownian motion baseline doesn't get to see \\emph{both} the endpoints? If I am mistaken about this, then more exposition is required here because I checked both the paper and the appendix carefully for this.\n\nTable 5 shows mixed results. More discussion and analysis here would be helpful.\n\nFor clarification: please make explicit whether the triplets have a notion of distance or not i.e. it is sensitive to different value of t depending on which sentence in the middle was sampled. From the context, I am assuming this is the case but clarification would be helpful. Also, notation in equation 2's denominator is confusing. Are you summing over all the negative x_{t'}? ", "summary_of_the_paper": "This paper proposes to model the evolution of sentences in a document via a stochastic process; specifically a Brownian Bridge process. The paper start off by assuming that the generated sequences by autoregressive models like GPT-2 follow Brownian motion in that they tend to get incoherent and \"meander\" in the semantic space. This paper aims to reduce this random behavior by pinning the endpoints of the trajectory and model the generation by Brownian bridge process instead. The key intuition behind this process is that given two endpoints z0 and zT, the evolution of z along time t is a Gaussian with mean that is some linear combination of z0 and zT. This paper models text by training an encoder for sentences x that produces the embedding z by training over triplets (x0, xt, xT) where 0<t<T that encourages zt to follow Brownian bridge dynamics and uses contrastive loss with a negatively sampled x't for training.\n\nThe approach is tested for local coherence, long range order sensitivity, and generation of long sequences and is compared against ablative and external baselines. The proposed approach does lead to learning of embeddings that are obtainable via linear combination and this leads to improved performance on sensitivity to sentence order in documents and document generation. ", "summary_of_the_review": "Overall, I think this paper is well motivated and proposes a reasonable solution to improve coherence of model generated text. This is supported by ample experiments but I have serious concerns about some of the crucial experiments and baselines that I have detailed in my main review. Also, I think that the paper could be clearer about its contributions and implementation details.\n-----\nPost rebuttal: thanks to the authors for the detailed response addressing many of my concerns. My biggest concern about the prior in the VAE baseline is somewhat alleviated given that the the authors used different fixed priors for the two settings. While this could be improved by having learnable priors/better priors, I think the current setting makes the experiments reasonably sound. I have raised my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7_d4nPUism2", "reviewer": "Reviewer_zz52", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n\n- The paper is well structured and easy to follow, the idea of modeling sentences to a Brownian bridge latent space is neat and generic enough to (1) allow for noise given its stochasticity (2) doesn't require explicit domain knowledge for planning. \n\n- Well Structured Experiments sections with 4 RQs and results that confirm each of the hypotheses\n\n- Reproducibility and Transparency in reporting of experiments in terms of available source code, dataset information, details about human evaluation, generation examples. \n\n**Areas of Enhancement & Questions to authors**\n\n- The information about each of the ablations (ID, BM) could be explained better. namely the section \"ablations\".  \n\n- There's a clear Inconsistency in the best TC method between different latent dimensions (8,6,32), in most of the experiments there's at least one of the 3 that is performing drastically worse than the other baselines, while there's overall no clear winner. I wonder if you have thoughts about this. \n\n- Table 5 the VAE(32) method performs the best overall in \"Wiki section\" although the TC (16) method has been highlighted as the best. Is there a reason behind this?\n\n- During the training of the decoder how do you make sure that the decoder uses the information given by the latent plan?\n\n- Overall the paper would have benefited from an intrinsic visualization of the latent space, to make sure for example that there's no  Information collapse of the embeddings when dealing with long sentences. This could be done by visualizing the planning trajectory difference between coherent and incoherent text.\n\n\n", "summary_of_the_paper": "This paper introduces a method to enhance the global coherence of text generated from Language models. The proposed method (Time Control). Under the assumption in the latent space of sentence embeddings, the incoherent text can be seen as \"Brownian motion\" in the latent space. In order to enforce a goal to the generated text authors by fixing a start and end to this Brownian motion the process of text generation can be modeled as a Brownian bridge. \n\nFrom this assumption, the authors drive a method that consists of three steps (1) training an encoder to map sentences to a latent plan defined as Brownian bridge (2) training a decoder to reconstruct sentences from the given context + the true encoded vector of the target sentence from planning latent space using the trained encoder (3) at inference time: given a start and endpoint, a target trajectory of vectors $z_0, ..., z_t, ..., z_T$ is sampled and use the decoder to generate a sentence based on this bridge. \n\nAuthors run several experiments to (1) evaluate the hypothesis that the encoder can capture local text dynamics using sentence order prediction task (2) evaluate the decoder to generate local incoherent text using the text-infilling task. (3) capture global text statistics by measuring the statistics (length of Wikipedia sections for city articles) of the generated text and compare them to the ground truth. (4) Evaluate the overall coherence of the long-generated text. Overall the results look convincing except for some caveats (see the areas of enhancement) \n", "summary_of_the_review": "The paper introduces a simple method of preserving coherence in language modeling it builds on previous work that tried to implicitly model planning dynamics. The introduced solution is effective and general enough to not need domain-specific planning information. It is a good paper to accept overall. I advise the authors to clarify the information about the used baselines in a more clear manner. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1T7HoNYk9ae", "reviewer": "Reviewer_HWGg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really like the main modelling contribution of this paper. It is this reviewer's personal opinion that to do long-form text generation, it is not enough to generate token-by-token, but that some high-level planning is required, and the Brownian bridge process model (Time Control; TC) the authors propose is definitely a good candidate to model the latent drift of discourse (indeed, papers like [1] already used random walk-style models to explain properties of word vectors). There are some prior works on using structured probabilistic models, such as switching latent dynamical systems, for text generation [2], which should also be cited.\n\nThe motivation of the model present is clear, and the description of how the model is trained is generally clear enough to reimplement. It wasn't immediately clear that training the model on triples only is enough to guarantee general Brownian bridge dynamics for the entire text trajectory, I feel a note should be added to clarify this. My other quibble here is with how the model is presented: although the general probabilistic model is written down in Equation 1, the likelihood function (i.e. the functional form of p(z_t | x_0, x_t, x_T)) is not explicitly written down anywhere, which leads to confusing things like the variance of the process \\sigma^2 being used in Equation 3 without prior introduction. I feel like explicitly writing down the likelihood would make the equations in the paper flow much better. \n\nI feel the major weakness of this paper is with the experimental sections. For various reasons, I have objections to each of the experiments, which I will go through below:\n\n- The first experiment attempts to show that TC is a better model of local discourse coherence. The authors take two sentences from a document k steps apart, embeds them and them attempts to predict the sentence ordering from the embeddings. They say that for k=1 all models considered perform at chance level on all datasets, and only show results for k=5 and k=10. However, models trained using the k=1 objective (such as ALBERT [3] and StructBERT[4]) seem to be able to perform the task better than chance, so theoretically this should be possible. Therefore, I think the baselines should at least include an ALBERT model to show the performance upper bound on this problem. Further, k=5 (or even 10) starts meaning the sentences start becoming very far apart (10 dialogue turns is more than enough to complete some of the simple dialogue agent tasks!), so it's questionable whether the model is really modelling 'local' dynamics at this point.\n\n- The second experiment looks at text infilling on the ROCStories dataset, and use BLEU and BLEURT to automatically evaluate their models (although the BLEURT results do not appear to be anywhere in the paper). The reported BLEU results are really low, to the extent that it's unclear whether an improvement from 2 to 5 BLEU is really meaningful. Part of the issue is that BLEU measures precision, which penalises text generation where there are a variety of possible outputs; for this reason, [2] report ROUGE results on ROCStories, which are much better. The missing BLEURT results would help contextualise model performance here. The human evaluation shows the model performs about as well as the ILM baseline from [5], which is ok I guess? \n\n  In addition, the table ordering is incredibly confusing. Table 6, which shows the human evaluation for experiment 2, appears much later in the text, after tables for the later experiments. It took me a long time to find it. Can you group the tables a bit better, in thematic order?\n\n- The third experiment attempts to measure 'global text dynamics' by measuring length mismatch per section on Wikisections. It's unclear what notion of 'global text dynamics' the authors are referring to - there are many theories on discourse coherence of long text, and none of them easily map onto a simple measure of section length. If the authors simply mean whether the model has learnt a notion of document structure, I think it would be better to be more explicit about this: showing that fine-tuned GPT2 can't even replicate the structure of a homogenous document corpus is an interesting negative result. \n\n- The fourth experiment forces models to generate beyond the expected document length by suppressing generation of the EOD token. I'm really not a fan of this experiment, because I don't even expect TC to perform well on it. Do the authors just keep on conditioning the decoder on z_T, and force the model to generate from this? At this point, the model is just a standard autoregressive model, so the modelling contribution should have no effect. Alternatively, do the authors resample z_{t+1} each time the model finishes generating a sentence? In which case, how do the authors preserve the Brownian bridge dynamics, conditioning on hitting a target state z_T? There are a few methodological issues with this experiment. A better experiment to run would be to simply ask the human annotators to score texts freely generated from GPT2 and TC for coherence, as a measure of how well TC can generate coherent text.\n\nOverall, while the experimental section is weak, I really believe the core idea of directed Brownian dynamics for planning is a cool one, and deserves to be shared more widely. This is why I recommend acceptance.\n\nReferences: \n- [1]: RAND-WALK: A latent variable model approach to word embeddings, Sanjeev Arora et al. 2015\n- [2]: Generating Narrative Text in a Switching Dynamical System, Noah Weber et al. 2020\n- [3]: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations, Zhenzhong Lan et al. 2021\n- [4]: StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding, Wei Wang et al. 2021\n- [5]: Enabling Language Models to Fill in the Blanks, Chris Donahue et al. 2020\n\n\n==================\n\nPost author response:\n\n> Nonetheless, we think these observations fit well with the intuition our work proposes: Neighboring sentences are close to each other and act like Brownian motion where ordering is difficult to infer, and goal-orientedness / discourse structure emerges on longer stretches of sentences in a document.\n\nI like this framing - currently it's implicit in the paper, but maybe it can be made more explicit that we expect the larger k results to be better, and that this verifies the Brownian bridge approach towards modelling text dynamics.\n\n> Nonetheless, the end arbiter of this task is a human (how coherent do the generations sound to a human?) and we care about at least matching ILM, a method developed specifically for text-infilling. So, it’s promising that our method performs better and/or competitively with ILM on human-based metrics (BLEURT and Human evaluations in Table 6).\n\nI think it should be made explicit then that ILM is in effect an upper bound for model performance, as it is a model trained specifically to do the task, and that matching the performance of ILM is actually a strong result for the TC model.\n\n> So, to directly answer the reviewer’s question: we do not condition the decoder on z_T and do not resample new latents during generation.\n\nThe model is thus primed to generate much longer text than it was typically exposed to? Thank you for the clarification.\n\n> We in fact do already ask human annotators to score the generation (rf. Table 7). In this setup, we remove the middle section of the generated output as the text is extremely long. See Figures 3-6 for examples of the full forced long text generation results.\n\nI believe the stronger (and more realistic) human evaluation is to not just evaluate the tail coherence on forced long text generation, but  instead directly sample from the model naturalistically and evaluate that output using human annotators. If TC better captures global coherence, this should be visible even in this setting.\n\nOverall, I would like to thank the authors for their response. Many of my concerns have been addressed, and I am happy to increase my score.", "summary_of_the_paper": "The authors propose to use a Brownian bridge process to model global coherence of a long piece of text. They show how to train such a model in an encoder-decoder style setup, using a contrastive loss to model the Brownian bridge dynamics. The authors then verify aspects of their model with a series of experiments to show that their model with an underlying generative process outperforms competing approaches on a variety of local and global coherence and generation tasks.", "summary_of_the_review": "Interesting modelling contribution to ensure global coherence of generated text. The proposed modelling approach could have wide applicability, which is why I recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers found that the proposed LM with Brownian motion is interesting and novel. Several reviewers raised (minor) concerns about experiments, but have been generally resolved by the authors.
Decision: Accept (Oral)