SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Results on IWSLT14 DE→EN test set following the setting of SEQDIFFUSEQ. “NFE” indicates the Number of Function Evaluations [Ye et al., 2023].
Methods	Pattern	BLEU	Steps	NFE (Steps×k)
Transformer [Vaswani et al., 2017]	AR	34.74	-	-
CNAT [Bao et al., 2021]	NAR	29.81	-	-
SeqDiffuSeq [Yuan et al., 2022] (k=1)	Diffusion	29.83	2,000	2,000 (2,000 × 1)
AR-DIFFUSION (k=1)	30.19	20	20 (20 × 1)
GENIE [Lin et al., 2023] (k=50)	Diffusion	30.08	20	1,000 (20 × 50)
AR-DIFFUSION (k=50)	34.95	20	1,000 (20 × 50)
AR-DIFFUSION (k=500)	35.62	20	10,000 (20 × 500)
Table 4: Results on COMMONGEN dev set. Results of NAR and AR are from Lin et al. [2020].
Methods	Pattern	ROUGE-2/L	BLEU-3/4	METEOR	SPICE
bRNN-CopyNet [Gu et al., 2016]	AR	9.23	30.57	13.60	7.80	17.40	16.90
Trans-CopyNet [Lin et al., 2020]	11.08	32.57	17.20	10.60	18.80	18.00
MeanPooling-CopyNet [Lin et al., 2020]	11.36	34.63	14.80	8.90	19.20	20.20
LevT [Gu et al., 2019]	NAR	12.22	35.42	23.10	15.00	22.10	21.40
ConstLeven [Susanto et al., 2020]	13.47	35.19	21.30	12.30	25.00	23.20
GENIE [Lin et al., 2023] (k=50)	Diffusion	12.89	35.21	22.00	13.30	24.30	23.00
AR-DIFFUSION (k=50)	13.93	37.36	25.60	16.40	25.00	24.20
Common Sense Generation As depicted in Table 4, AR-DIFFUSION achieves superior performance compared to the current AR, NAR, and other diffusion methods across all the metrics on the COMMONGEN dataset.
Table 5: Experimental results of GENIE and AR-DIFFUSION with inference steps of 2 and 3 on XSUM test set. Take  $k = 10$  to apply the MBR decoding strategy. (·) indicates the drop score compared to the 20-step.
Methods	Steps	NFE	ROUGE-1	ROUGE-2	ROUGE-L	AVG Drop
GENIE	2,000	20,000	30.36	8.78	23.31	-
20	200	28.33	7.46	21.15	-
3	30	25.03 (-3.30)	5.32 (-2.14)	18.17 (-2.98)	2.81
2	20	23.45 (-4.88)	3.95 (-3.51)	16.94 (-4.21)	4.20
AR-DIFFUSION	20	200	30.99	9.32	23.95	-
3	30	30.23 (-0.76)	8.68 (-0.64)	23.43 (-0.52)	0.64
2	20	29.28 (-1.71)	7.99 (-1.33)	22.98 (-0.97)	1.34
4.3 Inference Efficiency
First, we use the number of function evaluations (NFE) as a measure to compare inference efficiency [Ye et al., 2023] in machine translation. From Table 3, it is evident that even when the NFE is reduced to  $1\%$  of SeqDiffuSeq (equivalent to  $100 \times$  faster), AR-DIFFUSION still outperforms SeqDiffuSeq. Moreover, increasing the number of generated candidate samples ( $k = 500$ ) leads to further performance improvements, albeit with increased time consumption.
Second, we conduct experiments with an extremely limited number of inference steps (2 and 3) $^{12}$  and compare the performance with that of GENIE in XSUM. The results are presented in Table 5. When reducing the number of steps to 2, GENIE experiences a significant decline, with an average score of 4.20 in the AVG Drop column, while AR-DIFFUSION exhibits a comparatively smaller
12The time consumed by each step in the inference process is exactly the same.
decrease of 1.34. Furthermore, with 3 steps, although the performance deterioration of GENIE is somewhat reduced, the average score still shows a decline of 2.81. In contrast, AR-DIFFUSION maintains a high performance level, with an average score differing from the 20-step result by only 0.64. Notably, the results of AR-DIFFUSION at 3 steps are comparable to the results of GENIE at 2,000 steps. Therefore, compared to GENIE, the inference speed of AR-DIFFUSION can be accelerated by up to  $600\times$ .
Table 6: Diversity of 10 generated samples on XSUM test set and average of 10 results. The results of BART and GENIE are quoted from Lin et al. [2023].
Methods	BART	GENIE	AR-DIFFUSION
Sampling	Greedy Search	Beam Search	Diverse Beam Search	Typical Sample	Top-k Sample	Nucleus Sample	Diffusion
SELF-BLEU ↓	100.0	93.4	75.6	76.9	80.2	79.1	29.3	30.4
Diversity of Samples Diversity is a key advantage of diffusion models. To measure the diversity of generated samples, We adopt the SELF-BLEU [Zhu et al., 2018] metric, in which a lower score indicates higher diversity. In Lin et al. [2023], various sampling methods were applied to the pretrained auto-regressive model BART<sup>13</sup>. As shown in Table 6, AR-DIFFUSION achieves significantly higher diversity compared to the auto-regressive model. Furthermore, the diversity can be comparable to GENIE with a better performance.
(c)
Figure 2: Ablation experiments on XSUM test set and taking  $k = 5$ . The horizontal axis is the number of inference steps and the vertical axis is AVG-ROUGE = (ROUGE-1 + ROUGE-2 + ROUGE-L) / 3.
Ablation Study To demonstrate the effectiveness of our proposed method, we perform ablation experiments on the XSUM dataset. Our results show that both our proposed multi-level diffusion and skipping mechanism are essential for achieving the high performance of AR-DIFFUSION.
Maintaining the skipping inference method, we remove the token-level diffusion during the training process, which degenerates to GENIE w/ skipping. The comparison results are shown in Figure 2(a). It can be observed that after removing, the AVG-ROUGE score is greatly lower after 2 steps.
The performance of applying our proposed skipping mechanism and DDIM [Song et al., 2021] to AR-DIFFUSION is shown in Figure 2(b). The results demonstrate that the skipping mechanism consistently outperforms DDIM in various inference steps. Additionally, the skipping mechanism can be easily applied to GENIE. As depicted in Figure 2(c), DDIM suffers a significant drop in performance when the number of inference steps is less than 40. In contrast, the skipping mechanism consistently maintains good performance across all inference steps.
13Specific details are written in supplementary material G.
Figure 3: The intermediate state of AR-DIFFUSION gradually generating real text from a standard Gaussian noise through 20 steps. The brightness of the color represents the magnitude of the logits, with darker colors indicating larger logits. More cases are shown in the supplementary materials H.
Case Study By mapping the state to the token with the highest logits, we visualize the intermediate states of AR-DIFFUSION. As depicted in Figure 3, AR-DIFFUSION undergoes a denoising process, transforming the random Gaussian noise into a coherent sentence over 20 steps, and we present 5 of them. With the progression of each timestep, compared to the tokens on the right side of the sentence, the tokens on the left side demonstrate faster determination and a rapid increase in the corresponding logits. This behavior is consistent with our principle of dynamic movement speed from left to right.
AR and NAR Language Models AR models have been the dominant approach for text generation [OpenAI, 2023, Touvron et al., 2023, Dong et al., 2023], but their token-by-token generation nature often leads to unsatisfactory inference speed. To address this issue, NAR models have been developed in recent years. The NAR method is initially proposed by Gu et al. [2017], its objective is generate the entire output sequence in parallel, thereby improving generation speed and efficiency. Subsequently, LevT [Gu et al., 2019] adopts insertion and deletion to address the lack of flexibility in NAR generation, CMLM [Ghazvininejad et al., 2019] utilizes a masked language model to improve the quality of NAR generation through a constant number of iterations, and CNAT [Bao et al., 2021] introduces latent variables to represent the category information of the target word to make full use of the latent representation. However, these NAR methods are hard to model inter-token position dependency and deficient in generation performance.
Continuous Text Diffusion The application of diffusion models to continuous text space is first introduced by Li et al. [2022a]. Through the embedding and rounding processes, the direct integration of continuous noise into word embeddings was accomplished. After that, more people attempt to adopt continuous text diffusion model to solve sequence-to-sequence tasks. DiffuSeq [Gong et al., 2022] divides the input into two parts, utilizing one part as a condition, and perturbs the other part with noise. CDCD [Dieleman et al., 2022] proposes score interpolation and time warping to allow diffusion model and Euclidean embedding to share the same loss function for training. SeqDiffuSeq [Yuan et al., 2022], GENIE [Lin et al., 2023] and DINOISER [Ye et al., 2023] incorporate diffusion model into the encoder-decoder structure through cross-attention mechanisms.
It is important to highlight the differences between our method and both ARDMs [Hoogeboom et al., 2022] and TimeGrad [Rasul et al., 2021], despite the common references to autoregression and diffusion in all these. ARDMs employ an order-agnostic technique, leveraging masking and prediction for generation in arbitrary orders. On the other hand, TimeGrad integrates RNN and diffusion to model the conditional distribution of future steps of multivariate time series. In contrast, our research focuses on implementing the diffusion process within a continuous embedding space, with the primary aim of generating text in a left-to-right sequence.
This paper introduces AR-DIFFUSION, which exhibits AR-like generation behavior but enables efficient parallel decoding. Embracing the inherent sequential nature of language, we propose a multi-
level diffusion model, consisting of sentence-level and token-level components, to assign dynamic movement speeds to tokens. Consequently, compared to those on the right, the left tokens undergo fewer denoising steps and generate earlier to subsequently influence the later ones. Furthermore, we introduce a skipping mechanism to facilitate parallel generation within the multi-level diffusion framework. The experimental results across various tasks demonstrate that AR-DIFFUSION surpasses existing diffusion models in terms of quality while maintaining diversity. Additionally, compared to existing diffusion language models, AR-DIFFUSION achieves comparable results while being  $100 \times \sim 600 \times$  faster.
This research is supported by National Natural Science Foundation of China (Grant No.62276154), Research Center for Computer Network (Shenzhen) Ministry of Education, the Natural Science Foundation of Guangdong Province (Grant No. 2023A1515012914), Basic Research Fund of Shenzhen City (Grant No. JCYJ20210324120012033 and JSGG20210802154402007), the Major Key Project of PCL for Experiments and Applications (PCL2021A06), and Overseas Cooperation Research Fund of Tsinghua Shenzhen International Graduate School (HW2021008).
OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. doi: 10.48550/arXiv.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023. doi: 10.48550/arXiv.2302.13971. URL https://doi.org/10.48550/arXiv.2302.13971.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and HsuanTien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/bit/4c5bcfec8584af0d967f1ab10179ca4b-AAbstract.html.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-LM improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022a. URL https://openreview.net/forum?id=3s9IrEsjLyk.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. CoRR, abs/2210.08933, 2022. doi: 10.48550/arXiv.2210.08933. URL https://doi.org/10.48550/arXiv.2210.08933.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8857-8868. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/rasul21a.html.
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-2012. URL https://aclanthology.org/D18-2012.
Calvin Luo. Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970, 2022.
Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, and Tie-Yan Liu. A survey on non-autoregressive generation for neural machine translation and beyond. CoRR, abs/2204.09269, 2022.
Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. CoRR, abs/1610.02424, 2016.
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Typical decoding for natural language generation. CoRR, abs/2202.00666, 2022.
Angela Fan, Mike Lewis, and Yann N. Dauphin. Hierarchical neural story generation. In ACL (1), pages 889-898. Association for Computational Linguistics, 2018.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR. OpenReview.net, 2020.
A Limitation
A primary limitation of our work lies in the requirement of generating a large number of candidate samples for optimal performance. As an illustration in Table 2 of CNN/DAILYMAIL dataset, AR-DIFFUSION  $(k = 50)$  achieves a 0.8 lower ROUGE-2 score compared to AR-DIFFUSION  $(k = 500)$ . We anticipate exploring more efficient sampling strategies to minimize the number of generated samples without performance drop.
Model Setup Our model configuration is implemented based on Transformer-base [Vaswani et al., 2017]. In particular, For XSUM and CNN/DAILYMAIL, we set the diffusion embedding dimension to 128. For IWSLT14, we use 64-dimensional diffusion embedding, 4 attention heads and 1024-dimensional feed-forward layers. For COMMONGEN, we adopt 64-dimensional diffusion embedding, 8 attention heads and 512-dimensional feed-forward layers.
Training and Inference In the training phase, we employ a square-root noise schedule and 2,000 diffusion steps [Li et al., 2022a]. Specially, we use the tokenizer and vocabulary constructed by Byte Pair Encoding (BPE) $^{14}$  [Kudo and Richardson, 2018] for translation tasks. For other tasks, we adopt the tokenizer and vocabulary of bert-base-uncased.
Baselines We set four groups of baselines:
- NAR: NAT [Gu et al., 2017], iNAT [Lee et al., 2018], CMLM [Ghazvininejad et al., 2019], LevT [Gu et al., 2019] and CNAT [Bao et al., 2021];
- Semi-NAR: InsT [Stern et al., 2019], iNAT [Lee et al., 2018], CMLM [Ghazvininejad et al., 2019] and LevT [Gu et al., 2019];
- AR: bRNN [Gu et al., 2016], LSTM [Greff et al., 2017] and Transformer [Vaswani et al., 2017];
- Diffusion: DiffusionLM [Li et al., 2022a], CDCD [Dieleman et al., 2022], SeqDiffuSeq [Yuan et al., 2022], DINOISER [Ye et al., 2023] and GENIE [Lin et al., 2023].
Metrics We follow the approach of Qi et al.  $[2020]^{15}$  to evaluate the ROUGE-1/2/L of the summarization task. For the evaluation of translation tasks, we adopt the setting of SeqDiffuSeq [Yuan et al., 2022] to report BLEU score. In addition, we also calculate the SacreBLEU score according to the setting of DINOISER [Ye et al., 2023] for comparison. For COMMONGEN, we employ ROUGE-2/L, BLEU-3/4, METEOR and SPICE under the evaluation methods of Lin et al.  $[2020]^{16}$ .
Table 7: Training Parameter Settings. Batch Size = mini batch size ×  $N_{gc} \times$  GPU number, Optimized Steps = total steps /  $N_{gc}$ , and  $N_{gc}$  is gradient accumulation number.
Dataset	Lr & Schedule	Batch Size	Optimized Steps	Target Length
XSUM	8e-4 & Cosine	128×3×8	80,000 / 3	50
CNN/DAILYMAIL	8e-4 & Cosine	80×5×8	100,000 / 5	180
IWSLT14 DE→EN	2e-3 & Cosine	192×2×8	160,000 / 2	90
IWSLT14 EN→DE	1.8e-3 & Cosine	768×1×8	60,000	90
COMMONGEN	3e-4 & Constant	384×1×8	40,000	54
Training Parameters Our training parameters on different datasets are shown in Table 7. Our linear schedule warm up steps is  $4,000 \times N_{gc}$ , where  $N_{gc}$  denotes gradient accumulation number. In addition, we use the AdamW (weight decay = 0.0) optimizer and dropout is 0.2. All experiments are implemented on 8 Tesla V100-32G. It takes about 20 hours to train XSUM and CNN/DAILYMAIL, about 5 hours to train IWSLT14, and about 2 hours to train COMMENGEN.
<sup>14</sup>We train bpe on the training set, and follow the vocabulary size of fairseq, IWSLT14 is set to 10,000.
<sup>15</sup>https://github.com/microsoft/ProphetNet/tree/master/GLGE;baseline
$^{16}$ https://github.com/INK-UGC/CommonGen/tree/master/evaluation/Traditional/eval_metric
C More Results on IWSLT14
In Table 8 we give the SacreBLEU score according to the setting of DINOISER. AR-DIFFUSION has notable improvements over non-auto-regressive CMLM. Besides, AR-DIFFUSION achieves excellent performance among text diffusion models for both EN→DE and DE→EN tasks. Specifically, AR-DIFFUSION is far superior to GENIE and comparable to the newly proposed DINOISER at  $n = 50$ . Nevertheless, the performance is stronger than DINOISER when  $k = 500^{17}$ .
Table 8: SacreBLEU on the IWSLT14 test set. This result follows the setting of DINOISER.
Methods	IWSLT14
DE→EN	EN→DE
Transformer (AR, beam = 5) [Vaswani et al., 2017]	33.61	28.30
CMLM (NAR, k = 5) [Ghazvininejad et al., 2019]	29.41	24.34
DiffusionLM (k = 50) [Li et al., 2022a]	29.11	22.91
DINOISER (k = 50) [Ye et al., 2023]	31.61	26.14
GENIE (k = 50) [Lin et al., 2023]	29.45	23.89
AR-DIFFUSION (k = 50)	31.80	26.01
AR-DIFFUSION (k = 500)	32.35	26.51
D Comparison of Runtime Speed
We also compare the runtime speeds of GENIE and AR-DIFFUSION by measuring the time taken for each sample generation, denoted as seconds per iteration (s/it). To ensure precision in our measurements, we sample 50 cases from the CNN/DAILYMAIL dataset. The generation process speed comprises two main components: function evaluation (FE, which is equivalent to the model forward computation) and MBR. All experiments are conducted on an A100-40G GPU and 50 CPUs.
The results are presented in Table 9. It is evident that for  $k \leq 50$ , the MBR duration is minimal and can be negligible relative to NFE. Furthermore, even when  $k = 500$  and step  $= 20$ , AR-DIFFUSION remains approximately twice as efficient as GENIE ( $k = 10$  and step  $= 2000$ ).
Table 9: Comparison of inference speed between GENIE and AR-DIFFUSION.
GENIE	AR-DIFFUSION
k	10	10	10	50	500
Steps	2000	3	20	20	20
Speed of NFE (s/it)	47.54	0.25	0.61	2.12	21.03
Speed of MBR (s/it)	0.02	0.02	0.02	0.08	7.14
Total Speed (s/it)	47.56	0.27	0.63	2.20	28.17
ROUGE-2 in XSUM	8.78	8.68	9.32	10.1	10.6
E Impact of Minimum Bayes Risk and Anchor Point
Minimum Bayes Risk To investigate the relationship between the number of generated candidate samples  $(k)$  and the quality of generation, we generate varying numbers of samples, ranging up to 1,000, on the IWSLT14 De→En test set and present the results in Figure 4. The curve demonstrates an initial gain of approximately 0.5 SacreBLEU within the first 200 samples, after which the gain becomes insignificant with generating more samples.
Anchor Point We conduct experiments on AR-DIFFUSION using different anchor points  $(n_e, t_e)$ . These anchor points vary in terms of  $n_e$  values, namely  $1.0 \times N$ ,  $2.0 \times N$  and  $3.0 \times N$ , where  $N$  denotes the target sentence length. Additionally, they share a common  $t_e$  value of  $T$ , which represents
$^{17}$ DINOISER has shown in their Figure 4 that their method is not better with a larger  $k$ .
the total time step of diffusion. We present the results in Table 10, and determine that the best result is achieved at  $(n_e, t_e) = (2.0 \times N, T)$ .
Figure 4: Relationship between the number of candidate samples for applying MBR and SacreBLEU on IWSLT14 DE  $\rightarrow$  EN test set.
Table 10: Effect of anchor point at different positions on the IWSLT14 DE  $\rightarrow$  EN test set. "N" indicates the target sequence length and "T" represents the total time step of diffusion.
ne	te	SacreBLEU
1.0 × N	T	31.23
2.0 × N	T	31.80
3.0 × N	T	31.58
F Proof of Inference with Skipping
During the inference process, skipping strategy requires the model  $g_{\theta}$  to infer the state  $z_{t_{i+1}}^{n_2}$  at a far-off timestep  $t_{i+1}$  compared to the current state  $z_{t_i}^{n_2}$ , where  $t_{i+1} \ll t_i$ . In our model, due to the dynamic speed setting, token  $z_{t_{i+1}}^{n_1}$  with smaller timestep  $t_{i+1} \leq t_i$ , which is closer to  $t_{i+1}$ , and positions  $n_1 \leq n_2$  can provide stronger auxiliary information than  $z_{t_i}^{n_1}$ . This reduces the difficulty of inferring states for tokens in the end, making our multi-level diffusion model particularly suitable for accelerating the generation process.
Through maximizing the evidence lower bound (ELBO) of  $p(z_0)$ , the training object is equivalent to minimize the divergence between  $q(z_t | z_{t-1}, z_0)$  and  $p_\theta(z_{t-1} | z_t)$  following [Luo, 2022].
By converting the joint probability distribution into a conditional probability distribution, we obtain the following formula for  $q(\pmb{z}_{t_{i+1}} | \pmb{z}_{t_i}, \pmb{z}_0)$ .
$$
\begin{array}{l} q (\boldsymbol {z} _ {t _ {i + 1}} | \boldsymbol {z} _ {t _ {i}}, \boldsymbol {z} _ {0}) = q (\boldsymbol {z} _ {t _ {i + 1}} | \boldsymbol {z} _ {t _ {i - 1}}, \boldsymbol {z} _ {t _ {i}}, \boldsymbol {z} _ {0}) q (\boldsymbol {z} _ {t _ {i + 1} - 1} | \boldsymbol {z} _ {t _ {i}}, \boldsymbol {z} _ {0}) \\ = q \left(\boldsymbol {z} _ {t _ {i + 1}} \mid \boldsymbol {z} _ {t _ {i} - 1}, \boldsymbol {z} _ {0}\right) q \left(\boldsymbol {z} _ {t _ {i + 1} - 1} \mid \boldsymbol {z} _ {t _ {i}}, \boldsymbol {z} _ {0}\right) \\ = q \left(\boldsymbol {z} _ {t _ {i + 1}} \mid \boldsymbol {z} _ {t _ {i} - 2}, \boldsymbol {z} _ {0}\right) q \left(\boldsymbol {z} _ {t _ {i + 1} - 2} \mid \boldsymbol {z} _ {t _ {i} - 1}, \boldsymbol {z} _ {0}\right) q \left(\boldsymbol {z} _ {t _ {i + 1} - 1} \mid \boldsymbol {z} _ {t _ {i}}, \boldsymbol {z} _ {0}\right) \tag {13} \\ t _ {i} - t _ {i + 1} \\ = \prod_ {k = 1} q \left(\boldsymbol {z} _ {t _ {i} - k} \mid \boldsymbol {z} _ {t _ {i} - k + 1}, \boldsymbol {z} _ {0}\right) \\ \end{array}
$$
Similarly, we reach the same conclusion regarding  $p_{\theta}(\mathbf{z}_{t_{i + 1}}|\mathbf{z}_{t_i})$