SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Based on equation (13), which consists of  $q(\pmb{z}_t|\pmb{z}_{t-1},\pmb{z}_0)$ , and the interchangeability between  $q(\pmb{z}_t|\pmb{z}_{t-1},\pmb{z}_0)$  and  $p_\theta(\pmb{z}_{t-1}|\pmb{z}_t)$ , we can decompose  $q(\pmb{z}_{t_{i+1}}|\pmb{z}_{t_i},\pmb{z}_0)$  by incorporating  $\pmb{z}_{t_i}$  and  $\pmb{z}_0$ , and utilize our estimated  $\pmb{z}_0$  to determine the expression of  $p_\theta(\pmb{z}_{t_{i+1}}|\pmb{z}_{t_i})$ .
$$
q \left(\boldsymbol {z} _ {t _ {i + 1}} \mid \boldsymbol {z} _ {t _ {i}}, \boldsymbol {z} _ {0}\right) = \prod_ {n = 1} ^ {N} q \left(z _ {f (n, t _ {i + 1})} ^ {n} \mid z _ {f (n, t _ {i})} ^ {n}, z _ {0} ^ {n}\right) \tag {14}
$$
Next, we obtain the explicit expression  $q\big(z_{f(n,t_i + 1)}^n\mid z_{f(n,t_i)}^n,z_0^n\big)$  through linear interpolation between  $z_{f(n,t_i)}^n$  and  $z_0^n$ .
$$
\begin{array}{l} q \big (z _ {f (n, t _ {i + 1})} ^ {n} \mid z _ {f (n, t _ {i})} ^ {n}, z _ {0} ^ {n} \big) = \frac {q (z _ {f (n , t _ {i})} ^ {n} \mid z _ {f (n , t _ {i + 1})} ^ {n} , z _ {0} ^ {n}) q (z _ {f (n , t _ {i + 1})} ^ {n} \mid z _ {0} ^ {n})}{q (z _ {f (n , t _ {i})} ^ {n} \mid z _ {0} ^ {n})} \\ = \frac {\mathcal {N} \left(z _ {f (n , t _ {i})} ^ {n} ; \sqrt {\frac {\bar {\alpha} _ {f (n , t _ {i})}}{\bar {\alpha} _ {f (n , t _ {i + 1})}}} z _ {f (n , t _ {i + 1})} ^ {n} , \left(1 - \frac {\bar {\alpha} _ {f (n , t _ {i})}}{\bar {\alpha} _ {f (n , t _ {i + 1})}}\right) I\right) \mathcal {N} \left(z _ {f (n , t _ {i + 1})} ^ {n} ; \sqrt {\bar {\alpha} _ {f (n , t _ {i + 1})}} z _ {0} ^ {n} , \left(1 - \bar {\alpha} _ {f (n , t _ {i + 1})}\right) I\right)}{\mathcal {N} \left(z _ {f (n , t _ {i})} ^ {n} ; \sqrt {\bar {\alpha} _ {f (n , t _ {i})}} z _ {0} ^ {n} , \left(1 - \bar {\alpha} _ {t _ {i}}\right) I\right)} \\ \propto \exp \Bigg \{- \frac {\left(z _ {f (n , t _ {i})} ^ {n} - \sqrt {\frac {\bar {\alpha} _ {f (n , t _ {i})}}{\bar {\alpha} _ {f (n , t _ {i + 1})}}} z _ {f (n , t _ {i + 1})} ^ {n}\right) ^ {2}}{2 \left(1 - \frac {\bar {\alpha} _ {f (n , t _ {i})}}{\bar {\alpha} _ {f (n , t _ {i + 1})}}\right)} - \frac {\left(z _ {f (n , t _ {i + 1})} ^ {n} - \sqrt {\bar {\alpha} _ {t _ {i + 1}}} z _ {0} ^ {n}\right) ^ {2}}{1 - \bar {\alpha} _ {f (n , t _ {i + 1})}} \\ \left. + \frac {\left(z _ {f (n , t _ {i})} ^ {n} - \sqrt {\bar {\alpha} _ {f (n , t _ {i})}} z _ {0} ^ {n}\right) ^ {2}}{1 - \bar {\alpha} _ {f (n , t _ {i})}} \right\} \\ = \exp \left\{ \right.- \frac {1 - \bar {\alpha} _ {t _ {i}}}{2 \left(1 - \frac {\bar {\alpha} _ {t _ {i}}}{\bar {\alpha} _ {t _ {i + 1}}}\right)\left(1 - \bar {\alpha} _ {t _ {i + 1}}\right)} \left[ z _ {f (n, t _ {i + 1})} ^ {n} \right. ^ {2} - 2 \left(\frac {\sqrt {\frac {\bar {\alpha} _ {f (n , t _ {i})}}{\bar {\alpha} _ {f (n , t _ {i + 1})}}} \left(1 - \bar {\alpha} _ {f (n, t _ {i + 1})}\right)}{1 - \bar {\alpha} _ {f (n , t _ {i})}} z _ {f (n, t _ {i})} ^ {n} \right. \\ \left. \left. + \frac {\sqrt {\bar {\alpha} _ {f (n , t _ {i + 1})}} \left(1 - \frac {\bar {\alpha} _ {f (n , t _ {i})}}{\bar {\alpha} _ {f (n , t _ {i + 1})}}\right)}{1 - \bar {\alpha} _ {f (n , t _ {i})}} z _ {0} ^ {n}\right) z _ {f (n, t _ {i + 1})} ^ {n} \right] \Bigg \} \\ \propto \mathcal {N} \left( \right.z _ {f (n, t _ {i + 1})} ^ {n}; \frac {\sqrt {\frac {\bar {\alpha} _ {f (n , t _ {i})}}{\bar {\alpha} _ {f (n , t _ {i + 1})}}} (1 - \bar {\alpha} _ {f (n, t _ {i + 1})})}{1 - \bar {\alpha} _ {f (n, t _ {i})}} z _ {f (n, t _ {i})} ^ {n} + \frac {\sqrt {\bar {\alpha} _ {f (n , t _ {i + 1})}} (1 - \frac {\bar {\alpha} _ {f (n , t _ {i})}}{\bar {\alpha} _ {f (n , t _ {i + 1})}})}{1 - \bar {\alpha} _ {f (n, t _ {i})}} z _ {0} ^ {n}, \\ \left. \frac {\left(1 - \frac {\bar {\alpha} _ {t _ {i}}}{\bar {\alpha} _ {t _ {i + 1}}}\right) \left(1 - \bar {\alpha} _ {t _ {i + 1}}\right)}{1 - \bar {\alpha} _ {t _ {i}}} I\right) \\ = \mathcal {N} \left(z _ {f \left(n, t _ {i + 1}\right)} ^ {n}; \lambda z _ {f \left(n, t _ {i}\right)} ^ {n} + \mu z _ {0} ^ {n}, \sigma I\right) \tag {15} \\ \end{array}
$$
where we have the following notations for simplification.
$$
\lambda = \frac {\sqrt {\frac {\bar {\alpha} _ {f (n , t _ {i})}}{\bar {\alpha} _ {f (n , t _ {i + 1})}}} (1 - \bar {\alpha} _ {f (n , t _ {i + 1})})}{1 - \bar {\alpha} _ {f (n , t _ {i})}}, \mu = \frac {\sqrt {\bar {\alpha} _ {f (n , t _ {i + 1})}} (1 - \frac {\bar {\alpha} _ {f (n , t _ {i})}}{\bar {\alpha} _ {f (n , t _ {i + 1})}})}{1 - \bar {\alpha} _ {f (n , t _ {i})}}, \sigma = \frac {(1 - \alpha_ {f (n , t _ {i})}) (1 - \bar {\alpha} _ {f (n , t _ {i + 1})})}{1 - \bar {\alpha} _ {f (n , t _ {i})}}
$$
Building upon equation (15), we substitute  $z_0^n$  with  $g_{\theta}(z_{f(n,t)}^n, f(n,t); \boldsymbol{x})$ , yielding the final formula for  $p_{\theta}\left(z_{f(n,t_i+1)}^n \mid z_{f(n,t_i)}^n; \boldsymbol{x}\right)$  as the following equation.
$$
p _ {\theta} \left(\boldsymbol {z} _ {f (n, t _ {i + 1})} ^ {n} \mid \boldsymbol {z} _ {f (n, t _ {i})} ^ {n}; \boldsymbol {x}\right) \sim \mathcal {N} \left(\boldsymbol {z} _ {f (n, t _ {i + 1})} ^ {n}; \lambda \boldsymbol {z} _ {f (n, t _ {i})} ^ {n} + \mu \boldsymbol {g} _ {\theta} \left(\boldsymbol {z} _ {f (n, t)} ^ {n}, f (n, t); \boldsymbol {x}\right), \sigma \mathbf {I}\right) \tag {16}
$$
G Different Sampling Methods of BART
The sampling methods used by Lin et al. [2023] including Greedy Search, Beam Search Xiao et al. [2022], Diverse Beam Search(diversity strength  $= 0.8$ ) Vijayakumar et al. [2016], Typical Sample  $(\tau = 1.2)$  Meister et al. [2022], Top-k Sample  $(k = 50)$  Fan et al. [2018] and Nucleus Sample  $(p = 0.92)$  Holtzman et al. [2020].
Specifically, greedy search is to select the token with the highest probability at each step. Beam search is to select the largest token from among the beams with higher probability at each step. Diverse beam search is to divide the beams into multiple groups at each step and ensure the difference between groups by calculating the diversity score between groups. Typical sampling selects samples through a discrete random process. Top-k sampling is to randomly select one of the  $k$  candidate tokens with the highest probability at each step. Nucleus sampling is to randomly select one token at each step from the candidate tokens whose probability density is greater than  $p$ .
H More Cases
##ft hmm ` northern hacker support by yells lion on [unused698] tennis bars named! ##s 1898 1682α limp
a british soldier who was killed by an army in 43 lifted losers requested ##ged a prosecutor of verbal of rogers #.
a british soldier who was killed by an army in intersect has been named upon a prosecutor of deep ofise process.
a british soldier who was killed by an army in ash has been named by the ministry of crucial of championships.
a british soldier who was killed by an army in afghanistan has been named by the ministry of defence of moddra.
proxy oxidation ##h ##r o l ##ilaonate ##rwinpuri saskatoon amplitude 1702 ## the ##rialstraße ##: barclona
a plaque for spain 's first ##alis war amid been be in the historic fleet of serummik say, the ##l of 2003 dso g fee. ##p
a plaque for spain 's first world war has been added in the century city of liverpoal say, the certain of britain in g fee.
a plaque for spain 's first world war has been be in the historic city of liverpoal say, the amazing of serum in g post.
a plaque for spain 's first world war has been unveiled in the historic city of liverpoal , the first of britain in the years .
```c
##x angelica6 breach ##nostic [unused557] fires sale why organizational interception originates #inial ##pa knots
china 's prime minister creators it concerning the " 800 ## beach ' 1728 withdrawal [unused308] ## attending
china 's prime minister says it is the "ity emissions " in the country 's [unused697] foundation , the engine of " .
china 's prime minister says it is the " emissions " in the country 's tq crisis , the engine of " . # [unused887]#
china 's prime minister says it is the " emissions " in the country 's economic crisis , the engine of parliament .