SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yanyu Li $^{1,2,\dagger}$  Huan Wang $^{1,2,\dagger}$  Qing Jin $^{1,\dagger}$  Ju Hu $^{1}$  Pavlo Chemerys $^{1}$  Yun Fu $^{2}$  Yanzhi Wang $^{2}$  Sergey Tulyakov $^{1}$  Jian Ren $^{1,\dagger}$  1Snap Inc. 2Northeastern University Project Page: https://snap-research.github.io/SnapFusion
Figure 1: Example generated images by using our efficient text-to-image diffusion model.
Text-to-image diffusion models can create stunning images from natural language descriptions that rival the work of professional artists and photographers. However, these models are large, with complex network architectures and tens of denoising iterations, making them computationally expensive and slow to run. As a result, high-end GPUs and cloud-based inference are required to run diffusion models at scale. This is costly and has privacy implications, especially when user data is sent to a third party. To overcome these challenges, we present a generic approach that, for the first time, unlocks text-to-image diffusion models on mobile devices in less than 2 seconds. We achieve so by introducing efficient network architecture and improving step distillation. Specifically, we propose an efficient UNet by identifying the redundancy of the original model and reducing the computation of the image decoder via data distillation. Further, we enhance the step distillation by exploring training strategies and introducing regularization from classifier-free guidance. Our extensive experiments on MS-COCO show that our model with 8 denoising steps achieves better FID and CLIP scores than Stable Diffusion v1.5 with 50 steps. Our work democratizes content creation by bringing powerful text-to-image diffusion models to the hands of users.
Diffusion-based text-to-image models [1, 2, 3, 4] show remarkable progress in synthesizing photorealistic content using text prompts. They profoundly impact the content creation [5, 6], image
editing and in-painting [7, 8, 9, 10, 11], super-resolution [12], video synthesis [13, 14], and 3D assets generation [15, 16, 17], to name a few. This impact comes at the cost of the substantial increase in the computation requirements to run such models [18, 19, 20, 21]. As a result, to satisfy the necessary latency constraints large scale, often cloud-based inference platforms with high-end GPU are required. This incurs high costs and brings potential privacy concerns, motivated by the sheer fact of sending private images, videos, and prompts to a third-party service.
Not surprisingly, there are emerging efforts to speed up the inference of text-to-image diffusion models on mobile devices. Recent works use quantization [22, 23] or GPU-aware optimization to reduce the run time, i.e., accelerating the diffusion pipeline to 11.5s on Samsung Galaxy S23 Ultra [24]. While these methods effectively achieve a certain speed-up on mobile platforms, the obtained latency does not allow for a seamless user experience. Besides, none of the existing studies systematically examine the generation quality of on-device models through quantitative analysis.
In this work, we present the first text-to-image diffusion model that generates an image on mobile devices in less than 2 seconds. To achieve this, we mainly focus on improving the slow inference speed of the UNet and reducing the number of necessary denoising steps. First, the architecture of UNet, which is the major bottleneck for the conditional diffusion model (as we show in Tab. 1), is rarely optimized in the literature. Existing works primarily focus on post-training optimizations [25, 26]. Conventional compression techniques, e.g., model pruning [27, 28] and architecture search [29, 30], reduce the performance of pre-trained diffusion models [31], which is difficult to recover without heavy fine-tuning. Consequently, the architecture redundancies are not fully exploited, resulting in a limited acceleration ratio. Second, the flexibility of the denoising diffusion process is not well explored for the on-device model. Directly reducing the number of denoising steps impacts the generative performance, while progressively distilling the steps can mitigate the impacts [32, 33]. However, the learning objectives for step distillation and the strategy for training the on-device model have yet to be thoroughly studied, especially for models trained using large-scale datasets.
This work proposes a series of contributions to address the aforementioned challenges:
- We provide an in-depth analysis of the denoising UNet and identify the architecture redundancies.
- We propose a novel evolving training framework to obtain an efficient UNet that performs better than the original Stable Diffusion v1.5<sup>1</sup> while being significantly faster. We also introduce a data distillation pipeline to compress and accelerate the image decoder.
- We improve the learning objective during step distillation by proposing additional regularization, including losses from the  $\mathbf{v}$ -prediction and classifier-free guidance [34].
- Finally, we explore the training strategies for step distillation, especially the best teacher-student paradigm for training the on-device model.
Through the improved Step distillation and network architecture development for the diffFusion model, our introduced model, SnapFusion, generates a  $512 \times 512$  image from the text on mobile devices in less than 2 seconds, while with image quality similar to Stable Diffusion v1.5 [4] (see example images from our approach in Fig. 1).
2 Model Analysis of Stable Diffusion
2.1 Prerequisites of Stable Diffusion
Diffusion Models gradually convert the sample  $\mathbf{x}$  from a real data distribution  $p_{\mathrm{data}}(\mathbf{x})$  into a noisy version, i.e., the diffusion process, and learn to reverse this process by denoising the noisy data step by step [35]. Therefore, the model transforms a simple distribution, e.g., random Gaussian noise, to the desired more complicated distribution, e.g., real images. Specifically, given a (noise-prediction) diffusion model  $\hat{\epsilon}_{\theta}(\cdot)$  parameterized by  $\theta$ , which is typically structured as a UNet [36, 1], the training can be formulated as the following noise prediction problem [35, 1, 2]:
$$
\min  _ {\boldsymbol {\theta}} \mathbb {E} _ {t \sim U [ 0, 1 ], \mathbf {x} \sim p _ {\text {d a t a}} (\mathbf {x}), \boldsymbol {\epsilon} \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} \| \hat {\epsilon} _ {\boldsymbol {\theta}} (t, \mathbf {z} _ {t}) - \boldsymbol {\epsilon} \| _ {2} ^ {2}, \tag {1}
$$
where  $t$  refers to the time step;  $\epsilon$  is the ground-truth noise;  $\mathbf{z}_t = \alpha_t\mathbf{x} + \sigma_t\epsilon$  is the noisy data;  $\alpha_{t}$  and  $\sigma_{t}$  are the strengths of signal and noise, respectively, decided by a noise scheduler. A trained
<https://github.com/runwayml/stable-diffusion>
diffusion model can generate samples from noise with various samplers. In our experiments, we use DDIM [37] to sample with the following iterative denoising process from  $t$  to a previous time step  $t'$ ,
$$
\mathbf {z} _ {t ^ {\prime}} = \alpha_ {t ^ {\prime}} \frac {\mathbf {z} _ {t} - \sigma_ {t} \hat {\epsilon} _ {\boldsymbol {\theta}} (t , \mathbf {z} _ {t})}{\alpha_ {t}} + \sigma_ {t ^ {\prime}} \hat {\epsilon} _ {\boldsymbol {\theta}} (t, \mathbf {z} _ {t}), \tag {2}
$$
where  $\mathbf{z}_{t'}$  will be fed into  $\hat{\epsilon}_{\pmb{\theta}}(\cdot)$  again until  $t'$  becomes 0, i.e., the denoising process finishes.
Latent Diffusion Model / Stable Diffusion. The recent latent diffusion model (LDM) [4] reduces the inference computation and steps by performing the denoising process in the latent space, which is encoded from a pre-trained variational autoencoder (VAE) [38, 39]. During inference, the image is constructed through the decoder from the latent. LDM also explores the text-to-image generation, where a text prompt embedding  $\mathbf{c}$  is fed into the diffusion model as the condition. When synthesizing images, an important technique, classifier-free guidance (CFG) [34], is adopted to improve quality,
$$
\tilde {\boldsymbol {\epsilon}} _ {\boldsymbol {\theta}} (t, \mathbf {z} _ {t}, \mathbf {c}) = w \hat {\boldsymbol {\epsilon}} _ {\boldsymbol {\theta}} (t, \mathbf {z} _ {t}, \mathbf {c}) - (w - 1) \hat {\boldsymbol {\epsilon}} _ {\boldsymbol {\theta}} (t, \mathbf {z} _ {t}, \varnothing), \tag {3}
$$
where  $\hat{\epsilon}_{\theta}(t,\mathbf{z}_t,\varnothing)$  represents the unconditional output obtained by using null text  $\varnothing$ . The guidance scale  $w$  can be adjusted to control the strength of conditional information on the generated images to achieve the trade-off between quality and diversity. LDM is further trained on large-scale datasets [40], delivering a series of Stable Diffusion (SD) models [4]. We choose Stable Diffusion v1.5 (SD-v1.5) as the baseline. Next, we perform detailed analyses to diagnose the latency bottleneck of SD-v1.5.
2.2 Benchmark and Analysis
Here we comprehensively study the parameter and computation intensity of the SD-v1.5. The in-depth analysis helps us understand the bottleneck to deploying text-to-image diffusion models on mobile devices from the scope of network architecture and algorithm paradigms. Meanwhile, the micro-level breakdown of the networks serves as the basis of the architecture redesign and search.
Table 1: Latency Comparison between Stable Diffusion v1.5 and our proposed efficient diffusion models (UNet and Image Decoder) on iPhone 14 Pro.
Stable Diffusion v1.5	Text Encoder	UNet	VAE Decoder
Input Resolution	77 tokens	64 × 64	64 × 64
#Parameters (M)	123	860	50
Latency (ms)	4	~1,7002	369
Inference Steps	2	50	1
Total Latency (ms)	8	85,000	369
Our Model	Text Encoder	Our UNet	Our Image Decoder
Input Resolution	77 tokens	64 × 64	64 × 64
#Parameters (M)	123	848	13
Latency (ms)	4	230	116
Inference Steps	2	8	1
Total Latency (ms)	8	1,840	116
Figure 2: Latency (iPhone 14 Pro, ms) and parameter (M) analysis for cross-attention (CA) and ResNet blocks in the UNet of Stable Diffusion.
Macro Prospective. As shown in Tab. 1 and Fig. 3, the networks of stable diffusion consist of three major components. Text encoder employs a ViT-H model [41] for converting input text prompt into embedding and is executed in two steps (with one for CFG) for each image generation process, constituting only a tiny portion of inference latency (8 ms). The VAE decoder takes the latent feature to generate an image, which runs as  $369~\mathrm{ms}$ . Unlike the above two models, the denoising UNet is not only intensive in computation (1.7 seconds latency) but also demands iterative forwarding steps to ensure generative quality. For instance, the total denoising timesteps is set to 50 for inference in SD-v1.5, significantly slowing down the on-device generation process to the minute level.
Breakdown for UNet. The time-conditional  $(t)$  UNet consists of cross-attention and ResNet blocks. Specifically, a cross-attention mechanism is employed at each stage to integrate text embedding (c) into spatial features: Cross-Attention $(Q_{\mathbf{z}_t}, K_{\mathbf{c}}, V_{\mathbf{c}}) = \text{Softmax}\left(\frac{Q_{\mathbf{z}_t} \cdot K_{\mathbf{c}}^\top}{\sqrt{d}}\right) \cdot V_{\mathbf{c}}$ , where  $Q$  is projected from noisy data  $\mathbf{z}_t$ ,  $K$  are projected from text condition, and  $d$  is the feature dimension. UNet also uses ResNet blocks to capture locality, and we can formulate the forward of UNet as:
$$
\hat {\epsilon} _ {\boldsymbol {\theta}} (t, \mathbf {z} _ {t}) = \prod \left\{\text {C r o s s - A t t e n t i o n} (\mathbf {z} _ {t}, \mathbf {c}), \operatorname {R e s N e t} (\mathbf {z} _ {t}, t) \right\}. \tag {4}
$$
2We notice the latency varies depending on the phones and use three phones to get the average speed.
Figure 3: Workflow of text-to-image diffusion model (left) and the proposed step distillation (right).
The distribution of parameters and computations of UNet is illustrated in Fig. 2, showing that parameters are concentrated on the middle (downsampled) stages because of the expanded channel dimensions, among which ResNet blocks constitute the majority. In contrast, the slowest parts of UNet are the input and output stages with the largest feature resolution, as spatial cross-attentions have quadratic computation complexity with respect to feature size (tokens).
3 Architecture Optimizations
Here we investigate the architecture redundancy of SD-v1.5 to obtain efficient neural networks. However, it is non-trivial to apply conventional pruning [42, 43, 44, 45] or architecture search [46, 47, 30] techniques, given the tremendous training cost of SD. Any permutation in architecture may lead to degraded performance that requires fine-tuning with hundreds or thousands of GPUs days. Therefore, we propose an architecture-evolving method that preserves the performance of the pre-trained UNet model while gradually improving its efficacy. As for the deterministic image decoder, we apply tailored compression strategies and a simple yet effective prompt-driven distillation approach.
3.1 Efficient UNet
From our empirical observation, the operator changes resulting from network pruning or searching lead to degraded synthesized images, asking for significant training costs to recover the performance. Thus, we propose a robust training, and evaluation and evolving pipeline to alleviate the issue.
Robust Training. Inspired by the idea of elastic depth [48, 49], we apply stochastic forward propagation to execute each cross-attention and ResNet block by probability  $p(\cdot, I)$ , where  $I$  refers to identity mapping that skips the corresponding block. Thus, we have Eq. (4) becomes as follows:
$$
\hat {\epsilon} _ {\boldsymbol {\theta}} (t, \mathbf {z} _ {t}) = \prod \left\{p \left(\text {C r o s s - A t t e n t i o n} (\mathbf {z} _ {t}, \mathbf {c}), I\right), p \left(\text {R e s N e t} (\mathbf {z} _ {t}, t), I\right) \right\}. \tag {5}
$$
With this training augmentation, the network is robust to architecture permutations, which enables an accurate assessment of each block and a stable architectural evolution (more examples in Fig. 5).
Evaluation and Architecture Evolving. We perform online network changes of UNet using the model from robust training with the constructed evolution action set:  $A \in \{A_{Cross-Attention[i,j]}^{+, - }, A_{ResNet[i,j]}^{+, - }\}$ , where  $A^{+, - }$  denotes the action to remove  $(-)$  or add  $(+)$  a cross-attention or ResNet block at the corresponding position (stage  $i$ , block  $j$ ). Each action is evaluated by its impact on execution latency and generative performance. For latency, we use the lookup table built in Sec. 2.2 for each possible configuration of cross-attention and ResNet blocks. Note we improve the UNet for on-device speed; the optimization of model size can be performed similarly and is left as future work. For generative performance, we choose CLIP score [41] to measure the correlation between generated images and the text condition. We use a small subset (2K images) of MS-COCO validation set [50], fixed steps (50), and CFG scale as 7.5 to benchmark the score, and it takes about 2.5 A100 GPU hours to test each action. For simplicity, the value score of each action is defined as  $\frac{\Delta CLIP}{\Delta Latency}$ , where a block with lower latency and higher contribution to CLIP tends to be preserved, and the opposite is removed in
architecture evolving (more details in Alg. 1). To further reduce the cost for network optimization, we perform architecture evolving, i.e., removing redundant blocks or adding extra blocks at valuable positions by executing a group of actions at a time. Our training paradigm successfully preserves the performance of pre-trained UNet while tolerating large network permutations (Fig. 5). The details of our final architecture is presented in Sec. A.
3.2 Efficient Image Decoder
For the image decoder, we propose a distillation pipeline that uses synthetic data to learn the efficient image decoder obtained via channel reduction, which has  $3.8 \times$  fewer parameters and is  $3.2 \times$  faster than the one from SD-v1.5. The efficient image decoder is obtained by applying  $50\%$  uniform channel pruning to the original image decoder, resulting in a compressed efficient image decoder with approximately  $1/4$  size and MACs of the original one. Here we only train the efficient decoder instead of following the training of VAE [4, 38, 39] that also learns the image encoder. We use text prompts to get the latent representation from the UNet of SD-v1.5 after 50 denoising steps with DDIM and forward it to our efficient image decoder and the one of SD-v1.5 to generate two images. We then optimize the decoder by minimizing the mean squared error between the two images. Using synthetic data for distillation brings the advantage of augmenting the dataset on-the-fly where each prompt be used to obtain unlimited images by sampling various noises. Quantitative analysis of the compressed decoder can be found in Sec. B.2.
Algorithm 1 Optimizing UNet Architecture   
Require:UNet:  $\hat{\epsilon}_{\theta}$  ; validation set:  $\mathbb{D}_{val}$  latency lookup table  $\mathbb{T}$  : {Cross-Attention[i,j],ResNet[i,j]}.   
Ensure:  $\hat{\epsilon}_{\theta}$  converges and satisfies latency objective  $S$  while  $\hat{\epsilon}_{\theta}$  not converged do Perform robust training.  $\rightarrow$  Architecture optimization: if perform architecture evolving at this iteration then  $\rightarrow$  Evaluate blocks: for each block[i,j] do  $\Delta CLIP\gets \mathrm{eval}(\hat{\epsilon}_{\theta},A_{block[i,j]}^{-},\mathbb{D}_{val}),$ $\Delta Latency\leftarrow \mathrm{eval}(\hat{\epsilon}_{\theta},A_{block[i,j]}^{-},\mathbb{T})$  end for  $\rightarrow$  Sort actions based on  $\frac{\Delta CLIP}{\Delta Latency}$  , execute action, and evolve architecture to get latency  $T$  .. if latency objective  $S$  is not satisfied then  $\{\hat{A}^{-}\} \gets \arg \min_{A^{-}}\frac{\Delta CLIP}{\Delta Latency},$  else  $\{\hat{A}^{+}\} \gets \mathrm{copy}(\arg \max_{A^{-}}\frac{\Delta CLIP}{\Delta Latency}),$ $\hat{\epsilon}_{\theta}\gets \mathrm{evolve}(\hat{\epsilon}_{\theta},\{\hat{A}\})$  end if end if end while
4 Step Distillation
Besides proposing the efficient architecture of the diffusion model, we further consider reducing the number of iterative denoising steps for UNet to achieve more speedup. We follow the research direction of step distillation [33], where the inference steps are reduced by distilling the teacher, e.g., at 32 steps, to a student that runs at fewer steps, e.g., 16 steps. This way, the student enjoys  $2 \times$  speedup against the teacher. Here we employ different distillation pipelines and learning objectives from existing works [33, 32] to improve the image quality, which we elaborate on as follows.
4.1 Overview of Distillation Pipeline
Citing the wisdom from previous studies [33, 32], step distillation works best with the v-prediction type, i.e., UNet outputs velocity  $\mathbf{v}$  [33] instead of the noise  $\epsilon$ . Thus, we fine-tune SD-v1.5 to v-prediction (for notation clarity, we use  $\hat{\mathbf{v}}_{\theta}$  to mean the SD model in v-prediction vs. its  $\epsilon$ -prediction counterpart  $\hat{\epsilon}_{\theta}$ ) before step distillation, with the following original loss  $\mathcal{L}_{\mathrm{ori}}$ :
$$
\mathcal {L} _ {\mathrm {o r i}} = \mathbb {E} _ {t \sim U [ 0, 1 ], \mathbf {x} \sim p _ {\text {d a t a}} (\mathbf {x}), \epsilon \sim \mathcal {N} (\mathbf {0}, \mathbf {I})} | | \hat {\mathbf {v}} _ {\boldsymbol {\theta}} (t, \mathbf {z} _ {t}, \mathbf {c}) - \mathbf {v} | | _ {2} ^ {2}, \tag {6}
$$
where  $\mathbf{v}$  is the ground-truth target velocity, which can be derived analytically from the clean latent  $\mathbf{x}$  and noise  $\epsilon$  given time step  $t$ :  $\mathbf{v} \equiv \alpha_t \epsilon - \sigma_t \mathbf{x}$ .
Our distillation pipeline includes three steps. First, we do step distillation on SD-v1.5 to obtain the UNet with 16 steps that reaches the performance of the 50-step model. Note here we use a 32-step SD-v1.5 to perform distillation directly, instead of doing it progressively, e.g., using a 128-step model as a teacher to obtain the 64-step model and redo the distillation progressively. The reason is that we empirically observe that progressive distillation is slightly worse than direct distillation (see Fig. 6(a)
for details). Second, we use the same strategy to get our 16-step efficient UNet. Finally, we use the 16-step SD-v1.5 as the teacher to conduct step distillation on the efficient UNet that is initialized from its 16-step counterpart. This will give us the 8-step efficient UNet, which is our final UNet model.
4.2 CFG-Aware Step Distillation
We introduce the vanilla step distillation loss first, then elaborate more details on our proposed CFG-aware step distillation (Fig. 3).
Vanilla Step Distillation. Given the UNet inputs, time step  $t$ , noisy latent  $\mathbf{z}_t$ , and text embedding  $\mathbf{c}$ , the teacher UNet performs two DDIM denoising steps, from time  $t$  to  $t'$  and then to  $t''$  ( $0 \leq t'' < t' < t \leq 1$ ). This process can be formulated as (see the Sec. C for detailed derivations),
$$
\begin{array}{l} \hat {\mathbf {v}} _ {t} = \hat {\mathbf {v}} _ {\boldsymbol {\theta}} (t, \mathbf {z} _ {t}, \mathbf {c}) \Rightarrow \mathbf {z} _ {t ^ {\prime}} = \alpha_ {t ^ {\prime}} \left(\alpha_ {t} \mathbf {z} _ {t} - \sigma_ {t} \hat {\mathbf {v}} _ {t}\right) + \sigma_ {t ^ {\prime}} \left(\sigma_ {t} \mathbf {z} _ {t} + \alpha_ {t} \hat {\mathbf {v}} _ {t}\right), \\ \hat {\mathbf {v}} _ {t ^ {\prime}} = \hat {\mathbf {v}} _ {\boldsymbol {\theta}} \left(t ^ {\prime}, \mathbf {z} _ {t ^ {\prime}}, \mathbf {c}\right) \Rightarrow \mathbf {z} _ {t ^ {\prime \prime}} = \alpha_ {t ^ {\prime \prime}} \left(\alpha_ {t ^ {\prime}} \mathbf {z} _ {t ^ {\prime}} - \sigma_ {t ^ {\prime}} \hat {\mathbf {v}} _ {t ^ {\prime}}\right) + \sigma_ {t ^ {\prime \prime}} \left(\sigma_ {t ^ {\prime}} \mathbf {z} _ {t ^ {\prime}} + \alpha_ {t ^ {\prime}} \hat {\mathbf {v}} _ {t ^ {\prime}}\right). \tag {7} \\ \end{array}
$$
The student UNet, parameterized by  $\pmb{\eta}$ , performs only one DDIM denoising step,
$$
\hat {\mathbf {v}} _ {t} ^ {(s)} = \hat {\mathbf {v}} _ {\boldsymbol {\eta}} (t, \mathbf {z} _ {t}, \mathbf {c}) \Rightarrow \hat {\mathbf {x}} _ {t} ^ {(s)} = \alpha_ {t} \mathbf {z} _ {t} - \sigma_ {t} \hat {\mathbf {v}} _ {t} ^ {(s)}, \tag {8}
$$
where the super-script  $(s)$  indicates these variables are for the student UNet. The student UNet is supposed to predict the teacher's noisy latent  $\mathbf{z}_{t^{\prime \prime}}$  from  $\mathbf{z}_t$  with just one denoising step. This goal translates to the following vanilla distillation loss objective calculated in the x-space [33, 32],
$$
\mathcal {L} _ {\text {v a n i} \_ \text {d s t l}} = \varpi (\lambda_ {t}) \| \hat {\mathbf {x}} _ {t} ^ {(s)} - \frac {\mathbf {z} _ {t ^ {\prime \prime}} - \frac {\sigma_ {t ^ {\prime \prime}}}{\sigma_ {t}} \mathbf {z} _ {t}}{\alpha_ {t ^ {\prime \prime}} - \frac {\sigma_ {t ^ {\prime \prime}}}{\sigma_ {t}} \alpha_ {t}} \| _ {2} ^ {2}, \tag {9}
$$
where  $\varpi (\lambda_t) = \max \left(\frac{\alpha_t^2}{\sigma_t^2},1\right)$  is the truncated SNR weighting coefficients [33].
CFG-Aware Step Distillation. The above vanilla step distillation can improve the inference speed with no (or only little) FID compromised. However, we do observe the CLIP score turns obviously worse. As a remedy, this section introduces a classifier-free guidance-aware (CFG-aware) distillation loss objective function, which will be shown to improve the CLIP score significantly.
We propose to perform classifier-free guidance to both the teacher and student before calculating the loss. Specifically, for Eq. (7) and (8), after obtaining the  $\mathbf{v}$ -prediction output of UNet, we add the CFG step. Take Eq. (8) for an example,  $\hat{\mathbf{v}}_t^{(s)}$  is replaced with the following guided version,
$$
\tilde {\mathbf {v}} _ {t} ^ {(s)} = w \hat {\mathbf {v}} _ {\boldsymbol {\eta}} (t, \mathbf {z} _ {t}, \mathbf {c}) - (w - 1) \hat {\mathbf {v}} _ {\boldsymbol {\eta}} (t, \mathbf {z} _ {t}, \emptyset), \tag {10}
$$