SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \nabla_ {\mathbf {u}} D _ {\mathrm {K L}} \left[ \nu_ {\mathbf {x}} \mid \nu_ {\mathbf {u}} \right] \approx - \nabla_ {\mathbf {u}} \mathbb {E} _ {q _ {\mathbf {x}} \left(\mathbf {f} _ {\mathbf {u}}\right)} \left[ \log p (\tilde {\mathbf {y}} \mid \mathbf {f} _ {\mathbf {u}}) \right] + \mathbb {E} _ {q _ {\mathbf {u}} \left(\mathbf {f} _ {\mathbf {u}}\right)} \left[ \nabla_ {\mathbf {u}} \log p (\tilde {\mathbf {y}} \mid \mathbf {f} _ {\mathbf {u}}) \right] \\ = - \nabla_ {\mathbf {u}} \mathbb {E} _ {p \left(\varepsilon_ {\mathbf {x}}\right)} \left[ \log p \left(\tilde {\mathbf {y}} \mid \hat {\mu} _ {\mathbf {x}} + \hat {\Psi} _ {\mathbf {x}} ^ {1 / 2} \varepsilon_ {\mathbf {x}}\right) \right] + \mathbb {E} _ {p \left(\varepsilon_ {\mathbf {u}}\right)} \left[ \nabla_ {\mathbf {u}} \log p \left(\tilde {\mathbf {y}} \mid \hat {\mu} _ {\mathbf {u}} + \hat {\Psi} _ {\mathbf {u}} ^ {1 / 2} \varepsilon_ {\mathbf {u}}\right) \right] \tag {17} \\ \approx \frac {1}{S} \sum_ {s = 1} ^ {S} \left(- \nabla_ {\mathbf {u}} \log p \Big (\tilde {\mathbf {y}} \mid \hat {\mu} _ {\mathbf {x}} + \hat {\Psi} _ {\mathbf {x}} ^ {1 / 2} \varepsilon_ {\mathbf {x}} ^ {(s)} \Big) + \nabla_ {\mathbf {u}} \log p \Big (\tilde {\mathbf {y}} \mid \hat {\mu} _ {\mathbf {u}} + \hat {\Psi} _ {\mathbf {u}} ^ {1 / 2} \varepsilon_ {\mathbf {u}} ^ {(s)} \Big)\right), \\ \end{array}
$$
where  $p(\varepsilon_{\mathbf{x}})$  and  $p(\varepsilon_{\mathbf{u}})$  are standard Gaussians,  $(\varepsilon_{\mathbf{x}}^{(s)})_{s = 1}^{S}$  and  $(\varepsilon_{\mathbf{u}}^{(s)})_{s = 1}^{S}$  are i.i.d. samples from them.
Algorithm 1 Multi-architecture Function space Bayesian Pseudocoreset
Require: Set of architectures  $\mathcal{A}$ , expert trajectories  $\{\mathcal{E}^{(a)}: a \in \mathcal{A}\}$ , prior distributions of parameters  $\{\pi_0^{(a)}: a \in \mathcal{A}\}$ , an optimizer opt. Initialize u with random minibatch of coreset size  $m$ .  
for  $i = 1, \dots, N$  do  
Initialize the gradient of pseudocoreset,  $\mathbf{g} \gets 0$ .  
for  $a \in \mathcal{A}$  do  
Sample the MAP solution computed for  $\mathbf{x}$ ,  $\theta_{\mathbf{x}} \in \mathcal{E}^{(a)}$ .  
Sample an initial random parameter  $\theta_0 \sim \pi_0^{(a)}(\theta)$ .  
repeat  
 $\theta_t \gets \mathrm{opt}(\theta_{t-1}, (\mathbf{u}, \tilde{\mathbf{y}}))$   
until converges to obtain the MAP solution computed for  $\mathbf{u}$ ,  $\theta_{\mathbf{u}}$ .  
Obtain  $\hat{\mu}_{\mathbf{x}}, \hat{\mu}_{\mathbf{u}}, \hat{\Psi}_{\mathbf{x}}, \hat{\Psi}_{\mathbf{u}}$  by Eq. 16.  
Compute the pseudocoreset gradient  $\mathbf{g}^{(a)}$  using Eq. 17.  
 $\mathbf{g} \gets \mathbf{g} + \mathbf{g}^{(a)}$ .  
end for  
Update the pseudocoreset  $\mathbf{u}$  by using the gradient  $\mathbf{g}$ .  
end for
3.4 Multiple architectures FBPC training
One significant advantage of function space posterior matching is that the function is typically of much lower dimension compared to the weight. This makes it more likely for function spaces to exhibit similar posterior shapes in the vicinity of the MAP solutions. This characteristic of function space encourages the exploration of function space pseudocoreset training in the context of multiple architectures. Because, the task of training a coreset that matches the highly complex weight space posterior across multiple architectures is indeed challenging, while the situation becomes relatively easier when dealing with architectures that exhibit similar function posteriors.
Therefore we propose a novel multi-architecture FBPC algorithm in Algorithm 1. The training procedure involves calculating the FBPC losses for each individual architecture separately and then summing them together to update. This approach allows us to efficiently update the pseudocoreset by considering the contributions of each architecture simultaneously. We will empirically demonstrate that this methodology significantly enhances the architecture generalization ability of pseudocoresets in Section 5.
3.5 Compare to weight space Bayesian pseudocoresets
By working directly on the function space, our method could bypass several challenges that may arise when working on a weight space. Indeed, a legitimate concern arises regarding multi-modality, as the posterior distributions of deep neural networks are highly complex. It makes the optimization of pseudocoresets on weight space difficult. Moreover, minimization of weight space divergence does not necessarily guarantee proximity in the function space. Consequently, although we try to minimize the weight space divergence, there is a possibility that the obtained function posterior may significantly deviate from the true posterior. However, if we directly minimize the divergence between the function distributions, we can effectively address this issue.
On the other hand, there is an additional concern related to memory limitations. While it has been demonstrated in Kim et al. [15] that the memory usage of Bayesian pseudocoresets employing forward KL divergence is not excessively high, we can see that Eq. 7 requires Monte-Carlo samples of weights, which requires  $\mathcal{O}(Sp)$  where  $S$  and  $p$  represent the number of Monte-Carlo samples and the dimensionality of the weights, respectively. This dependence on Monte-Carlo sampling poses a limitation for large-scale networks when memory resources are constrained. In contrast, our proposed method requires significantly less memory,  $\mathcal{O}(Sd)$  where  $d$  represents the dimensionality of the functions. Indeed, all the results presented in this paper were obtained using a single NVIDIA RTX-3090 GPU with 24GB VRAM.
Bayesian coresets Bayesian Coreset [3-5, 14] is a field of research aimed at addressing the computational challenges of MCMC and VI on large datasets in terms of time and space complexity [9, 1, 26]. It aims to approximate the energy function of the entire dataset using a weighted sum of a small subset. However for high-dimensional data, Manousakas et al. [19] demonstrates that considering only subsets as Bayesian coreset is not sufficient, as the KL divergence between the approximated coreset posterior and the true posterior increases with the data dimension, and they proposed Bayesian pseudocoresets. There are recent works on constructing pseudocoreset variational posterior to be more flexible [8] or how to effectively optimize the divergences between posteriors [15, 8, 20, 21]. However, there is still a limitation in constructing high-dimensional Bayesian pseudocoresets specifically for deep neural networks.
Dataset distillation Dataset distillation also aims to synthesize the compact datasets that capture the essence of the original dataset. However, the dataset distillation places particular on optimizing the test performance of the distilled dataset. Consequently, the primary objective in dataset distillation is to maximize the performance of models trained using the distilled dataset, and researchers provides how to effectively solve this bi-level optimization [33, 23, 22, 40, 36, 39, 7]. In recent work, Kim et al. [15] established a link between specific dataset distillation methods and optimizing certain divergence measures associated with Bayesian pseudocoresets.
Function space variational inference Although Bayesian neural networks exhibit strong capabilities in performing variational inference, defining meaningful priors or efficiently inferring the posterior on weight space is still challenging due to their over-parametrization. To overcome this issue, researchers have increasingly focused on function space variational inference [6, 2, 16, 30, 25, 32, 27, 17]. For instance, Sun et al. [31] introduced a framework that formulates the KL divergence between functions as the supremum of marginal KL divergences over finite sets of inputs. Wang et al. [34] utilizes particle-based optimization directly in the function space. Furthermore, Rudner et al. [28] recently proposed a scalable method for function space variational inference on deep neural networks.
In our study, we employed the CIFAR10, CIFAR100 and Tiny-ImageNet datasets to create Bayesian pseudocoresets of coreset size  $m \in \{1, 10, 50\}$  images per class (ipc). These pseudocoresets were then evaluated by conducting the Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) [9] algorithm on those pseudocoresets. We measured the top-1 accuracy and negative log-likelihood of the SGHMC algorithm on the respective test datasets. Following the experimental setup of previous works [15, 7, 39], we use the differentiable siamese augmentation [37]. For the network architectures, we use 3-layer ConvNet for CIFAR10 and CIFAR100, and 4-layer ConvNet for Tiny-ImageNet.
We employed three baseline methods to compare the effectiveness of function space Bayesian pseudocoresets. The first baseline is the random coresets, which involves selecting a random minibatch of the coreset size. The others two baseline methods, BPC-rKL [15, 19] and BPC-fKL [15], are Bayesian pseudocoresets on weight space. BPC-rKL and BPC-fKL employ reverse KL divergence and forward KL divergence as the divergence measures for their training, respectively.
Table 1 and Table 2 show the results of each baseline and our method for each dataset. For BPC-rKL and BPC-fKL, we used the official code from [15] for training the pseudocoresets, and only difference is that we used our own SGHMC hyperparameters during evaluation. For detailed experiment setting, please refer to Appendix C.
As discussed earlier, we utilized the empirical covariance to variational posterior instead of using naive isotropic Gaussian for the function space variational posterior. To assess the effectiveness of using sample covariance, we compare these two in Table 1, as we also presented the results for
Table 1: Averaged test accuracy and negative log-likelihoods of models trained on each Bayesian pseudocoreset from scratch using SGHMC on the CIFAR10 dataset. Bold is the best and underline is the second best. These values are averaged over 5 random seeds.
ipc	SGHMC	Random	BPC-rKL [15, 19]	BPC-fKL [15]	FBPC-isotropic (Ours)	FBPC (Ours)
1	Acc (↑)	16.30±0.74	20.44±1.06	34.50±1.62	32.00±0.75	35.45±0.31
NLL (↓)	4.66±0.03	4.51±0.10	3.86±0.13	3.40±0.27	3.79±0.04
10	Acc (↑)	32.48±0.34	37.92±0.66	56.19±0.61	61.43±0.35	62.33±0.34
NLL (↓)	2.98±0.03	2.47±0.04	1.48±0.02	1.35±0.02	1.31±0.02
50	Acc (↑)	49.68±0.46	51.86±0.38	64.74±0.32	71.33±0.19	71.23±0.17
NLL (↓)	2.06±0.02	1.95±0.02	1.26±0.01	1.03±0.01	1.03±0.05
Table 2: Averaged test accuracy and negative log-likelihoods of models trained on each Bayesian pseudocoreset from scratch using SGHMC on the CIFAR100 and Tiny-ImageNet datasets. These values are averaged over 3 random seeds.
ipc	1	CIFAR100	Tiny-ImageNet
10	50	1	10	50
Random	Acc (↑)	4.82±0.47	18.0±0.31	35.1±0.23	1.90±0.08	7.21±0.04	19.15±0.12
NLL (↓)	5.55±0.07	4.57±0.01	3.35±0.01	6.18±0.04	5.77±0.02	4.88±0.01
BPC-fKL	Acc (↑)	14.7±0.16	28.1±0.60	37.1±0.33	3.98±0.13	11.4±0.45	22.18±0.32
NLL (↓)	4.17±0.05	3.53±0.05	3.28±0.24	5.63±0.03	5.08±0.05	4.65±0.02
FBPC (Ours)	Acc (↑)	21.0±0.76	39.7±0.31	44.47±0.35	10.14±0.68	19.42±0.51	26.43±0.31
NLL (↓)	3.76±0.11	2.67±0.02	2.63±0.01	4.69±0.05	4.14±0.02	4.30±0.05
FBPC-isotropic, which represents the FBPC trained with a unit covariance Gaussian posteriors. The results clearly demonstrate that using sample covariance captures valuable information from the posterior distribution, resulting in improved performance. Overall, the results presented in Table 1 and Table 2 demonstrate that our method also outperforms the baseline approaches, including random coresets, BPC-rKL and BPC-fKL, in terms of both accuracy and negative log-likelihood, especially on the large-scale datasets in Table 2.
Furthermore, the Bayesian pseudocoreset can be leveraged to enhance robustness against distributional shifts when combined with Bayesian model averaging. To assess the robustness of our function space Bayesian pseudocoresets on out-of-distribution inputs, we also conducted experiments using the CIFAR10-C dataset [11]. This dataset involves the insertion of image corruptions into the CIFAR10 images. By evaluating the performance of the pseudocoresets on CIFAR10-C, we can see the model's ability to handle input data that deviates from the original distribution. In Table 3, we provide the results for top-1 accuracy and degradation scores, which indicate the extent to which accuracy is reduced compared to the in-distribution's test accuracy. The result demonstrates that our FBPC consistently outperforms the weight space Bayesian pseudocoreset, BPC-fKL.
5.3 Architecture generalization
In this section, we aim to demonstrate the architecture generalizability of FBPC and emphasize the utility of multi-architecture training as we discussed in the previous section. We specifically focus on investigating the impact of varying normalization layers on the generalizability of the pseudocoreset, since it is widely recognized that a pseudocoreset trained using one architecture may struggle to generalize effectively to a model that employs different normalization layers. We have also included the results of cross-architecture experiments that involve changing the architecture itself in Appendix D.1.
To assess this, we compare a single architecture trained pseudocoreset and a multiple architecture trained pseudocoreset. For single architecture training, we initially train a pseudocoreset using one architecture with a specific normalization layer, for instance we use instance normalization. Subsequently, we evaluate the performance of this pseudocoreset on four different types of normalization layers: instance normalization, group normalization, layer normalization, and batch normalization. For multiple architecture training, we aggregate four losses for single architecture training of each architecture, and train the pseudocoreset with the sum of all losses, as mentioned in previous section.
(a) Performance evaluation with different normalization layers. Color represents the test architectures. The multiple architecture FBPC training enhances the generalization ability to other architectures.
Figure 2: Results for multiple architecture FBPC training.
(b) A sample image and its corresponding function values. Despite variations in the normalization layers of different architectures, the function values exhibit similarity across the architectures.
Table 3: Test accuracy and degradation scores of models trained on each Bayesian pseudocoreset from scratch using SGHMC on the CIFAR10-C. Degradation refers to the extent by which a model's accuracy decreases when evaluated on the CIFAR10-C dataset compared to the CIFAR10 test dataset.
corruption	BN	DB	ET	FT	GB	JPEG	MB	PIX	SN	SP	Avg.
BPC-fKL	Acc (↑)	33.5	34	35.9	25.1	33.7	39.1	32.7	38.3	28.9	41.2	34.3
Degradation (↓)	40.3	39.4	36	55.2	39.9	30.3	41.6	31.6	48.4	26.5	38.9
FBPC	Acc (↑)	48.9	46.4	47.6	41.7	44.0	52.0	44.3	51.0	47.1	52.3	47.5
Degradation (↓)	21.5	25.7	23.7	33.0	29.3	16.4	28.8	18.1	24.4	16.1	23.7
As depicted in Fig. 2a, we observe that both WBPC (Weight space Bayesian pseudocoresets) and FBPC-single (Function space Bayesian pseudocoresets trained on a single architecture) exhibit a notable trend, that they tend to not perform well when evaluated on the architecture that incorporates different normalizations, regardless of whether it is trained on weight space or function space. On the other hand, when trained with multiple architectures, both WBPC-multi and FBPC-multi perform well across the all architectures, while notably FBPC-multi significantly outperforms WBPC-multi.
As mentioned in the previous section, we hypothesize that the superior performance of FBPC compared to WBPC can be attributed to the likelihood of having similar function space posterior across architectures. To validate this, we conduct an examination of the logit values for each sample across different architectures. As an illustration, we provide an example pseudocoreset image belonging to the class label "dog" along with its corresponding logits for all four architectures. As Fig. 2b shows, it can be observed that the logits display a high degree of similarity, indicating a strong likelihood of matching function posterior distributions. Our analysis confirms that, despite architectural disparities, the function spaces generated by these architectures exhibit significant similarity and it contributes to superiority of FBPC in terms of architecture generalizability.
In this paper, we explored the function space Bayesian pseudocoreset. We constructed it by minimizing forward KL divergence between the function posteriors of pseudocoreset and the entire dataset. To optimize the divergence, we proposed a novel method to effectively approximate the function posteriors with an efficient training procedure. Finally, we empirically demonstrated the superiority of our function space Bayesian pseudocoresets compared to weight space Bayesian pseudocoresets, in terms of test performance, uncertainty quantification, OOD robustness, and architectural robustness.
**Limitation** Despite showing promising results on function space Bayesian pseudocoresets, there still exist a few limitations in the training procedure. Our posterior approximation strategy requires
the MAP solutions, which necessitates training them prior to each update step or preparing expert trajectories. This can be time-consuming and requires additional memory to store the expert trajectories.
Societal Impacts Our work is hardly likely to bring any negative societal impacts.
This work was supported by KAIST-NAVER Hypercreative AI Center, Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST), No.2022-0-00184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics, and No.2022-0-00713, Meta-learning Applicable to Real-world Problems) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2022R1A5A708390812)
[1] J. Baker, P. Fearnhead, E. B. Fox, and C. Nemeth. Control variates for stochastic gradient mcmc. Statistics and Computing, 29:599-615, 2019.
[2] D. R. Burt, S. W. Ober, A. Garriga-Alonso, and M. van der Wilk. Understanding variational inference in function-space. In Third Symposium on Advances in Approximate Bayesian Inference, 2020.
[3] T. Campbell and B. Beronov. Sparse variational inference: Bayesian coresets from scratch. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019.
[4] T. Campbell and T. Broderick. Bayesian coreset construction via greedy iterative geodesic ascent. In Proceedings of The 35th International Conference on Machine Learning (ICML 2018), 2018.
[5] T. Campbell and T. Broderick. Automated scalable bayesian inference via hilbert coresets. The Journal of Machine Learning Research, 20(1):551-588, 2019.
[6] E. D. Carvalho, R. Clark, A. Nicastro, and P. H. Kelly. Scalable uncertainty for computer vision with functional variational inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12003–12013, 2020.
[7] G. Cazenavette, T. Wang, A. Torralba, A. A. Efros, and J.-Y. Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[8] N. Chen, Z. Xu, and T. Campbell. Bayesian inference via sparse hamiltonian flows. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022.
[9] T. Chen, E. B. Fox, and C. Guestrin. Stochastic gradient Hamiltonian Monte Carlo. In Proceedings of the 31st International Conference on Machine Learning (ICML 2014), 2014.
[10] A. G. de G. Matthews, J. Hensman, R. E. Turner, and Z. Ghahramani. On sparse variational methods and the Kullback-Leibler divergence between stochastic processes. Proceedings of Machine Learning Research, 51:231-239, 2016.
[11] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
[12] J. Hensman, N. Fusi, and N. D. Lawrence. Gaussian processes for big data. In Uncertainty in Artificial Intelligence, 2013.
[13] J. Hensman, A. Matthews, and Z. Ghahramani. Scalable variational gaussian process classification. In Artificial Intelligence and Statistics, 2015.
[33] T. Wang, J.-Y. Zhu, A. Torralba, and A. A. Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018.
[34] Z. Wang, T. Ren, J. Zhu, and B. Zhang. Function space particle optimization for Bayesian neural networks. In International Conference on Learning Representations, 2019.
[35] R. Yu, S. Liu, and X. Wang. Dataset distillation: A comprehensive review. arXiv preprint arXiv:2301.07014, 2023.
[36] B. Zhao and H. Bilen. Dataset condensation with differentiable siamese augmentation. In Proceedings of The 38th International Conference on Machine Learning (ICML 2021), 2021.
[37] B. Zhao and H. Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages 12674-12685. PMLR, 2021.
[38] B. Zhao and H. Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6514–6523, 2023.
[39] B. Zhao, K. R. Mopuri, and H. Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021.
[40] Y. Zhou, E. Nezhadarya, and J. Ba. Dataset distillation using neural feature regression. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2022.
Proposition A.1. The gradient of the forward KL divergence with respect to the coreset  $\mathbf{u}$  is
$$
\nabla_ {\mathbf {u}} D _ {\mathrm {K L}} \left[ \nu_ {\mathbf {x}} \| \nu_ {\mathbf {u}} \right] = - \nabla_ {\mathbf {u}} \mathbb {E} _ {[ \nu_ {\mathbf {x}} ] _ {\mathbf {u}}} \left[ \log p (\tilde {\mathbf {y}} \mid \mathbf {f} _ {\mathbf {u}}) \right] + \mathbb {E} _ {[ \nu_ {\mathbf {u}} ] _ {\mathbf {u}}} \left[ \nabla_ {\mathbf {u}} \log p (\tilde {\mathbf {y}} \mid \mathbf {f} _ {\mathbf {u}}) \right], \tag {10}
$$
where  $[\nu_{\mathbf{x}}]_{\mathbf{u}}$  and  $[\nu_{\mathbf{u}}]_{\mathbf{u}}$  are finite-dimensional distributions of the stochastic processes  $\nu_{\mathbf{x}}$  and  $\nu_{\mathbf{u}}$ , respectively,  $\mathbf{f}_{\mathbf{u}} \coloneqq (f(u_j))_{j=1}^{m}$  and  $p(\tilde{\mathbf{y}} \mid \mathbf{f}_{\mathbf{u}}) = \prod_{j=1}^{m} p(\tilde{\mathbf{y}}_j \mid f(u_j))$ .
Proof. We follow the arguments in de G. Matthews et al. [10], Rudner et al. [29]. The forward KL divergence is defined as,
$$
D _ {\mathrm {K L}} \left[ \nu_ {\mathbf {x}} \| \nu_ {\mathbf {u}} \right] = \int \log \frac {\mathrm {d} \nu_ {\mathbf {x}}}{\mathrm {d} \nu_ {\mathbf {u}}} (f) \mathrm {d} \nu_ {\mathbf {x}} (f). \tag {18}
$$
By the chain rule for the Radon-Nikodym derivative, we have
$$
D _ {\mathrm {K L}} \left[ \nu_ {\mathbf {x}} \| \nu_ {\mathbf {u}} \right] = \int \log \frac {\mathrm {d} \nu_ {\mathbf {x}}}{\mathrm {d} \nu_ {0}} (f) \mathrm {d} \nu_ {\mathbf {x}} (f) - \int \log \frac {\mathrm {d} \nu_ {\mathbf {u}}}{\mathrm {d} \nu_ {0}} (f) \mathrm {d} \nu_ {\mathbf {x}} (f). \tag {19}
$$
The first term does not depend on  $\mathbf{u}$ , so we investigate the second term. By the measure theoretic Bayes' rule,
$$
\frac {\mathrm {d} \nu_ {\mathbf {u}}}{\mathrm {d} \nu_ {0}} (f) = \frac {p (\tilde {\mathbf {y}} \mid f , \mathbf {u})}{p (\tilde {\mathbf {y}} \mid \mathbf {u})}, \tag {20}
$$
where  $p(\tilde{\mathbf{y}}\mid f,\mathbf{u}):= \prod_{j = 1}^{m}p(\tilde{y}_j\mid u_j,f)$  and,
$$
p (\tilde {\mathbf {y}} | \mathbf {u}) = \int p (\tilde {\mathbf {y}} | f, \mathbf {u}) \mathrm {d} \nu_ {0} (f). \tag {21}
$$
Now let  $\rho_A:(\mathcal{X}\to \mathbb{R}^d)\to (A\to \mathbb{R}^d)$  be a projection function that takes a function  $f$  and returns its restriction on a set  $A\subseteq \mathcal{X}$ . Assuming that the likelihood depends only on the finite index set  $\mathbf{u}$ , we can write