SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Stellantis, France
Ankit Singh
Stellantis, India
Thomas Hannagan
The elusive nature of gradient-based optimization in neural networks is tied to their loss landscape geometry, which is poorly understood. However recent work has brought solid evidence that there is essentially no loss barrier between the local solutions of gradient descent, once accounting for weight-permutations that leave the network's computation unchanged. This raises questions for approximate inference in Bayesian neural networks (BNNs), where we are interested in marginalizing over multiple points in the loss landscape. In this work, we first extend the formalism of marginalized loss barrier and solution interpolation to BNNs, before proposing a matching algorithm to search for linearly connected solutions. This is achieved by aligning the distributions of two independent approximate Bayesian solutions with respect to permutation matrices. We build on the results of Ainsworth et al. (2023), reframing the problem as a combinatorial optimization one, using an approximation to the sum of bilinear assignment problem. We then experiment on a variety of architectures and datasets, finding nearly zero marginalized loss barriers for linearly connected solutions.
Throughout the last decade, deep neural networks (DNNs) have achieved significant success in a wide range of practical applications, becoming the fundamental ingredient for e.g., computer vision [e.g., 57, 25, 41, 64], language models [e.g., 24, 82, 17] and generative models [e.g., 54, 96, 97, 98, 102, 34]. Despite recent important advancements, understanding the loss landscape of DNNs is still challenging. The characterization of its highly non-convex nature, its relation with architectural choices like depth and width and the connection with optimization and generalization are just some of the problems which have been the focus of extensive research in the last few years [e.g., 76, 26, 36, 32, 2, 30, 38, 79]. It is well known, for example, that one of the fundamental characteristics of deep neural networks is their ability to learn hierarchical features, and in this regards deeper networks seem to be exponentially more expressive than shallower models [e.g., 3, 5, 7, 8, 18, 109], leading the loss landscape to have many optima due to symmetries and over-parameterization [76, 26, 111, 94]. At the same time, the role of the depth of a model in relation with its width is far less understood [80], despite wide neural networks exhibiting important theoretical properties in their infinite limit behavior [e.g., 73, 23, 29, 53, 48, 37, 20].
Two notions that have been useful to shed light on the geometry of loss landscapes are that of loss barriers and mode connectivity [36, 26]. The mode connectivity hypothesis states that given two points in the landscape, there exists a path connecting them such that the loss is constant or near constant (or, said differently, the loss barrier is null). We refer to linear mode connectivity when the path connecting the two solutions is linear [32]. Recently, evidence has surfaced that stochastic gradient descent (SGD) solutions to the loss minimization problem can be linearly connected. Indeed, Entezari et al. [30] discuss the role of permutation symmetries from a loss connectivity viewpoint, conjecturing the possibility that mode connectivity is actually linear once accounting for all permutation invariances.
Figure 1: Permutation symmetries for regression on the Snelson dataset [95]. (Left) Two different solutions, with similar function-space behavior (showing  $\mu \pm 2\sigma$ ). (Right) Functions obtained when using two different strategies to interpolate between solutions. When we align the solutions, by taking into account the permutation symmetries, we retain the capability to model the data, indicating that there are solutions that once linearly interpolated exhibit no loss in performance. Note that, in weight space, the solution found with alignment is neither equal to solution 1 nor 2 (see the black curve which is a single function sample with fixed randomness).
Additionally, [2] gathers compelling empirical evidence across several network architectures and tasks, that under such a permutation symmetry the loss landscape often contains a single, nearly convex basin.
In this work, we are taking a different perspective on this analysis. We are interested in the Bayesian treatment of neural networks, which results in a natural form of regularization and allows to reason about uncertainty in the predictions [101, 73, 66]. Bayesian inference for deep neural networks is notoriously challenging, as we wish to marginalize over multi-modal distributions with high dimensionality [47]. For this reason, there are various ways to approximate the posterior, involving techniques like variational inference [39, 14, 35, 62, 77], Markov chain Monte Carlo (MCMC) methods [75, 72, 27], possibly with stochastic gradients [19, 112, 33, 105, 70] and the Laplace approximation [68, 68, 84]. Indeed, fundamentally the Bayesian posterior and the loss landscapes are tightly interconnected: (i) solutions to the loss minimization problem are equivalent to maximum-a-posteriori (MAP) solutions, (ii) the loss landscape is equivalent to the un-normalized negative log-posterior. While in theory, given a dataset, the posterior is unique and the solution is global, many approximations will only explore local properties of the true posterior<sup>1</sup>. It's worth noting that a posterior over the parameters of the neural network induces a posterior on the functions generated by the model. Permutation symmetries play an important role in the geometry of the weight-space posterior, which are generally not reflected in function-space. While it is possible to carry out inference directly in function space, this poses a number of challenges [99, 71, 89, 61]. Fig. 1 illustrates this situation for a regression task on the Snelson dataset using a 3-layer DNN: on the left we compare two (approximate) solutions which have different weight-space posterior but similar function-space behavior. Notably, when we interpolate these two solutions (Fig. 1 on the right), we completely lose all capability of modeling the data. However, when we account for permutation symmetries in the posterior, we end up with solutions that once interpolated are still good approximations. This suggests that for any weight-space distribution, there exists a class of solutions which are functionally equivalent and linearly connected. This example motivates an informal generic conjecture:
Solutions of approximate Bayesian inference for neural networks are linearly connected after accounting for functionally equivalent permutations.
While being similar to the one in [30, 2], if this conjecture was to hold true for approximate Bayesian neural networks (BNNs) it would represent an important step in further characterizing the properties of the Bayesian posterior and the effect of various approximations. We purposely leave the previous conjecture broadly open regarding the choice of the approximation method to allow for a more general discussion. More specifically, in this paper we will analyze and focus our discussion on the variational inference framework, making a more specific conjecture:
Conjecture 1. Variational inference solutions for approximate Bayesian inference in neural networks are linearly connected after accounting for functionally equivalent permutations.
<sup>1</sup>By local properties, we mean that despite theoretical convergence guarantees of many methods like variational inference and MCMC, in practice the true posterior for deep neural networks is still highly elusive; see for instance the empirical convergence analysis of Hamiltonian Monte Carlo (HMC) in [47].
Figure 2: Permutations in multi-modal posterior. Log-posterior for MLP/CIFAR10, showing the two solutions  $(q_{0}$  and  $q_{1})$  for which we can find a permutation map such that  $P_{\#}q_{1}$  can be linearly connected to  $q_{0}$  with low barrier (brighter regions).
Contributions. With this work, we aim at studying the linear connectivity properties of approximate solutions to the Bayesian inference problem and we make several contributions. (i) We extend the formalism of loss barrier and solution interpolation to BNNs. (ii) For the variational inference setting, propose a matching algorithm to search for linearly connected solutions by aligning the distributions of two independent solutions with respect to permutation matrices. Inspired by [2], we frame the problem as a combinatorial optimization problem using approximation to the linear sum assignment problem. (iii) We then experiment on a variety of architectures and datasets, finding nearly zero-loss barriers for linearly connected solutions. In Fig. 2 we present a sneak-peek and a visualization of our findings, where we show that after weight distribution alignment we can find a permutation map  $P$  of the solution  $q_{1}$  such that it can be linearly connected through high density regions to  $q_{0}$ .
2 Preliminaries on Bayesian deep learning
In this section, we review some basic notations on BNNs and we review stochastic variational inference (SVI), which is the main approximation method that we analyze in this paper. Let's consider a generic multilayer perceptron (MLP) with  $L$  layers, where the output of the  $l$ -th layer  $\pmb{f}_l(\pmb{\theta}_l, \pmb{x})$  is a vector-valued function of the previous layer output  $\pmb{f}_{l-1}$  as follows,
$$
\boldsymbol {f} _ {l} \left(\boldsymbol {\theta} _ {l}, \boldsymbol {x}\right) = \boldsymbol {W} _ {l} a \left(\boldsymbol {f} _ {l - 1} \left(\boldsymbol {\theta} _ {l - 1}, \boldsymbol {x}\right)\right) + \boldsymbol {b} _ {l} \tag {1}
$$
where  $a(\cdot)$  is a non-linearity,  $\pmb{W}_l$  is a  $D_l\times D_{l - 1}$  weight matrix and  $\pmb{b}_{l}$  the corresponding bias vector. We shall refer to the parameters of the layer  $l$  as  $\pmb{\theta}_l = \{\pmb {b}_l,\pmb {W}_l\}$ , and the union of all trainable parameters as  $\pmb {\theta} = \{\pmb {\theta}_l\}_{l = 1}^L$
The objective of using Bayesian inference on deep neural networks [67, 65] involves inferring a posterior distribution over the parameters of the neural network given the available dataset  $\{X,Y\} = \{(x_i,y_i)\}_{i = 1}^N$ . This requires choosing a likelihood and a prior [73, 74, 103]:
$$
p (\boldsymbol {\theta} \mid \boldsymbol {Y}, \boldsymbol {X}) = Z ^ {- 1} p (\boldsymbol {Y} \mid \boldsymbol {\theta}, \boldsymbol {X}) p (\boldsymbol {\theta}) \tag {2}
$$
where the normalization constant  $Z$  is the marginal likelihood  $p(\mathbf{Y} \mid \mathbf{X})$ . As usually done, we assume that the likelihood factorizes over observations, i.e.  $p(\mathbf{Y} \mid \boldsymbol{\theta}, \mathbf{X}) = \prod_{i=1}^{N} p(\mathbf{y}_i \mid \boldsymbol{\theta}, \mathbf{x}_i)$ .
Bayesian deep learning is intractable due to the non-conjugacy likelihood-prior and thus we don't have access to closed form solutions. Variational inference (VI) is a common technique to handle intractable Bayesian neural networks [12, 43, 39, 50]. Let  $\mathcal{P}(\mathbb{R}^d)$  be the space of probability measures on  $\mathbb{R}^d$ ; VI reframes the inference problem into an optimization one, commonly by introducing a parameterized distribution  $q(\pmb{\theta}) \in \mathcal{P}(\mathbb{R}^d)$  which is optimized to minimize the Kullback-Leibler (KL) divergence with respect to the true posterior  $p(\pmb{\theta} \mid \pmb{Y}, \pmb{X})$ . In practice, this involves the maximization of the evidence lower bound (ELBO) defined as
$$
\mathcal {L} _ {\mathrm {E L B O}} (q) \stackrel {\text {d e f}} {=} \int \log p (\boldsymbol {Y} \mid \boldsymbol {\theta}, \boldsymbol {X}) \mathrm {d} q (\boldsymbol {\theta}) - \mathrm {K L} [ q (\boldsymbol {\theta}) \| p (\boldsymbol {\theta}) ] \tag {3}
$$
whose gradients can be unbiasedly estimated with mini-batches of data [43] and the reparameterization trick [54, 55]. Despite its simple formulation, the optimization of the ELBO hides several challenges, like the initialization of the variational parameters [87], the effects of over-parameterization on the quality of the approximation [88, 44, 58]. Here we are interested in how different solutions to Eq. (3) relate to each other in terms of loss barrier, which we will define formally in the following section.
3 Loss barriers
In the context of Bayesian inference, we are interested in the loss computed by marginalization of the model parameters with respect to (an approximation of) the posterior. As such, we use the predictive
Log-posterior for CIFAR10
likelihood, a proper scoring method for probabilistic models [83], defined as
$$
\log p \left(\boldsymbol {y} ^ {\star} \mid \boldsymbol {x} ^ {\star}\right) = \log \int p \left(\boldsymbol {y} ^ {\star} \mid \boldsymbol {\theta}, \boldsymbol {x} ^ {\star}\right) p \left(\boldsymbol {\theta} \mid \boldsymbol {Y}, \boldsymbol {X}\right) \mathrm {d} \boldsymbol {\theta} \approx \log \int p \left(\boldsymbol {y} ^ {\star} \mid \boldsymbol {\theta}, \boldsymbol {x} ^ {\star}\right) \mathrm {d} q (\boldsymbol {\theta}) \tag {4}
$$
where  $q(\pmb{\theta})$  is an approximation of the true posterior (parametric or otherwise),  $\{\pmb{x}^{\star},\pmb{y}^{\star}\}$  are respectively the input and its corresponding label a data point under evaluation. To keep the notation uncluttered for the remaining of the paper, we write the predictive likelihood computed for a set of points  $\{\pmb{x}_i^\star ,\pmb{y}_i^\star \}_{i = 1}^N$  as a functional  $\mathcal{L}:\mathcal{P}(\mathbb{R}^d)\to \mathbb{R}$ , defined as
$$
\mathcal {L} (q) \stackrel {\text {d e f}} {=} \sum_ {i = 1} ^ {N} \log \int p \left(\boldsymbol {y} _ {i} ^ {\star} \mid \boldsymbol {\theta}, \boldsymbol {x} _ {i} ^ {\star}\right) \mathrm {d} q (\boldsymbol {\theta}) \tag {5}
$$
Let's assume two models trained with VI with two different initializations, random seeds, and batch ordering. Variational inference in the classic inverse sense KL  $[q\parallel p]$  is mode seeking, thus we expect the two runs to converge to different solutions, say  $q_{0}$  and  $q_{1}$ . To test the loss barrier as we interpolate between the two solutions we need to decide on the interpolation rule. We decide to interpolate the solutions following the Wasserstein geodesics between  $q_{0}$  and  $q_{1}$ . First, let's start with a few definitions. Let  $q\in \mathcal{P}(\mathbb{R}^d)$  be a probability measure on  $\mathbb{R}^d$  and  $T:\mathbb{R}^d\to \mathbb{R}^d$  a measurable map; we denote  $T_{\#}q$  the push-forward measure of  $q$  through  $T$ . Now we can introduce the Wasserstein geodesics, as follows.
Definition 1. The Wasserstein geodesics between  $q_{0}$  and  $q_{1}$  is defined as the path
$$
q _ {\tau} = \left((1 - \tau) \mathrm {I d} + \tau T _ {q _ {0}} ^ {q _ {1}}\right) _ {\#} q _ {0}, \quad \tau \in [ 0, 1 ] \tag {6}
$$
where  $\mathrm{Id}$  is the identity map and  $T_{q_0}^{q_1}$  is the optimal transport map between  $q_0$  and  $q_1$ , which for Brenier's theorem [16], is unique.
While we could interpolate using a mixture of the two solutions, we argue that this choice is trivial and does not fully give us a picture of the underlying loss landscape. Indeed, Eq. (6) is fundamentally different from a naive mixture path  $\tilde{q}_{\tau} = (1 - \tau)q_{0} + \tau q_{1}$ . In case of Gaussian distributions, when  $q_{0} = \mathcal{N}(m_{0},S_{0})$  and  $q_{1} = \mathcal{N}(m_{1},S_{1})$ ,  $q_{\tau}$  is Gaussian as well [100] with mean and covariance computed as follows:
$$
\boldsymbol {m} _ {\tau} = (1 - \tau) \boldsymbol {m} _ {1} + \tau \boldsymbol {m} _ {2}
$$
$$
\boldsymbol {S} _ {\tau} = \boldsymbol {S} _ {1} ^ {- 1 / 2} \left((1 - \tau) \boldsymbol {S} _ {1} + \tau \left(\boldsymbol {S} _ {1} ^ {1 / 2} \boldsymbol {S} _ {2} \boldsymbol {S} _ {1} ^ {1 / 2}\right) ^ {1 / 2}\right) ^ {2} \boldsymbol {S} _ {1} ^ {- 1 / 2} \tag {7}
$$
which simplifies even further when the covariances are diagonal.
Now, we can define convexity along Wasserstein geodesics [4] as follows.
Definition 2. Let  $\mathcal{L}:\mathcal{P}(\mathbb{R}^d)\to \mathbb{R}$ ,  $\mathcal{L}$  is  $\lambda$  geodesics convex with  $\lambda >0$  if for any  $q_{0},q_{1}\in \mathcal{P}(\mathbb{R}^{d})$  it holds that
$$
\mathcal {L} \left(q _ {\tau}\right) \leq (1 - \tau) \mathcal {L} \left(q _ {0}\right) + \tau \mathcal {L} \left(q _ {1}\right) - \frac {\lambda \tau (1 - \tau)}{2} \mathcal {W} _ {2} ^ {2} \left(q _ {0}, q _ {1}\right) \tag {8}
$$
where  $\mathcal{W}_2^2 (q_0,q_1)$  is the Wasserstein distance defined as [104, 51, 52]
$$
\mathcal {W} _ {2} ^ {2} \left(q _ {1}, q _ {0}\right) = \inf  _ {\gamma \in \Pi \left(q _ {1}, q _ {0}\right)} \int \left\| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {0} \right\| _ {2} ^ {2} \mathrm {d} \gamma \left(\boldsymbol {\theta} _ {1}, \boldsymbol {\theta} _ {0}\right) \tag {9}
$$
with  $\Pi (\cdot ,\cdot)$  being the space of measure with  $q_{0}$  and  $q_{1}$  as marginals.
While mathematically proving the geodesics convexity of the predictive likelihood for arbitrary architectures and densities is currently beyond the scope of this work, we can empirically define a proxy using the functional loss barrier, defined as follows.
Definition 3. The functional loss barrier along the Wasserstein geodesics from  $q_{0}$  and  $q_{1}$  is defined as the highest difference between the marginal loss computed when interpolating two solutions  $q_{0}$  and  $q_{1}$  and the linear interpolation of the loss at  $q_{0}$  and  $q_{1}$ :
$$
\mathcal {B} \left(q _ {0}, q _ {1}\right) = \max  _ {\tau} \mathcal {L} \left(q _ {\tau}\right) - \left((1 - \tau) \mathcal {L} \left(q _ {0}\right) + \tau \mathcal {L} \left(q _ {1}\right)\right) \tag {10}
$$
where  $q_{\tau}$  follows the definition in Eq. (6).
This definition is a more general than the ones in [2, 30, 32] but we can recover [30] by assuming delta posteriors  $q_{i} = \delta (\pmb{\theta} - \pmb{\theta}_{i})$  and we can further recover [2, 32] by also assuming  $\mathcal{L}(q_0) = \mathcal{L}(q_1)$ .
A comment on mixtures. In previous paragraphs, we argued that the mixture of distributions is not sufficient to capture the underlying complex geometry of the posterior. Now, we want to better illustrate this choice with a simple example. In Fig. 3 we plot the test likelihood with two interpolation strategies between two solutions (MLP on CIFAR10): the Wasserstein geodesics and the mixture. With mixtures, we see that the likelihood is pretty much constant during the interpolation, but this is very miss-leading: we don't see barriers not because they don't exist, but because the mixture simply re-weights the distributions, without continuously transporting mass in the parameter space.
Figure 3: Wasserstein geodesics and mixtures. Test likelihood for mixture and the Wasserstein geodesics interpolation. Solutions are MLPs trained on CIFAR10.
4 Aligning distributions by looking for permutation symmetries
In this section, we formalize the algorithm that aligns the solutions of Bayesian inference through permutation symmetries of weight matrices and biases. Let  $\mathbb{S}(d)$  be the set of valid  $d\times d$  permutation matrices. Given a generic distribution  $q(\pmb {\theta})$ , we can apply a permutation matrix  $\pmb {P}\in \mathbb{S}(D_l)$  to a hidden layer output at layer  $l$ , and if we define  $\pmb{\theta}^{\prime}$  to be equivalent to  $\pmb{\theta}$  with the exception of
$$
\boldsymbol {W} _ {l} ^ {\prime} = \boldsymbol {P} \boldsymbol {W} _ {l}, \quad \boldsymbol {b} _ {l} ^ {\prime} = \boldsymbol {P} \boldsymbol {b} _ {l}, \quad \boldsymbol {W} _ {l + 1} ^ {\prime} = \boldsymbol {W} _ {l + 1} \boldsymbol {P} ^ {\top}, \tag {11}
$$
then  $P_{\#}q$  is the equivalent push-forward distribution for  $\theta'$ , where  $P$  is the associated permutation map. Let us define the distribution over the functional output of the model as
$$
q (\boldsymbol {f} (\boldsymbol {\theta}, \cdot)) = \int \delta (\boldsymbol {f} (\boldsymbol {\theta}, \cdot) - \boldsymbol {f} (\widehat {\boldsymbol {\theta}}, \cdot)) \mathrm {d} q (\widehat {\boldsymbol {\theta}}), \tag {12}
$$
and, equivalently, the distribution on the function using the permuted parameters as
$$
q \left(\boldsymbol {f} \left(\boldsymbol {\theta} ^ {\prime}, \cdot\right)\right) = \int \delta \left(\boldsymbol {f} \left(\boldsymbol {\theta} ^ {\prime}, \cdot\right) - \boldsymbol {f} \left(\widehat {\boldsymbol {\theta}} ^ {\prime}, \cdot\right)\right) \mathrm {d} P _ {\#} q \left(\widehat {\boldsymbol {\theta}} ^ {\prime}\right), \tag {13}
$$
where in both cases  $\delta (\cdot)$  is the Dirac function. Then, it is simple to verify that the two models are functionally equivalent for any inputs,
$$
q (\boldsymbol {f} (\boldsymbol {\theta}, \cdot)) = q (\boldsymbol {f} (\boldsymbol {\theta} ^ {\prime}, \cdot)). \tag {14}
$$
This implies that for any weight-space distribution  $q$ , there exists a class of functionally equivalent solutions  $P_{\#}q$ , in the sense of Eq. (14). These same considerations can be easily extended to other layers, by considering multiple permutation matrices  $P_l$ . For our analysis, given two solutions  $q_0$  and  $q_1$  we are interested in finding the permuted distribution  $P_{\#}q_1$ , functionally equivalent to  $q_1$ , in such a way that once interpolating using Eq. (6) we observe similar performance to  $q_0$  and  $q_1$ . Formally, we can write
$$
\underset {P} {\arg \min } \mathcal {D} \left(q _ {1} \left(\boldsymbol {f} \left(\boldsymbol {\theta} ^ {\prime}, \cdot\right)\right), q _ {0} \left(\boldsymbol {f} \left(\boldsymbol {\theta}, \cdot\right)\right)\right) = \underset {P} {\arg \min } \mathcal {D} \left(P _ {\#} q _ {1} (\boldsymbol {\theta}), q _ {0} (\boldsymbol {\theta})\right), \tag {15}
$$
where  $\mathcal{D}$  is a generic measure of discrepancy.
4.1 Problem setup for permutation of vectors
We start from a single vector of parameters, disregarding for the moment the functional equivalence constraint. We will extend these results to matrices and multiple layers later. In practice considering
2To be formally correct, the l.h.s. is a discrepancy defined on stochastic processes while the r.h.s. is defined on random vectors. Under mild assumptions on the distribution on the parameters and the architectures,  $\mathcal{D}$  is well defined in both cases [see e.g., 81].
Gaussian distributions, we know that if  $q = \mathcal{N}(\boldsymbol{m}, \boldsymbol{S})$ , then  $P_{\#}q = \mathcal{N}(\boldsymbol{P}\boldsymbol{m}, \boldsymbol{P}\boldsymbol{S}\boldsymbol{P}^{\top})$  and if  $q = \mathcal{N}(\boldsymbol{m}, \mathrm{diag}(s^{2}))$  then  $P_{\#}q = \mathcal{N}(\boldsymbol{P}\boldsymbol{m}, \mathrm{diag}(\boldsymbol{P}\boldsymbol{s}^{2}))$  [11]. With the KL divergence KL  $[P_{\#}q_1 \| q_0]$  it's easy to verify that it leads to just a distance between means, disregarding any covariance information. While certainly this represents a valid choice, we argue that we can find a better solution by using the Wasserstein distance. For Gaussian measures, the Wasserstein distance has analytic solution:
$$
\begin{array}{l} \mathcal {W} _ {2} ^ {2} \left(q _ {1}, q _ {0}\right) = \left\| \boldsymbol {m} _ {0} - \boldsymbol {m} _ {1} \right\| _ {2} ^ {2} + \operatorname {T r} \left(\boldsymbol {S} _ {1} + \boldsymbol {S} _ {0} - 2 \left(\boldsymbol {S} _ {1} ^ {1 / 2} \boldsymbol {S} _ {0} \boldsymbol {S} _ {1} ^ {1 / 2}\right) ^ {1 / 2}\right) = \\ = \left\| \boldsymbol {m} _ {0} - \boldsymbol {m} _ {1} \right\| _ {2} ^ {2} + \left\| \boldsymbol {S} _ {0} - \boldsymbol {S} _ {1} \right\| _ {F} ^ {2}, \tag {16} \\ \end{array}
$$
where  $\| \cdot \| _F$  denotes the Frobenius norm,  $\| A\| = \sum_{ij}a_{ij}^2$ , and where the second line is valid only if the covariances commute ( $S_{1}S_{0} = S_{0}S_{1}$ ). In our case, then, we can simplify as follows:
$$
\mathcal {W} _ {2} ^ {2} \left(P _ {\#} q _ {1}, q _ {0}\right) = \| \boldsymbol {m} _ {0} - \boldsymbol {P m} _ {1} \| _ {2} ^ {2} + \| \boldsymbol {s} _ {0} - \boldsymbol {P s} _ {1} \| _ {2} ^ {2}. \tag {17}
$$
To summarize, the problem now can be written as: