SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tejas Jayashankar<sup>1</sup> Gary C.F. Lee<sup>1</sup> Alejandro Lancho<sup>1,2</sup> Amir Weiss<sup>1</sup>
Yury Polyanskiy<sup>1</sup> Gregory W. Wornell<sup>1</sup>
<sup>1</sup>Massachusetts Institute of Technology <sup>2</sup>Universidad Carlos III de Madrid
We propose a new method for separating superimposed sources using diffusion-based generative models. Our method relies only on separately trained statistical priors of independent sources to establish a new objective function guided by maximum a posteriori estimation with an  $\alpha$ -posterior, across multiple levels of Gaussian smoothing. Motivated by applications in radio-frequency (RF) systems, we are interested in sources with underlying discrete nature and the recovery of encoded bits from a signal of interest, as measured by the bit error rate (BER). Experimental results with RF mixtures demonstrate that our method results in a BER reduction of  $95\%$  over classical and existing learning-based methods. Our analysis demonstrates that our proposed method yields solutions that asymptotically approach the modes of an underlying discrete distribution. Furthermore, our method can be viewed as a multi-source extension to the recently proposed score distillation sampling scheme, shedding additional light on its use beyond conditional sampling. The project webpage is available at https://alpha-rgs.github.io.
The problem of single-channel source separation (SCSS) is ubiquitous and arises in many different applications, ranging from the cocktail party problem in the audio domain to interference mitigation in the digital communications domain. Broadly speaking, the goal in SCSS is to decompose a mixture  $\mathbf{y} = \kappa_1\mathbf{x}_1 + \dots +\kappa_K\mathbf{x}_K\in \mathcal{Y}^N$  into its  $K$  constituent components,  $\{\mathbf{x}_i\}_{i = 1}^K,\mathbf{x}_i\in \mathcal{X}_i^N$ . Motivated by applications in modern engineering, e.g., intelligent communication systems, we are particularly interested in the SCSS problem for heterogeneous sources with underlying discrete structure.
Prior art that uses deep learning for source separation problems has been well-studied in the audio and image domain by leveraging domain-specific structures. For example, recent end-to-end methods in the audio domain leverage spectrogram masking techniques to separate sparse time-frequency representations [1, 2], while in the visual domain, natural images may be separable by exploiting local features and "smoothness" assumptions in the color space [3, 4]. More recent efforts have tried to solve this problem by leveraging independently trained statistical priors using annealed Langevin dynamics sampling [5-7]. However, these methods have demonstrated shortcomings on discrete sources that exhibit intricate temporal structures with equiprobable multimodal distributions [8].
These additional challenges motivate us to develop a novel general Bayesian framework for source separation that relies on statistical priors over sources with underlying discreteness. Specifically, we focus on learning data-driven priors for two main reasons—i) Unknown system parameters, e.g., the underlying signal generation model may not be available to create hand-crafted priors; and ii) Automation, facilitated by learning methods to create plug-and-play priors for versatile use.
In this paper, we study the SCSS problem involving a mixture of two signals,  $\mathbf{s}$  and  $\mathbf{b}$ , which interfere at a different relative scale to produce the mixture signal  $\mathbf{y} = \mathbf{s} + \kappa \mathbf{b}$ , where  $\kappa \in \mathbb{R}_+$  is the scaling coefficient. We additionally impose restrictions on access to paired training data samples ( $\mathbf{s},\mathbf{b},\mathbf{y}$ ) during training. Our motivation for this restriction arises from the following consideration: given  $n$  sources, the number of two-component source separation models leveraging joint-statistics grows as  $\mathcal{O}(n^2)$ . Any changes to the training data—even for a single source—would require  $\mathcal{O}(n)$  model updates. In contrast, solutions that leverage independently trained priors over the sources need to update only a single model (i.e.,  $\mathcal{O}(1)$ ). Additionally, these priors can also be used to solve more generalized source-separation problems, e.g., with  $K > 2$  components or with a different mixture model, without requiring any additional training.
We demonstrate the successful operation of our method in the digital radio-frequency (RF) setting. Signals in the RF domain are often discrete in nature as their randomness arises from bits/symbols that modulate a continuous waveform. As such, source separation performance is measured not only through continuous reconstruction metrics, e.g., the mean squared error (MSE), but also (and primarily) by the fidelity of digital communication, commonly measured by the bit error rate (BER).
Broader Impact. The crux of our work is motivated by challenges arising in the next generation of wireless systems. The proliferation of intelligent wireless devices—utilizing finite spectrum resources—has called for better interoperability strategies, so as to ensure reliable operations in a rapidly growing heterogeneous wireless networking ecosystem [9-11]. Interference from other sources operating in the same channel, e.g., 5G waveforms or WiFi signals, can lead to deterioration in quality of service. While conventional approaches treat such interference as Gaussian noise, there could be significant performance gains if we were able to learn and exploit intricate statistical structures of these co-channel signals [12]. We take an important step towards this goal by developing data-driven score-based SCSS solutions that demonstrate significant gains over modern and conventional approaches, thereby unveiling novel intelligent and effective interference mitigation strategies. We further elaborate on the broader impact of our work in Appendix A.
Contributions. We start from first principles and propose a new Bayesian-inspired method for SCSS. Our main contributions are as follows.
- We present a new method for SCSS that leverages the (approximate) score from pre-trained diffusion models to extend MAP estimation using generalized Bayes' theorem with an  $\alpha$ -posterior [13, 14] across different levels of Gaussian smoothing. Our method, termed as  $\alpha$ -RGS ( $\alpha$ -posterior with Randomized Gaussian Smoothing) (see §3), is easy to implement by randomizing the predetermined training noise levels without cumbersome tuning of a special annealing noise schedule, as in Langevin-dynamics-based approaches [5].
- We show that our formulation is a multi-source extension of the recently proposed score distillation sampling (SDS) objective [15]. Our analysis (see §4) shows that despite randomizing across multiple noise levels, the local extrema of our loss (and hence of SDS) asymptotically approach the modes of the underlying unsmoothed source distribution(s) (see §3.2.1).
- We demonstrate through experimental results (see §5) that  $\alpha$ -RGS outperforms classical signal processing and annealed Langevin-dynamics-based approaches for RF source separation, with a  $96\%$  and  $94.5\%$  improvement in BER (averaged across different levels of interference) over traditional and existing learning-based methods, respectively. While score-based methods have been adopted to solve problems such as symbol detection [16] and channel estimation [17], to the best of our knowledge, this is the first work that applies score-based models for the task of source separation in the RF domain.
Paper Organization. The paper is organized as follows. The necessary prerequisites are introduced in §2. §3 meticulously develops our proposed method, starting from basic principles. We provide analytical characterizations of our method in §4, with additional details available in Appendix B. In §5, we introduce our problem in the context of digital communication signals and describe our experiments along with results. A detailed summary of our results is available in Appendix G. A primer on digital communication signals and details about our datasets are available in Appendix C. Details related to diffusion models in the RF context, a primer on conventional baselines, and a description of score-based separation baselines can be found in Appendices D, E and F, respectively. We conclude with §6, where we comment on future work and on the broader impact of our method.
2.1 Finite Alphabet Signal Processing
In many engineering systems, signals commonly exhibit continuous magnitudes. However, in particular cases, these signals may possess discrete properties, leading to a finite (but possibly large) number of possible realizations. Such signals are often expressed as,
$$
s (t) = \sum_ {p = - \infty} ^ {\infty} c _ {p} g _ {p} \left(t - t _ {p}\right), \tag {1}
$$
where  $c_{p} \in S$  are discrete symbols drawn from a finite set  $S \subset \mathbb{C}$  (or  $\mathbb{R}$ ) and  $g_{p}(\cdot)$  is a continuous filter that "carries" contributions from the symbols. For example, in optical or RF communications, the symbols could correspond to complex-valued mappings of the underlying bits [18], while in the discrete tomography domain, the symbols might correspond to measurements obtained from different materials with a finite number of phases or absorption values [19]. In this work, we will address the challenges associated with finding the global optimum within the optimization landscape formulated for separating a superposition of such discrete sources.
Diffusion models are a class of generative models introduced by Sohl-Dickstein et al. [20] based on the principles of thermodynamic diffusion. The model is parametrized by two Markov chains of length  $T$  —i) the forward chain  $q(\mathbf{x}_t|\mathbf{x}_{t-1})$ , in which noise is gradually added to the data  $\mathbf{x}_0$  such that at the end of the forward process,  $q(\mathbf{x}_T) = \mathcal{N}(\mathbf{x}_T; 0, \mathbf{I})$ ; and ii) the learned reverse chain  $p_{\phi}(\mathbf{x}_{t-1}|\mathbf{x}_t)$ , in which noise is gradually removed (i.e., denoising) in order to generate new samples.
These models are trained by minimizing a variational lower bound on the log-likelihood. Recent work by Ho et al. [21] in the image domain shows that this optimization problem translates to learning a denoiser  $r_{\phi}(\mathbf{x}_t,t)$  parametrized as a neural network with weights  $\phi$  that estimates the noise term in  $\mathbf{x}_t\coloneqq \sqrt{\alpha_t}\mathbf{x}_0 + \sqrt{1 - \alpha_t}\mathbf{z},\mathbf{z}\sim \mathcal{N}(0,\mathbf{I})$  across different timesteps  $t\sim \mathrm{Unif}(\{1,\dots ,T\})$
$$
\phi^ {*} = \underset {\phi} {\arg \min } \mathbb {E} _ {\mathbf {t}, \mathbf {x}, \mathbf {z}} \left[ \| r _ {\phi} (\mathbf {x} _ {t}, t) - \mathbf {z} \| _ {2} ^ {2} \right], \tag {2}
$$
where  $\{1 - \alpha_i\}_{i = 1}^T$  defines an increasing signal variance-preserving noise schedule with  $0\leq \alpha_{i}\leq 1$ . The works in [21-23] show that objective (2) is equivalent to minimizing a loss whose minimizer is an estimate of the marginal score  $S_{\mathbf{x}_t}(\mathbf{x}_t) = \nabla_{\mathbf{x}_t}\log q(\mathbf{x}_t)$  such that,
$$
S _ {\mathbf {x} _ {t}} (\mathbf {x} _ {t}) \approx - (1 - \alpha_ {t}) ^ {- 1 / 2} r _ {\phi} (\mathbf {x} _ {t}, t). \tag {3}
$$
Other works have extended diffusion models to continuous time using stochastic differential equations [23], while others have used them for signals in the audio domain [24] and forecasting problems [25].
3 A Bayesian Framework for Source Separation
We now develop the Bayesian framework underlying our proposed source separation method. We are particularly motivated in developing a generalized framework that relies on statistical priors only, without any other form of domain-specific regularization.
3.1 Maximum a Posteriori (MAP) Source Separation
Signal statistical structure. Let  $\mathbf{s} \in S \subset \mathbb{C}^d$  and  $\mathbf{b} \in \mathbb{C}^d$  be two statistically independent complex-valued vector sources, where  $S$  is a countable set of all realizations  $\mathbf{s}$ . We assume that  $\mathbf{s}$  has PMF  $P_{\mathrm{s}}$ , and we let  $\mathbf{b}$  be an arbitrary source (potentially discrete with some noise) with PDF  $p_{\mathrm{b}}$ . We assume that the latter distributions are multimodal, where the probability is generally (close to) zero except at the modes. Additionally, we seek to be robust to the challenging setting where the distributions have multiple equiprobable modes, so as to develop novel methods that can tackle the finite alphabet source separation problem as is motivated in §1. We emphasize that these assumptions do not completely characterize the often complicated fine-grained statistical source structure, and we therefore rely on generative models to learn such unknown structures from data.
MAP problem formulation. Consider a mixture composed of two superimposed sources,
$$
\mathbf {y} = \mathbf {s} + \kappa \mathbf {b}, \tag {4}
$$
where  $\kappa \in \mathbb{R}_+$  is a relative scaling coefficient between the two signals.
Given  $\mathbf{y}$  and assuming  $\kappa$  is known, in order to separate the sources, it is sufficient to estimate  $\mathbf{s}$  since  $\mathbf{b} = (\mathbf{y} - \mathbf{s}) / \kappa$ . The MAP estimate of  $\mathbf{s}$  is then given by,
$$
\widehat {\mathbf {s}} = \underset {\mathbf {s} \in \mathcal {S} \text {s . t} \mathbf {y} = \mathbf {s} + \kappa \mathbf {b}} {\arg \max } p _ {\mathbf {s} | \mathbf {y}} (\mathbf {s} | \mathbf {y}). \tag {5}
$$
Using Bayes' theorem, (5) can be equivalently expressed as,
$$
\begin{array}{l} \underset {\mathbf {s} \in \mathcal {S} \text {s . t} \mathbf {y} = \mathbf {s} + \kappa \mathbf {b}} {\arg \max } p _ {\mathbf {s} | \mathbf {y}} (\mathbf {s} | \mathbf {y}) = \underset {\mathbf {s} \in \mathcal {S} \text {s . t} \mathbf {y} = \mathbf {s} + \kappa \mathbf {b}} {\arg \max } p _ {\mathbf {y} | \mathbf {s}} (\mathbf {y} | \mathbf {s}) P _ {\mathbf {s}} (\mathbf {s}) (6a) \\ = \underset {\mathbf {s} \in \mathcal {S}} {\arg \min } - \log P _ {\mathbf {s}} (\mathbf {s}) - \log p _ {\mathbf {b}} ((\mathbf {y} - \mathbf {s}) / \kappa), (6b) \\ \end{array}
$$
where the likelihood  $p_{\mathbf{y}|\mathbf{s}}(\mathbf{y}|\mathbf{s}) = p_{\mathbf{b}}((\mathbf{y} - \mathbf{s}) / \kappa)$  under the constraint (4), and we convert to negative log probabilities in (6b). Due to the underlying discreteness of s, and the potential underlying discreteness of b as well, the objective function in (6b) is not differentiable. Hence, in its current form, gradient-based techniques cannot be used to solve this problem, and we must instead resort to combinatorial methods, which are computationally infeasible even for moderate dimension size  $d$ .
3.2 Proposed Method  $(\alpha$  RGS)
To overcome the computational complexity of combinatorial-based methods, our goal is to develop new gradient-based SCSS solutions that leverage diffusion models trained on discrete sources. To this end, we propose to use multiple levels of Gaussian smoothing with an extended MAP framework with an  $\alpha$ -posterior, such that the optimization landscape of the resulting objective function is smoothened.
3.2.1 The Smoothing Model and  $\alpha$ -posterior Generalized Bayes'
Surrogate distribution. One can almost perfectly approximate  $P_{s}$  (in some well-defined sense) with the surrogate distribution  $p_{\bar{s}}$ , where  $\bar{\mathbf{s}} = \mathbf{s} + \varepsilon_{s}$  for  $\varepsilon_{s} \sim \mathcal{N}(0, \sigma_{s}^{2}\mathbf{I})$ ,  $\sigma_{s} \to 0$ . While now theoretically amenable to optimization via gradient descent, the sharp modes, which constitute numerical pitfalls, often cause gradient-based methods to get stuck in local extrema.
Gaussian Smoothing Model. We adopt a variance-preserving smoothing model, with adjustable noise levels, based on the formulation of the stochastic process proposed in [21]. Let  $\{1 - \alpha_t\}_{t=1}^T$  be a sequence of noise levels such that  $\alpha_i > \alpha_{i+1}$  and  $\alpha_i \in (0,1]$ . Define the "smoothened sources" as
$$
\tilde {\mathbf {s}} _ {t} (\bar {\mathbf {s}}) := \sqrt {\alpha_ {t} \bar {\mathbf {s}}} + \sqrt {1 - \alpha_ {t}} \mathbf {z} _ {s}, \tag {7a}
$$
$$
\tilde {\mathbf {b}} _ {u} (\bar {\mathbf {s}}, \mathbf {y}) := \sqrt {\alpha_ {u}} \left(\mathbf {y} - \bar {\mathbf {s}}\right) / \kappa + \sqrt {1 - \alpha_ {u}} \mathbf {z} _ {b}, \tag {7b}
$$
where  $t, u \sim \mathrm{Unif}\left(\{1, \dots, T\}\right)$  and  $\mathbf{z}_s, \mathbf{z}_b \sim \mathcal{N}(0, \mathbf{I}_d)$ . Observe that  $\tilde{\mathbf{s}}_t$  and  $\tilde{\mathbf{b}}_u$  the continuous-valued proxies for s and b, respectively—have PDFs rather than PMFs, and in particular, their PDFs have infinite support, and they are differentiable. More importantly, a direct consequence is that it smoothens the optimization landscape and helps in preventing gradient-based algorithms from getting stuck at spurious local minima.
Generalized Bayes' with an  $\alpha$ -posterior. We found it useful to replace the likelihood in (6a) with a distribution proportional to  $p_{\mathrm{y|s}}(\mathbf{y}|\mathbf{s})^{\omega}$ ,  $\omega > 1$ . Intuitively, under the constraint (4), this sharpens the distribution of  $\mathbf{b}$  and gives a higher weight to the modes of  $p_{\mathrm{b}}$  relative to the natural weighting that arises from the MAP criterion. This is beneficial, for example, when  $p_{\mathrm{b}}$  is more complicated and has many more modes than  $P_{s}$ .
This aforementioned reweighting has been used as an implementation trick in diffusion sampling with classifier conditioning [26], but we recognize this as MAP estimation with an  $\alpha$ -posterior<sup>1</sup> where  $\alpha = \omega$  which is expressed through the generalized Bayes' rule [13, 14, 27, 28] as,
$$
\underbrace {p _ {\mathrm {s} | \mathrm {y}} (\mathbf {s} | \mathbf {y} ; \omega)} _ {\alpha - \text {p o s t e r i o r w i t h} \alpha = \omega} \propto \underbrace {p _ {\mathrm {y} | \mathrm {s}} (\mathbf {y} | \mathbf {s}) ^ {\omega}} _ {\text {t e m p e r e d l i k e l i h o o d}} \underbrace {P _ {\mathrm {s}} (\mathbf {s})} _ {\text {p r i o r}}. \tag {8}
$$
$^{1}\alpha$  should not be confused with, and has no relation to,  $\alpha_{t}$  from the Gaussian smoothing model.
Algorithm 1 Proposed Method:  $\alpha$ -posterior with Randomized Gaussian Smoothing ( $\alpha$ -RGS)
1: function SEPARATION(y,  $\kappa$ ,  $N$ ,  $\{\eta_i\}_{i=0}^{N-1}$ ,  $\pmb{\theta}^{(0)}$ ) ▷  $N$  total steps, learning rate  $\eta_i$  at step  $i$   
2: for  $i \gets 0, N-1$  do  
3:  $t, u \sim \mathrm{Uni}\{1, \dots, T\}$ ,  $\mathbf{z}_s, \mathbf{z}_b \sim \mathcal{N}(0, \mathbf{I})$   
4:  $\tilde{\mathbf{s}}_t(\pmb{\theta}^{(i)}) = \sqrt{\alpha_t} \pmb{\theta}^{(i)} + \sqrt{1 - \alpha_t} \mathbf{z}_s$   
5:  $\tilde{\mathbf{b}}_u(\pmb{\theta}^{(i)}, \mathbf{y}) = \sqrt{\alpha_u} (\mathbf{y} - \pmb{\theta}^{(i)}) / \kappa + \sqrt{1 - \alpha_u} \mathbf{z}_b$   
6:  $\widehat{\mathbf{z}}_s, \widehat{\mathbf{z}}_b = r_{\phi_s}(\tilde{\mathbf{s}}_t(\pmb{\theta}^{(i)}), t)$ ,  $r_{\phi_b}(\tilde{\mathbf{b}}_u(\pmb{\theta}^{(i)}, \mathbf{y}), u)$   
7:  $\pmb{\theta}^{(i+1)} = \pmb{\theta}^{(i)} - \eta_i\left[\frac{\omega}{\kappa} \frac{\sqrt{\alpha_u}}{\sqrt{1 - \alpha_u}} (\widehat{\mathbf{z}}_b - \mathbf{z}_b) - \frac{\sqrt{\alpha_t}}{\sqrt{1 - \alpha_t}} (\widehat{\mathbf{z}}_s - \mathbf{z}_s)\right]$   
8: end for  
9: return  $\widehat{\mathbf{s}} = \pmb{\theta}^{(N)}$ ,  $\widehat{\mathbf{b}} = (\mathbf{y} - \widehat{\mathbf{s}})/\kappa$   
10: end function
In practice, using an  $\alpha$ -posterior has demonstrated increased learning speeds [13, 14] and could potentially help in resolving model mismatch arising from the use of approximate densities or scores (rather than exact ones) during optimization via gradient descent.
Single noise level estimation loss. Let  $\widehat{\mathbf{s}} (\theta) = \theta$  be our estimate of s. Motivated by the form in (6b), we generalize using the smoothing (7a)-(7b) in conjunction with the  $\alpha$ -posterior with  $\alpha = \omega$  to define a new single noise level estimation loss,
$$
\mathcal {L} _ {t, u} (\boldsymbol {\theta}) := - \log p _ {\tilde {\mathbf {s}} _ {t}} \left(\tilde {\mathbf {s}} _ {t} (\boldsymbol {\theta})\right) - \omega \log p _ {\tilde {\mathbf {b}} _ {u}} \left(\tilde {\mathbf {b}} _ {u} (\boldsymbol {\theta}, \mathbf {y})\right). \tag {9}
$$
While departing from the original MAP (6b), the newly-defined approximated loss (9) thereof facilitates gradient-based methods, which is key to our solution approach. Particularly, by varying the level of smoothing, (9) is more easily explored in regions between the modes via gradient descent.
3.2.2 Estimation Rule Across Multiple Noise Levels
Intuitively, larger noise variances allow us to move between modes during gradient descent. In contrast, at lower noise levels the modes are sharper, which is beneficial in resolving the solution, assuming the iterative procedure starts at the basin of attraction of the correct local extremum point, or, alternatively, another "good" local extremum point. We propose a new estimation rule that uses (9) across multiple noise levels. Randomizing over multiple levels of Gaussian smoothing, our proposed gradient update asymptotically (in the number of iterations) takes the form,
$$
\nabla_ {\boldsymbol {\theta}} \mathcal {L} (\boldsymbol {\theta}) := \underbrace {- \mathbb {E} _ {\mathrm {t} , \mathbf {z} _ {t}} \left[ \sqrt {\alpha_ {t}} S _ {\tilde {\mathbf {s}} _ {t}} \left(\tilde {\mathbf {s}} _ {t} (\boldsymbol {\theta})\right) \right] + \frac {\omega}{\kappa} \mathbb {E} _ {\mathrm {u}, \mathbf {z} _ {u}} \left[ \sqrt {\alpha_ {u}} S _ {\tilde {\mathbf {b}} _ {u}} \left(\tilde {\mathbf {b}} _ {u} (\boldsymbol {\theta}, \mathbf {y})\right) \right]} _ {\mathbb {E} _ {\mathrm {t}, \mathrm {u}} [ \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {t, u} (\boldsymbol {\theta}) ]}, \tag {10}
$$
where  $t, u \sim \mathrm{Unif}(\{1, \ldots, T\})$ , and the true score is defined as
$$
\left. S _ {\tilde {\mathbf {s}} _ {t}} \left(\tilde {\mathbf {s}} _ {t} (\boldsymbol {\theta})\right) := \nabla_ {\mathbf {x}} \log p _ {\tilde {\mathbf {s}} _ {t}} (\mathbf {x}) \right| _ {\mathbf {x} = \tilde {\mathbf {s}} _ {t} (\boldsymbol {\theta})}, \tag {11}
$$
and similarly for  $S_{\tilde{b}_u}$
Our proposed updates can be implemented using stochastic gradient descent as shown in Algorithm 1. We use pre-trained diffusion models (§2.2) to approximate the score when deriving the analytical score is not possible. Using (3), we relate the learned score to the denoiser available from the diffusion model, and also use a zero mean noise corrected estimate,
$$
\mathbb {E} _ {\mathbf {z} _ {s}} \left[ r _ {\phi_ {\mathbf {s}}} \left(\tilde {\mathbf {s}} _ {t} (\boldsymbol {\theta}), t\right) \right] = \mathbb {E} _ {\mathbf {z} _ {s}} \left[ r _ {\phi_ {\mathbf {s}}} \left(\tilde {\mathbf {s}} _ {t} (\boldsymbol {\theta}), t\right) - \mathbf {z} _ {s} \right], \tag {12}
$$
in our updates as shown in Algorithm 1. The above rule is motivated to introduce numerical stability and reduces the variance of the updates, since the diffusion models were trained to minimize the squared value of the same error term in (2). The state-of-the-art method of score distillation sampling (SDS) [15] that uses diffusion models as critics, also uses the same term in their gradient update step.
$^2$ Owing to the continuous nature of the optimization landscape,  $\pmb{\theta}$  is in fact an estimate of  $\overline{\mathbf{s}}$  (see §3.2.1), and does not affect BER and MSE estimates significantly.
Figure 1: Left: Two discrete sources, with infinitesimal additive noise, superimposed to produce a joint distribution with 8 equiprobable modes. An observed mixture  $\mathbf{y}$ , imposes a linear constraint in this space. Middle: Extending vanilla MAP  $(\omega = 1)$  to multiple noise levels still has a relatively large local minima. Right: By using  $\omega = \kappa^2$ , we are able to accentuate the correct mode and smooth the landscape even further. Colored curves correspond to (15) evaluated with  $T = 1$  and  $t = u$ .