SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Victor Letzelter $^{1,2}$
Mathieu Fontaine
Mickael Chen
Slim Essid
Gael Richard
$^{2}$  LTCI, Télécom Paris, Institut Polytechnique de Paris, France
We introduce Resilient Multiple Choice Learning (rMCL), an extension of the MCL approach for conditional distribution estimation in regression settings where multiple targets may be sampled for each training input. Multiple Choice Learning is a simple framework to tackle multimodal density estimation, using the Winner-Takes-All (WTA) loss for a set of hypotheses. In regression settings, the existing MCL variants focus on merging the hypotheses, thereby eventually sacrificing the diversity of the predictions. In contrast, our method relies on a novel learned scoring scheme underpinned by a mathematical framework based on Voronoi tessellations of the output space, from which we can derive a probabilistic interpretation. After empirically validating rMCL with experiments on synthetic data, we further assess its merits on the sound source localization task, demonstrating its practical usefulness and the relevance of its interpretation.
Machine learning models are commonly trained to produce, for any given input, a single prediction. In most cases, for instance, when minimizing the empirical risk with quadratic loss, this prediction can be interpreted as the conditional output expectation given the input. However, there are many tasks for which the conditional output distribution can be multimodal, either by the nature of the task or due to various sources of uncertainty. Temporal tracking and forecasting, for instance, are problems of this type [5, 32, 44]. In such cases, the mean might fall in a low-density region of the conditional probability function, and it would be beneficial to predict multiple hypotheses instead [34, 20]. In this context, Multiple Choice Learning (MCL) has emerged as a simple pragmatic solution [17, 29]. Thanks to a network architecture with multiple heads, MCL models can produce multiple hypotheses, one per head. During supervised training, the gradients are only computed for the head that provides the best prediction given the current input sample. This Winner-Takes-All (WTA) training scheme allows each head to specialize in a region of the output space. Accordingly, Rupprecht et al. [34] have proposed a probabilistic interpretation of MCL based on Voronoi tessellations.
On that account, MCL suffers from two significant issues: hypotheses collapse and overconfidence. Hypotheses collapse occurs during training when a head takes the lead, the others being almost never selected under WTA and thus not updated. This leads to a situation where most heads are not trained effectively and produce meaningless outputs during inference. Overconfidence can also be observed at inference time when looking at MCL under the lens of its probabilistic interpretation. Hypotheses that correspond to rare events tend to be over-represented, thus not reflecting the true distribution of
outputs well. Multiple variants of MCL have been developed to alleviate these issues, in particular for classification or segmentation tasks [29, 27, 42, 13].
In this paper, we are interested in MCL in the context of regression settings, which has been mostly overlooked. More specifically, we propose rMCL, for resilient Multiple Choice Learning, a MCL variant with a learned scoring scheme to mitigate overconfidence. Unlike prior work, rMCL is designed to handle more general settings where multiple targets may be sampled for each input during training. Moreover, we propose a probabilistic interpretation of rMCL, and show that rMCL's scoring fixes overconfidence by casting multi-choice regression as conditional distribution estimation. While introduced in the context of the original WTA loss, the proposed scoring approach is also compatible with other variants of WTA. We validate our claims with experiments on synthetic data, and we further evaluate the performance of rMCL on the sound source localization problem. The accompanying code with a rMCL implementation is made available.<sup>1</sup>
Uncertainty estimation. The estimation of machine learning model's uncertainty can be studied from different perspectives depending on the type of ambiguity being considered. The uncertainty can concern either the model (epistemic) or the data (aleatoric) [7, 12]. Kendall and Gal [22] showed that those types of uncertainty can be combined in deep learning settings, placing both a distribution on the weights and the outputs of the model. Epistemic uncertainty estimation may, for instance, be achieved by variational approximation of the weights posterior [31, 14]. Independent ensembles (IE) of neural networks [18, 11] is another widespread technique for epistemic uncertainty approximation. Several neural networks, trained from different random initializations, provide samples for approximating the weights posterior distribution [26].
Multiple Choice learning. Originally introduced by Guzman-Rivera et al. [17] and adapted to deep learning settings by Lee et al. [28, 29], MCL is a framework in which multiple output heads propose different possible predictions. Since many datasets contain only a single output realization for each input, the heads are trained with a Winner-Takes-All scheme where only the head that made the best prediction is updated. It can be understood as a specific case of dependent ensembles [8, 46, 4, 47] where the members interact with each other during training. Alternatively, [34] have shown that MCL can be adapted successfully for multi-label classification [23].
The Winner-Takes-All training scheme, however, can cause two main issues known as hypotheses collapse and overconfidence [34, 6, 10, 19, 10, 27, 42]. Most related previous works tackle either or both of those questions. Hypotheses collapse occurs during training when some heads receive little or no updates due to bad initial values under the WTA loss and other hypotheses taking the lead. To fix this issue, Rupprecht et al. [34] proposed a relaxed Winner-Takes-All (RWTA, or  $\varepsilon$ -WTA) loss that allows the update of non-winner heads, albeit with gradients scaled by a small constant  $\varepsilon$ . While this approach ensures that every head receives gradients updates, it also tends to level the different heads, which counters the initial goal of WTA and thus needs careful tuning. Finally, one can leverage the evolving Winner-Takes-All loss [32], with the top- $n$  heads getting updated (top- $n$ -WTA) instead of only the best one. The authors validate that their approach, with a scheduled decreasing value for  $n$ , achieves improved conditioning of the outputs by reducing inconsistent hypotheses in the context of future frames prediction.
On the other hand, overconfidence is an issue that can be observed in inference when evaluating a model as a density estimator. In this case, one would want the different outputs to be distributed across the different modes of the true conditional probability of the outputs. Empirical observations [27, 42] show that this is not usually the case, and rare events tend to be overly represented, rendering such a model inadequate for integration within real-world decision-making systems. In this context, Lee et al. [27] proposed Confident Multiple Choice Learning (CMCL) for solving overconfidence in a classification setup. Additionally to the WTA loss, CMCL is trained by maximizing the entropy of the class distributions predicted by the non-selected hypotheses. The classifier's final prediction is based on a simple average of the discrete class distributions predicted by each of the heads. Although designed for tackling the overconfidence problem, CMCL reduces the diversity of the hypotheses. On that account, Versatile Multiple Choice Learning [42] (vMCL) proposed to address the issue by leveraging a choice network aiming at predicting a score for each hypothesis head. This may be seen
<sup>1</sup>https://github.com/Victorletzelter/code-rMCL
as a variant of Mixture-of-Experts approaches [33], in which the choice network is supervised with explicit targets. The final prediction in vMCL is derived by weighting the class distributions from each hypothesis by their respective scores. These works focus on classification tasks and aggregate the hypotheses, thereby losing individual information from each head. In contrast, we address regression tasks and demonstrate how to benefit from the diversity of predictions without aggregation.
We propose to revisit the vMCL approach for regression, single-target or multi-target, and to extend accordingly the mathematical interpretation of the WTA proposed by Rupprecht et al. [34].
3 rMCL regression and its probabilistic interpretation
After reviewing MCL, this section introduces resilient Multiple Choice Learning, dubbed rMCL. This proposed variant handles multi-target settings thanks to its scoring scheme. We then provide a distribution learning interpretation of rMCL that is relevant to regression tasks, thus completing the MCL probabilistic interpretations of prior works [9, 34].
3.1 Fixing the overconfidence issue in Multiple Choice Learning
Let  $\mathcal{X} \subseteq \mathbb{R}^d$  and  $\mathcal{Y} \subseteq \mathbb{R}^q$  be the input and output spaces in a supervised learning setting, such that the training set  $\mathcal{D}$  is composed of samples  $(\pmb{x}_s, \pmb{y}_s)$  from an underlying joint distribution  $p(\pmb{x}, \pmb{y})$  over  $\mathcal{X} \times \mathcal{Y}$ .
Multiple choice learning was proposed to address tasks with ambiguous outputs, i.e., for which the ground-truth distribution  $p(\boldsymbol{y} \mid \boldsymbol{x})$  is multimodal [17, 29, 10]. Adapted by Lee et al. [29] for deep-learning settings in Stochastic Multiple Choice Learning (sMCL), it leverages several models  $f_{\theta} \triangleq (f_{\theta}^{1}, \ldots, f_{\theta}^{K}) \in \mathcal{F}(\mathcal{X}, \mathcal{Y}^{K})$ , referred to as  $K$  hypotheses, trained using the Winner-takes-all (or Oracle) loss. It consists, given an underlying loss function  $\ell$  and for each sample  $(\boldsymbol{x}_s, \boldsymbol{y}_s)$  in the current batch, of first the computation of
$$
\mathcal {L} \left(f _ {\theta} \left(\boldsymbol {x} _ {s}\right), \boldsymbol {y} _ {s}\right) \triangleq \min  _ {k \in [ 1, K ]} \ell \left(f _ {\theta} ^ {k} \left(\boldsymbol {x} _ {s}\right), \boldsymbol {y} _ {s}\right) \tag {1}
$$
after each forward pass, followed by backpropagation on the winner hypothesis, that is, the minimizing one. Assuming now that a set  $\mathbf{Y}_s$  of targets is available in the training set for input  $\mathbf{x}_s$ , Firman et al. [10] have shown that (1) can be generalized by updating the best hypothesis per target using the meta-loss
$$
\mathcal {L} \left(f _ {\theta} \left(\boldsymbol {x} _ {s}\right), \boldsymbol {Y} _ {s}\right) = \sum_ {\boldsymbol {y} \in \boldsymbol {Y} _ {s}} \sum_ {k = 1} ^ {K} \mathbf {1} \left(\boldsymbol {y} \in \mathcal {Y} ^ {k} \left(\boldsymbol {x} _ {s}\right)\right) \ell \left(f _ {\theta} ^ {k} \left(\boldsymbol {x} _ {s}\right), \boldsymbol {y}\right), \tag {2}
$$
$$
\mathcal {Y} ^ {k} (\boldsymbol {x}) \triangleq \left\{\boldsymbol {y} \in \mathcal {Y}, \ell \left(f _ {\theta} ^ {k} (\boldsymbol {x}), \boldsymbol {y}\right) <   \ell \left(f _ {\theta} ^ {r} (\boldsymbol {x}), \boldsymbol {y}\right), \forall r \neq k \right\}. \tag {3}
$$
This loss can, however, lead to poor conditioning of the output set of predictions (hypotheses collapse), with only one or a few hypotheses being exclusively selected for backpropagation. Furthermore, sMCL is subject to the overconfidence problem. Following Theorem 1 of [34], a necessary condition for minimizing the risk
$$
\int_ {\mathcal {X}} \sum_ {k = 1} ^ {K} \int_ {\mathcal {Y} ^ {k} (\boldsymbol {x})} \ell \left(f _ {\theta} ^ {k} (\boldsymbol {x}), \boldsymbol {y}\right) p (\boldsymbol {x}, \boldsymbol {y}) d \boldsymbol {y} d \boldsymbol {x}, \tag {4}
$$
is that each prediction  $f_{\theta}^{k}(\boldsymbol{x})$  coincides with the centroid of  $\mathcal{Y}^k(\boldsymbol{x})$  with respect to  $p(\boldsymbol{y} \mid \boldsymbol{x})$ . We say that the components  $\{\mathcal{Y}^k(\boldsymbol{x})\}_k$  form a centroidal Voronoi tessellation [9]. If  $\ell$  is for instance the  $\ell_2$ -loss, this condition means that for each non-zero probability cell  $k$ ,  $f_{\theta}^{k}(\boldsymbol{x})$  amounts to the conditional mean of  $p(\boldsymbol{y} \mid \boldsymbol{x})$  within the Voronoi cell  $\mathcal{Y}^k(\boldsymbol{x})$ . However, this theorem tells us nothing about the predictions in very low probability zones; in such regions, the inference-time predictions  $f_{\theta}^{k}(\boldsymbol{x})$  will be meaningless. During inference, the sMCL indeed faces a limitation: it cannot solely rely on the predicted hypotheses to identify Voronoi cells in the output space with low probability mass (see Figure 1). This observation leads us to propose hypothesis-scoring heads  $\gamma_{\theta}^{1}, \ldots, \gamma_{\theta}^{K} \in \mathcal{F}(\mathcal{X}, [0,1])$ . Those aim to predict, for unseen input  $\boldsymbol{x}$ , the probability  $\mathbb{P}(Y_{\boldsymbol{x}} \in \mathcal{Y}^k(\boldsymbol{x}))$  where  $Y_{\boldsymbol{x}} \sim p(\boldsymbol{y} \mid \boldsymbol{x})$  with the aim of mitigating this overconfidence issue.
While MCL variants have often been proposed for classification tasks, we are, to the best of our knowledge, the first to propose to solve this issue for multi-target regression, interpreting the problem as a multimodal conditional distribution estimation.
3.2 Resilient Multiple Choice Learning
We consider hereafter a problem of estimation of a multimodal conditional distribution denoted  $p(\boldsymbol{y} \mid \boldsymbol{x})$  for each input  $\boldsymbol{x} \in \mathcal{X}$ . In a real-world setting, only one or several samples (the targets) drawn from  $p(\boldsymbol{y} \mid \boldsymbol{x})$  are usually accessible. Sound source localization is a concrete instance of such multimodal prediction for whose  $p(\boldsymbol{y} \mid \boldsymbol{x}_s)$  represents the sound source position for an input audio clip  $\boldsymbol{x}_s \in \mathcal{X}$  at a given time  $t$ .
In this last example, each target sample in  $\mathbf{Y}_s$  may represent the location of a mode of the ground-truth multimodal distribution. For such multi-output regression tasks, the MCL training of a randomly initialized multi-hypotheses model with scoring functions,  $(f_{\theta}^{1},\dots ,f_{\theta}^{K},\gamma_{\theta}^{1},\dots ,\gamma_{\theta}^{K})$  can be adapted as follows.
For each training sample  $(\pmb{x}_s, \pmb{Y}_s)$ , let
$$
\mathcal {K} _ {+} \left(\boldsymbol {x} _ {s}\right) \triangleq \left\{k ^ {+} \in [ 1, K ]: \exists \boldsymbol {y} \in \boldsymbol {Y} _ {s}, k ^ {+} \in \operatorname * {a r g m i n} _ {k} \ell \left(f _ {\theta} ^ {k} \left(\boldsymbol {x} _ {s}\right), \boldsymbol {y}\right) \right\} \tag {5}
$$
and  $\mathcal{K}_{-}(\pmb{x}_s) \triangleq [[1, K]] - \mathcal{K}_{+}(\pmb{x}_s)$  be the set of positive (or winner) and negative hypotheses respectively. It is then possible to combine the multi-target WTA loss  $\mathcal{L}$  in (2) with a hypothesis scoring loss
$$
\mathcal {L} _ {\text {s c o r i n g}} (\theta) \triangleq - \left(\sum_ {k ^ {+} \in \mathcal {K} _ {+} \left(\boldsymbol {x} _ {s}\right)} \log \boldsymbol {\gamma} _ {\theta} ^ {k ^ {+}} \left(\boldsymbol {x} _ {s}\right) + \sum_ {k ^ {-} \in \mathcal {K} _ {-} \left(\boldsymbol {x} _ {s}\right)} \log \left(1 - \boldsymbol {\gamma} _ {\theta} ^ {k ^ {-}} \left(\boldsymbol {x} _ {s}\right)\right)\right), \tag {6}
$$
in a compound loss  $\mathcal{L} + \beta \mathcal{L}_{\mathrm{scoring}}$ . This novel approach differs from previous MCL variants [42] in its ability to predict multimodal distributions in regression settings and by the introduction of separated scoring branches that are updated based on loss (6), such that the target for the scoring branch  $k$  is the probability that hypothesis  $k$  is among the winners for that sample.
A resource-efficient implementation of rMCL is achieved by deriving the hypotheses and score heads from a shared representation, with distinct parameters at the final stages of the architecture (e.g., through independent fully connected layers). Designed as such, it is also possible to reduce memory cost by updating only a fraction of score heads associated with the negative hypotheses at each training step, typically with distinct samples  $k^{-} \sim \mathcal{U}\left(\mathcal{K}_{-}\left(\boldsymbol{x}_{s}\right)\right)$ . This trick could also alleviate the imbalanced binary classification task that the scoring heads face with a large number of hypotheses  $|\mathcal{K}_{-}\left(\boldsymbol{x}_{s}\right)| \gg |\mathcal{K}_{+}\left(\boldsymbol{x}_{s}\right)|$  (typically, the number of target samples at disposal is small relative to the number of hypotheses  $K$ ). The variants of the WTA (e.g., top-n-WTA,  $\varepsilon$ -WTA, see Sec. 2) are also compatible with rMCL. Inference with the proposed rMCL model is outlined in Algorithm 1.
As highlighted above, the hypothesis output can be interpreted, in the context of risk minimization, as the conditional mean of the Voronoi cell  $\mathcal{Y}^k (\pmb {x})$  it defines, providing that the cell has non-zero probability. Furthermore, given (6), the output of the score head  $\gamma_{\theta}^{k}(\pmb {x})$  can be interpreted as an approximation of the probability of a sample from  $p(\pmb {y}\mid \pmb {x})$  to belong to this cell.
Algorithm 1 Inference in the rMCL model
Input: Unlabelled input  $\pmb{x} \in \mathcal{X}$ . Trained hypotheses and score heads  $(f_{\theta}^{1}, \dots, f_{\theta}^{K}, \gamma_{\theta}^{1}, \dots, \gamma_{\theta}^{K}) \in \mathcal{F}(\mathcal{X}, \mathcal{Y})^{K} \times \mathcal{F}(\mathcal{X}, [0, 1])^{K}$ .
Output: Prediction of the output conditional distribution  $p(\mathbf{y} \mid \mathbf{x})$ .
1: Perform a forward pass by computing  $f_{\theta}^{1}(\pmb{x}), \dots, f_{\theta}^{K}(\pmb{x}), \gamma_{\theta}^{1}(\pmb{x}), \dots, \gamma_{\theta}^{K}(\pmb{x})$ .
2: Construct the associated Voronoi components  $\mathcal{V}^k (\pmb {x})$  (3) with  $\mathcal{V} = \cup_{k = 1}^{K}\overline{\mathcal{V}^k(\pmb{x})}$ .
3:Normalize the predicted scores  $\gamma_{\theta}^{k}(\pmb {x})\gets \gamma_{\theta}^{k}(\pmb {x}) / \sum_{k = 1}^{K}\gamma_{\theta}^{k}(\pmb {x}).$
4: If  $Y_{\pmb{x}} \sim p(\pmb{y} \mid \pmb{x})$ , then for  $k$  such that  $\gamma_{\theta}^{k}(\pmb{x}) > 0$ , interpret the predictions as estimations of
$$
\left\{ \begin{array}{l} \gamma_ {\theta} ^ {k} (\boldsymbol {x}) = \mathbb {P} \left(Y _ {\boldsymbol {x}} \in \mathcal {Y} ^ {k} (\boldsymbol {x})\right) \\ f _ {\theta} ^ {k} (\boldsymbol {x}) = \mathbb {E} \left[ Y _ {\boldsymbol {x}} \mid Y _ {\boldsymbol {x}} \in \mathcal {Y} ^ {k} (\boldsymbol {x}) \right] \end{array} \right. \tag {7}
$$
3.3 Probabilistic interpretation at inference time
Let us consider a trained rMCL model, such as the one described in the previous section. Following the theoretical interpretation of [34] and the motivation of solving the overconfidence issue, as explained in Section 3.1, the output objectives of the model can be summarized in (7). Although it is possible, assuming that those properties are verified, to deduce the following law of total expectation
$$
\mathbb {E} \left[ Y _ {\boldsymbol {x}} \right] = \sum_ {k = 1} ^ {K} \gamma_ {\theta} ^ {k} (\boldsymbol {x}) f _ {\theta} ^ {k} (\boldsymbol {x}), \tag {8}
$$
the above quantity may not be informative enough, especially if the law of  $Y_{\pmb{x}} \sim p(\pmb{y}|\pmb{x})$  is multimodal. Indeed, to be able to derive a complete probabilistic interpretation, we still need to characterize how rMCL approximates the full conditional density of  $Y_{\pmb{x}}$ . For that purpose, it is sufficient to fix a law  $\pi_k$  for  $Y_{\pmb{x}}$  within each Voronoi cell  $k$  to accurately represent the predicted distribution in each region of the output space as per the rMCL model, as shown in Proposition 1.
Proposition 1 (Probabilistic interpretation of rMCL). With the above notations, let us consider a multi-hypothesis model with properties of (7). Let us furthermore assume that
$$
Y _ {\boldsymbol {x}} ^ {k} \sim \pi_ {k} (\cdot | \boldsymbol {x}) \tag {9}
$$
i.e., the law  $Y_{\pmb{x}}^{k}$  of  $Y_{\pmb{x}}$  conditioned by the event  $\{Y_{\pmb{x}} \in \mathcal{Y}^{k}(\pmb{x})\}$  is described by  $\pi_k$  for each input  $\pmb{x} \in \mathcal{X}$  with constraint  $\mathbb{E}[Y_{\pmb{x}}^{k}] = f_{\theta}^{k}(\pmb{x})$ . Then, for each measurable set  $A \subseteq \mathcal{V}$ ,
$$
\mathbb {P} \left(Y _ {\boldsymbol {x}} \in A\right) = \int_ {\boldsymbol {y} \in A} \sum_ {k = 1} ^ {K} \gamma_ {\theta} ^ {k} (\boldsymbol {x}) \pi_ {k} (\boldsymbol {y} \mid \boldsymbol {x}) \mathrm {d} \boldsymbol {y}. \tag {10}
$$
$$
\begin{array}{l} \mathbb {P} (Y _ {\boldsymbol {x}} \in A) = \sum_ {k} \mathbb {P} (Y _ {\boldsymbol {x}} \in A \cap \mathcal {Y} ^ {k} (\boldsymbol {x})) \\ = \sum_ {k} ^ {n} \mathbb {P} \left(Y _ {\boldsymbol {x}} \in \mathcal {Y} ^ {k} (\boldsymbol {x})\right) \mathbb {P} \left(Y _ {\boldsymbol {x}} \in A \mid Y _ {\boldsymbol {x}} \in \mathcal {Y} ^ {k} (\boldsymbol {x})\right) \\ = \sum_ {k} ^ {n} \gamma_ {\theta} ^ {k} (\boldsymbol {x}) \int_ {\boldsymbol {y} \in A} \pi_ {k} (\boldsymbol {y} \mid \boldsymbol {x}) \mathrm {d} \boldsymbol {y} \quad (\text {b y (9)}). \\ \end{array}
$$
Following the principle of maximum entropy [38], the uniform distribution inside each cell  $Y_{\pmb{x}}^{k} \sim \mathcal{U}(\mathcal{Y}^{k}(\pmb{x}))$  is the least-informative distribution, assuming that the output space  $\mathcal{V}$  is with finite volume and the hypotheses lie in the geometric center of their associated cell. With this prior, the predicted distribution can be interpreted as
$$
\hat {p} (\boldsymbol {y} \mid \boldsymbol {x}) = \sum_ {k = 1} ^ {K} \gamma_ {\theta} ^ {k} (\boldsymbol {x}) \frac {\mathbf {1} \left(\boldsymbol {y} \in \mathcal {Y} ^ {k} (\boldsymbol {x})\right)}{\mathcal {V} \left(\mathcal {Y} ^ {k} (\boldsymbol {x})\right)}, \tag {11}
$$
where  $\mathcal{V}(\mathcal{Y}^k (\pmb {x}))\triangleq \int_{\pmb {y}\in \mathcal{Y}^k (\pmb {x})}\mathrm{d}\pmb {y}$  is the volume of  $\mathcal{V}^k (\pmb {x})$  . Similarly, if we model the output distribution as a mixture of Dirac deltas such that in each cell  $k$ $\gamma_{\theta}^{k}(\pmb {x}) > 0\Rightarrow Y_{\pmb{x}}^{k}\sim \delta_{f_{\theta}^{k}(\pmb {x})}$  , then
$$
\hat {p} (\boldsymbol {y} \mid \boldsymbol {x}) = \sum_ {k = 1} ^ {K} \gamma_ {\theta} ^ {k} (\boldsymbol {x}) \delta_ {f _ {\theta} ^ {k} (\boldsymbol {x})} (\boldsymbol {y}). \tag {12}
$$
3.4 Toy example
We build upon the toy example formalized in [34, §4.1] to validate the proposed algorithm and its interpretation. In particular, we seek to illustrate how rMCL handles the overconfidence problem in a regression task. The toy problem involves the prediction of a 2D distribution  $p(\boldsymbol{y} \mid x)$  based on scalar input  $x \in \mathcal{X}$ , where  $\mathcal{X} = [0,1]$ . The training dataset is constructed by selecting for each random input  $x \sim \mathcal{U}([0,1])$  one of the four sections  $S_{1} = [-1,0) \times [-1,0)$ ,  $S_{2} = [-1,0) \times [0,1]$ ,  $S_{3} = [0,1] \times [-1,0)$ ,  $S_{4} = [0,1] \times [0,1]$ , with probabilities  $p(S_{1}) = p(S_{4}) = \frac{1 - x}{2}$ ,  $p(S_{2}) = p(S_{3}) = \frac{x}{2}$ . Whenever a region is selected, a point is then sampled uniformly in this region. To slightly increase the complexity of the dataset, it has been chosen to enable the sampling of two (instead of one) target training samples with probability  $q(x)$  for each input  $x$ . Here,  $q$  is a piece-wise affine function with  $q(0) = 1$ ,  $q\left(\frac{1}{2}\right) = 0$ , and  $q(1) = 1$ .