<figure><img src="image_1.png" /></figure>

HAL
open science

# Biquality Learning: a Framework to Design Algorithms Dealing with Closed-Set Distribution Shifts

Pierre Nodet, Vincent Lemaire, Alexis Bondu, Antoine Cornuéjols

<figure><img src="image_2.png" /></figure>

## To cite this version:

Pierre Nodet, Vincent Lemaire, Alexis Bondu, Antoine Cornuéjols. Biquality Learning: a Framework to Design Algorithms Dealing with Closed-Set Distribution Shifts. 2023. hal-04338679v1

HAL Id: hal-04338679

https://hal.science/hal-04338679v1

Preprint submitted on 12 Dec 2023 (v1), last revised 24 Dec 2025 (v2)

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Biquality Learning: a Framework to Design Algorithms Dealing with Closed-Set Distribution Shifts

Pierre Nodet¹,², Vincent Lemaire¹, Alexis Bondu¹, and Antoine Cornuéjols²

¹ Orange Innovation, France

² AgroParisTech, France

**Abstract.** Training machine learning models from data with weak supervision and dataset shifts is still challenging. Designing algorithms when these two situations arise has not been explored much, and existing algorithms cannot always handle the most complex distributional shifts. We think the biquality data setup is a suitable framework for designing such algorithms. Biquality Learning assumes that two datasets are available at training time: a trusted dataset sampled from the distribution of interest and the untrusted dataset with dataset shifts and weaknesses of supervision (aka distribution shifts). The trusted and untrusted datasets available at training time make designing algorithms dealing with any distribution shifts possible. We propose two methods, one inspired by the label noise literature and another by the covariate shift literature for biquality learning. We experiment with two novel methods to synthetically introduce concept drift and class-conditional shifts in real-world datasets across many of them. We opened some discussions and assessed that developing biquality learning algorithms robust to distributional changes remains an interesting problem for future research.

**Keywords:** Biquality Learning, Distribution Shift, Weakly Supervised Learning, Dataset Shift, Machine Learning

## 1 Introduction

Supervised machine learning has been studied and explored extensively during the last decades, and both theoretical and experimental solutions exist to accomplish this task [13]. Weakly supervised machine learning (WSL) has not reached this state yet. WSL is the machine learning field where algorithms learn a model from data with weak supervision instead of strong supervision. Multiple weak supervisions have been identified in [46] such as *inaccurate supervision* when samples are mislabeled, *inexact supervision* when labels are not adapted to the classification task, or *incomplete supervision* when labels are missing which reflects the inadequacy of the available labels in the real world. For every kind of weak supervision, assumptions are needed to design sound algorithms, especially on the corruption model, the generative process behind the weakness of supervision.
Weaknesses in supervision are one facet of weakly supervised learning, and dataset shifts are another. Dataset shifts happen when the data distribution observed at training time is different from what is expected from the data distribution at testing time [22]. This distribution change can take multiple forms, such as a change in the distribution of a single feature, a combination of features, or the concept to be learned. Thus, the common assumption that the training and testing data follow the same distributions is often violated in real-world applications. Again, designing algorithms to handle dataset shifts usually requires assumptions on the nature of the shift [7].

We believe that the biquality data setup proposed in [26] is a suitable framework to design algorithms capable of handling both dataset shifts and weaknesses of supervision simultaneously.

Biquality Learning assumes that two datasets are available at training time: a trusted dataset $D_T$ and an untrusted dataset $D_U$ both composed of labeled samples $(x_i, y_i) \in \mathcal{X} \times \mathcal{Y}$. These datasets share the same features $\mathcal{X}_T = \mathcal{X}_U$ and the same set of labels $\mathcal{Y}_T = \mathcal{Y}_U$, having a closed-set of features $\mathcal{X}$ and labels $\mathcal{Y}$ with $K = |\mathcal{Y}|$ classes. However, the two datasets differ in terms of the joint distribution $\mathbb{P}_T(X, Y) \neq \mathbb{P}_U(X, Y)$ where the trusted distribution is the distribution of interest.

In practice, it is often the case that the trusted dataset is not large enough to learn an efficient model to estimate $\mathbb{P}_T(X, Y)$. By contrast, it is generally easy to get a large enough untrusted dataset that enables to correctly estimate $\mathbb{P}_U(X, Y)$. However, this data distribution could be completely different from the one of interest, $\mathbb{P}_T(X, Y)$. In the biquality data setup, there is no assumption on the difference in joint distribution between the two datasets, and it can cover a wide range of known problems. From the Bayes Formula:

$$ \mathbb{P}(X, Y) = \mathbb{P}(X | Y)\mathbb{P}(Y) = \mathbb{P}(Y | X)\mathbb{P}(X) \quad (1) $$

distribution shift covers covariate shift $\mathbb{P}_T(X) \neq \mathbb{P}_U(X)$, concept drift $\mathbb{P}_T(Y | X) \neq \mathbb{P}_U(Y | X)$, class-conditional shift $\mathbb{P}_T(X | Y) \neq \mathbb{P}_U(X | Y)$ and prior shift $\mathbb{P}_T(Y) \neq \mathbb{P}_U(Y)$.

The Biquality Data setup typically occurs in three scenarios in practice.

1. The first scenario corresponds to the case where annotating samples is expansive to the point of being prohibitive to label an entire dataset, but labeling a small part of the dataset is doable. It is typically the case in Fraud Detection and in Cyber Security where labeling samples require complex forensics from domain experts. There, the rest of the dataset is usually labeled by hand-engineered rules which might not perfectly fit the classification task and the labels can not properly be trusted [28].

2. The second scenario happens when there are data shifts during the labeling process over the course of time. For example in MLOps [19], when a model is first learned on clean data and then deployed in production, predictions can be used to learn an updated model [36]; these predictions may be faulty and need to be dealt with. A second example in MLOps occurs when newly
clean data is acquired to retrain a deployed model; the most recent clean
data is considered trusted and the old clean data is considered untrusted
when dataset shifts occurred [10].

3. The third scenario occurs when multiple annotators are responsible for dataset labeling. In natural language processing (NLP) products, for example, multiple annotators follow labeling guidelines to annotate verbatims. The efficiency of annotators to follow these guidelines may vary and might not be trusted. However, if one annotator can be trusted, all the other annotators can be set as untrusted and the biquality data setup may apply. Especially, considering each untrusted annotator against the trusted annotator can be viewed as a biquality learning task [42].

Having the trusted and untrusted datasets available at training time makes it possible to design algorithms dealing with closed-set distribution shifts. If algorithms designed for biquality data with concept drift only have been explored recently [25], algorithms that deal with distribution shift are still behind. In this paper, we propose two biquality approaches that adapt methods from either the covariate shift literature or the concept drift literature in order to deal with both corruptions simultaneously.

Multiple biquality learning algorithm designs have been identified in [26]. They have been divided into three main families based on how they modify instances to correct the global learning procedure. Untrusted instances can be (i) relabeled correctly, (ii) modified in the feature space, or (iii) reweighted such that the untrusted dataset seems sampled from the trusted distribution $P_T(X, Y)$. We propose here two corresponding algorithms for the third case of importance reweighting.

In Section 2, a brief state-of-the-art relates what has already been achieved in biquality learning. Then, Section 3 further focuses on the state-of-the-art of importance reweighting on biquality learning. Section 4 and Section 5 introduce our proposals to use classifiers to reweight untrusted instances for biquality learning with distribution shifts. Then, Section 6 describes the experiments that evaluate the efficiency of our proposed approaches on real datasets and corruptions. Section 7 presents the results of the proposed experiments. Finally, Section 8 open some discussions about the presented results before concluding.

## 2 Related work

Machine learning algorithms on biquality data have been developed in many different sub-domains of weakly supervised learning. Some of these sub-domains are robust learning to label noise, learning under covariate shift, or transfer learning. Because these subdomains expect different corruptions, not all algorithms designed for some subcases of biquality learning will work in a more general setting with distribution shifts.

For example, Gold Loss Correction (GLC) [14] and Importance Reweighting for Biquality Learning (IRBL) [26] are algorithms designed to specifically deal with a concept drift between the trusted and the untrusted datasets. GLC, on
the one hand, corrects the learning procedure on the untrusted dataset using a noise transition matrix between the trusted and untrusted concept. IRBL, on the other hand, reweight untrusted instances using an estimation of the ratio of both concepts. These algorithms are not theoretically designed to handle covariate shift; nevertheless, they could be empirically efficient on this task and serve as a reference.

Another group of algorithms, such as Kernel Mean Matching (KMM) [11], and Probabilistic Density Ratio Estimation (PDR) [1], only deals with covariate shift between the two datasets. These algorithms seek to reweight untrusted instances such that the distribution of features between the two datasets is equivalent. KMM minimizes the difference of the features mean in a reproducing kernel Hilbert space. PDR learns the classification task of predicting if an instance is untrusted or not and uses the predicted probability of being an untrusted instance as the weight. These algorithms are not designed to handle concept drift and will serve as references too.

However, a recent proposal aims to adapt these algorithms to the biquality framework with distribution shift [9]. They proposed to use one density ratio estimation algorithm per class, which, when combined, corrects the distribution shift. They also proposed to transform the joint density ratio estimation problem by combining the features and labels of the data into a new feature space that allows for a single density ratio estimation algorithm. These adapted algorithms will also serve as competitors.

Finally, recent approaches such as Learning to Reweight (L2RW) [29], or Meta-Weight-Net (MWNet) [32] based on deep learning and meta-learning have not been tested in this paper. Indeed, they require a class of algorithms with a differentiable and incremental learning procedure that does not fit most popular families of classifiers, such as gradient boosting trees. They are left to be tested in future works.

Biquality Data is not a new setup per see, as previous work exists on this setup going back to [17] to the best of our knowledge. Each previous work was carried out in different sub-domains of weakly supervised learning and thus achieved different goals based on different setups [14,32,29,45,17]. These setups used different terms, definitions, hypotheses, and requirements but still sought to solve the same fundamental problem of biquality learning. Only recently, some efforts have been done to provide clear and concise definitions of the biquality learning framework [26]. We propose in this paper to extend it to include dataset shifts. This extension is, to the best of our knowledge, a new problem to tackle that few tried already [9], limiting existing prior literature.

## 3 Reweighting for distribution shift

The previous Section introduced the most common algorithms used in machine learning for biquality data. Most of them are based on instance reweighting, and specifically, on estimating the Radon-Nikodym Derivative (RND) [24] of $P_T(X, Y)$ with respect to $P_U(X, Y)$.
**Theorem 1 (Radon-Nikdoym-Lebesgue theorem [31]).** Let $\mu$ and $\nu$ two positive $\sigma$-finite measures defined on a measurable space $(X, \mathcal{A})$ with $\nu$ being absolutely continuous with respect to $\mu$. Then there exists a unique positive measurable function $f$ defined on $X$ such that:

$$ \forall A \in \mathcal{A}, \nu(A) = \int_A f d\mu \qquad (2) $$

Typically, machine learning datasets form measurable spaces, and probability densities are positive finite measures on these measurable spaces. Assuming that $\mathbb{P}_T(X, Y)$ is absolutely continuous with respect to $\mathbb{P}_U(X, Y)$, then the RND exists, is unique and equals to $\frac{d\mathbb{P}_T(X, Y)}{d\mathbb{P}_U(X, Y)}$.

Equation 3 shows that minimizing the reweighted empirical risk by the RND on the untrusted data is equivalent to minimizing the empirical risk on trusted data.

$$ \begin{aligned} R_{(X,Y)\sim T,L}(f) &= \mathbb{E}_{(X,Y)\sim T}[L(f(X), Y)] \\ &= \int L(f(X), Y) d\mathbb{P}_T(X, Y) \\ &= \int \frac{d\mathbb{P}_T(X, Y)}{d\mathbb{P}_U(X, Y)} L(f(X), Y) d\mathbb{P}_U(X, Y) \qquad (3) \\ &= \mathbb{E}_{(X,Y)\sim U}[\frac{\mathbb{P}_T(X, Y)}{\mathbb{P}_U(X, Y)} L(f(X), Y)] \\ &= \mathbb{E}_{(X,Y)\sim U}[\beta L(f(X), Y)] \\ &= R_{(X,Y)\sim U, \beta L}(f) \end{aligned} $$

However, estimating the RND can be a difficult task, especially in the case of distribution shift where the joint distribution ratio $\beta$ needs to be estimated. Proposals have been made to ease this estimation.

A first proposal has been made in IRBL [26] which focused first on the concept drift between datasets using the Bayes Formula:

$$ \beta(X, Y) = \frac{\mathbb{P}_T(X, Y)}{\mathbb{P}_U(X, Y)} = \frac{\mathbb{P}_T(Y | X)\mathbb{P}_T(X)}{\mathbb{P}_U(Y | X)\mathbb{P}_U(X)} \qquad (4) $$

Their proposed algorithm is based on the decomposition of the joint density ratio estimation task into three sub-tasks. The first one is to estimate the trusted concept $\mathbb{P}_T(Y | X)$, and is done by learning a classifier on the trusted dataset. The second task is to estimate the untrusted concept $\mathbb{P}_U(Y | X)$, which is done by learning a classifier on the untrusted dataset. And the third task about density ratio estimation $\frac{\mathbb{P}_T(X)}{\mathbb{P}_U(X)}$ was skipped as no covariate shift was introduced in their benchmark, but it is a well known and solved machine learning task [34].

A second proposal has been made in [9] which focused on the covariate shift between datasets using the Bayes Formula differently:

$$ \beta(X, Y) = \frac{\mathbb{P}_T(X, Y)}{\mathbb{P}_U(X, Y)} = \frac{\mathbb{P}_T(X | Y)\mathbb{P}_T(Y)}{\mathbb{P}_U(X | Y)\mathbb{P}_U(Y)} \qquad (5) $$
In their proposed algorithm, the joint density ratio estimation task has been decomposed into K-tasks where K is the number of classes to predict. For each class, only samples of the given class are selected on both datasets, such that the samples are drawn from the $P(X | Y)$ distribution. Then, a density ratio estimation procedure usually employed to estimate $\frac{P_T(X)}{P_U(X)}$ is learned on these sub-datasets to estimate $\frac{P_T(X|Y)}{P_U(X|Y)}$, effectively handling distribution shift from Equation 5. As it uses K density ratio algorithms, this generic approach will be named K-DensityRatio (K-DR) in the rest of the paper.

Finally, a last approach is to focus on the density ratio estimation task by finding a deterministic and invertible transformation $f$ as proposed in [9]:

$$ \beta(X, Y) = \frac{\mathbb{P}_T(X, Y)}{\mathbb{P}_U(X, Y)} = \frac{\mathbb{P}_T(Z)}{\mathbb{P}_U(Z)}, \quad Z = f(X, Y) \quad (6) $$

An example of such transformation [9] is the classification loss of a model learned on the biquality data. One density ratio estimation procedure is done on these new features $\mathcal{Z}$ to directly estimate $\frac{P_T(\mathcal{Z})}{P_U(\mathcal{Z})}$.

IRBL has experimentally proved to efficiently solve the biquality learning task on tabular data [26]. However, the experiments were conducted on corruptions only affecting the untrusted concept $P(Y | X)$ and not the joint distribution $P(X, Y)$. We propose here to adapt IRBL to handle distribution shifts by solving the third task of density ratio estimation with a probabilistic classifier. Moreover, we propose a new version of K-DR using probabilistic classifiers to solve the K density ratio estimation tasks. This proposition is driven by the desire to reuse efficient tricks from IRBL and to rely on a non parametric approach by contrast to the original proposal [9].

# 4 First proposed approach: IRBL2

Importance Reweighting for Biquality Learning (IRBL) [26] is a biquality learning algorithm designed to handle closed-set concept-drift. The algorithm is based on using two probabilistic classifiers: first, to estimate both concepts $P_T(Y | X)$ and $P_U(Y | X)$ and, second, using these classifiers' outputs to estimate the RND between both data distributions. In the particular case of label noise, especially instance dependent label noise, it has been shown to be the best approach experimentally on a wide variety of datasets.

We propose to adapt it to handle covariate shift by estimating the ratio $\frac{P_T(X)}{P_U(X)}$ by using a third probabilistic classifier based on Discriminative Learning [1]. This algorithm works by defining a new supervised classification task by learning to predict if a sample is trusted or untrusted by only using its features. If there exists covariate shift between the datasets, the classifier should be able to discriminate between the two datasets.

Let's introduce $\mathcal{S}$ as the new target:

$$ s_i(x_i) = \begin{cases} 0, & \text{if } x_i \in D_U \\ 1, & \text{if } x_i \in D_T \end{cases} \quad (7) $$
Estimating $\mathbb{P}(S | X)$ allows us to estimate $\mathbb{P}_{\mathbb{P}_U(X)}$ directly without estimating both distributions:

$$ \frac{\mathbb{P}_T(X)}{\mathbb{P}_U(X)} = \frac{\mathbb{P}(X | S = 1)}{\mathbb{P}(X | S = 0)} = \frac{\mathbb{P}(S = 1 | X)\mathbb{P}(X)}{\mathbb{P}(S = 1)} \times \frac{\mathbb{P}(S = 0)}{\mathbb{P}(S = 0 | X)\mathbb{P}(X)} \\ = \frac{\mathbb{P}(S = 1 | X)}{1 - \mathbb{P}(S = 1 | X)} \times \frac{1 - \mathbb{P}(S = 1)}{\mathbb{P}(S = 1)} \quad (8) $$

Combining Equation 4 and 8:

$$ \frac{\mathbb{P}_T(Y | X)\mathbb{P}_T(X)}{\mathbb{P}_U(Y | X)\mathbb{P}_U(X)} = \frac{\mathbb{P}_T(Y | X)}{\mathbb{P}_U(Y | X)} \times \frac{\mathbb{P}(S = 1 | X)}{1 - \mathbb{P}(S = 1 | X)} \times \frac{1 - \mathbb{P}(S = 1)}{\mathbb{P}(S = 1)} \quad (9) $$

We propose to estimate Equation 9 by learning probabilistic classifiers $f \in \mathcal{F}$ to estimate each of its terms. A probabilistic classifier $f_T$ is learned on $D_T$ to estimate $\mathbb{P}_T(Y | X)$, $f_U$ is learned on $D_U$ to estimate $\mathbb{P}_U(Y | X)$, and $f_S$ is learned on $\{(x, s(x)) | \forall x \in D_T \cup D_U\}$ to estimate $\mathbb{P}_U(S | X)$, leading to the following Algorithm 1.

<caption>**Algorithm 1:** Importance Reweighting for Biquality Learning 2 (IRBL2)</caption>

**Input:** Trusted Dataset $D_T$, Untrusted Dataset $D_U$, Probabilistic Classifier Family $\mathcal{F}$

Learn $f_U \in \mathcal{F}$ on $D_U$

Learn $f_T \in \mathcal{F}$ on $D_T$

Learn $f_S \in \mathcal{F}$ on $\{(x, s(x)) | \forall x \in D_T \cup D_U\}$

**for** $(x_i, y_i) \in D_U$ **do**

$$ \hat{\beta}(x_i, y_i) = \frac{f_T(x_i)y_i f_S(x_i)1}{f_U(x_i)y_i |D_T| f_S(x_i)0} $$

**for** $(x_i, y_i) \in D_T$ **do**

$$ \hat{\beta}(x_i, y_i) = 1 $$

Learn $f \in \mathcal{F}$ on $D_T \cup D_U$ with weights $\hat{\beta}$

**Output:** $f$

## 5 Second proposed approach: K-PDR

K-DensityRatio (K-DR) [9] is an alternative approach to design a biquality learning algorithm able to handle distribution shift. The focus is made on the covariate shift between the two datasets. It handles the covariate shift in a class conditional fashion to deal with distribution shifts by using covariate shift correction once per class.
From Equation 5, K-DR evaluates the ratio $\frac{\mathbb{P}_T(X|Y)}{\mathbb{P}_U(X|Y)}$ with density ratio estimation algorithms. To do so, it first samples data from the $X$ | $Y$ distribution by selecting only samples from a given class $k \in [1, K]$ in both datasets $D_T$ and $D_U$. Then, it uses density ratio estimation algorithms $e \in \mathcal{E}$ on these sub-datasets to estimate $\frac{\mathbb{P}_T(X|Y=k)}{\mathbb{P}_U(X|Y=k)}$ independently $k$ times. The class priors $\mathbb{P}_T(Y)$ and $\mathbb{P}_U(Y)$ are estimated empirically from both training sets. See Algorithm 2.

<caption>**Algorithm 2: K-DensityRatio (K-DR)**</caption>

**Input:** Trusted Dataset $D_T$, Untrusted Dataset $D_U$, Density Ratio Estimator Family $\mathcal{E}$, Probabilistic Classifier Family $\mathcal{F}$
**for** $k \in [1, K]$ **do**
Let $D_T^k = \{\forall(x, y) \in D_T | y = k\}$
Let $D_U^k = \{\forall(x, y) \in D_U | y = k\}$
Learn $e^k \in \mathcal{E}$ on $D_T^k$ and $D_U^k$
**for** $(x_i, y_i) \in D_U$ **do**
$\hat{\beta}(x_i, y_i) = e^{y_i}(x_i) \frac{|D_T^{y_i}|}{|D_T|} \frac{|D_U^{y_i}|}{|D_U|}$
**for** $(x_i, y_i) \in D_T$ **do**
$\hat{\beta}(x_i, y_i) = 1$
Learn $f \in \mathcal{F}$ on $D_T \cup D_U$ with weights $\hat{\beta}$
**Output:** $f$

In [9] Kernel Mean Matching (KMM) [15,11] has been used as the Density Ratio algorithm $e$ to handle covariate shift. Empirically, KMM is an algorithm that matches with quadratic programming [39] the mean of both datasets in a feature space induced by a kernel $k$ on the domain $\mathcal{X} \times \mathcal{X}$:

$$ \begin{array}{ll} \min_{\beta_i} & \left\| \frac{1}{|D_U|} \sum_{i=0}^{|D_U|} \beta_i \Phi(x_i) - \frac{1}{|D_T|} \sum_{i=0}^{|D_T|} \Phi(x_i) \right\|_{\mathcal{H}} \\ \text{s.t.} & 0 \le \beta_i \le B \\ & \left| \frac{1}{|D_U|} \sum_{i=0}^{|D_U|} \beta_i - 1 \right| < \epsilon \end{array} \quad (10) $$

where $\Phi: \mathcal{X} \to \mathcal{H}$ denotes the canonical feature map, $\mathcal{H}$ is the reproducing kernel Hilbert space induced by the kernel $k$, $| \cdot |_{\mathcal{H}}$ is the norm on $\mathcal{H}$ and $B$ and $\epsilon$ are regularization and normalization constraints.

As such, KMM is a parametric algorithm based on kernels. We propose to use instead a probabilistic classifier to handle covariate shift, in the same fashion as in Equations 7 and 8 to make a non-parametric version of K-DR as shown in Equation 8.b.
$$ 
\begin{aligned}
\frac{\mathbb{P}_T(X | Y)\mathbb{P}_T(Y)}{\mathbb{P}_U(X | Y)\mathbb{P}_U(Y)} &= \frac{\mathbb{P}(X | Y, S = 1)}{\mathbb{P}(X | Y, S = 0)} \times \frac{\mathbb{P}(Y | S = 1)}{\mathbb{P}(Y | S = 0)} \\
&= \frac{\mathbb{P}(S = 1 | X, Y)\mathbb{P}(X, Y)}{\mathbb{P}(Y | S = 1)\mathbb{P}(S = 1)} \times \frac{\mathbb{P}(Y | S = 0)\mathbb{P}(S = 0)}{\mathbb{P}(S = 0 | X, Y)\mathbb{P}(X, Y)} \quad (8.b) \\
&\quad \times \frac{\mathbb{P}(Y | S = 1)}{\mathbb{P}(Y | S = 0)} \\
&= \frac{\mathbb{P}(S = 1 | X, Y)}{1 - \mathbb{P}(S = 1 | X, Y)} \times \frac{1 - \mathbb{P}(S = 1)}{\mathbb{P}(S = 1)}
\end{aligned}
$$

The main advantage of the non-parametric approach is that it does not require assumptions about the data distribution, which may not be satisfied in many real-world datasets and could lead to poor performances. Moreover, the scalability of K-PDR is better than the scalability of K-KMM both in space and time complexity. K-PDR has K times the same complexity as the complexity of learning the chosen probabilistic classifier, which is $O(K \times |X| \times |D_u^k| \log(|D_u^k|))$ for Decisions Trees [6]. Meanwhile, K-KMM memory complexity is $O(|D_u^k|^2 + |D_u^k| \times |D_t^k|)$ to sequentially build matrices necessary for the quadratic program, and a worst-case time complexity of $O(K \times |D_u^k|^3)$ to solve the quadratic program [41]. Finally, the proposed approach is even more flexible than the previous one, as any family of machine learning classifiers could be used instead of kernels. This leads to Algorithm 3.

<caption>**Algorithm 3: K-Probabilistic Density Ratio (K-PDR)**</caption>

**Input:** Trusted Dataset $D_T$, Untrusted Dataset $D_U$, Probabilistic Classifier Family $\mathcal{F}$

**for** $k \in [1, K]$ **do**

Let $D_T^k = \{\forall(x, y) \in D_T | y = k\}$

Let $D_U^k = \{\forall(x, y) \in D_U | y = k\}$

Learn $f_S^k \in \mathcal{F}$ on $\{(x, s(x)) | \forall x \in D_T^k \cup D_U^k\}$

**for** $(x_i, y_i) \in D_U$ **do**

$$ \hat{\beta}(x_i, y_i) = \frac{f_S^{y_i(x_i)}_1 |D_U|}{f_S^{y_i(x_i)}_0 |D_T|} $$

**for** $(x_i, y_i) \in D_T$ **do**

$$ \hat{\beta}(x_i, y_i) = 1 $$

Learn $f \in \mathcal{F}$ on $D_T \cup D_U$ with weights $\hat{\beta}$

**Output:** $f$

# 6 Experiments

Benchmarking biquality learning algorithms means evaluating their efficiency and resilience on both dataset shifts and weaknesses of supervision in a joint
manner. Introducing these corruptions synthetically in usual public multi-class classification datasets allows a fine-grained and controlled evaluation of these algorithms.

From Equation 1, introducing distribution shift can be done in four ways: by introducing covariate shift, concept drift, class-conditional shift, or prior shift. Especially modifying both concept drift and covariate shift or class-conditional shift and prior shift at the same time leads to particularly complex distribution shifts. Table 1 sums up the hierarchy of distribution shift sources.

<table><caption>Table 1: Hierarchy of Distribution Shift sources</caption><thead><tr><th colspan="4">Distribution Shift<br>P(X, Y)</th></tr><tr><th colspan="2">P(Y | X)P(X)</th><th colspan="2">P(X | Y)P(Y)</th></tr><tr><th>Concept Drift<br>P(Y | X)</th><th>Covariate Shift<br>P(X)</th><th>Class-Conditional Shift<br>P(X | Y)</th><th>Prior Shift<br>P(Y)</th></tr></thead><tbody><tr><td></td><td></td><td></td><td></td></tr></tbody></table>

We chose two methods to synthetically generate distribution shifts, one in each sub-tree of distribution shift sources: concept drift and class-conditional shift. We propose two novel ways to generate such shifts in real-world datasets that are detailed in the following Subsection 6.1 and Subsection 6.2.

## 6.1 Concept Drift

Concept drift corresponds to changes in the decision boundary $P(Y | X)$ of a classifier in some parts of the feature space $\mathcal{X}$.

To synthetically generate concept drift in real-world datasets, we propose to model the feature space with a Decision Tree classifier [3] learned on the original dataset $D$, such that each leaf of the decision tree will correspond to a patch of the dataset grid. By restricting the minimum samples per leaf in the decision tree, here set as 10% of the dataset per class, we will be able to control the finesse of the grid.

Then in each leaf of the decision tree, we are going to change the class distribution, essentially by generating noisy labels $\tilde{Y}$ from clean labels $Y$ thanks to a transition matrix $T$, such that $\forall (i, j) \in [1, K]^2$, $T_{i,j} = P(\tilde{Y} = j | Y = i)$. We chose to restrict the transition matrix to a permutation matrix $P$, meaning that each clean class will be associated in a bijective way with a noisy class, different from the clean class. These permutation matrices $P$ will be generated randomly once per dataset. For example, for three-class classification, one of these permutation matrices could be the following:

$$P = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0 \end{pmatrix}$$
To decide how much concept drift will occur, we rank the leaves in the de-
cision tree by their purity, starting from the purest leaves, and chose enough
leaves such that r% of the untrusted dataset fall in these leaves. These samples
will be given a noisy label, and the rest of the samples will be left untouched.

The benefits of this methodology are twofold:

- Having all samples assigned the same noisy label given their class in a whole subspace of the feature space will avoid the class overlap usually created with uniform (completely at random) label noise.

- Choosing to add label noise in the purest leaves of the decision tree first creates new patterns in otherwise completely clean and easy parts of the dataset.

6.2 Class-Conditional Shift

Class-conditional shift corresponds to changes in the feature distribution $\mathbb{P}(X)$, with different changes per class $\mathbb{P}(X | Y)$.

To synthetically generate class-conditional shift in a real-world dataset, we propose to split the original dataset $D$ into sub-datasets $D^k$, such that $\forall k \in [1, K]$, $D^k = \{(x, y) \in D \mid y = k\}$, where $D^k$ corresponds to samples from the same class $k$ out of all the classes $K$.

For each sub-dataset, we train a K-Means [20] clustering to learn a division of the feature space $\mathcal{X}$ given the class $k$. The number of clusters is chosen by cross-validation to maximize the average silhouette score [30].

Then we sub-sample some of these clusters to modify the class-conditional
distribution $\mathbb{P}(X | Y = k)$. We sort the clusters by their size and separate them
in two groups of equal size. The group formed by the smallest clusters will be
subsampled according to a subsampling ratio $\rho$. The group formed by the biggest
clusters will be left untouched. For example, if the number of clusters for a class
is 4 and $\rho = 10$, the two smallest clusters will be subsampled by a ratio of 10,
the two bigests will not be subsampled.

A higher $\rho$ will lead to more data being missing in the untrusted dataset, especially samples in sparse regions of the feature space, represented by small clusters.

The final size of the untrusted dataset relative to the non-sub-sampled un-
trusted dataset is reported for each scenario in Table 5 in the Appendices.

6.3 Illustration on a Toy Dataset

We propose to first see the experimental protocol on a toy dataset with two
features to visualize better how the corruption mechanisms will act. We chose
the two moons dataset, composed of two croissant-shaped classes that are almost
linearly separable.

Figure 1 shows that a Support Vector classifier [2] with a Polynomial kernel of degree 3 can adequately separate the two classes.
<figure><img src="image_4.png" /><figcaption>Fig. 1: Two moons dataset with the decision boundary of a Support Vector Machine classifier using a polynomial kernel.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 2: Corrupted two moons dataset with the decision boundary of a Support Vector Machine classifier using a polynomial kernel.</figcaption></figure>

Now we can apply our synthetic corruptions to this dataset and see the effect on the decision boundary of the Support Vector classifier.

The proposed synthetic corruptions affect the learned decision boundary of the Support Vector classifier but each in their own way. The concept drift dramatically changes the decision boundary, and the classifier cannot recognize the two original croissants illustrated in Figure 2. Instead, it emphasizes the new class patterns created with the noisy patches. The class-conditional shift, however, does not inherently change the shape of the decision boundary. We can still see that the classifier somewhat separates the two croissants. However, the margin is way tighter on the yellow croissant. This classifier would fail to classify some data points that would be considered out-of-distribution for the corrupted dataset but completely normal in the original dataset.

Now that we verified that the synthetic corruptions produce the expected effects on classifiers, we can look at how the previous method of the state-of-the-art can correct these corrupted datasets, especially IRBL and PDR that are expected to work only on concept drift or covariate shift, and our two proposed
algorithm, IRBL2, and *K*-PDR, that should work on distribution shifts. We propose to keep 5% of the origin two moons dataset as trusted data that will be left untouched and corrupt the rest of the dataset with both corruptions.

<figure><img src="image_6.png" /><figcaption>Fig. 3: Trusted and untrusted two moons dataset. Trusted data points are represented with a □ marker and untrusted data points are represented with a ○ marker and with a more transparent color. The markers' sizes are proportional to the weights given by the corresponding algorithm.</figcaption></figure>

In Figure 3, we can see the limitations of IRBL and PDR. IRBL can detect samples belonging to the wrong part of the feature space given their label but cannot take into account the uncertainty of some patches of the feature space that are not present in the trusted dataset. For example, in the top part of the dataset, there is a zone where no trusted points exist for the purple moon. IRBL does not take into account and re-weights equally all these, as long as they are from the correct color, purple. Meanwhile, PDR does re-weight the samples differently depending if they belong to the trusted distribution, but indifferently depending on the sample color.

In the second part of Figure 3, we can observe that IRBL2 and *K*-PDR can handle both cases and, thus, distribution shift. Their re-weighting schemes are reasonably similar, and to assess which one is better, we need to benchmark them on a broader range of real-world datasets.
## 6.4 Datasets

We randomly picked supervised classification datasets, see Table 2, from different sources: UCI [8], libsvm [4], active learning challenge [12] and openML [35]. A part of these datasets comes from past challenges in active learning, where high performances with a low number of labeled samples have proved challenging to obtain, which makes leveraging the untrusted dataset necessary. With this choice of datasets, an extensive range of class ratios, number of classes, number of features, and dataset sizes are covered.

<table><caption>Table 2: Multi-class classification datasets used for the evaluation. Columns: number of samples (|D|), number of features (|X|), number of classes (|Y|), and ratio of the number of samples from the minority class over the number of samples from the majority class (min).</caption><thead><tr><th>Datasets</th><th>|D|</th><th>|X|</th><th>|Y|</th><th>min</th><th>Datasets</th><th>|D|</th><th>|X|</th><th>|Y|</th><th>min</th></tr></thead><tbody><tr><td>eeg</td><td>15K</td><td>14</td><td>2</td><td>0.81</td><td>ad</td><td>3.28K</td><td>1558</td><td>2</td><td>0.16</td></tr><tr><td>ibn-sina</td><td>20.7K</td><td>92</td><td>2</td><td>0.61</td><td>nomao</td><td>34.5K</td><td>118</td><td>2</td><td>0.40</td></tr><tr><td>zebra</td><td>61.5K</td><td>154</td><td>2</td><td>0.05</td><td>click</td><td>39.9K</td><td>9</td><td>2</td><td>0.20</td></tr><tr><td>musk</td><td>6.6K</td><td>167</td><td>2</td><td>0.18</td><td>jm1</td><td>10.9K</td><td>21</td><td>2</td><td>0.24</td></tr><tr><td>phishing</td><td>11.1K</td><td>30</td><td>2</td><td>0.80</td><td>pendigits</td><td>11K</td><td>16</td><td>10</td><td>0.92</td></tr><tr><td>spam</td><td>4.6K</td><td>57</td><td>2</td><td>0.65</td><td>japanese-vowels</td><td>9.96K</td><td>14</td><td>9</td><td>0.48</td></tr><tr><td>ijcnn1</td><td>192K</td><td>22</td><td>2</td><td>0.11</td><td>gas-drift</td><td>13.9K</td><td>128</td><td>6</td><td>0.55</td></tr><tr><td>hiva</td><td>42.7K</td><td>1617</td><td>2</td><td>0.04</td><td>connect-4</td><td>67.6K</td><td>42</td><td>3</td><td>0.15</td></tr><tr><td>svmguide3</td><td>1.24K</td><td>22</td><td>2</td><td>0.31</td><td>satimage</td><td>6.43K</td><td>36</td><td>6</td><td>0.41</td></tr><tr><td>web</td><td>37K</td><td>123</td><td>2</td><td>0.32</td><td>dna</td><td>3.19K</td><td>180</td><td>3</td><td>0.46</td></tr><tr><td>mushroom</td><td>8.12K</td><td>22</td><td>2</td><td>0.93</td><td>first-ord-theo-prov</td><td>6.12K</td><td>51</td><td>6</td><td>0.19</td></tr><tr><td>skin-seg</td><td>245K</td><td>3</td><td>2</td><td>0.26</td><td>artificial-chars</td><td>10.2K</td><td>7</td><td>10</td><td>0.42</td></tr><tr><td> mozilla4</td><td>15.5K</td><td>5</td><td>2</td><td>0.49</td><td>splice</td><td>3.19K</td><td>60</td><td>3</td><td>0.46</td></tr><tr><td>electricity</td><td>45.3K</td><td>8</td><td>2</td><td>0.74</td><td>har</td><td>10.3K</td><td>561</td><td>6</td><td>0.72</td></tr><tr><td>bank-marketing</td><td>45.2K</td><td>16</td><td>2</td><td>0.13</td><td>usps</td><td>9.3K</td><td>256</td><td>10</td><td>0.46</td></tr><tr><td>magic-telescope</td><td>19K</td><td>10</td><td>2</td><td>0.54</td><td>mnist</td><td>70K</td><td>784</td><td>10</td><td>0.80</td></tr><tr><td>phoeneme</td><td>5.4K</td><td>5</td><td>2</td><td>0.42</td><td>letter</td><td>20K</td><td>16</td><td>26</td><td>0.90</td></tr><tr><td>poker</td><td>1.03M</td><td>10</td><td>2</td><td>1.00</td><td>isolet</td><td>7.8K</td><td>617</td><td>26</td><td>0.99</td></tr></tbody></table>

Each dataset is first jumbled, then split in a stratified fashion with 80% of samples used for training set, and 20% used for the test set.

Each training set is again split in a stratified fashion into a trusted and untrusted set with a ratio of trusted *p*. To find a ratio of trusted data that is coherent between all datasets, we compute a learning curve on the training set and choose a ratio of trusted data such that the classifier's performance is equal to *p%* of the classifier learned on the entire training set. The actual ratio of trusted data for each dataset is provided in Table 4 in the Appendices.

Finally, we synthetically corrupt the untrusted set with the methods described earlier in this Section.
## 6.5 Competitors

We compare IRBL2 and K-PDR against multiple state-of-the-art competitors and baselines:

- K-KMM, the original version of K-DR using KMM as proposed in [9];

- IRBL [26], which is IRBL2 without covariate shift correction;

- PDR [1] the covariate-shift only baseline;

- Trusted-Only baseline, when the model is learned using only the trusted dataset;

- No-Correction baseline, when the model is learned on both datasets without correction applied.

For every competitors, we use the same probabilistic classifier family, histogram-based gradient boosting (HGBT) trees [18] from Scikit-Learn [27] with their default hyperparameters, each time an algorithm required a base classifier. When it required a well-calibrated classifier, the HGBT trees where calibrated thanks to an Isotonic Regression [43] from Scikit-Learn using Zadrozny's heuristic [44] for multiclass calibration.

For KMM and K-KMM we use the Radial Basis Function (RBF) kernel [37] with the default $\gamma = \frac{1}{|X|}$ value from Scikit-Learn. In order to scale KMM to the bigger datasets, we used an ensembling version [21] of the original KMM algorithm with a batch size of 100.

## 7 Results

We conducted the previously defined experiments of the proposed datasets with the label noise strength $r$ varying from 0% to 50% of the dataset and the maximum cluster subsampling $\rho$ from 1 to 100. We also tested three different values for the ratio of trusted data $p$ equals to 25%, 50%, and 75%. The predictive performance we used in these experiments is Cohen's kappa coefficient $\kappa$ [5]. It is a measure of agreement between assignments relative to the agreement between two random assignments and is fitted to evaluate classifier performance in unbalanced settings. Before looking at the whole grid of experiments over $r$ and $\rho$, we can look at the two axes independently to independently analyze the impact of concept drift and class-conditional shift.

The primary metric to analyze to assess the efficiency of biquality learning algorithms is the evolution of the predictive performance $\kappa$ given the corruption strength ($\rho$ or $r$). For example, having a constant predictive performance given the corruption strength means being robust to this corruption.

We propose to illustrate this function by plotting the averaged predictive performance over all datasets for all tested corruption strength and ratio of trusted data.

Figure 4 shows that with very few trusted data, $p = 0.25$, biquality-learning algorithms have trouble differentiating themselves from one another and with baselines, showing their limitations on low trusted data regimes. However, $p =$
<figure><img src="image_7.png" /><figcaption>Fig. 4: Average Cohen's kappa $\kappa$ with different corruption strengths (noise ratio $r$, or cluster imbalance $\rho$) and ratio of trusted data $p$ over all datasets. The first row corresponds to experiments without noise ($r = 0$), and the second row corresponds to experiments without cluster imbalance ($\rho = 0$). Each column corresponds to a different case of ratio of trusted data ($p = 0.25, p = 0.5, p = 0.75$).</figcaption></figure>

$0.25$ leads to having around only thousandths of the data as trusted, which we consider to be a particularly pessimistic scenario in practice. With more trusted data, some competitors start to pull apart, especially K-PDR on class-conditional shift and IRBL and IRBL2 on concept drift.

In order to summarize these plots, we propose to compute the area under the curve (AUC) of the previous error curves, which is primarily a fancier average when the data points are not evenly spaced. This AUC is then normalized by the domain length we integrate the curve on.

<table><caption>Table 3: Averaged area under the Cohen's kappa $\kappa$ curve for all datasets.</caption><thead><tr><th></th><th>p IRBL</th><th>IRBL2</th><th>K-KMM</th><th>K-PDR</th><th>No Corr.</th><th>PDR</th><th>Trusted</th></tr></thead><tbody><tr><td>0</td><td>0.25</td><td>0.676</td><td>0.675</td><td>0.575</td><td><strong>0.694</strong></td><td>0.693</td><td>0.624</td><td>0.266</td></tr><tr><td>||</td><td>0.50</td><td>0.699</td><td>0.698</td><td>0.610</td><td><strong>0.709</strong></td><td>0.700</td><td>0.630</td><td>0.484</td></tr><tr><td>~</td><td>0.75</td><td>0.722</td><td>0.722</td><td>0.658</td><td><strong>0.729</strong></td><td>0.719</td><td>0.718</td><td>0.619</td></tr><tr><td>0</td><td>0.25</td><td>0.457</td><td>0.457</td><td>0.444</td><td>0.457</td><td><strong>0.459</strong></td><td>0.422</td><td>0.266</td></tr><tr><td>||</td><td>0.50</td><td><strong>0.515</strong></td><td><strong>0.515</strong></td><td>0.476</td><td>0.474</td><td>0.462</td><td>0.461</td><td>0.484</td></tr><tr><td>~</td><td>0.75</td><td><strong>0.630</strong></td><td><strong>0.630</strong></td><td>0.534</td><td>0.539</td><td>0.486</td><td>0.486</td><td>0.619</td></tr></tbody></table>
Table 3 mostly confirmed what could be observed in Figure 4, but more particularly highlights the mere improvements brought by IRBL2 over IRBL with both corruptions.

As the previous comparison are made using averaged predictive measures that might not be relatable from one dataset to the other, we propose in a second time to compute the average ranking instead thanks to critical diagrams presented in Figure 5 as a more robust way to compare competitors.

The Nemenyi test [23] is used to rank the approaches in terms of AUC of the predictive metric over the corruption strength. The Nemenyi test consists of two successive steps. First, the Friedman test is applied to the AUC of competing approaches to determine whether their overall performance is similar. Second, if not, the post-hoc test is applied to determine groups of approaches whose overall performance is significantly different from that of the other groups.

<figure><img src="image_8.png" /><figcaption>Fig. 5: Critical diagrams with different ratio of trusted data $p$ over all datasets. The first row corresponds to experiments without noise ($r = 0$), and the second row corresponds to experiments without cluster imbalance ($\rho = 0$). Each column corresponds to a different case of ratio of trusted data ($p = 0.25, p = 0.5, p = 0.75$).</figcaption></figure>

Figure 5 confirms all previous observations. K-PDR seems to be the best approach on class-conditional noise but is still somewhat robust to label noise, especially at medium and high levels of trusted data ratio as its able to detach from PDR in terms of average rank in these situations. IRBL and IRBL2 seem to be the best approaches for learning with label noise, even though they struggle with a low ratio of trusted data. However, IRBL2 struggles to improve IRBL performances on class-conditional shifts.

Finally, to study the combined effect of both axes, we extend the experiments on a grid with varying strength of label noise $r$ and sub-sampling $\rho$.

Figure 6 presents six graphics, each reporting the Wilcoxon test that evaluates one competitor against another based on the accuracy over all datasets. These graphics form a grid with the horizontal axis representing the label noise strength $r$ and the vertical axis representing the sub-sampling ratio $\rho$. On each point of a grid $(r, \rho)$, a Wilcoxon rank-signed test [38] is conducted on two
competitors for all datasets to determine if there is a significant difference in accuracy between them. If the first competitor wins, “o” is placed on the grid at this location, “.” and “●” indicate a tie or a loss, respectively.

<figure><img src="image_9.png" /><figcaption>Fig. 6: Results of the Wilcoxon signed rank test computed on all datasets. Each figure compares one competitor versus another for a given trusted data ratio. Figures in the same row are the same competitors against different cases of trusted data ratio: *p* = 0.25, *p* = 0.5, *p* = 0.75. In each figure “o”, “.” and “●” indicate respectively a win, a tie, or a loss of the first competitor compared to the second competitor, the vertical axis is *r*, and the horizontal axis is *p*.</figcaption></figure>

Figure 6 allows a point-wise analysis between different competitors, but most notably, it uses a robust statistical test to draw statistically significant conclusions. IRBL2 is not able to improve on IRBL in all tested cases, as seen by the numerous number of ties. *K*-PDR is a better approach than *K*-KMM as the only times *K*-PDR loses to *K*-KMM, *K*-KMM get also beaten by the Trusted Only baseline (see Figure 7 in Appendices). Finally, IRBL2 (or IRBL) is a better approach than *K*-PDR when untrusted data are noisy as soon as there is enough trusted data such that the trusted concept can be learned accurately enough to reweight untrusted samples.
# 8 Discussions

The previously presented results can be quite underwhelming for both proposed methods, so we propose to discuss these results in this section.

First, results showed that K-PDR could substantially improve on K-KMM, proving the advantage of using classifiers instead of kernels, which are more powerful tools as they require fewer assumptions about the data distribution and are fitter for learning from heterogeneous data such as tabular data. It also highlights a typical result in real-life ML applications where non-parametric methods work better than parametric methods.

Secondly, results showed no improvements brought by IRBL2 over IRBL on class-conditional shift. Indeed, IRBL might already be able to take into account the uncertainty about $P(X)$ provided by PDR in the predicted probabilities $P(Y|X)$ thanks to well-calibrated classifiers, nullifying the effect of IRBL2. For example, in unusual regions of the trusted feature space seen in the untrusted dataset, the value of $P_T(Y|X)$ might tend to be around $\frac{1}{K}$ to account the classifier uncertainty and will assign low weights to such samples already without the aid of a specific classifier for that. Meanwhile, K-PDR was clearly able to improve from PDR on label noise corruptions.

Furthermore, the proposed experiments for class-conditional shifts might only reproduce some cases of dataset shifts, especially shifts with out-of-distribution data [40]. With the proposed design of synthetic class-conditional shift in our per-class cluster sub-sampling, we did not introduce data points in foreign feature spaces. In our experiments, all points from the untrusted dataset are in-distribution. We only under-represented some sub-populations. This corruption could lead to biased models for individuals from minority groups. However, it is a more manageable situation to deal with than having different groups in the trusted and untrusted dataset. Moreover, the chosen evaluation metric, Cohen's kappa $\kappa$, might not be able to detect such biases, as this metric only evaluates predictive performance.

Finally, the calibration efficiency of the HGBT trees in these experiments greatly impacts the efficiency of every tested biquality-learning algorithm. This benchmark has not explored different calibration techniques other than Isotonic Regression.

Obviously, this benchmark is not a definitive answer to the problem as we could have extended the experiments to other corruptions in real-world untrusted datasets, such as data poisoning [33] or class imbalance [16].

# 9 Conclusion

In this paper, we have shown the capabilities of the biquality learning framework to design algorithms able to handle closed-set distribution shifts by having access to a trusted and untrusted dataset at training time. We proposed two biquality learning algorithms, IRBL2 and K-PDR, inspired respectively by the *label noise* and the *covariate shift* literature. We reviewed distribution shift sources and
their hierarchy and proposed two novel methods to synthetically create concept drift and class-conditional shifts in real-world datasets. Throughout extensive experiments, we benchmarked many competitors from the state-of-the-art. We opened some discussions on the results and assessed that the development of biquality learning algorithms robust to distributional changes of closed sets, despite the presented results, remains an interesting problem for future research in this area.
# Declarations

## Funding

Pierre Nodet, Vincent Lemaire, Alexis Bondu, and Antoine Cornuéjols received funding from Orange SA.

## Competing Interests

Pierre Nodet, Vincent Lemaire, and Alexis Bondu have received research support from Orange SA. Antoine Cornuéjols has received research support from AgroParisTech and Orange SA.

## Ethics approval

Not Applicable.

## Consent to participate

All authors have read and approved the final manuscript.

## Consent for publication

Not Applicable.

## Availability of data and materials

All data used in this study are available publicly online on openML: https://www.openml.org/

## Code availability

Code is available at the following url: https://github.com/pierrenodet/blds

## Authors’ contributions

Pierre Nodet, Vincent Lemaire, Alexis Bondu, and Antoine Cornuéjols contributed to the manuscript equally.
References

1. Bickel, S., Brückner, M., Scheffer, T.: Discriminative learning for differing training and test distributions. In: Proceedings of the 24th international conference on Machine learning. pp. 81–88 (2007)

2. Boser, B.E., Guyon, I.M., Vapnik, V.N.: A training algorithm for optimal margin classifiers. In: Proceedings of the fifth annual workshop on Computational learning theory. pp. 144–152 (1992)

3. Breiman, L.: Classification and regression trees. Routledge (1984)

4. Chang, C.C., Lin, C.J.: Libsvm: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST) 2(3), 1–27 (2011)

5. Cohen, J.: A coefficient of agreement for nominal scales. Educational and psychological measurement 20(1), 37–46 (1960)

6. Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to algorithms. MIT press (2022)

7. David, S.B., Lu, T., Luu, T., Pál, D.: Impossibility theorems for domain adaptation. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. pp. 129–136. JMLR Workshop and Conference Proceedings (2010)

8. Dua, D., Graff, C.: Uci machine learning repository (2017), http://archive.ics.uci.edu/ml

9. Fang, T., Lu, N., Niu, G., Sugiyama, M.: Rethinking importance weighting for deep learning under distribution shift. Advances in Neural Information Processing Systems 33, 11996–12007 (2020)

10. Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., Bouchachia, A.: A survey on concept drift adaptation. ACM computing surveys (CSUR) 46(4), 1–37 (2014)

11. Gretton, A., Smola, A., Huang, J., Schmittfull, M., Borgwardt, K., Schölkopf, B.: Covariate shift by kernel mean matching. Dataset shift in machine learning 3(4), 5 (2009)

12. Guyon, I.: Datasets of the active learning challenge. Tech. rep., University of Wisconsin-Madison Department of Computer Sciences (2010)

13. Hastie, T., Tibshirani, R., Friedman, J.H., Friedman, J.H.: The elements of statistical learning: data mining, inference, and prediction, vol. 2. Springer (2009)

14. Hendrycks, D., Mazeika, M., Wilson, D., Gimpel, K.: Using trusted data to train deep networks on labels corrupted by severe noise. In: Advances in Neural Information Processing Systems 31, pp. 10456–10465 (2018)

15. Huang, J., Gretton, A., Borgwardt, K., Schölkopf, B., Smola, A.J.: Correcting sample selection bias by unlabeled data. In: Advances in neural information processing systems. pp. 601–608 (2007)

16. Japkowicz, N., Stephen, S.: The class imbalance problem: A systematic study. Intelligent data analysis 6(5), 429–449 (2002)

17. Jiang, L., Zhou, Z., Leung, T., Li, L.J., Fei-Fei, L.: Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In: ICML (2018)

18. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., Liu, T.Y.: Lightgbm: A highly efficient gradient boosting decision tree. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 30. Curran Associates, Inc. (2017), https://proceedings.neurips.cc/paper/2017/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf
19. Kreuzberger, D., Kühl, N., Hirschl, S.: Machine learning operations (mlops): Overview, definition, and architecture. arXiv preprint arXiv:2205.02302 (2022)

20. Lloyd, S.: Least squares quantization in pcm. IEEE transactions on information theory 28(2), 129–137 (1982)

21. Miao, Y.Q., Farahat, A.K., Kamel, M.S.: Ensemble kernel mean matching. In: 2015 IEEE International Conference on Data Mining. pp. 330–338. IEEE (2015)

22. Moreno-Torres, J.G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N.V., Herrera, F.: A unifying view on dataset shift in classification. Pattern recognition 45(1), 521–530 (2012)

23. Nemenyi, P.: Distribution-free multiple comparisons. Biometrics 18(2), 263 (1962)

24. Nikodym, O.: Sur une généralisation des intégrales de m. j. radon. Fundamenta Mathematicae 15(1), 131–179 (1930), http://eudml.org/doc/212339

25. Nodet, P., Lemaire, V., Bondu, A., Cornuéjols, A., Ouorou, A.: From Weakly Supervised Learning to Biquality Learning: an Introduction. In: International Joint Conference on Neural Networks (IJCNN). IEEE (2021)

26. Nodet, P., Lemaire, V., Bondu, A., Cornuejols, A., Ouorou, A.: Importance reweighting for biquality learning. In: International Joint Conference on Neural Networks (IJCNN). IEEE (2021)

27. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: Machine learning in python. Journal of machine learning research 12, 2825–2830 (2011)

28. Ratner, A., Bach, S.H., Ehrenberg, H., Fries, J., Wu, S., Ré, C.: Snorkel: Rapid training data creation with weak supervision. The VLDB Journal 29(2), 709–730 (2020)

29. Ren, M., Zeng, W., Yang, B., Urtasun, R.: Learning to reweight examples for robust deep learning. In: International conference on machine learning. pp. 4334–4343. PMLR (2018)

30. Rousseeuw, P.J.: Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics 20, 53–65 (1987). https://doi.org/10.1016/0377-0427(87)90125-7, https://www.sciencedirect.com/science/article/pii/0377042787901257

31. Rudin, W.: Analyse réelle et complexe. Dunod (1975)

32. Shu, J., Xie, Q., Yi, L., Zhao, Q., Zhou, S., Xu, Z., Meng, D.: Meta-weight-net: Learning an explicit mapping for sample weighting. Advances in neural information processing systems 32 (2019)

33. Steinhardt, J., Koh, P.W.W., Liang, P.S.: Certified defenses for data poisoning attacks. Advances in neural information processing systems 30 (2017)

34. Sugiyama, M., Suzuki, T., Kanamori, T.: Density ratio estimation in machine learning. Cambridge University Press (2012)

35. Vanschoren, J., van Rijn, J.N., Bischl, B., Torgo, L.: Openml: Networked science in machine learning. SIGKDD Explor. Newsl. 15(2), 49–60 (jun 2014). https://doi.org/10.1145/2641190.2641198, https://doi.org/10.1145/2641190.2641198

36. Veeramachaneni, K., Arnaldo, I., Korrapati, V., Bassias, C., Li, K.: Ai2: Training a big data machine to defend. In: 2016 IEEE 2nd International Conference on Big Data Security on Cloud (BigDataSecurity). pp. 49–54 (2016)

37. Vert, J.P., Tsuda, K., Schölkopf, B.: A primer on kernel methods. Kernel methods in computational biology 47, 35–70 (2004)

38. Wilcoxon, F.: Individual comparisons by ranking methods. In: Breakthroughs in statistics, pp. 196–202. Springer (1992)
39. Wright, S.J.: Continuous optimization (nonlinear and linear programming). Foundations of Computer-Aided Process Design (1999)

40. Yang, J., Zhou, K., Li, Y., Liu, Z.: Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334 (2021)

41. Ye, Y., Tse, E.: An extension of karmarkar's projective algorithm for convex quadratic programming. Mathematical programming **44**(1), 157-179 (1989)

42. Yuen, M.C., King, I., Leung, K.S.: A survey of crowdsourcing systems. In: 2011 IEEE third international conference on privacy, security, risk and trust and 2011 IEEE third international conference on social computing. pp. 766-773. IEEE (2011)

43. Zadrozny, B., Elkan, C.: Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In: Proceedings of the Eighteenth International Conference on Machine Learning. p. 609-616. ICML '01, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (2001)

44. Zadrozny, B., Elkan, C.: Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 694-699 (2002)

45. Zheng, G., Awadallah, A.H., Dumais, S.: Meta label correction for noisy label learning. Proceedings of the AAAI Conference on Artificial Intelligence **35** (2021)

46. Zhou, Z.H.: A brief introduction to weakly supervised learning. National Science Review **5**(1), 44-53 (08 2017)
A Additional Tables
<table><caption>Table 4: Table of conversion between *p* and the actual ratio of trusted data (actual).</caption><thead><tr><th>Dataset</th><th>p</th><th>actual</th><th>Dataset</th><th>p</th><th>actual</th></tr></thead><tbody><tr><td rowspan="3">ad</td><td>0.25</td><td>0.028</td><td rowspan="3">magic-telescope</td><td>0.25</td><td>0.003</td></tr><tr><td>0.50</td><td>0.041</td><td>0.50</td><td>0.004</td></tr><tr><td>0.75</td><td>0.112</td><td>0.75</td><td>0.015</td></tr><tr><td rowspan="3">artificial-chars</td><td>0.25</td><td>0.010</td><td rowspan="3">mnist</td><td>0.25</td><td>0.001</td></tr><tr><td>0.50</td><td>0.048</td><td>0.50</td><td>0.002</td></tr><tr><td>0.75</td><td>0.273</td><td>0.75</td><td>0.005</td></tr><tr><td rowspan="3">bank-marketing</td><td>0.25</td><td>0.003</td><td rowspan="3">moszilla4</td><td>0.25</td><td>0.003</td></tr><tr><td>0.50</td><td>0.004</td><td>0.50</td><td>0.005</td></tr><tr><td>0.75</td><td>0.032</td><td>0.75</td><td>0.013</td></tr><tr><td rowspan="3">click</td><td>0.25</td><td>0.165</td><td rowspan="3">mushroom</td><td>0.25</td><td>0.006</td></tr><tr><td>0.50</td><td>0.328</td><td>0.50</td><td>0.008</td></tr><tr><td>0.75</td><td>0.045</td><td>0.75</td><td>0.009</td></tr><tr><td rowspan="3">connect-4</td><td>0.25</td><td>0.003</td><td rowspan="3">musk</td><td>0.25</td><td>0.009</td></tr><tr><td>0.50</td><td>0.009</td><td>0.50</td><td>0.019</td></tr><tr><td>0.75</td><td>0.041</td><td>0.75</td><td>0.120</td></tr><tr><td rowspan="3">dna</td><td>0.25</td><td>0.020</td><td rowspan="3">nomao</td><td>0.25</td><td>0.002</td></tr><tr><td>0.50</td><td>0.032</td><td>0.50</td><td>0.003</td></tr><tr><td>0.75</td><td>0.044</td><td>0.75</td><td>0.005</td></tr><tr><td rowspan="3">eeg</td><td>0.25</td><td>0.010</td><td rowspan="3">pendigits</td><td>0.25</td><td>0.007</td></tr><tr><td>0.50</td><td>0.032</td><td>0.50</td><td>0.008</td></tr><tr><td>0.75</td><td>0.093</td><td>0.75</td><td>0.011</td></tr><tr><td rowspan="3">electricity</td><td>0.25</td><td>0.003</td><td rowspan="3">phishing</td><td>0.25</td><td>0.006</td></tr><tr><td>0.50</td><td>0.004</td><td>0.50</td><td>0.008</td></tr><tr><td>0.75</td><td>0.024</td><td>0.75</td><td>0.009</td></tr><tr><td rowspan="3">first-ord-theo-prov</td><td>0.25</td><td>0.015</td><td rowspan="3">phoeneme</td><td>0.25</td><td>0.013</td></tr><tr><td>0.50</td><td>0.082</td><td>0.50</td><td>0.016</td></tr><tr><td>0.75</td><td>0.271</td><td>0.75</td><td>0.115</td></tr><tr><td rowspan="3">gas-drift</td><td>0.25</td><td>0.003</td><td rowspan="3">poker</td><td>0.25</td><td>0.002</td></tr><tr><td>0.50</td><td>0.005</td><td>0.50</td><td>0.005</td></tr><tr><td>0.75</td><td>0.013</td><td>0.75</td><td>0.013</td></tr><tr><td rowspan="3">har</td><td>0.25</td><td>0.007</td><td rowspan="3">satimage</td><td>0.25</td><td>0.007</td></tr><tr><td>0.50</td><td>0.009</td><td>0.50</td><td>0.009</td></tr><tr><td>0.75</td><td>0.015</td><td>0.75</td><td>0.020</td></tr><tr><td rowspan="3">hiva</td><td>0.25</td><td>0.015</td><td rowspan="3">skin-seg</td><td>0.25</td><td>0.000</td></tr><tr><td>0.50</td><td>0.070</td><td>0.50</td><td>0.001</td></tr><tr><td>0.75</td><td>0.214</td><td>0.75</td><td>0.001</td></tr><tr><td rowspan="3">ibn-sina</td><td>0.25</td><td>0.002</td><td rowspan="3">spam</td><td>0.25</td><td>0.013</td></tr><tr><td>0.50</td><td>0.003</td><td>0.50</td><td>0.015</td></tr><tr><td>0.75</td><td>0.004</td><td>0.75</td><td>0.033</td></tr><tr><td rowspan="3">ijcnn1</td><td>0.25</td><td>0.002</td><td rowspan="3">splice</td><td>0.25</td><td>0.017</td></tr><tr><td>0.50</td><td>0.003</td><td>0.50</td><td>0.019</td></tr><tr><td>0.75</td><td>0.008</td><td>0.75</td><td>0.034</td></tr><tr><td rowspan="3">isolet</td><td>0.25</td><td>0.015</td><td rowspan="3">svmguide3</td><td>0.25</td><td>0.160</td></tr><tr><td>0.50</td><td>0.015</td><td>0.50</td><td>0.364</td></tr><tr><td>0.75</td><td>0.048</td><td>0.75</td><td>0.607</td></tr><tr><td rowspan="3">japanese-vowels</td><td>0.25</td><td>0.007</td><td rowspan="3">usps</td><td>0.25</td><td>0.007</td></tr><tr><td>0.50</td><td>0.009</td><td>0.50</td><td>0.009</td></tr><tr><td>0.75</td><td>0.027</td><td>0.75</td><td>0.014</td></tr><tr><td rowspan="3">jm1</td><td>0.25</td><td>0.016</td><td rowspan="3">web</td><td>0.25</td><td>0.002</td></tr><tr><td>0.50</td><td>0.016</td><td>0.50</td><td>0.004</td></tr><tr><td>0.75</td><td>0.149</td><td>0.75</td><td>0.009</td></tr><tr><td rowspan="3">ldpa</td><td>0.25</td><td>0.001</td><td rowspan="3">zebra</td><td>0.25</td><td>0.033</td></tr><tr><td>0.50</td><td>0.004</td><td>0.50</td><td>0.122</td></tr><tr><td>0.75</td><td>0.026</td><td>0.75</td><td>0.272</td></tr><tr><td rowspan="3">letter</td><td>0.25</td><td>0.007</td><td></td><td></td><td></td></tr><tr><td>0.50</td><td>0.012</td><td></td><td></td><td></td></tr><tr><td>0.75</td><td>0.040</td><td></td><td></td><td></td></tr></tbody></table>
<table><caption>Table 5: Table of conversion between $\rho$ and the actual ratio between the number of samples after and before subsampling (actual).</caption><thead><tr><th>Dataset</th><th>ρ</th><th>actual</th><th>Dataset</th><th>ρ</th><th>actual</th><th>Dataset</th><th>ρ</th><th>actual</th></tr></thead><tbody><tr><td rowspan="6">ad</td><td>1</td><td>1.000</td><td rowspan="6">ijcnn1</td><td>1</td><td>1.000</td><td rowspan="6">phishing</td><td>1</td><td>1.000</td></tr><tr><td>2</td><td>0.922</td><td>2</td><td>0.801</td><td>2</td><td>0.851</td></tr><tr><td>5</td><td>0.877</td><td>5</td><td>0.682</td><td>5</td><td>0.761</td></tr><tr><td>10</td><td>0.861</td><td>10</td><td>0.642</td><td>10</td><td>0.731</td></tr><tr><td>20</td><td>0.853</td><td>20</td><td>0.623</td><td>20</td><td>0.716</td></tr><tr><td>50</td><td>0.849</td><td>50</td><td>0.611</td><td>50</td><td>0.708</td></tr><tr><td rowspan="6">artificial-chars</td><td>100</td><td>0.848</td><td rowspan="6">isolet</td><td>100</td><td>0.607</td><td rowspan="6">phoeneme</td><td>100</td><td>0.704</td></tr><tr><td>1</td><td>1.000</td><td>1</td><td>1.000</td><td>1</td><td>1.000</td></tr><tr><td>2</td><td>0.814</td><td>2</td><td>0.828</td><td>2</td><td>0.802</td></tr><tr><td>5</td><td>0.703</td><td>5</td><td>0.726</td><td>5</td><td>0.683</td></tr><tr><td>10</td><td>0.666</td><td>10</td><td>0.691</td><td>10</td><td>0.643</td></tr><tr><td>20</td><td>0.647</td><td>20</td><td>0.674</td><td>20</td><td>0.623</td></tr><tr><td>50</td><td>0.636</td><td>50</td><td>0.663</td><td>50</td><td>0.611</td></tr><tr><td rowspan="6">bank-marketing</td><td>100</td><td>0.632</td><td rowspan="6">japanese-vowels</td><td>100</td><td>0.662</td><td rowspan="6">poker</td><td>100</td><td>0.607</td></tr><tr><td>1</td><td>1.000</td><td>1</td><td>1.000</td><td>1</td><td>1.000</td></tr><tr><td>2</td><td>0.896</td><td>2</td><td>0.791</td><td>2</td><td>0.762</td></tr><tr><td>5</td><td>0.834</td><td>5</td><td>0.666</td><td>5</td><td>0.620</td></tr><tr><td>10</td><td>0.814</td><td>10</td><td>0.624</td><td>10</td><td>0.572</td></tr><tr><td>20</td><td>0.803</td><td>20</td><td>0.604</td><td>20</td><td>0.549</td></tr><tr><td>50</td><td>0.797</td><td>50</td><td>0.591</td><td>50</td><td>0.534</td></tr><tr><td rowspan="6">click</td><td>100</td><td>0.795</td><td rowspan="6">jml</td><td>100</td><td>0.587</td><td rowspan="6">satimage</td><td>100</td><td>0.530</td></tr><tr><td>1</td><td>1.000</td><td>1</td><td>1.000</td><td>1</td><td>1.000</td></tr><tr><td>2</td><td>0.889</td><td>2</td><td>0.960</td><td>2</td><td>0.855</td></tr><tr><td>5</td><td>0.823</td><td>5</td><td>0.936</td><td>5</td><td>0.768</td></tr><tr><td>10</td><td>0.800</td><td>10</td><td>0.928</td><td>10</td><td>0.740</td></tr><tr><td>20</td><td>0.789</td><td>20</td><td>0.924</td><td>20</td><td>0.725</td></tr><tr><td rowspan="6">connect-4</td><td>50</td><td>0.783</td><td rowspan="6">ldpa</td><td>50</td><td>0.921</td><td rowspan="6">skin-seg</td><td>50</td><td>0.717</td></tr><tr><td>100</td><td>0.780</td><td>100</td><td>0.920</td><td>100</td><td>0.714</td></tr><tr><td>1</td><td>1.000</td><td>1</td><td>1.000</td><td>1</td><td>1.000</td></tr><tr><td>2</td><td>0.778</td><td>2</td><td>0.791</td><td>2</td><td>0.865</td></tr><tr><td>5</td><td>0.645</td><td>5</td><td>0.665</td><td>5</td><td>0.784</td></tr><tr><td>10</td><td>0.601</td><td>10</td><td>0.624</td><td>10</td><td>0.757</td></tr><tr><td rowspan="6">dna</td><td>100</td><td>0.643</td><td rowspan="6">ldpa</td><td>100</td><td>0.603</td><td rowspan="6">satimage</td><td>100</td><td>0.733</td></tr><tr><td>1</td><td>1.000</td><td>1</td><td>1.000</td><td>1</td><td>1.000</td></tr><tr><td>2</td><td>0.820</td><td>2</td><td>0.806</td><td>2</td><td>0.957</td></tr><tr><td>5</td><td>0.711</td><td>5</td><td>0.690</td><td>5</td><td>0.931</td></tr><tr><td>10</td><td>0.675</td><td>10</td><td>0.651</td><td>10</td><td>0.922</td></tr><tr><td>20</td><td>0.657</td><td>20</td><td>0.632</td><td>20</td><td>0.918</td></tr><tr><td rowspan="6">eeg</td><td>50</td><td>0.646</td><td rowspan="6">ldpa</td><td>50</td><td>0.620</td><td rowspan="6">satimage</td><td>50</td><td>0.915</td></tr><tr><td>100</td><td>0.643</td><td>100</td><td>0.617</td><td>100</td><td>0.914</td></tr><tr><td>1</td><td>1.000</td><td>1</td><td>1.000</td><td>1</td><td>1.000</td></tr><tr><td>2</td><td>0.913</td><td>2</td><td>0.832</td><td>2</td><td>0.821</td></tr><tr><td>5</td><td>0.860</td><td>5</td><td>0.732</td><td>5</td><td>0.713</td></tr><tr><td>10</td><td>0.843</td><td>10</td><td>0.698</td><td>10</td><td>0.676</td></tr><tr><td rowspan="6">electricity</td><td>20</td><td>0.834</td><td rowspan="6">ldpa</td><td>20</td><td>0.681</td><td rowspan="6">satimage</td><td>20</td><td>0.658</td></tr><tr><td>50</td><td>0.829</td><td>50</td><td>0.671</td><td>50</td><td>0.648</td></tr><tr><td>100</td><td>0.827</td><td>100</td><td>0.668</td><td>100</td><td>0.646</td></tr><tr><td>1</td><td>1.000</td><td>1</td><td>1.000</td><td>1</td><td>1.000</td></tr><tr><td>2</td><td>0.775</td><td>2</td><td>0.861</td><td>2</td><td>0.855</td></tr><tr><td>5</td><td>0.641</td><td>5</td><td>0.777</td><td>5</td><td>0.769</td></tr><tr><td rowspan="6">first-ord-theo-prov</td><td>100</td><td>0.555</td><td rowspan="6">ldpa</td><td>100</td><td>0.725</td><td rowspan="6">satimage</td><td>100</td><td>0.716</td></tr><tr><td>2</td><td>0.922</td><td>1</td><td>1.000</td><td>1</td><td>1.000</td></tr><tr><td>5</td><td>0.875</td><td>2</td><td>0.861</td><td>2</td><td>0.810</td></tr><tr><td>10</td><td>0.859</td><td>5</td><td>0.777</td><td>5</td><td>0.697</td></tr><tr><td>20</td><td>0.852</td><td>10</td><td>0.749</td><td>10</td><td>0.658</td></tr><tr><td>50</td><td>0.849</td><td>20</td><td>0.735</td><td>20</td><td>0.639</td></tr><tr><td rowspan="6">gas-drift</td><td>100</td><td>0.847</td><td rowspan="6">ldpa</td><td>100</td><td>0.724</td><td rowspan="6">satimage</td><td>100</td><td>0.624</td></tr><tr><td>1</td><td>1.000</td><td>1</td><td>1.000</td><td>1</td><td>1.000</td></tr><tr><td>2</td><td>0.876</td><td>2</td><td>0.852</td><td>2</td><td>0.802</td></tr><tr><td>5</td><td>0.801</td><td>5</td><td>0.763</td><td>5</td><td>0.683</td></tr><tr><td>10</td><td>0.777</td><td>10</td><td>0.734</td><td>10</td><td>0.643</td></tr><tr><td>20</td><td>0.764</td><td>20</td><td>0.719</td><td>20</td><td>0.623</td></tr><tr><td rowspan="6">har</td><td>50</td><td>0.757</td><td rowspan="6">ldpa</td><td>50</td><td>0.710</td><td rowspan="6">satimage</td><td>50</td><td>0.612</td></tr><tr><td>100</td><td>0.754</td><td>100</td><td>0.707</td><td>100</td><td>0.608</td></tr><tr><td>1</td><td>1.000</td><td>1</td><td>1.000</td><td>1</td><td>1.000</td></tr><tr><td>2</td><td>0.894</td><td>2</td><td>0.814</td><td>2</td><td>0.903</td></tr><tr><td>5</td><td>0.830</td><td>5</td><td>0.703</td><td>5</td><td>0.846</td></tr><tr><td>10</td><td>0.809</td><td>10</td><td>0.666</td><td>10</td><td>0.826</td></tr><tr><td rowspan="6">hiva</td><td>20</td><td>0.798</td><td rowspan="6">ldpa</td><td>20</td><td>0.647</td><td rowspan="6">satimage</td><td>20</td><td>0.817</td></tr><tr><td>50</td><td>0.792</td><td>50</td><td>0.636</td><td>50</td><td>0.811</td></tr><tr><td>100</td><td>0.790</td><td>100</td><td>0.632</td><td>100</td><td>0.809</td></tr><tr><td>1</td><td>1.000</td><td>1</td><td>1.000</td><td>1</td><td>1.000</td></tr><tr><td>2</td><td>0.940</td><td>2</td><td>0.931</td><td>2</td><td>0.903</td></tr><tr><td>5</td><td>0.905</td><td>5</td><td>0.890</td><td>5</td><td>0.846</td></tr><tr><td rowspan="6">ibn-sina</td><td>10</td><td>0.893</td><td rowspan="6">ldpa</td><td>10</td><td>0.877</td><td rowspan="6">satimage</td><td>10</td><td>0.826</td></tr><tr><td>20</td><td>0.887</td><td>20</td><td>0.870</td><td>20</td><td>0.817</td></tr><tr><td>50</td><td>0.883</td><td>50</td><td>0.866</td><td>50</td><td>0.811</td></tr><tr><td>100</td><td>0.882</td><td>100</td><td>0.864</td><td>100</td><td>0.809</td></tr><tr><td>1</td><td>1.000</td><td>1</td><td>1.000</td><td>1</td><td>1.000</td></tr><tr><td>2</td><td>0.929</td><td>2</td><td>0.830</td><td>2</td><td>0.809</td></tr><tr><td rowspan="6">ibn-sina</td><td>5</td><td>0.886</td><td rowspan="6">ldpa</td><td>5</td><td>0.728</td><td rowspan="6">satimage</td><td>5</td><td>0.728</td></tr><tr><td>10</td><td>0.872</td><td>10</td><td>0.694</td><td>10</td><td>0.694</td></tr><tr><td>20</td><td>0.865</td><td>20</td><td>0.677</td><td>20</td><td>0.677</td></tr><tr><td>50</td><td>0.861</td><td>50</td><td>0.667</td><td>50</td><td>0.667</td></tr><tr><td>100</td><td>0.859</td><td>100</td><td>0.664</td><td>100</td><td>0.664</td></tr></tbody></table>
<figure><img src="image_10.png" /><figcaption>Fig. 7: Additional results of the Wilcoxon signed rank test computed on all datasets. Each figure compares one competitor versus another for a given trusted data ratio. Figures in the same row are the same competitors against different cases of trusted data ratio: *p* = 0.25, *p* = 0.5, *p* = 0.75. In each figure “o”, “.” and “●” indicate respectively a win, a tie, or a loss of the first competitor compared to the second competitor, the vertical axis is *r*, and the horizontal axis is *ρ*.</figcaption></figure>