Repository: scikit-activeml/scikit-activeml (language: python)

Issue:
FR: Implement the pool-based query strategy `Clue`
### Describe the workflow you want to enable

`Clue` (Clustering Uncertainty-weighted Embeddings) [1] is a pool-based strategy that selects samples nearest to the centroids of a $k$-means clustering in the (classifier's learned) feature space using uncertainties as sample weights.

[[1] Prabhu, Viraj, Arjun Chandrasekaran, Kate Saenko, and Judy Hoffman. "Active Domain Adaptation via Clustering Uncertainty-weighted Embeddings." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8505-8514. 2021](https://openaccess.thecvf.com/content/ICCV2021/papers/Prabhu_Active_Domain_Adaptation_via_Clustering_Uncertainty-Weighted_Embeddings_ICCV_2021_paper.pdf)

### Describe your proposed solution

The authors detail their algorithm in the paper [1] and provide a Pytorch implementation at [GitHub](https://github.com/virajprabhu/CLUE).

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_


Discussion / hints:

Fix patch:
diff --git a/README.rst b/README.rst
index f6ddda33..83e49c2c 100644
--- a/README.rst
+++ b/README.rst
@@ -1,6 +1,18 @@
 .. intro_start
 
-|Doc| |Codecov| |PythonVersion| |PyPi| |Paper| |Black| |Downloads|
+|
+
+.. raw:: html
+
+   <p align="center">
+      <img src="https://raw.githubusercontent.com/scikit-activeml/scikit-activeml/master/docs/logos/scikit-activeml-logo.png" width="300px"\>
+   </p>
+|
+
+=====================================================================
+scikit-activeml: A Library and Toolbox for Active Learning Algorithms
+=====================================================================
+|Doc| |Codecov| |PythonVersion| |PyPi| |Black| |Downloads| |Paper|
 
 .. |Doc| image:: https://img.shields.io/badge/docs-latest-green
    :target: https://scikit-activeml.github.io/scikit-activeml-docs/
@@ -23,25 +35,13 @@
 .. |Downloads| image:: https://static.pepy.tech/badge/scikit-activeml
    :target: https://www.pepy.tech/projects/scikit-activeml
 
-|
-
-.. image:: https://raw.githubusercontent.com/scikit-activeml/scikit-activeml/master/docs/logos/scikit-activeml-logo.png
-   :width: 200
-
-|
-
-Machine learning applications often need large amounts of training data to
+Machine learning models often need large amounts of training data to
 perform well. Whereas unlabeled data can be easily gathered, the labeling process
 is difficult, time-consuming, or expensive in most applications. Active learning can help solve
-this problem by querying labels for those data samples that will improve the performance
+this problem by querying labels for those data samples improving the performance
 the most. Thereby, the goal is that the learning algorithm performs sufficiently well with
-fewer labels
-
-With this goal in mind, **scikit-activeml** has been developed as a Python module for active learning
-on top of `scikit-learn <https://scikit-learn.org/stable/>`_. The project was initiated in 2020 by the
-`Intelligent Embedded Systems Group <https://www.uni-kassel.de/eecs/en/sections/intelligent-embedded-systems/home>`_
-at the University of Kassel and is distributed under the `3-Clause BSD licence
-<https://github.com/scikit-activeml/scikit-activeml/blob/master/LICENSE.txt>`_.
+fewer labels. With this goal in mind, **scikit-activeml** has been developed as a Python library for active learning
+on top of `scikit-learn <https://scikit-learn.org/stable/>`_.
 
 .. intro_end
 
@@ -56,16 +56,17 @@ The easiest way of installing scikit-activeml is using ``pip``:
 
     pip install -U scikit-activeml
 
-.. note::
-    The installation via `pip` defines only minimum requirements to avoid
-    potential package downgrades withing your installation. If you encounter
-    any incompatibility issues, you can downgrade packages by installing the
-    `maximum requirements <https://github.com/scikit-activeml/scikit-activeml/blob/master/requirements_max.txt>`_,
-    we tested at the release of the current package
-    version:
-    ::
+The installation via `pip` defines only minimum requirements to avoid
+potential package downgrades withing your installation. If you encounter
+any incompatibility issues, you can downgrade packages by installing the
+`maximum requirements <https://github.com/scikit-activeml/scikit-activeml/blob/master/requirements_max.txt>`_,
+we tested at the release of the current package
+version:
+
+::
+
+    pip install -r requirements_max.txt
 
-        pip install -r requirements_max.txt
 .. user_installation_end
 
 .. examples_start
@@ -85,7 +86,7 @@ We provide a broad overview of different use-cases in our `tutorial section <htt
 - `Batch Stream-based Active Learning with Pool Query Strategies <https://scikit-activeml.github.io/scikit-activeml-docs/generated/tutorials/21_stream_batch_with_pool_al.html>`_,
 - and `Stream-based Active Learning With River <https://scikit-activeml.github.io/scikit-activeml-docs/generated/tutorials/22_river_classifier.html>`_.
 
-Two simple examples illustrating the straightforwardness of implementing active learning cycles with our Python package ``skactiveml`` are given in the following.
+Two simple code snippets illustrating the straightforwardness of implementing active learning cycles with our Python package ``skactiveml`` are given in the following.
 
 Pool-based Active Learning
 ##########################
@@ -188,16 +189,16 @@ Query Strategy Overview
 #######################
 
 For better orientation, we provide an `overview <https://scikit-activeml.github.io/scikit-activeml-docs/generated/strategy_overview.html>`_
-(incl. paper references and `visualizations <https://scikit-activeml.github.io/scikit-activeml-docs/generated/sphinx_gallery_examples/index.html>`_)
+(incl. paper references and `visual examples <https://scikit-activeml.github.io/scikit-activeml-docs/generated/sphinx_gallery_examples/index.html>`_)
 of the query strategies implemented by ``skactiveml``.
 
 |Overview| |Visualization|
 
 .. |Overview| image:: https://raw.githubusercontent.com/scikit-activeml/scikit-activeml/master/docs/logos/strategy-overview.gif
-   :width: 400
+   :width: 365
 
 .. |Visualization| image:: https://raw.githubusercontent.com/scikit-activeml/scikit-activeml/master/docs/logos/example-overview.gif
-   :width: 400
+   :width: 365
 
 .. examples_end
 
diff --git a/docs/examples/pool/_clue.json b/docs/examples/pool/_clue.json
new file mode 100644
index 00000000..b29854c1
--- /dev/null
+++ b/docs/examples/pool/_clue.json
@@ -0,0 +1,18 @@
+[
+{
+    "class" : "Clue",
+    "package" : "pool",
+    "method" : "Clustering Uncertainty-weighted Embeddings (CLUE)",
+    "category" : "Uncertainty Sampling",
+    "template" : "examples/template_pool_batch.py",
+    "tags" : ["pool", "classification", "single-annotator"],
+    "title" : "Clustering Uncertainty-weighted Embeddings (CLUE)",
+    "refs" : ["prabhu2021active"],
+    "sequence" : ["title", "text_0", "plot", "refs"],
+    "text_0" : "",
+    "import_misc" : "from skactiveml.pool import Clue",
+    "init_qs" : "Clue()",
+    "query_params" : "X=X, y=y, clf=clf"
+}
+]
+
diff --git a/docs/generate.py b/docs/generate.py
index c28a0721..1f346cc2 100644
--- a/docs/generate.py
+++ b/docs/generate.py
@@ -622,11 +622,11 @@ def format_plot(data, template_path):
 
             if "FULLEXAMPLES" not in os.environ:
                 if key == "n_samples":
-                    data[key] = "10"
+                    data[key] = "50"
                 elif key == "n_cycles":
-                    data[key] = "2"
+                    data[key] = "5"
                 elif key == "res":
-                    data[key] = "3"
+                    data[key] = "8"
             if key in data.keys():
                 if isinstance(data[key], list):
                     new_str = ""
diff --git a/docs/index.rst b/docs/index.rst
index 3c71439d..2802e6a3 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -1,6 +1,3 @@
-Welcome to scikit-activeml's documentation!
-===========================================
-
 .. toctree::
    :maxdepth: 2
    :hidden:
@@ -24,8 +21,6 @@ Welcome to scikit-activeml's documentation!
    :start-after: examples_start
    :end-before: examples_end
 
-You can find more examples under :doc:`Examples </generated/sphinx_gallery_examples/index>`.
-
 .. include:: ../README.rst
    :start-after: dev_start
    :end-before: dev_end
diff --git a/docs/refs.bib b/docs/refs.bib
index 8f48c59c..62f036ab 100644
--- a/docs/refs.bib
+++ b/docs/refs.bib
@@ -324,4 +324,11 @@ @inproceedings{burbidge2007active
   booktitle={Internation Conference on Intelligent Data Engineering and Automated Learning},
   pages={209--218},
   year={2007},
+}
+@inproceedings{prabhu2021active,
+  title={{Active Domain Adaptation via Clustering Uncertainty-weighted Embeddings}},
+  author={Prabhu, Viraj and Chandrasekaran, Arjun and Saenko, Kate and Hoffman, Judy},
+  booktitle={ICCV},
+  pages={8505--8514},
+  year={2021}
 }
\ No newline at end of file
diff --git a/skactiveml/pool/__init__.py b/skactiveml/pool/__init__.py
index 41303cc6..d745ab03 100644
--- a/skactiveml/pool/__init__.py
+++ b/skactiveml/pool/__init__.py
@@ -38,6 +38,7 @@
 from ._badge import Badge
 from ._prob_cover import ProbCover
 from ._contrastive_al import ContrastiveAL
+from ._clue import Clue
 from ._wrapper import SubSamplingWrapper, ParallelUtilityEstimationWrapper
 
 __all__ = [
@@ -67,6 +68,7 @@
     "GreedySamplingTarget",
     "DiscriminativeAL",
     "BatchBALD",
+    "Clue",
     "batch_bald",
     "CoreSet",
     "k_greedy_center",
diff --git a/skactiveml/pool/_clue.py b/skactiveml/pool/_clue.py
new file mode 100644
index 00000000..b295e909
--- /dev/null
+++ b/skactiveml/pool/_clue.py
@@ -0,0 +1,203 @@
+"""
+Module implementing Clustering Uncertainty-weighted Embeddings (CLUE).
+
+CLUE is a deep active learning strategy, which performs a clustering with
+uncertainties as sample weights.
+"""
+
+import numpy as np
+
+from ..base import SingleAnnotatorPoolQueryStrategy, SkactivemlClassifier
+from ..pool import uncertainty_scores
+from ..utils import (
+    MISSING_LABEL,
+    rand_argmax,
+    check_type,
+    check_equal_missing_label,
+)
+from sklearn.base import clone
+from sklearn.cluster import KMeans
+
+
+class Clue(SingleAnnotatorPoolQueryStrategy):
+    """Clustering Uncertainty-weighted Embeddings (CLUE)
+
+    This class implements the Clustering Uncertainty-weighted Embeddings (CLUE)
+    query strategy [1]_, which considers both diversity and uncertainty of the
+    samples.
+
+    Parameters
+    ----------
+    missing_label : scalar or string or np.nan or None, default=np.nan
+        Value to represent a missing label.
+    random_state : None or int or np.random.RandomState, default=None
+        The random state to use.
+    cluster_algo : ClusterMixin.__class__, default=KMeans
+        The cluster algorithm to be used. It must implement a `fit_transform`
+        method, which takes samples `X` and `sample_weight` as inputs, e.g.,
+        sklearn.clustering.KMeans and sklearn.clustering.MiniBatchKMeans.
+    cluster_algo_dict : dict, default=None
+        The parameters passed to the clustering algorithm `cluster_algo`,
+        excluding the parameter for the number of clusters.
+    n_cluster_param_name : string, default="n_clusters"
+        The name of the parameter for the number of clusters.
+    method : 'least_confident' or 'margin_sampling' or 'entropy', \
+            default="entropy"
+        - `method='least_confident'` queries the sample whose maximal posterior
+          probability is minimal.
+        - `method='margin_sampling'` queries the sample whose posterior
+          probability gap between the most and the second most probable class
+          label is minimal.
+        - `method='entropy'` queries the sample whose posterior's have the
+          maximal entropy.
+    clf_embedding_flag_name : str or None, default=None
+        Name of the flag, which is passed to the `predict_proba` method for
+        getting the (learned) sample representations.
+            - If `clf_embedding_flag_name=None` and `predict_proba` returns
+              only one output, the input samples `X` are used.
+            - If `predict_proba` returns two outputs or `clf_embedding_name` is
+              not `None`, `(proba, embeddings)` are expected as outputs.
+
+    References
+    ----------
+    .. [1] Prabhu, Viraj, Arjun Chandrasekaran, Kate Saenko, and Judy Hoffman,
+       "Active Domain Adaptation via Clustering Uncertainty-weighted
+       Embeddings", ICCV, 2022.
+    """
+
+    def __init__(
+        self,
+        missing_label=MISSING_LABEL,
+        random_state=None,
+        cluster_algo=KMeans,
+        cluster_algo_dict=None,
+        n_cluster_param_name="n_clusters",
+        method="entropy",
+        clf_embedding_flag_name=None,
+    ):
+        super().__init__(
+            missing_label=missing_label, random_state=random_state
+        )
+        self.cluster_algo = cluster_algo
+        self.cluster_algo_dict = cluster_algo_dict
+        self.n_cluster_param_name = n_cluster_param_name
+        self.method = method
+        self.clf_embedding_flag_name = clf_embedding_flag_name
+
+    def query(
+        self,
+        X,
+        y,
+        clf,
+        fit_clf=True,
+        sample_weight=None,
+        candidates=None,
+        batch_size=1,
+        return_utilities=False,
+    ):
+        """Query the next samples to be labeled
+
+        X : array-like of shape (n_samples, n_features)
+            Training data set, usually complete, i.e. including the labeled and
+            unlabeled samples.
+        y : array-like of shape (n_samples,)
+            Labels of the training data set (possibly including unlabeled ones
+            indicated by `self.missing_label`.)
+        clf : skactiveml.base.SkactivemlClassifier
+            Classifier implementing the methods `fit` and `predict_proba`.
+        fit_clf : bool, default=True
+            Defines whether the classifier `clf` should be fitted on `X`, `y`,
+            and `sample_weight`.
+        sample_weight: array-like of shape (n_samples,), default=None
+            Weights of training samples in `X`.
+        candidates : None or array-like of shape (n_candidates, ) of type \
+                int, default=None
+            - If `candidates` is `None`, the unlabeled samples from
+              `(X,y)` are considered as `candidates`.
+            - If `candidates` is of shape `(n_candidates,)` and of type
+              `int`, `candidates` is considered as the indices of the
+              samples in `(X,y)`.
+        batch_size : int, default=1
+            The number of samples to be selected in one AL cycle.
+        return_utilities : bool, default=False
+            If true, also return the utilities based on the query strategy.
+
+        Returns
+        -------
+        query_indices : numpy.ndarray of shape (batch_size)
+            The query_indices indicate for which candidate sample a label is
+            to queried, e.g., `query_indices[0]` indicates the first selected
+            sample. The indexing refers to the samples in `X`.
+        utilities : numpy.ndarray of shape (batch_size, n_samples)
+            The utilities of samples after each selected sample of the batch,
+            e.g., `utilities[0]` indicates the utilities used for selecting
+            the first sample (with index `query_indices[0]`) of the batch.
+            Utilities for labeled samples will be set to np.nan. The indexing
+            refers to the samples in `X`.
+        """
+        # Check `__init__` and `query` parameters.
+        X, y, candidates, batch_size, return_utilities = self._validate_data(
+            X, y, candidates, batch_size, return_utilities, reset=True
+        )
+        X_cand, mapping = self._transform_candidates(
+            candidates, X, y, enforce_mapping=True
+        )
+        check_type(
+            self.cluster_algo_dict, "cluster_algo_dict", (dict, type(None))
+        )
+        cluster_algo_dict = (
+            {}
+            if self.cluster_algo_dict is None
+            else self.cluster_algo_dict.copy()
+        )
+        check_type(self.n_cluster_param_name, "n_cluster_param_name", str)
+        check_type(clf, "clf", SkactivemlClassifier)
+        check_type(fit_clf, "fit_clf", bool)
+        check_equal_missing_label(clf.missing_label, self.missing_label_)
+
+        # Fit the classifier.
+        if fit_clf:
+            if sample_weight is not None:
+                clf = clone(clf).fit(X, y, sample_weight)
+            else:
+                clf = clone(clf).fit(X, y)
+
+        # Compute class-membership predictions and optionally embeddings.
+        if self.clf_embedding_flag_name is not None:
+            probas, X_cand = clf.predict_proba(
+                X_cand, **{self.clf_embedding_flag_name: True}
+            )
+        else:
+            probas = clf.predict_proba(X_cand)
+            if isinstance(probas, tuple):
+                probas, X_cand = probas
+
+        # Compute uncertainties according to given `method`.
+        uncertainties = uncertainty_scores(probas=probas, method=self.method)
+
+        # Implement a fallback, if all uncertainties are zero.
+        if np.sum(uncertainties) == 0:
+            uncertainties = np.ones_like(uncertainties)
+
+        # Perform clustering to get centroids.
+        cluster_algo_dict[self.n_cluster_param_name] = batch_size
+        cluster_obj = self.cluster_algo(**cluster_algo_dict)
+        dist = cluster_obj.fit_transform(
+            X_cand, y=None, sample_weight=uncertainties
+        )
+
+        # Determine `query_indices` of the samples being closest to the
+        # respective centroids.
+        query_indices = []
+        utilities = np.full((batch_size, len(X)), fill_value=np.nan)
+        for b in range(batch_size):
+            utilities[b][mapping] = -dist[:, b]
+            utilities[b][query_indices] = np.nan
+            idx_b = rand_argmax(utilities[b], random_state=self.random_state_)
+            query_indices.append(idx_b[0])
+        query_indices = np.array(query_indices, dtype=int)
+
+        if return_utilities:
+            return query_indices, utilities
+        else:
+            return query_indices


Test patch:
diff --git a/skactiveml/pool/tests/test_clue.py b/skactiveml/pool/tests/test_clue.py
new file mode 100644
index 00000000..b5137a62
--- /dev/null
+++ b/skactiveml/pool/tests/test_clue.py
@@ -0,0 +1,202 @@
+import unittest
+
+import numpy as np
+from sklearn.linear_model import LogisticRegression
+from sklearn.svm import SVC
+from sklearn.datasets import make_blobs
+from sklearn.cluster import SpectralClustering, KMeans, MiniBatchKMeans
+from skactiveml.pool import Clue
+from skactiveml.classifier import ParzenWindowClassifier, SklearnClassifier
+from skactiveml.utils import MISSING_LABEL
+from skactiveml.tests.template_query_strategy import (
+    TemplateSingleAnnotatorPoolQueryStrategy,
+)
+from skactiveml.tests.utils import (
+    ParzenWindowClassifierEmbedding,
+    ParzenWindowClassifierTuple,
+)
+
+
+class TestClue(TemplateSingleAnnotatorPoolQueryStrategy, unittest.TestCase):
+    def setUp(self):
+        X = np.linspace(0, 1, 20).reshape(10, 2)
+        y = np.hstack([[0, 1], np.full(8, MISSING_LABEL)])
+        self.classes = [0, 1]
+        self.query_default_params_clf = {
+            "X": X,
+            "y": y,
+            "clf": ParzenWindowClassifier(
+                classes=self.classes, random_state=42
+            ),
+        }
+        self.query_default_params_clf_embedding = {
+            "X": X,
+            "y": y,
+            "clf": ParzenWindowClassifierEmbedding(
+                classes=self.classes,
+            ),
+        }
+        cluster_dict = {"random_state": 0, "n_init": 1}
+        super().setUp(
+            qs_class=Clue,
+            init_default_params={"cluster_algo_dict": cluster_dict},
+            query_default_params_clf=self.query_default_params_clf,
+        )
+
+    def test_init_param_cluster_algo(self, test_cases=None):
+        test_cases = [] if test_cases is None else test_cases
+        test_cases += [
+            (1, TypeError),
+            ("string", TypeError),
+            (None, TypeError),
+            (SpectralClustering, AttributeError),
+            (Clue, TypeError),
+            (MiniBatchKMeans, None),
+            (KMeans, None),
+        ]
+        self._test_param(
+            "init",
+            "cluster_algo",
+            test_cases,
+            replace_init_params={"random_state": 0},
+        )
+
+    def test_init_param_cluster_algo_dict(self, test_cases=None):
+        test_cases = [] if test_cases is None else test_cases
+        test_cases += [
+            (1, TypeError),
+            ("string", TypeError),
+            (None, None),
+            ({}, None),
+            ({"n_init": "auto", "random_state": 0}, None),
+        ]
+        self._test_param("init", "cluster_algo_dict", test_cases)
+
+    def test_init_param_n_cluster_param_name(self, test_cases=None):
+        test_cases = [] if test_cases is None else test_cases
+        test_cases += [
+            (1, TypeError),
+            ("string", TypeError),
+            (None, TypeError),
+            ("n_clusters", None),
+        ]
+        self._test_param("init", "n_cluster_param_name", test_cases)
+
+    def test_init_param_method(self, test_cases=None):
+        test_cases = [] if test_cases is None else test_cases
+        test_cases += [
+            (1, ValueError),
+            ("string", ValueError),
+            ("entropy", None),
+            ("margin_sampling", None),
+            ("least_confident", None),
+        ]
+        self._test_param("init", "method", test_cases)
+
+    def test_init_param_clf_embedding_flag_name(self, test_cases=None):
+        test_cases = [] if test_cases is None else test_cases
+        test_cases += [
+            (1, TypeError),
+            (None, None),
+            (False, TypeError),
+            (True, TypeError),
+            ("return_embeddings", None),
+        ]
+        self._test_param(
+            "init",
+            "clf_embedding_flag_name",
+            replace_query_params=self.query_default_params_clf_embedding,
+            test_cases=test_cases,
+        )
+
+    def test_query_param_clf(self, test_cases=None):
+        test_cases = [] if test_cases is None else test_cases
+        test_cases += [
+            (SVC(), TypeError),
+            (SklearnClassifier(SVC()), AttributeError),
+            (SklearnClassifier(SVC(probability=True)), None),
+            (
+                SklearnClassifier(LogisticRegression(), classes=self.classes),
+                None,
+            ),
+        ]
+        super().test_query_param_clf(test_cases=test_cases)
+
+    def test_query(self):
+        X, y_true = make_blobs(n_samples=100, centers=4, random_state=0)
+        y_true = y_true.astype(float)
+        clf_orig = ParzenWindowClassifier(classes=np.unique(y_true))
+        clf_emb = ParzenWindowClassifierEmbedding(classes=np.unique(y_true))
+        clf_tuple = ParzenWindowClassifierTuple(classes=np.unique(y_true))
+        for candidates in [None, np.arange(10, 30)]:
+            prev_clf_utilities = None
+            for clf in [clf_orig, clf_emb, clf_tuple]:
+                qs_plus = Clue(
+                    cluster_algo_dict={"init": "k-means++", "random_state": 0},
+                    random_state=42,
+                )
+                qs_random = Clue(
+                    cluster_algo_dict={"init": "random", "random_state": 0},
+                    random_state=42,
+                )
+
+                # Without labeled samples, the initial selection is the same,
+                # even for different clustering initializations.
+                y = np.full(len(y_true), MISSING_LABEL)
+                query_indices_plus, utilities_plus = qs_plus.query(
+                    X, y, clf=clf, candidates=candidates, return_utilities=True
+                )
+                query_indices_random, utilities_random = qs_random.query(
+                    X, y, clf=clf, candidates=candidates, return_utilities=True
+                )
+                np.testing.assert_array_equal(
+                    query_indices_plus, query_indices_random
+                )
+                np.testing.assert_almost_equal(
+                    utilities_plus, utilities_random
+                )
+                if candidates is not None:
+                    self.assertTrue(np.isnan(utilities_plus[0, :10]).all())
+                    self.assertTrue(np.isnan(utilities_plus[0, 30:]).all())
+                    self.assertTrue((utilities_plus[30:50] <= 0).all())
+                else:
+                    self.assertTrue((utilities_plus <= 0).all())
+
+                # All utilities are non-positive values or np.nan.
+                is_unlabeled = np.random.RandomState(0).choice(
+                    [False, True], size=(len(X),), replace=True
+                )
+                y = y_true.copy()
+                y[is_unlabeled] = np.nan
+                prev_utilities = None
+                for qs in [qs_plus, qs_random]:
+                    query_indices, utilities = qs.query(
+                        X,
+                        y,
+                        clf=clf,
+                        candidates=candidates,
+                        batch_size=2,
+                        return_utilities=True,
+                    )
+                    utilities_copy = utilities.copy()
+                    is_nan = np.isnan(utilities)
+                    utilities_copy[is_nan] = 0.0
+                    is_non_positive = utilities_copy <= 0
+                    self.assertTrue(is_non_positive.all())
+                    if prev_utilities is not None:
+                        # Check that different clustering initializations
+                        # lead to different utilities.
+                        self.assertTrue(
+                            np.nansum(utilities) != np.nansum(prev_utilities)
+                        )
+                    else:
+                        # Check that the utilities are consistent across
+                        # classifiers using different syntax to return
+                        # embeddings.
+                        if prev_clf_utilities is not None:
+                            np.testing.assert_array_equal(
+                                prev_clf_utilities, utilities
+                            )
+                        prev_clf_utilities = utilities
+
+                    prev_utilities = utilities