Repository: alan-turing-institute/sktime
Pull request #3240 (2022-08-10 22:05:01)

[ENH] Cleaned up user docs - direct video links
https://github.com/alan-turing-institute/sktime/pull/3211, but tutorials linking directly to videos. Based on @aiwalter's statements, this should not be controversial.

review-notebook-app[bot]: Check out this pull request on&nbsp; <a href="https://app.reviewnb.com/alan-turing-institute/sktime/pull/3240"><img align="absmiddle"  alt="ReviewNB" height="28" class="BotMessageButtonImage" src="https://raw.githubusercontent.com/ReviewNB/support/master/images/button_reviewnb.png"/></a> 

 See visual diffs & provide feedback on Jupyter Notebooks. 

---

 <i>Powered by <a href='https://www.reviewnb.com/?utm_source=gh'>ReviewNB</a></i>

Patch:
diff --git a/.gitignore b/.gitignore
index ecf50eabc..86dba102b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -80,7 +80,6 @@ instance/
 docs/_build/
 docs/source/api_reference/auto_generated/
 docs/estimator_overview_table.md
-docs/source/examples
 
 # PyBuilder
 target/
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 66d3ffbdb..2b0bb07e9 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -10,6 +10,7 @@ repos:
       - id: check-yaml
       - id: debug-statements
       - id: end-of-file-fixer
+        exclude: "^docs/source/examples/"
       - id: fix-encoding-pragma
       - id: requirements-txt-fixer
       - id: trailing-whitespace
diff --git a/docs/source/examples/01a_forecasting_sklearn.ipynb b/docs/source/examples/01a_forecasting_sklearn.ipynb
new file mode 100644
index 000000000..00d7abb85
--- /dev/null
+++ b/docs/source/examples/01a_forecasting_sklearn.ipynb
@@ -0,0 +1,1 @@
+../../../examples/01a_forecasting_sklearn.ipynb
\ No newline at end of file
diff --git a/docs/source/examples/01b_forecasting_proba.ipynb b/docs/source/examples/01b_forecasting_proba.ipynb
new file mode 100644
index 000000000..f3d49782c
--- /dev/null
+++ b/docs/source/examples/01b_forecasting_proba.ipynb
@@ -0,0 +1,1 @@
+../../../examples/01b_forecasting_proba.ipynb
\ No newline at end of file
diff --git a/docs/source/examples/02_classification.ipynb b/docs/source/examples/02_classification.ipynb
new file mode 100644
index 000000000..cc6de2b26
--- /dev/null
+++ b/docs/source/examples/02_classification.ipynb
@@ -0,0 +1,1 @@
+../../../examples/02_classification.ipynb
diff --git a/docs/source/examples/AA_datatypes_and_datasets.ipynb b/docs/source/examples/AA_datatypes_and_datasets.ipynb
new file mode 100644
index 000000000..ceb6a0ac7
--- /dev/null
+++ b/docs/source/examples/AA_datatypes_and_datasets.ipynb
@@ -0,0 +1,1 @@
+../../../examples/AA_datatypes_and_datasets.ipynb
\ No newline at end of file
diff --git a/docs/source/examples/feature_extraction_with_tsfresh.ipynb b/docs/source/examples/feature_extraction_with_tsfresh.ipynb
deleted file mode 120000
index 15159e47c..000000000
--- a/docs/source/examples/feature_extraction_with_tsfresh.ipynb
+++ /dev/null
@@ -1,1 +0,0 @@
-../../../examples/feature_extraction_with_tsfresh.ipynb
\ No newline at end of file
diff --git a/docs/source/examples/loading_data.ipynb b/docs/source/examples/loading_data.ipynb
deleted file mode 120000
index 03568cedc..000000000
--- a/docs/source/examples/loading_data.ipynb
+++ /dev/null
@@ -1,1 +0,0 @@
-../../../examples/loading_data.ipynb
\ No newline at end of file
diff --git a/docs/source/examples/plateau_finder.ipynb b/docs/source/examples/plateau_finder.ipynb
deleted file mode 120000
index c66de0b6c..000000000
--- a/docs/source/examples/plateau_finder.ipynb
+++ /dev/null
@@ -1,1 +0,0 @@
-../../../examples/plateau_finder.ipynb
\ No newline at end of file
diff --git a/docs/source/examples/rocket.ipynb b/docs/source/examples/rocket.ipynb
deleted file mode 120000
index 1ad8cc230..000000000
--- a/docs/source/examples/rocket.ipynb
+++ /dev/null
@@ -1,1 +0,0 @@
-../../../examples/rocket.ipynb
\ No newline at end of file
diff --git a/docs/source/examples/shapelet_transform.ipynb b/docs/source/examples/shapelet_transform.ipynb
deleted file mode 120000
index cded2255e..000000000
--- a/docs/source/examples/shapelet_transform.ipynb
+++ /dev/null
@@ -1,1 +0,0 @@
-../../../examples/shapelet_transform.ipynb
\ No newline at end of file
diff --git a/docs/source/tutorials.rst b/docs/source/tutorials.rst
index e0cf7d94e..7440d0384 100644
--- a/docs/source/tutorials.rst
+++ b/docs/source/tutorials.rst
@@ -3,18 +3,62 @@
 Tutorials
 =========
 
-We have a number of tutorial notebooks. If you like to run them interactively, you can launch them on Binder without
-having to install anything.
+Below are introductory tutorials for ``sktime``.
 
-We assume basic familiarity with `scikit-learn`_. If you haven’t worked with scikit-learn before, check out their
-`getting-started guide`_.
+Each tutorial is located in its own repository, which contains notebooks and links to a youtube video walkthrough.
 
-The notebook files can be found `here <https://github.com/alan-turing-institute/sktime/blob/main/examples>`_.
+.. note::
 
-.. _scikit-learn: https://scikit-learn.org/stable/
-.. _getting-started guide: https://scikit-learn.org/stable/getting_started.html
+    The tutorials and videos are based on snapshots of ``sktime`` at earlier versions.
+    They explain stable functionality which rarely changes, and most of the content is continuously tested in ``sktime`` CI.
+    Nonetheless, there may be outdated parts - in such a case, we would be very thankful for bug reports.
 
-.. nbgallery::
-    :glob:
+For user guides specific to learning tasks, see our :ref:`user_guide` page.
+The user guide notebooks are always functional with the most recent stable version.
 
-    examples/*
+.. panels::
+    :card: + intro-card text-center
+
+    ---
+
+    Introductory tutorial
+    ^^^^^^^^^^^^^^^^^^^^^
+
+    A general introduction to ``sktime`` - for first time users.
+
+    +++
+
+    .. link-button:: https://www.youtube.com/watch?v=ODspi8-uWgo
+            :type: url
+            :text: Intro to sktime
+            :classes: btn-block btn-secondary stretched-link
+
+    ---
+
+    Implementing your own estimator
+    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+    How to implement your own ``sktime`` compatible estimator and check its validity.
+
+    +++
+
+    .. link-button:: https://youtu.be/S_3ewcvs_pg
+            :type: url
+            :text: Implementing estimators
+            :classes: btn-block btn-secondary stretched-link
+
+    ---
+
+    Advanced forecasting tutorial
+    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+    Probabilistic and hierarchical forecasting with ``sktime``
+
+    +++
+
+    .. link-button:: https://www.youtube.com/watch?v=4Rf9euAhjNc
+            :type: url
+            :text: Advanced forecasting
+            :classes: btn-block btn-secondary stretched-link
+
+    ---
diff --git a/docs/source/user_guide.rst b/docs/source/user_guide.rst
index 3fa729c61..de0e370c7 100644
--- a/docs/source/user_guide.rst
+++ b/docs/source/user_guide.rst
@@ -7,29 +7,23 @@ User Guide
 
 Welcome to sktime's user guide!
 
-The purpose of the user guide is to provide a scientific manual for the
-methods implemented in sktime. The :ref:`api_reference`, by contrast, is
-meant to provide a technical manual.
+The user guide consists of introductory notebooks, ordered by learning task.
 
+For guided tutorials with videos, see our :ref:`tutorials` page.
 
-.. note::
+To run the user guide notebooks interactively, you can
+`launch them on binder <https://mybinder.org/v2/gh/alan-turing-institute/sktime/main?filepath=examples>`_
+without having to install anything.
 
-    The user guide is under development. We have created a basic
-    structure and are looking for contributions to develop the user guide
-    further. For more details, please go to issue `#361 <https://github
-    .com/alan-turing-institute/sktime/issues/361>`_ on GitHub.
+We assume basic familiarity with `scikit-learn`_. If you haven’t worked with scikit-learn before, check out their
+`getting-started guide`_.
 
+The notebook files can be found `here <https://github.com/alan-turing-institute/sktime/blob/main/examples>`_.
 
-.. toctree::
-   :maxdepth: 1
+.. _scikit-learn: https://scikit-learn.org/stable/
+.. _getting-started guide: https://scikit-learn.org/stable/getting_started.html
 
-   user_guide/introduction
-   user_guide/learning_tasks
-   user_guide/transformations
-   user_guide/classification
-   user_guide/regression
-   user_guide/forecasting
-   user_guide/clustering
-   user_guide/annotation
-   user_guide/performance_metrics
-   user_guide/resources
+.. nbgallery::
+    :glob:
+
+    examples/*
diff --git a/docs/source/user_guide/annotation.rst b/docs/source/user_guide/annotation.rst
deleted file mode 100644
index 2bb1d03f6..000000000
--- a/docs/source/user_guide/annotation.rst
+++ /dev/null
@@ -1,11 +0,0 @@
-.. _user_guide_annotation:
-
-Annotation
-==========
-
-.. note::
-
-    The user guide is under development. We have created a basic
-    structure and are looking for contributions to develop the user guide
-    further. For more details, please go to issue `#361 <https://github
-    .com/alan-turing-institute/sktime/issues/361>`_ on GitHub.
diff --git a/docs/source/user_guide/classification.rst b/docs/source/user_guide/classification.rst
deleted file mode 100644
index 3fb80a0b5..000000000
--- a/docs/source/user_guide/classification.rst
+++ /dev/null
@@ -1,11 +0,0 @@
-.. _user_guide_classification:
-
-Time Series Classification
-==========================
-
-.. note::
-
-    The user guide is under development. We have created a basic
-    structure and are looking for contributions to develop the user guide
-    further. For more details, please go to issue `#361 <https://github
-    .com/alan-turing-institute/sktime/issues/361>`_ on GitHub.
diff --git a/docs/source/user_guide/clustering.rst b/docs/source/user_guide/clustering.rst
deleted file mode 100644
index 14a73bf71..000000000
--- a/docs/source/user_guide/clustering.rst
+++ /dev/null
@@ -1,11 +0,0 @@
-.. _user_guide_clustering:
-
-Clustering
-==========
-
-.. note::
-
-    The user guide is under development. We have created a basic
-    structure and are looking for contributions to develop the user guide
-    further. For more details, please go to issue `#361 <https://github
-    .com/alan-turing-institute/sktime/issues/361>`_ on GitHub.
diff --git a/docs/source/user_guide/forecasting.rst b/docs/source/user_guide/forecasting.rst
deleted file mode 100644
index 6a434dd6d..000000000
--- a/docs/source/user_guide/forecasting.rst
+++ /dev/null
@@ -1,57 +0,0 @@
-.. _user_guide_forecasting:
-
-Forecasting
-===========
-
-.. note::
-
-    The user guide is under development. We have created a basic
-    structure and are looking for contributions to develop the user guide
-    further. For more details, please go to issue `#361 <https://github
-    .com/alan-turing-institute/sktime/issues/361>`_ on GitHub.
-
-Introduction
-------------
-Forecasating is making forward temporal predictions based on past data. The simplest case is the univariate case.
-
-Inputs.
-
-* Univariate time series, :math:`y = y(0), y(1), y(2), ..., y(N)`
-* Forecasting horizon, :math:`fh = N+1, N+2, ..., N+h`
-
-Output.
-
-* Predictions of :math:`y` at the times in :math:`fh`, :math:`\hat{y} = \hat{y}(N+1), \hat{y}(N+2), ..., \hat{y}(N+h)`
-
-Examples.
-
-* Forecasting the global population
-* Forecasting the price of a stock
-* Forecasting the daily maximum temperature in a given location
-
-Example
--------
-Predicting flights. See the `Tutorial on Forecasting <https://github.com/alan-turing-institute/sktime/blob/main/examples/01_forecasting.ipynb>`_.
-
-Algorithms included in sktime
------------------------------
-See the `API Reference <https://www.sktime.org/en/latest/api_reference.html#sktime-forecasting-time-series-forecasting>`_.
-
-Reductions included in sktime
------------------------------
-
-
-Variations in generative setting
---------------------------------
-
-
-Evaluation and model selection
-------------------------------
-
-
-Algorithms not included in sktime
----------------------------------
-
-
-Further reading
----------------
diff --git a/docs/source/user_guide/learning_tasks.rst b/docs/source/user_guide/learning_tasks.rst
deleted file mode 100644
index f33962837..000000000
--- a/docs/source/user_guide/learning_tasks.rst
+++ /dev/null
@@ -1,11 +0,0 @@
-.. _user_guide_learning_tasks:
-
-Overview of Learning Tasks
-==========================
-
-.. note::
-
-    The user guide is under development. We have created a basic
-    structure and are looking for contributions to develop the user guide
-    further. For more details, please go to issue `#361 <https://github
-    .com/alan-turing-institute/sktime/issues/361>`_ on GitHub.
diff --git a/docs/source/user_guide/performance_metrics.rst b/docs/source/user_guide/performance_metrics.rst
deleted file mode 100644
index 8cb5de28c..000000000
--- a/docs/source/user_guide/performance_metrics.rst
+++ /dev/null
@@ -1,15 +0,0 @@
-.. _user_guide_performance_metrics:
-
-Performance Metrics
-===================
-
-.. note::
-
-    The user guide is under development. We have created a basic
-    structure and are looking for contributions to develop the user guide
-    further. For more details, please go to issue `#361 <https://github
-    .com/alan-turing-institute/sktime/issues/361>`_ on GitHub.
-
-
-Forecasting
------------
diff --git a/docs/source/user_guide/regression.rst b/docs/source/user_guide/regression.rst
deleted file mode 100644
index 73caead8d..000000000
--- a/docs/source/user_guide/regression.rst
+++ /dev/null
@@ -1,11 +0,0 @@
-.. _user_guide_regression:
-
-Time Series Regression
-======================
-
-.. note::
-
-    The user guide is under development. We have created a basic
-    structure and are looking for contributions to develop the user guide
-    further. For more details, please go to issue `#361 <https://github
-    .com/alan-turing-institute/sktime/issues/361>`_ on GitHub.
diff --git a/docs/source/user_guide/resources.rst b/docs/source/user_guide/resources.rst
deleted file mode 100644
index 5eeffdcb8..000000000
--- a/docs/source/user_guide/resources.rst
+++ /dev/null
@@ -1,15 +0,0 @@
-.. _user_guide_resources:
-
-Extra Resources
-===============
-
-.. note::
-
-    The user guide is under development. We have created a basic
-    structure and are looking for contributions to develop the user guide
-    further. For more details, please go to issue `#361 <https://github
-    .com/alan-turing-institute/sktime/issues/361>`_ on GitHub.
-
-
-Blog posts
-----------
diff --git a/docs/source/user_guide/transformations.rst b/docs/source/user_guide/transformations.rst
deleted file mode 100644
index e5ba148b9..000000000
--- a/docs/source/user_guide/transformations.rst
+++ /dev/null
@@ -1,29 +0,0 @@
-.. _user_guide_transformations:
-
-Time Series Transformations
-===========================
-
-.. note::
-
-    The user guide is under development. We have created a basic
-    structure and are looking for contributions to develop the user guide
-    further. For more details, please go to issue `#361 <https://github
-    .com/alan-turing-institute/sktime/issues/361>`_ on GitHub.
-
-
-.. contents:: :local:
-
-Tabular-to-tabular transformations
-----------------------------------
-
-Series-to-primitives transformations
-------------------------------------
-
-Series-to-series transformations
---------------------------------
-
-Panel-to-tabular transformations
---------------------------------
-
-Panel-to-panel transformations
-------------------------------
diff --git a/pyproject.toml b/pyproject.toml
index 55c864bd7..c26cb45b7 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -119,3 +119,8 @@ file = "LICENSE"
 [build-system]
 requires = ["setuptools", "wheel", "toml", "build"]
 build-backend = "setuptools.build_meta"
+
+[tool.nbqa.exclude]
+black = "^docs/source/examples/"
+flake8 = "^docs/source/examples/"
+isort = "^docs/source/examples/"
diff --git a/sktime/datatypes/_panel/_convert.py b/sktime/datatypes/_panel/_convert.py
index a784829f6..597b175bb 100644
--- a/sktime/datatypes/_panel/_convert.py
+++ b/sktime/datatypes/_panel/_convert.py
@@ -847,9 +847,7 @@ def from_nested_to_3d_numpy(X):
     # Then the multi-indexed DataFrame can be converted to 3d NumPy array
     else:
         X_mi = from_nested_to_multi_index(X)
-        X_3d = from_multi_index_to_3d_numpy(
-            X_mi, instance_index="instance", time_index="timepoints"
-        )
+        X_3d = from_multi_index_to_3d_numpy(X_mi)
 
     return X_3d
 
diff --git a/sktime/performance_metrics/forecasting/_classes.py b/sktime/performance_metrics/forecasting/_classes.py
index b50effdbd..436bb021b 100644
--- a/sktime/performance_metrics/forecasting/_classes.py
+++ b/sktime/performance_metrics/forecasting/_classes.py
@@ -8,6 +8,7 @@ Classes named as ``*Error`` or ``*Loss`` return a value to minimize:
 the lower the better.
 """
 from copy import deepcopy
+from inspect import getfullargspec, isfunction, signature
 from warnings import warn
 
 import numpy as np
@@ -121,7 +122,11 @@ class BaseForecastingErrorMetric(BaseMetric):
     def __init__(self, multioutput="uniform_average", multilevel="uniform_average"):
         self.multioutput = multioutput
         self.multilevel = multilevel
-        self.name = type(self).__name__
+
+        if not hasattr(self, "name"):
+            self.name = type(self).__name__
+
+        super(BaseForecastingErrorMetric, self).__init__()
 
     def __call__(self, y_true, y_pred, **kwargs):
         """Calculate metric value using underlying metric function.
@@ -509,10 +514,24 @@ class BaseForecastingErrorMetricFunc(BaseForecastingErrorMetric):
         """Evaluate the desired metric on given inputs."""
         # this dict should contain all parameters
         params = self.get_params()
+
         # adding kwargs to the metric, should not overwrite params (but does if clashes)
         params.update(kwargs)
+
+        # calls class variable func, if available, or dynamic (object) variable
         # we need to call type since we store func as a class attribute
-        res = type(self).func(y_true=y_true, y_pred=y_pred, **params)
+        if hasattr(type(self), "func") and isfunction(type(self).func):
+            func = type(self).func
+        else:
+            func = self.func
+
+        # if func does not catch kwargs, subset to args of func
+        if getfullargspec(func).varkw is None:
+            func_params = signature(func).parameters.keys()
+            func_params = set(func_params).difference(["y_true", "y_pred"])
+            params = {key: params[key] for key in func_params}
+
+        res = func(y_true=y_true, y_pred=y_pred, **params)
         return res
 
 
@@ -531,10 +550,43 @@ class _DynamicForecastingErrorMetric(BaseForecastingErrorMetricFunc):
         self.multilevel = multilevel
         self.func = func
         self.name = name
-        super().__init__()
+        self.lower_is_better = lower_is_better
+
+        super(_DynamicForecastingErrorMetric, self).__init__(
+            multioutput=multioutput, multilevel=multilevel
+        )
+
+        self.set_tags(**{"lower_is_better": lower_is_better})
+
+    @classmethod
+    def get_test_params(cls, parameter_set="default"):
+        """Return testing parameter settings for the estimator.
+
+        Parameters
+        ----------
+        parameter_set : str, default="default"
+            Name of the set of test parameters to return, for use in tests. If no
+            special parameters are defined for a value, will return `"default"` set.
+
+        Returns
+        -------
+        params : dict or list of dict, default = {}
+            Parameters to create testing instances of the class
+            Each dict are parameters to construct an "interesting" test instance, i.e.,
+            `MyClass(**params)` or `MyClass(**params[i])` creates a valid test instance.
+            `create_test_instance` uses the first (or only) dictionary in `params`
+        """
+
+        def custom_mape(y_true, y_pred) -> float:
+
+            eps = np.finfo(np.float64).eps
+
+            result = np.mean(np.abs(y_true - y_pred) / np.maximum(np.abs(y_true), eps))
+
+            return float(result)
 
-        if not lower_is_better:
-            self.set_tags(**{"lower_is_better": False})
+        params = {"func": custom_mape, "name": "custom_mape", "lower_is_better": False}
+        return params
 
 
 class _ScaledMetricTags:
diff --git a/sktime/performance_metrics/forecasting/_functions.py b/sktime/performance_metrics/forecasting/_functions.py
index 804d4c89a..252b648c9 100644
--- a/sktime/performance_metrics/forecasting/_functions.py
+++ b/sktime/performance_metrics/forecasting/_functions.py
@@ -54,7 +54,7 @@ def _get_kwarg(kwarg, metric_name="Metric", **kwargs):
     if kwarg_ is None:
         msg = "".join(
             [
-                f"{metric_name} requires `{kwarg}`.",
+                f"{metric_name} requires `{kwarg}`. ",
                 f"Pass `{kwarg}` as a keyword argument when calling the metric.",
             ]
         )
diff --git a/sktime/utils/estimator_checks.py b/sktime/utils/estimator_checks.py
index c290c3410..77c07898b 100644
--- a/sktime/utils/estimator_checks.py
+++ b/sktime/utils/estimator_checks.py
@@ -4,6 +4,8 @@
 __author__ = ["fkiraly"]
 __all__ = ["check_estimator"]
 
+from inspect import isclass
+
 
 def check_estimator(
     estimator,
@@ -96,7 +98,14 @@ def check_estimator(
         fixtures_to_exclude=fixtures_to_exclude,
     )
 
-    if isinstance(estimator, BaseEstimator) or issubclass(estimator, BaseEstimator):
+    def is_estimator(obj):
+        """Return whether obj is an estimator class or estimator object."""
+        if isclass(obj):
+            return issubclass(obj, BaseEstimator)
+        else:
+            return isinstance(obj, BaseEstimator)
+
+    if is_estimator(estimator):
         results_estimator = TestAllEstimators().run_tests(
             estimator=estimator,
             return_exceptions=return_exceptions,


Test patch:
diff --git a/sktime/performance_metrics/tests/test_metrics_classes.py b/sktime/performance_metrics/tests/test_metrics_classes.py
index 82c2b7a12..a4c436db0 100644
--- a/sktime/performance_metrics/tests/test_metrics_classes.py
+++ b/sktime/performance_metrics/tests/test_metrics_classes.py
@@ -6,7 +6,12 @@ import numpy as np
 import pandas as pd
 import pytest
 
-from sktime.performance_metrics.forecasting import MeanSquaredError, _classes
+from sktime.datasets import load_airline
+from sktime.performance_metrics.forecasting import (
+    MeanSquaredError,
+    _classes,
+    make_forecasting_scorer,
+)
 from sktime.utils._testing.hierarchical import _make_hierarchical
 from sktime.utils._testing.series import _make_series
 
@@ -96,3 +101,32 @@ def test_metric_hierarchical(multioutput, multilevel, n_columns):
             assert isinstance(res, np.ndarray)
             assert res.ndim == 1
             assert len(res) == len(y_true.columns)
+
+
+@pytest.mark.parametrize("greater_is_better", [True, False])
+def test_custom_metric(greater_is_better):
+    """Test custom metric constructor, integration _DynamicForecastingErrorMetric."""
+    from sktime.utils.estimator_checks import check_estimator
+
+    y = load_airline()
+
+    def custom_mape(y_true, y_pred) -> float:
+
+        eps = np.finfo(np.float64).eps
+
+        result = np.mean(np.abs(y_true - y_pred) / np.maximum(np.abs(y_true), eps))
+
+        return float(result)
+
+    fc_scorer = make_forecasting_scorer(
+        func=custom_mape,
+        name="custom_mape",
+        greater_is_better=False,
+    )
+
+    assert isinstance(fc_scorer, _classes._DynamicForecastingErrorMetric)
+
+    score = fc_scorer(y, y)
+    assert isinstance(score, float)
+
+    check_estimator(fc_scorer, return_exceptions=False)