Repository: aeon-toolkit/aeon
Pull request #954 (2023-12-01 21:31:09)

[ENH] New plot for significance relationships
Plotting tool under development to allow for the case where cliques can be deceiving. More to follow, PR so I dont forget where it is!

Edited David:
- Include new plot, that may be improved if any new idea is proposed. 
- Move statistical methods to performance_metrics module. It is used by CDDs and new significance plots. 
- Include more tests for stats (in performance_metrics module) and for CDDs and Significance plots (in visualisation module). 

Changes for a future PR: 
There is a common block of code for CDDs and Significance plots, the one that extracts cliques and/or p_values. 
1) The possibility of returning the p_values: as we give the option of using Nemenyi, which doesn't return p_values, and Wilcoxon, which does. This can be solved by raising an error (already in the CDD function).
2) I've thought in creating a new function named "get_cliques(test, ...), but is not easy to be designed as the way for building cliques differ from CDDs and Significance plots. CDD stops forming a clique when a gap is found, i.e. a method with significant differences, whereas Significance plots does not. For this I have to give a second thought. 



aeon-actions-bot[bot]: 
## Thank you for contributing to `aeon`!

I have added the following labels to this PR based on the title: **[ $\\color{#FEF1BE}{\\textsf{enhancement}}$ ]**.
I have added the following labels to this PR based on the changes made: **[ $\\color{#264F59}{\\textsf{benchmarking}}$ ]**. Feel free to change these if they do not properly represent the PR.

The [Checks](https://github.com/aeon-toolkit/aeon/pull/954/checks) tab will show the status of our automated tests. You can click on individual test runs in the tab or "Details" in the panel below to see more information if there is a failure.

If our `pre-commit` code quality check fails, any trivial fixes will automatically be pushed to your PR unless it is a draft.

Don't hesitate to ask questions on the `aeon` [Slack](https://join.slack.com/t/aeon-toolkit/shared_invite/zt-22vwvut29-HDpCu~7VBUozyfL_8j3dLA) channel if you have any!
    
TonyBagnall: will wait for #1002 for this
MatthewMiddlehurst: Good stuff moving this forward, think its good to try out at least. May conflict with #1092 but probably nothing major.
dguijo: Some examples of the new plot:

For the bake off redux. In this case, there is no difference as no gaps are present between classifiers. 
![CD_redux](https://github.com/aeon-toolkit/aeon/assets/47889499/28f4b01f-6b20-407c-9678-977bbb1a042f)
![significance_redux](https://github.com/aeon-toolkit/aeon/assets/47889499/132591a7-7deb-40fc-a917-aba55bca9827)

Including more classifiers shows the potential of this new plot:
![CD_redux_complete](https://github.com/aeon-toolkit/aeon/assets/47889499/76817ab8-4d94-41dc-bdd1-3e894212551f)
![significance_redux_complete](https://github.com/aeon-toolkit/aeon/assets/47889499/948931ac-4c9a-48ef-94e0-1af88d0e968c)

More examples with TSER this time:
![CD_TSER](https://github.com/aeon-toolkit/aeon/assets/47889499/df716737-0a6e-49c8-8b31-be006d906712)
![significance_TSER](https://github.com/aeon-toolkit/aeon/assets/47889499/48ed18ac-796d-4ac4-b2a3-d53341564fdd)

A more comprehensive comparison of TSER approaches:
![CD_TSER_complete](https://github.com/aeon-toolkit/aeon/assets/47889499/20468d6a-de29-4916-b835-6fc65ce58e92)
![significance_TSER_complete](https://github.com/aeon-toolkit/aeon/assets/47889499/97d78d00-ab13-47ec-8281-6a1fa8a87255)



Patch:
diff --git a/aeon/performance_metrics/stats.py b/aeon/performance_metrics/stats.py
new file mode 100644
index 00000000..672a4ce9
--- /dev/null
+++ b/aeon/performance_metrics/stats.py
@@ -0,0 +1,129 @@
+"""Functions to compute stats and get p-values."""
+
+__author__ = ["SveaMeyer13", "dguijo", "TonyBagnall"]
+
+__all__ = ["check_friedman", "nemenyi_test", "wilcoxon_test"]
+
+import warnings
+
+import numpy as np
+from scipy.stats import distributions, find_repeats, wilcoxon
+
+from aeon.benchmarking.utils import get_qalpha
+
+
+def check_friedman(ranks):
+    """
+    Check whether Friedman test is significant.
+
+    Parameters
+    ----------
+    ranks : np.array
+      Rank of estimators on datasets, shape (n_estimators, n_datasets).
+
+    Returns
+    -------
+    float
+      p-value of the test.
+    """
+    n_datasets, n_estimators = ranks.shape
+
+    if n_estimators < 3:
+        raise ValueError(
+            "At least 3 sets of measurements must be given for Friedmann test, "
+            f"got {n_estimators}."
+        )
+
+    # calculate c to correct chisq for ties:
+    ties = 0
+    for i in range(n_datasets):
+        replist, repnum = find_repeats(ranks[i])
+        for t in repnum:
+            ties += t * (t * t - 1)
+    c = 1 - ties / (n_estimators * (n_estimators * n_estimators - 1) * n_datasets)
+
+    ssbn = np.sum(ranks.sum(axis=0) ** 2)
+    chisq = (
+        12.0 / (n_estimators * n_datasets * (n_estimators + 1)) * ssbn
+        - 3 * n_datasets * (n_estimators + 1)
+    ) / c
+    p_value = distributions.chi2.sf(chisq, n_estimators - 1)
+    return p_value
+
+
+def nemenyi_test(ordered_avg_ranks, n_datasets, alpha):
+    """
+    Find cliques using post hoc Nemenyi test.
+
+    Parameters
+    ----------
+    ordered_avg_ranks : np.array
+        Average ranks of estimators.
+    n_datasets : int
+        Mumber of datasets.
+    alpha : float
+        alpha level for Nemenyi test.
+
+    Returns
+    -------
+    list of lists
+        List of cliques. A clique is a group of estimators within which there is no
+        significant difference.
+    """
+    n_estimators = len(ordered_avg_ranks)
+    qalpha = get_qalpha(alpha)
+    # calculate critical difference with Nemenyi
+    cd = qalpha[n_estimators] * np.sqrt(
+        n_estimators * (n_estimators + 1) / (6 * n_datasets)
+    )
+    # compute statistically similar cliques
+    cliques = np.tile(ordered_avg_ranks, (n_estimators, 1)) - np.tile(
+        np.vstack(ordered_avg_ranks.T), (1, n_estimators)
+    )
+    cliques[cliques < 0] = np.inf
+    cliques = cliques < cd
+
+    return cliques
+
+
+def wilcoxon_test(results, labels, lower_better=False):
+    """
+    Perform Wilcoxon test.
+
+    Parameters
+    ----------
+    results: np.array
+      results of estimators on datasets
+
+    lower_better : bool, default = False
+        Indicates whether smaller is better for the results in scores. For example,
+        if errors are passed instead of accuracies, set ``lower_better`` to ``True``.
+
+    Returns
+    -------
+    np.array
+        p-values of Wilcoxon sign rank test.
+    """
+    n_estimators = results.shape[1]
+
+    p_values = np.eye(n_estimators)
+
+    for i in range(n_estimators - 1):
+        for j in range(i + 1, n_estimators):
+            # if the difference is zero, the p-value is 1
+            if np.all(results[:, i] == results[:, j]):
+                p_values[i, j] = 1
+                # raise warning
+                warnings.warn(
+                    f"Estimators {labels[i]} and {labels[j]} have the same performance"
+                    "on all datasets. This may cause problems when forming cliques.",
+                    stacklevel=2,
+                )
+            else:
+                p_values[i, j] = wilcoxon(
+                    results[:, i],
+                    results[:, j],
+                    zero_method="wilcox",
+                    alternative="less" if lower_better else "greater",
+                )[1]
+    return p_values
diff --git a/aeon/visualisation/__init__.py b/aeon/visualisation/__init__.py
index af946b45..f710f5e9 100644
--- a/aeon/visualisation/__init__.py
+++ b/aeon/visualisation/__init__.py
@@ -12,6 +12,7 @@ __all__ = [
     "plot_series_with_change_points",
     # Results plotting
     "plot_critical_difference",
+    "plot_significance",
     "plot_boxplot_median",
     "plot_scatter_predictions",
     "plot_pairwise_scatter",
@@ -38,6 +39,7 @@ from aeon.visualisation.results._scatter import (
     plot_scatter_predictions,
     plot_score_vs_time_scatter,
 )
+from aeon.visualisation.results._significance import plot_significance
 from aeon.visualisation.series._collections import (
     plot_collection_by_class,
     plot_series_collection,
diff --git a/aeon/visualisation/results/_critical_difference.py b/aeon/visualisation/results/_critical_difference.py
index d7f3c865..0857c630 100644
--- a/aeon/visualisation/results/_critical_difference.py
+++ b/aeon/visualisation/results/_critical_difference.py
@@ -7,12 +7,11 @@ __all__ = [
 ]
 
 import math
-import warnings
 
 import numpy as np
-from scipy.stats import distributions, find_repeats, rankdata, wilcoxon
+from scipy.stats import rankdata
 
-from aeon.benchmarking.utils import get_qalpha
+from aeon.performance_metrics.stats import check_friedman, nemenyi_test, wilcoxon_test
 from aeon.utils.validation._dependencies import _check_soft_dependencies
 
 
@@ -199,11 +198,12 @@ def plot_critical_difference(
     else:
         colours = ["#000000"] * len(ordered_labels)
     # Step 3 : check whether Friedman test is significant
-    p_value_friedman = _check_friedman(ranks)
+    p_value_friedman = check_friedman(ranks)
     # Step 4: If Friedman test is significant find cliques
     if p_value_friedman < alpha:
         if test == "nemenyi":
-            cliques = _nemenyi_test(ordered_avg_ranks, n_datasets, alpha)
+            cliques = nemenyi_test(ordered_avg_ranks, n_datasets, alpha)
+            cliques = _build_cliques(cliques)
         elif test == "wilcoxon":
             if correction == "bonferroni":
                 adjusted_alpha = alpha / (n_estimators * (n_estimators - 1) / 2)
@@ -213,7 +213,7 @@ def plot_critical_difference(
                 adjusted_alpha = alpha
             else:
                 raise ValueError("correction available are None, Bonferroni and Holm.")
-            p_values = _wilcoxon_test(ordered_scores, ordered_labels, lower_better)
+            p_values = wilcoxon_test(ordered_scores, ordered_labels, lower_better)
             cliques = _build_cliques(p_values > adjusted_alpha)
         else:
             raise ValueError("tests available are only nemenyi and wilcoxon.")
@@ -455,125 +455,6 @@ def plot_critical_difference(
         return fig, ax
 
 
-def _check_friedman(ranks):
-    """
-    Check whether Friedman test is significant.
-
-    Parameters
-    ----------
-    ranks : np.array
-      Rank of estimators on datasets, shape (n_estimators, n_datasets).
-
-    Returns
-    -------
-    float
-      p-value of the test.
-    """
-    n_datasets, n_estimators = ranks.shape
-
-    if n_estimators < 3:
-        raise ValueError(
-            "At least 3 sets of measurements must be given for Friedmann test, "
-            f"got {n_estimators}."
-        )
-
-    # calculate c to correct chisq for ties:
-    ties = 0
-    for i in range(n_datasets):
-        replist, repnum = find_repeats(ranks[i])
-        for t in repnum:
-            ties += t * (t * t - 1)
-    c = 1 - ties / (n_estimators * (n_estimators * n_estimators - 1) * n_datasets)
-
-    ssbn = np.sum(ranks.sum(axis=0) ** 2)
-    chisq = (
-        12.0 / (n_estimators * n_datasets * (n_estimators + 1)) * ssbn
-        - 3 * n_datasets * (n_estimators + 1)
-    ) / c
-    p_value = distributions.chi2.sf(chisq, n_estimators - 1)
-    return p_value
-
-
-def _nemenyi_test(ordered_avg_ranks, n_datasets, alpha):
-    """
-    Find cliques using post hoc Nemenyi test.
-
-    Parameters
-    ----------
-    ordered_avg_ranks : np.array
-        Average ranks of estimators.
-    n_datasets : int
-        Mumber of datasets.
-    alpha : float
-        alpha level for Nemenyi test.
-
-    Returns
-    -------
-    list of lists
-        List of cliques. A clique is a group of estimators within which there is no
-        significant difference.
-    """
-    n_estimators = len(ordered_avg_ranks)
-    qalpha = get_qalpha(alpha)
-    # calculate critical difference with Nemenyi
-    cd = qalpha[n_estimators] * np.sqrt(
-        n_estimators * (n_estimators + 1) / (6 * n_datasets)
-    )
-    # compute statistically similar cliques
-    cliques = np.tile(ordered_avg_ranks, (n_estimators, 1)) - np.tile(
-        np.vstack(ordered_avg_ranks.T), (1, n_estimators)
-    )
-    cliques[cliques < 0] = np.inf
-    cliques = cliques < cd
-
-    cliques = _build_cliques(cliques)
-
-    return cliques
-
-
-def _wilcoxon_test(results, labels, lower_better=False):
-    """
-    Perform Wilcoxon test.
-
-    Parameters
-    ----------
-    results: np.array
-      results of estimators on datasets
-
-    lower_better : bool, default = False
-        Indicates whether smaller is better for the results in scores. For example,
-        if errors are passed instead of accuracies, set ``lower_better`` to ``True``.
-
-    Returns
-    -------
-    np.array
-        p-values of Wilcoxon sign rank test.
-    """
-    n_estimators = results.shape[1]
-
-    p_values = np.eye(n_estimators)
-
-    for i in range(n_estimators - 1):
-        for j in range(i + 1, n_estimators):
-            # if the difference is zero, the p-value is 1
-            if np.all(results[:, i] == results[:, j]):
-                p_values[i, j] = 1
-                # raise warning
-                warnings.warn(
-                    f"Estimators {labels[i]} and {labels[j]} have the same performance"
-                    "on all datasets. This may cause problems when forming cliques.",
-                    stacklevel=2,
-                )
-            else:
-                p_values[i, j] = wilcoxon(
-                    results[:, i],
-                    results[:, j],
-                    zero_method="wilcox",
-                    alternative="less" if lower_better else "greater",
-                )[1]
-    return p_values
-
-
 def _build_cliques(pairwise_matrix):
     """
     Build cliques from pairwise comparison matrix.
diff --git a/aeon/visualisation/results/_significance.py b/aeon/visualisation/results/_significance.py
new file mode 100644
index 00000000..d524fafe
--- /dev/null
+++ b/aeon/visualisation/results/_significance.py
@@ -0,0 +1,294 @@
+import warnings
+
+import numpy as np
+from scipy.stats import rankdata
+
+from aeon.performance_metrics.stats import check_friedman, nemenyi_test, wilcoxon_test
+from aeon.utils.validation._dependencies import _check_soft_dependencies
+
+
+def plot_significance(
+    scores,
+    labels,
+    alpha=0.1,
+    lower_better=False,
+    test="wilcoxon",
+    correction="holm",
+    fontsize=12,
+    reverse=True,
+    return_p_values=False,
+):
+    """
+    Plot that allow the case where cliques can be deceiving.
+
+    It is a visual representation of the results of a statistical test to compare the
+    performance of different classifiers. The plot is based on the work of Demsar [1]_.
+    The plot shows the average rank of each classifier and the average score of each
+    method. The plot also shows the cliques of classifiers that are not significantly
+    different from each other.
+
+    Parameters
+    ----------
+    scores : np.array
+        Array of shape (n_datasets, n_estimators) with the performance of each estimator
+        in each dataset.
+    labels : list of str
+        List of length n_estimators with the name of each estimator.
+    alpha : float, default=0.1
+        The significance level used in the statistical test.
+    lower_better : bool, default=False
+        Whether lower scores are better than higher scores.
+    test : str, default="wilcoxon"
+        The statistical test to use. Available tests are "nemenyi" and "wilcoxon".
+    correction : str, default="holm"
+        The correction to apply to the p-values. Available corrections are "bonferroni",
+        "holm" and None.
+    fontsize : int, default=12
+        The fontsize of the text in the plot.
+    reverse : bool, default=True
+        Whether to reverse the order of the labels.
+    return_p_values : bool, default=False
+        Whether to return the pairwise matrix of p-values.
+
+    Returns
+    -------
+        fig : matplotlib.figure
+            Figure created.
+        ax : matplotlib.axes
+            Axes of the figure.
+        p_values : np.ndarray (optional)
+            if return_p_values is True, returns a (n_estimators, n_estimators) matrix of
+            unadjusted p values for the pairwise Wilcoxon sign rank test.
+
+    References
+    ----------
+    .. [1] Demsar J., "Statistical comparisons of classifiers over multiple data sets."
+    Journal of Machine Learning Research 7:1-30, 2006.
+
+    Example
+    -------
+    >>> from aeon.visualisation import plot_significance
+    >>> from aeon.benchmarking.results_loaders import get_estimator_results_as_array
+    >>> methods = ["IT", "WEASEL-Dilation", "HIVECOTE2", "FreshPRINCE"]
+    >>> results = get_estimator_results_as_array(estimators=methods) # doctest: +SKIP
+    >>> plot = plot_significance(results[0], methods, alpha=0.1)\
+        # doctest: +SKIP
+    >>> plot.show()  # doctest: +SKIP
+    >>> plot.savefig("significance_plot.pdf", bbox_inches="tight")  # doctest: +SKIP
+    """
+    _check_soft_dependencies("matplotlib")
+
+    import matplotlib.pyplot as plt
+
+    n_datasets, n_estimators = scores.shape
+    if isinstance(test, str):
+        test = test.lower()
+    if isinstance(correction, str):
+        correction = correction.lower()
+    # Step 1: rank data: in case of ties average ranks are assigned
+    if lower_better:  # low is good -> rank 1
+        ranks = rankdata(scores, axis=1)
+    else:  # assign opposite ranks
+        ranks = rankdata(-1 * scores, axis=1)
+
+    if return_p_values and test != "wilcoxon":
+        raise ValueError(
+            f"Cannot return p values for {test}, since it does "
+            "not calculate p-values."
+        )
+
+    # Step 2: calculate average rank per estimator
+    avg_ranks = ranks.mean(axis=0)
+    # Sort labels and ranks
+    ordered_labels_ranks = np.array(
+        [(l, float(r)) for r, l in sorted(zip(avg_ranks, labels))], dtype=object
+    )
+    ordered_labels = np.array([la for la, _ in ordered_labels_ranks], dtype=str)
+    ordered_avg_ranks = np.array([r for _, r in ordered_labels_ranks], dtype=np.float32)
+
+    indices = [np.where(np.array(labels) == r)[0] for r in ordered_labels]
+
+    ordered_scores = scores[:, indices]
+    ordered_avg_scores = ordered_scores.mean(axis=0).flatten()
+
+    # Step 3 : check whether Friedman test is significant
+    p_value_friedman = check_friedman(ranks)
+    # Step 4: If Friedman test is significant find cliques
+    if p_value_friedman < alpha:
+        if test == "nemenyi":
+            cliques = nemenyi_test(ordered_avg_ranks, n_datasets, alpha)
+        elif test == "wilcoxon":
+            if correction == "bonferroni":
+                adjusted_alpha = alpha / (n_estimators * (n_estimators - 1) / 2)
+            elif correction == "holm":
+                adjusted_alpha = alpha / (n_estimators - 1)
+            elif correction is None:
+                adjusted_alpha = alpha
+            else:
+                raise ValueError("correction available are None, Bonferroni and Holm.")
+            p_values = wilcoxon_test(ordered_scores, ordered_labels, lower_better)
+            cliques = _build_cliques(p_values > adjusted_alpha)
+        else:
+            raise ValueError("tests available are only nemenyi and wilcoxon.")
+    # If Friedman test is not significant everything has to be one clique
+    else:
+        p_values = np.triu(np.ones((n_estimators, n_estimators)))
+        cliques = [[1] * n_estimators]
+
+    if len(cliques) == 0:
+        warnings.warn(
+            "No significant difference between the classifiers. "
+            "Consider using critical difference diagrams.",
+            stacklevel=2,
+        )
+    if reverse:
+        ordered_labels = ordered_labels[::-1]
+        ordered_avg_ranks = ordered_avg_ranks[::-1]
+        ordered_avg_scores = ordered_avg_scores[::-1]
+        for i in range(len(cliques)):
+            cliques[i] = cliques[i][::-1]
+
+    label_lengths = np.array([len(i) for i in ordered_labels])
+    labels_space = 0.5 if max(label_lengths) > 14 else 0
+
+    # Compress the graph by reducing vertical spacing
+    spacing = 0.2 if len(cliques) >= 3 else 0.3
+
+    height = (
+        max(
+            len(cliques) * 0.6 if len(cliques) > 5 else len(cliques),
+            1.5 if n_estimators < 7 else 2.5,
+        )
+        + labels_space
+    )
+    width = max(n_estimators * 0.8, 7.5)
+
+    fig, ax = plt.subplots(figsize=(width, height), layout="tight")
+
+    # Draw the horizontal lines for each valid row
+    for row_index, row in enumerate(cliques[::-1]):
+        true_indices = np.where(row)[0]
+        if true_indices.size > 0:
+            y_pos = spacing * row_index
+            for col_index in true_indices:
+                # add circles for each 'True' value
+                ax.plot(
+                    col_index,
+                    y_pos,
+                    "o",
+                    markersize=fontsize - 2,
+                    color="black",
+                )
+
+            ax.hlines(
+                y=y_pos,
+                xmin=true_indices[-1],
+                xmax=true_indices[0],
+                color="black",
+                linewidth=2,
+            )
+
+    # Setting labels for x-axis. Rotate only if labels are too long.
+    ax.xaxis.tick_top()
+    ax.set_xticks(np.arange(len(ordered_labels)))
+    if (sum(label_lengths) > 40) or (max(label_lengths[:-1] + label_lengths[1:]) > 20):
+        ax.set_xticklabels(ordered_labels, rotation=45, ha="left", fontsize=fontsize)
+    else:
+        ax.set_xticklabels(ordered_labels, fontsize=fontsize)
+
+    ax.xaxis.set_label_position("top")
+
+    y_pos = spacing * (len(cliques) + 2.5)
+    # Add a horizontal line on the X axis under the labels
+    ax.hlines(
+        y=y_pos,
+        xmin=-0.5,
+        xmax=n_estimators - 0.5,
+        color="black",
+        linewidth=1,
+    )
+
+    for idx, (ran, val) in enumerate(zip(ordered_avg_ranks, ordered_avg_scores)):
+        ax.text(
+            idx,
+            y_pos - spacing / 2,
+            f"{ran:.3f}",
+            ha="center",
+            va="center",
+            fontsize=fontsize,
+        )
+
+        ax.text(
+            idx,
+            y_pos - spacing * 1.5,
+            f"{val:.2f}" if ordered_avg_scores.mean() > 9 else f"{val:.3f}",
+            ha="center",
+            va="center",
+            fontsize=fontsize - 2 if ordered_avg_scores.mean() > 9 else fontsize,
+        )
+
+    ax.text(
+        -1,
+        y_pos - spacing / 2,
+        "Avg. rank",
+        ha="right",
+        va="center",
+        fontsize=fontsize,
+        fontweight="bold",
+    )
+
+    ax.text(
+        -1,
+        y_pos - spacing * 1.5,
+        "Avg. value",
+        ha="right",
+        va="center",
+        fontsize=fontsize,
+        fontweight="bold",
+    )
+
+    # Adjust the y limits
+    ax.set_ylim(-0.1, y_pos)
+    ax.set_xlim(-1, n_estimators)
+
+    # Remove the spines, ticks, labels, and grid
+    ax.spines[["top", "right", "left", "bottom"]].set_visible(False)
+    ax.tick_params(left=False, labelleft=False, labelbottom=False)
+    ax.grid(False)
+
+    if return_p_values:
+        return fig, ax, p_values
+    else:
+        return fig, ax
+
+
+# this _build_cliques is different to the CD one.
+def _build_cliques(pairwise_matrix):
+    """
+    Build cliques from pairwise comparison matrix.
+
+    Parameters
+    ----------
+    pairwise_matrix : np.array
+        Pairwise matrix shape (n_estimators, n_estimators) indicating if there is a
+        significant difference between pairs. Assumed to be ordered by rank of
+        estimators.
+
+    Returns
+    -------
+    list of lists
+        cliques within which there is no significant different between estimators.
+    """
+    n = np.sum(pairwise_matrix, 1)
+    cliques = pairwise_matrix[n > 1, :]
+
+    for i in range(len(cliques) - 1, 0, -1):
+        for j in range(i - 1, -1, -1):
+            if np.all(cliques[j, cliques[i, :]]):
+                cliques[i, :] = 0
+                break
+
+    n = np.sum(cliques, 1)
+    cliques = cliques[n > 1, :]
+
+    return cliques


Test patch:
diff --git a/aeon/performance_metrics/tests/test_stats.py b/aeon/performance_metrics/tests/test_stats.py
new file mode 100644
index 00000000..7eef938f
--- /dev/null
+++ b/aeon/performance_metrics/tests/test_stats.py
@@ -0,0 +1,111 @@
+"""Tests for the critical difference diagram maker."""
+
+import os
+
+import numpy as np
+from numpy.testing import assert_almost_equal
+from scipy.stats import rankdata
+
+import aeon
+from aeon.benchmarking.results_loaders import get_estimator_results_as_array
+from aeon.datasets.tsc_data_lists import univariate_equal_length
+from aeon.performance_metrics.stats import check_friedman, nemenyi_test, wilcoxon_test
+
+data_path = os.path.join(
+    os.path.dirname(aeon.__file__),
+    "benchmarking/example_results/",
+)
+
+
+def test_nemenyi_test():
+    """Test Nemenyi test for differences in ranks."""
+    cls = ["HC2", "FreshPRINCE", "InceptionT", "WEASEL-D"]
+    alpha = 0.05
+    data_full = list(univariate_equal_length)
+    data_full.sort()
+
+    res = get_estimator_results_as_array(
+        estimators=cls, datasets=data_full, path=data_path, include_missing=True
+    )
+
+    ranked_data = rankdata(-1 * res, axis=1)
+
+    avranks = ranked_data.mean(axis=0)
+    avranks = np.array([s for s, _ in sorted(zip(avranks, cls))])
+
+    cliques = nemenyi_test(avranks, len(data_full), alpha)
+    test_clique = np.array(
+        [
+            [True, False, False, False],
+            [False, True, True, True],
+            [False, False, True, True],
+            [False, False, False, True],
+        ]
+    )
+    assert np.all(cliques == test_clique)
+
+    # to check the existence of a clique we select a subset of the datasets.
+    data = data_full[45:55]
+
+    res = get_estimator_results_as_array(
+        estimators=cls, datasets=data, path=data_path, include_missing=True
+    )
+
+    ranked_data = rankdata(-1 * res, axis=1)
+    avranks = ranked_data.mean(axis=0)
+    avranks = np.array([s for s, _ in sorted(zip(avranks, cls))])
+    cliques = nemenyi_test(avranks, len(data), 0.01)
+    test_clique = np.array(
+        [
+            [True, True, True, True],
+            [False, True, True, True],
+            [False, False, True, True],
+            [False, False, True, True],
+        ]
+    )
+
+    assert np.all(cliques == test_clique)
+
+
+def test_wilcoxon_test():
+    """Test Wilcoxon pairwise test for multiple estimators."""
+    cls = ["HC2", "InceptionT", "WEASEL-D", "FreshPRINCE"]
+    data_full = list(univariate_equal_length)
+    data_full.sort()
+    res = get_estimator_results_as_array(
+        estimators=cls, datasets=data_full, path=data_path, include_missing=True
+    )
+    p_vals = wilcoxon_test(res, cls)
+    assert_almost_equal(p_vals[0], np.array([1.0, 0.0, 0.0, 0.0]), decimal=2)
+
+    res = get_estimator_results_as_array(
+        estimators=cls, datasets=data_full, path=data_path, include_missing=True
+    )
+    p_vals = wilcoxon_test(res, cls, lower_better=True)
+    assert_almost_equal(p_vals[0], np.array([1, 0.99, 0.99, 0.99]), decimal=2)
+
+
+def test__check_friedman():
+    """Test Friedman test for overall difference in estimators."""
+    cls = ["HC2", "FreshPRINCE", "InceptionT", "WEASEL-D"]
+    data = univariate_equal_length
+    res = get_estimator_results_as_array(
+        estimators=cls, datasets=data, path=data_path, include_missing=True
+    )
+    ranked_data = rankdata(-1 * res, axis=1)
+    assert check_friedman(ranked_data) < 0.05
+
+    # test that approaches are not significantly different.
+    cls = ["HC2", "HC2", "HC2"]
+    res = get_estimator_results_as_array(
+        estimators=cls,
+        datasets=data,
+        path=data_path,
+        include_missing=True,
+    )
+    # add some tiny noise to the results.
+    np.random.seed(42)
+    random_values = np.random.uniform(-0.01, 0.01, size=res.shape)
+    res = np.clip(res + random_values, 0, 1)
+    ranked_data = rankdata(-1 * res, axis=1)
+    assert check_friedman(ranked_data) > 0.05
diff --git a/aeon/visualisation/results/tests/test_critical_difference.py b/aeon/visualisation/results/tests/test_critical_difference.py
index d0225799..f121b365 100644
--- a/aeon/visualisation/results/tests/test_critical_difference.py
+++ b/aeon/visualisation/results/tests/test_critical_difference.py
@@ -4,8 +4,6 @@ import os
 
 import numpy as np
 import pytest
-from numpy.testing import assert_almost_equal
-from scipy.stats import rankdata
 
 import aeon
 from aeon.benchmarking.results_loaders import get_estimator_results_as_array
@@ -13,9 +11,6 @@ from aeon.datasets.tsc_data_lists import univariate_equal_length
 from aeon.utils.validation._dependencies import _check_soft_dependencies
 from aeon.visualisation.results._critical_difference import (
     _build_cliques,
-    _check_friedman,
-    _nemenyi_test,
-    _wilcoxon_test,
     plot_critical_difference,
 )
 
@@ -24,86 +19,6 @@ data_path = os.path.join(
     "benchmarking/example_results/",
 )
 
-
-def test_nemenyi_test():
-    """Test Nemenyi test for differences in ranks."""
-    cls = ["HC2", "FreshPRINCE", "InceptionT", "WEASEL-D"]
-    alpha = 0.05
-    data_full = list(univariate_equal_length)
-    data_full.sort()
-
-    res = get_estimator_results_as_array(
-        estimators=cls, datasets=data_full, path=data_path, include_missing=True
-    )
-
-    ranked_data = rankdata(-1 * res, axis=1)
-
-    avranks = ranked_data.mean(axis=0)
-    avranks = np.array([s for s, _ in sorted(zip(avranks, cls))])
-
-    cliques = _nemenyi_test(avranks, len(data_full), alpha)
-
-    assert np.all(cliques == [False, True, True, True])
-
-    # to check the existence of a clique we select a subset of the datasets.
-    data = data_full[45:55]
-
-    res = get_estimator_results_as_array(
-        estimators=cls, datasets=data, path=data_path, include_missing=True
-    )
-
-    ranked_data = rankdata(-1 * res, axis=1)
-    avranks = ranked_data.mean(axis=0)
-    avranks = np.array([s for s, _ in sorted(zip(avranks, cls))])
-    cliques = _nemenyi_test(avranks, len(data), 0.01)
-
-    assert np.all(cliques == [True, True, True, True])
-
-
-def test_wilcoxon_test():
-    """Test Wilcoxon pairwise test for multiple estimators."""
-    cls = ["HC2", "InceptionT", "WEASEL-D", "FreshPRINCE"]
-    data_full = list(univariate_equal_length)
-    data_full.sort()
-    res = get_estimator_results_as_array(
-        estimators=cls, datasets=data_full, path=data_path, include_missing=True
-    )
-    p_vals = _wilcoxon_test(res, cls)
-    assert_almost_equal(p_vals[0], np.array([1.0, 0.0, 0.0, 0.0]), decimal=2)
-
-    res = get_estimator_results_as_array(
-        estimators=cls, datasets=data_full, path=data_path, include_missing=True
-    )
-    p_vals = _wilcoxon_test(res, cls, lower_better=True)
-    assert_almost_equal(p_vals[0], np.array([1, 0.99, 0.99, 0.99]), decimal=2)
-
-
-def test__check_friedman():
-    """Test Friedman test for overall difference in estimators."""
-    cls = ["HC2", "FreshPRINCE", "InceptionT", "WEASEL-D"]
-    data = univariate_equal_length
-    res = get_estimator_results_as_array(
-        estimators=cls, datasets=data, path=data_path, include_missing=True
-    )
-    ranked_data = rankdata(-1 * res, axis=1)
-    assert _check_friedman(ranked_data) < 0.05
-
-    # test that approaches are not significantly different.
-    cls = ["HC2", "HC2", "HC2"]
-    res = get_estimator_results_as_array(
-        estimators=cls,
-        datasets=data,
-        path=data_path,
-        include_missing=True,
-    )
-    # add some tiny noise to the results.
-    np.random.seed(42)
-    random_values = np.random.uniform(-0.01, 0.01, size=res.shape)
-    res = np.clip(res + random_values, 0, 1)
-    ranked_data = rankdata(-1 * res, axis=1)
-    assert _check_friedman(ranked_data) > 0.05
-
-
 test_clique1 = np.array(
     [
         [True, False, False, False],
diff --git a/aeon/visualisation/results/tests/test_significance.py b/aeon/visualisation/results/tests/test_significance.py
new file mode 100644
index 00000000..24f7ccbf
--- /dev/null
+++ b/aeon/visualisation/results/tests/test_significance.py
@@ -0,0 +1,224 @@
+"""Tests for the critical difference diagram maker."""
+
+import os
+
+import numpy as np
+import pytest
+
+import aeon
+from aeon.benchmarking.results_loaders import get_estimator_results_as_array
+from aeon.datasets.tsc_data_lists import univariate_equal_length
+from aeon.utils.validation._dependencies import _check_soft_dependencies
+from aeon.visualisation.results._significance import _build_cliques, plot_significance
+
+data_path = os.path.join(
+    os.path.dirname(aeon.__file__),
+    "benchmarking/example_results/",
+)
+
+
+test_clique1 = np.array(
+    [
+        [True, False, False, False],
+        [False, True, False, False],
+        [False, False, True, True],
+        [False, False, False, True],
+    ]
+)
+correct_clique1 = np.array(
+    [
+        [False, False, True, True],
+    ]
+)
+
+test_clique2 = np.array(
+    [
+        [True, False, False, False],
+        [False, True, False, False],
+        [False, False, True, True],
+        [False, False, False, True],
+    ]
+)
+correct_clique2 = np.array([[False, False, True, True]])
+
+test_clique3 = np.array(
+    [
+        [True, True, True, True, True, False, False, False, False, False, False],
+        [False, True, True, True, True, True, True, False, False, False, False],
+        [False, False, True, True, True, True, False, False, False, False, False],
+        [False, False, False, True, True, True, True, True, True, True, True],
+        [False, False, False, False, True, True, True, True, True, True, False],
+        [False, False, False, False, False, True, True, True, True, True, True],
+        [False, False, False, False, False, False, True, True, True, True, False],
+        [False, False, False, False, False, False, False, True, True, True, True],
+        [False, False, False, False, False, False, False, False, True, True, True],
+        [False, False, False, False, False, False, False, False, False, True, True],
+        [False, False, False, False, False, False, False, False, False, False, True],
+    ]
+)
+correct_clique3 = np.array(
+    [
+        [True, True, True, True, True, False, False, False, False, False, False],
+        [False, True, True, True, True, True, True, False, False, False, False],
+        [False, False, False, True, True, True, True, True, True, True, True],
+    ]
+)
+
+test_clique4 = np.array(
+    [
+        [True, True, True, True],
+        [False, True, False, False],
+        [False, False, True, True],
+        [False, False, False, True],
+    ]
+)
+correct_clique4 = np.array([[True, True, True, True]])
+
+
+test_clique5 = np.array(
+    [
+        [True, True, False, True],
+        [False, True, False, True],
+        [False, False, False, True],
+        [False, True, False, True],
+    ]
+)
+correct_clique5 = np.array([[True, True, False, True]])
+
+test_clique6 = np.array(
+    [
+        [True, True, False, True, True, True, True],
+        [False, True, True, True, True, False, False],
+        [False, False, True, True, True, False, False],
+        [False, False, False, True, False, True, False],
+    ]
+)
+correct_clique6 = np.array(
+    [
+        [True, True, False, True, True, True, True],
+        [False, True, True, True, True, False, False],
+    ]
+)
+
+CLIQUES = [
+    (test_clique1, correct_clique1),
+    (test_clique2, correct_clique2),
+    (test_clique3, correct_clique3),
+    (test_clique4, correct_clique4),
+    (test_clique5, correct_clique5),
+    (test_clique6, correct_clique6),
+]
+
+
+@pytest.mark.parametrize("cliques", CLIQUES)
+def test__build_cliques(cliques):
+    """Test build cliques with different cases."""
+    obs_clique = _build_cliques(cliques[0])
+    assert np.all(obs_clique == cliques[1])
+
+
+def test__build_cliques_empty():
+    """Test build cliques with empty return."""
+    test_clique = np.array([[True, False], [False, True]])
+    obs_clique = _build_cliques(test_clique)
+    assert obs_clique.size == 0
+
+
+@pytest.mark.skipif(
+    not _check_soft_dependencies("matplotlib", severity="none"),
+    reason="skip test if required soft dependency not available",
+)
+@pytest.mark.parametrize("correction", ["bonferroni", "holm", None])
+def test_plot_significance_corrections(correction):
+    """Test plot significance diagram."""
+    import matplotlib
+    import matplotlib.pyplot as plt
+
+    matplotlib.use("Agg")
+
+    cls = ["HC2", "FreshPRINCE", "InceptionT", "WEASEL-D"]
+    data_full = list(univariate_equal_length)
+    data_full.sort()
+
+    res = get_estimator_results_as_array(
+        estimators=cls, datasets=data_full, path=data_path, include_missing=True
+    )
+
+    fig, ax = plot_significance(
+        res,
+        cls,
+        test="wilcoxon",
+        correction=correction,
+    )
+    plt.gcf().canvas.draw_idle()
+
+    assert isinstance(fig, plt.Figure) and isinstance(ax, plt.Axes)
+
+
+@pytest.mark.skipif(
+    not _check_soft_dependencies("matplotlib", severity="none"),
+    reason="skip test if required soft dependency not available",
+)
+def test_plot_significance():
+    """Test plot critical difference diagram."""
+    _check_soft_dependencies("matplotlib")
+
+    import matplotlib
+    import matplotlib.pyplot as plt
+
+    matplotlib.use("Agg")
+
+    cls = ["HC2", "FreshPRINCE", "InceptionT", "WEASEL-D"]
+    data_full = list(univariate_equal_length)
+    data_full.sort()
+
+    res = get_estimator_results_as_array(
+        estimators=cls, datasets=data_full, path=data_path, include_missing=True
+    )
+
+    fig, ax = plot_significance(
+        res,
+        cls,
+        lower_better=False,
+        alpha=0.05,
+        reverse=True,
+        test="wilcoxon",
+        correction="holm",
+        fontsize=12,
+    )
+    plt.gcf().canvas.draw_idle()
+    assert isinstance(fig, plt.Figure) and isinstance(ax, plt.Axes)
+
+
+@pytest.mark.skipif(
+    not _check_soft_dependencies("matplotlib", severity="none"),
+    reason="skip test if required soft dependency not available",
+)
+def test_plot_significance_p_values():
+    """Test plot significance diagram."""
+    _check_soft_dependencies("matplotlib")
+    import matplotlib
+    import matplotlib.pyplot as plt
+
+    matplotlib.use("Agg")
+
+    cls = ["HC2", "FreshPRINCE", "InceptionT", "WEASEL-D"]
+    data_full = list(univariate_equal_length)
+    data_full.sort()
+
+    res = get_estimator_results_as_array(
+        estimators=cls, datasets=data_full, path=data_path, include_missing=True
+    )
+
+    fig, ax, p_values = plot_significance(
+        res,
+        cls,
+        lower_better=False,
+        alpha=0.05,
+        reverse=True,
+        return_p_values=True,
+    )
+    plt.gcf().canvas.draw_idle()
+
+    assert isinstance(fig, plt.Figure) and isinstance(ax, plt.Axes)
+    assert isinstance(p_values, np.ndarray)