Repository: sktime/sktime

Issue:
[ENH] Hierarchical agglomerative estimation: E-Agglo
### Details 
E-Agglo is a nonparametric method for performing hierarchical agglomerative estimation of multiple change points based on [1][2]. Neighboring segments are then sequentially merged to maximize a goodness-of-fit statistic. The goodness-of-fit statistic is the "between-within distance" [3] among adjacent segments i.e.

![image](https://user-images.githubusercontent.com/34525024/189269160-707c101a-6344-456b-8571-048344348322.png)

where || . || is the Euclidean norm. Which is the scaled sample measure of divergence below

![image](https://user-images.githubusercontent.com/34525024/189269751-d45fa4e1-5278-48ba-b9f5-033cc9137051.png)

i.e.
``e(A, B) = mn/(m+n) * [minimum energy clustering]`` as evidenced in [4].

Note that minimum energy clustering can be computed with `scipy.stats.energy_distance(...)` [5].

#### References
[1] Matteson, David S., and Nicholas A. James. "A nonparametric approach for multiple change point analysis of multivariate data." Journal of the American Statistical Association 109.505 (2014): 334-345. URL: https://www.tandfonline.com/doi/full/10.1080/01621459.2013.849605
[2] James, Nicholas A., and David S. Matteson. "ecp: An R package for nonparametric multiple change point analysis of multivariate data." arXiv preprint arXiv:1309.3295 (2013). URL: https://arxiv.org/pdf/1309.3295.pdf
[3] Szekely, Gabor J., and Maria L. Rizzo. "Hierarchical clustering via joint between-within distances: Extending Ward's minimum variance method." Journal of classification 22.2 (2005): 151-184. URL: https://home.ipipan.waw.pl/j.mielniczuk/Szekely-Rizzo_2005.pdf
[4] https://en.wikipedia.org/wiki/Hierarchical_clustering#cite_ref-7
[5] https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.energy_distance.html


## Option 1: Interface sklearn.cluster.AgglomerativeClustering

[sklearn.cluster.AgglomerativeClustering](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html#sklearn.cluster.AgglomerativeClustering) has parameters `linkage{‘ward’, ‘complete’, ‘average’, ‘single’}, default=’ward’`.

We need another linkage type not offered in this list. Therefore, this task involves writing another version of [ward_tree](https://github.com/scikit-learn/scikit-learn/blob/36958fb24/sklearn/cluster/_agglomerative.py#L169), with distnace between clusters `e(A,B)` from above instead of Ward's linkage defined as

![image](https://user-images.githubusercontent.com/34525024/189270752-245a4dc8-4747-4d28-959f-94f565bd0ba1.png)
(the visualisation of linkage parameters comes from [here](https://towardsdatascience.com/machine-learning-algorithms-part-12-hierarchical-agglomerative-clustering-example-in-python-1e18e0075019))

and interfacing `AgglomerativeClustering(ClusterMixin, BaseEstimator)` with this extended functionality.

### Dependencies

- [ ] `scipy.cluster.hierarchy.ward(X)` s called, where
![image](https://user-images.githubusercontent.com/34525024/189275455-a94e6ade-cd19-47a4-8948-7e31a596c10a.png)

which is (almost) the same as the desired incremental metric from [2]:
![image](https://user-images.githubusercontent.com/34525024/189278041-e2f4b226-fc38-4cd1-affa-37090de43bd9.png)


- [ ] `ward_dist` relies on c++ ` _hierarchical.compute_ward_dist` - can this be fully replaced with the scipy function? [5]
- [ ] `ward_dist` relies on c++  `_hierarchical._get_parents` - can this be called?


## Option 2: rewrite R code as stand alone

The R code is: https://github.com/cran/ecp/blob/master/R/e_agglomerative.R

- [x] getWithin, getBetween are C functions that need to be re-written
https://github.com/cran/ecp/blob/master/src/energyChangePoint.cpp

but maybe they can be replaced by `scipy.stats.energy_distance(...)` [5] - wouldn't take too long to verify.





Discussion:
My hot take: this is confusing. I think the safest option would be to directly re-write the R implementation, write unit tests, then (potentially) refactor this logic to interface AgglomerativeClustering.
Estimated time: 2 weeks for draft PR, just to the re-write R and unit test phase.
@KatieBuc this is great, thanks. Would you at this point be able to draft e-Agglo in pseudocode so we could see what component we have and what would have to be implemented or interfaced?
Pseudo code for the R-rewrite:

```
def process.data():

        #set up left and right neighbors, cluster size, checks
        C: getWithin() #create array of within distances 
        C: getBetween() #make distance matrix 
        #set initial GOF value
        #create matrix for change point progression
        #vector to specify the starting point of a cluster
```

```
def gof.update()
        #Function to calculate the new GOF value. The i argument is assumed to be the left 
        #cluster.

        #get new left and right clusters
        #get cluster sizes
        #add distance to new left cluster
        #add distance to new right cluster
```

```
def find.closest()
        #Determine which clusters will be merged. Returns a vector of length 3. The first 
        #element of the vector is the left cluster. The third element of the vector is the 
        #newest GOF value.
        
        #iterate to see how the GOF value changes
        for i in 1:K:
           gof.update() #get updated gof value
           #if better value found, update

```

```
def updateDistance():
        #Function to update the distance from the new cluster to the other clusters. 
        #i is assumed to be the left cluster. Also updates any other necessary 
        #information in the list (ret)
        
        #update left and right neighbors in ret
        #assign size to newly created cluster
        #update set of change points
        #update distances as per Lemma 3 in above comment
        for(k in 1:K):
              n3 = ret$sizes[k]
              n = n1+n2+n3
              ret$D[K+1,k] = ret$D[k,K+1] = ((n-n2)*ret$D[i,k]+(n-n1)*ret$D[j,k]-n3*ret$D[i,j])/n
```


```
def e.agglo():

    process.data() #initilize components
    # find which clusters optimize the GOF and then update the distances
    for K in N:...
        find.closest() #find which clusters to merge
        updateDistance() #update information after merger
    #penalize the GOF statistic
    #get the set of change points for the "best" clustering
    #create final membership vector
```





If this were to be interfaced with [sklearn.cluster.AgglomerativeClustering](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html#sklearn.cluster.AgglomerativeClustering), then all this would be refactored into an appropriated version of [ward_tree](https://github.com/scikit-learn/scikit-learn/blob/36958fb24/sklearn/cluster/_agglomerative.py#L169) within an instance of `AgglomerativeClustering(ClusterMixin, BaseEstimator):`. However, I'm still unsure of what would be useful in this class, since the tree building approach and node merging approach is different i.e. 

```
def ward_tree(X, *, connectivity=None, n_clusters=None, return_distance=False):
    """Ward clustering based on a Feature matrix.
    Recursively merges the pair of clusters that minimally increases
    within-cluster variance.
    The inertia matrix uses a Heapq-based representation.
    This is the structured version, that takes into account some topological
    structure between samples.
```
how this `inertia` matrix relates to the greedy agglomerative procedure outlined here:

![image](https://user-images.githubusercontent.com/34525024/189785485-6aad98dc-5d34-4704-a8e1-764b986c48c9.png)

How to write the R in python is clear to me, how to interface this with a seemingly similar class is very unclear still.







@lmmentel @Lovkush-A @fkiraly fyi
In terms of re-writing this R code, a simplification of the function (which calls C functions) is:
```
install.packages('ecp')
library(ecp)

set.seed(1234)
X <- c(rnorm(15, mean = -6), 0, rnorm(16, mean = 6))

#Initialize all the necessary components of the list.
process.data = function(member,X,alpha){
  ret = NULL #the list with the necessary information
  u = unique(member)
  N = ret$N = length(u)#number of clusters

  for(i in 1:N)#relabel clusters
    member[member==u[i]] = i
  
  ret$sizes = numeric(2*N)
  
  for(i in 1:N)#calculate initial cluster sizes
    ret$sizes[i] = sum(member==i)
  
  #array of within distances
  within = numeric(N)
  for(i in 1:N)
    within[i] = ecp::getWithin(alpha,matrix(X[member==i,],ret$sizes[i],ncol(X)))
  
  ret$within = within
  
  #make distance matrix
  ret$D = matrix(Inf,nrow=2*N,ncol=2*N)
  for(i in 1:N)
    for(j in i:N)
      if(j != i)
        ret$D[i,j] = ret$D[j,i] = ecp::getBetween(alpha,matrix(X[member==i,],ret$sizes[i],ncol(X)),matrix(X[member==j,],ret$sizes[j],ncol(X))) - within[i] - within[j]
  
  diag(ret$D) = 0

  return(ret)
}
ret = process.data(1:32,as.matrix(X),1)
ret$D

```

which gives distance matrix:
![image](https://user-images.githubusercontent.com/34525024/189818648-6a6a0646-d0c3-4a55-b340-d0f252ef08d9.png)

The equivalent python code is:
```
def get_within(X, alpha):
    n = X.shape[0]
    return list(sum(np.power(np.sqrt(sum((X.iloc[i,]-X.iloc[j,])*(X.iloc[i,]-X.iloc[j,]))),alpha) for j in range(n))/(n**2) for i in range(n))

within = [get_within(X.loc[[i,]], alpha)[0] for i in range(N)]

def get_between(X, Y, alpha):
    n = X.shape[0]
    m = X.shape[0]
    return list(sum(np.power(np.sqrt(sum((X.iloc[i,]-Y.iloc[j,])*(X.iloc[i,]-Y.iloc[j,]))),alpha) for j in range(m))/(m * n) for i in range(n))

D = pd.DataFrame(index = range(N), columns = range(N))

for i in range(N):
    for j in range(N):
        between = get_between(X.loc[[i,]], X.loc[[j,]], alpha)[0]
        D.iloc[i,j] = D.iloc[j, i] = 2*between - within[i] - within[j]

```

which gives the expected distance matrix:
![image](https://user-images.githubusercontent.com/34525024/189818972-22d4ba56-d97f-4c73-8f8b-d02d00153494.png)

Also, note that this distance matrix could also be obtained using `scipy.stats.energy_distance()`
![image](https://user-images.githubusercontent.com/34525024/189819472-fa898d1f-35e6-4e7d-b795-eb9ff47a44fd.png)

which I think is the way to go with this.

Resolving patch:
diff --git a/sktime/annotation/datagen.py b/sktime/annotation/datagen.py
--- a/sktime/annotation/datagen.py
+++ b/sktime/annotation/datagen.py
@@ -118,11 +118,9 @@ def get_covariances(var):
         # get covariance matrices from variance arrays
         covariances = [get_covariances(var) for var in variances]
 
-    # if covariance is specified, then check if symmetric (eigenvalues are real)
-    # and check if matrix is positive semidefinite (eigenvalues are nonnegative)
     else:
-        assert all([np.allclose(np.array(cov), np.array(cov).T) for cov in covariances])
-        assert all([np.all(np.linalg.eigvals(cov) >= 0) for cov in covariances])
+        assert all(np.allclose(np.array(cov), np.array(cov).T) for cov in covariances)
+        assert all(np.all(np.linalg.eigvals(cov) >= 0) for cov in covariances)
 
     assert np.array(covariances).shape[0] == L
     assert np.array(covariances).shape[1] == N
diff --git a/sktime/annotation/eagglo.py b/sktime/annotation/eagglo.py
new file mode 100644
--- /dev/null
+++ b/sktime/annotation/eagglo.py
@@ -0,0 +1,438 @@
+# -*- coding: utf-8 -*-
+"""E-Agglo: agglomerative clustering algorithm that preserves observation order."""
+
+import warnings
+from typing import Callable, Dict, List, Tuple
+
+import numpy as np
+import pandas as pd
+from scipy.spatial.distance import cdist
+
+from sktime.transformations.base import BaseTransformer
+
+__author__ = ["KatieBuc"]
+__all__ = ["EAgglo"]
+
+
+class EAgglo(BaseTransformer):
+    """
+    Hierarchical agglomerative estimation of multiple change points.
+
+    E-Agglo is a non-parametric clustering approach for multivariate timeseries[1]_,
+    where neighboring segments are sequentially merged_ to maximize a goodness-of-fit
+    statistic. Unlike most general purpose agglomerative clustering algorithms, this
+    procedure preserves the time ordering of the observations.
+
+    This method can detect distributional change within an independent sequence,
+    and does not make any distributional assumptions (beyond the existence of an
+    alpha-th moment). Estimation is performed in a manner that simultaneously
+    identifies both the number and locations of change points.
+
+    Parameters
+    ----------
+    member : array_like (default=None)
+        Assigns points to to the initial cluster membership, therefore the first
+        dimension should be the same as for data. If `None` it will be initialized
+        to dummy vector where each point is assigned to separate cluster.
+    alpha : float (default=1.0)
+        Fixed constant alpha in (0, 2] used in the divergence measure, as the
+        alpha-th absolute moment, see equation (4) in [1]_.
+    penalty : str or callable or None (default=None)
+        Function that defines a penalization of the sequence of goodness-of-fit
+        statistic, when overfitting is a concern. If `None` not penalty is applied.
+        Could also be an existing penalty name, either `len_penalty` or
+        `mean_diff_penalty`.
+
+    Attributes
+    ----------
+    merged_ : array_like
+        2D `array_like` outlining which clusters were merged_ at each step.
+    gof_ : float
+        goodness-of-fit statistic for current clsutering.
+    cluster_ : array_like
+        1D `array_like` specifying which cluster each row of input data
+        X belongs to.
+
+    Notes
+    -----
+    Based on the work from [1]_.
+
+    - source code based on: https://github.com/cran/ecp/blob/master/R/e_agglomerative.R
+    - paper available at: https://www.tandfonline.com/doi/full/10.1080/01621459.\
+        2013.849605
+
+    References
+    ----------
+    .. [1] Matteson, David S., and Nicholas A. James. "A nonparametric approach for
+    multiple change point analysis of multivariate data." Journal of the American
+    Statistical Association 109.505 (2014): 334-345.
+
+    .. [2] James, Nicholas A., and David S. Matteson. "ecp: An R package for
+    nonparametric multiple change point analysis of multivariate data." arXiv preprint
+    arXiv:1309.3295 (2013).
+
+    Examples
+    --------
+    >>> from sktime.annotation.datagen import piecewise_normal_multivariate
+    >>> X = piecewise_normal_multivariate(means=[[1, 3], [4, 5]], lengths=[3, 4],
+    ... random_state = 10)
+    >>> from sktime.annotation.eagglo import EAgglo
+    >>> model = EAgglo()
+    >>> model.fit_transform(X)
+    array([0, 0, 0, 1, 1, 1, 1])
+    """
+
+    _tags = {
+        "fit_is_empty": False,
+    }
+
+    def __init__(
+        self,
+        member=None,
+        alpha=1.0,
+        penalty=None,
+    ):
+        self.member = member
+        self.alpha = alpha
+        self.penalty = penalty
+        super(EAgglo, self).__init__()
+
+    def _fit(self, X: pd.DataFrame, y=None):
+        """Find optimally clustered segments.
+
+        First, by determining which pairs of adjacent clusters will be merged_. Then,
+        this process is repeated, recording the goodness-of-fit statistic at each step,
+        until all observations belong to a single cluster. Finally, the estimated number
+        of change points is estimated by the clustering that maximizes the goodness-of-
+        fit statistic over the entire merging sequence.
+
+        Parameters
+        ----------
+        X : pd.DataFrame
+            Data for anomaly detection (time series).
+        y : pd.Series, optional
+            Not used for this unsupervsed method.
+
+        Returns
+        -------
+        self :
+            Reference to self.
+        """
+        self._X = X
+
+        if self.alpha <= 0 or self.alpha > 2:
+            raise ValueError(f"alowed values for 'alpha' are (0, 2], got: {self.alpha}")
+
+        self._initialize_params(X)
+
+        # find which clusters optimize the gof_ and then update the distances
+        for K in range(self.n_cluster - 1, 2 * self.n_cluster - 2):
+            i, j = self._find_closest(K)
+            self._update_distances(i, j, K)
+
+        def filter_na(i):
+            return list(
+                filter(
+                    lambda v: v == v,
+                    self.progression[
+                        i,
+                    ],
+                )
+            )
+
+        # penalize the gof_ statistic
+        if self.penalty is not None:
+            penalty_func = self._get_penalty_func()
+            cps = [filter_na(i) for i in range(len(self.progression))]
+            self.gof_ += list(map(penalty_func, cps))
+
+        # get the set of change points for the "best" clustering
+        idx = np.argmax(self.gof_)
+        self._estimates = np.sort(filter_na(idx))
+
+        # remove change point N+1 if a cyclic merger was performed
+        self._estimates = (
+            self._estimates[:-1] if self._estimates[0] != 0 else self._estimates
+        )
+
+        # create final membership vector
+        def get_cluster(estimates):
+            return np.repeat(
+                range(len(np.diff(estimates))), np.diff(estimates).astype(int)
+            )
+
+        if self._estimates[0] == 0:
+            self.cluster_ = get_cluster(self._estimates)
+        else:
+            tmp = get_cluster(np.append([0], self._estimates))
+            self.cluster_ = np.append(tmp, np.zeros(X.shape[0] - len(tmp)))
+
+        return self
+
+    def _transform(self, X: pd.DataFrame, y=None):
+        """Transform X and return a transformed version.
+
+        private _transform containing core logic, called from transform
+
+        Parameters
+        ----------
+        X : Series of mtype X_inner_mtype
+            Data to be transformed
+        y : Series of mtype y_inner_mtype, default=None
+            Not required for this unsupervised transform.
+
+        Returns
+        -------
+        cluster
+            numeric representation of cluster membership for each row of X.
+        """
+        # fit again if indices not seen, but don't store anything
+        if not X.index.equals(self._X.index):
+            X_full = X.combine_first(self._X)
+            new_eagglo = EAgglo(
+                member=self.member,
+                alpha=self.alpha,
+                penalty=self.penalty,
+            ).fit(X_full)
+            warnings.warn(
+                "Warning: Input data X differs from that given to fit(). "
+                "Refitting with both the data in fit and new input data, not storing "
+                "updated public class attributes. For this, explicitly use fit(X) or "
+                "fit_transform(X)."
+            )
+            return new_eagglo.cluster_
+
+        return self.cluster_
+
+    def _initialize_params(self, X: pd.DataFrame) -> None:
+        """Initialize parameters and store to self."""
+        self._member = np.array(
+            self.member if self.member is not None else range(X.shape[0])
+        )
+
+        unique_labels = np.sort(np.unique(self._member))
+        self.n_cluster = len(unique_labels)
+
+        # relabel clusters to be consecutive numbers (when user specified)
+        for i in range(self.n_cluster):
+            self._member[np.where(self._member == unique_labels[i])[0]] = i
+
+        # check if sorted
+        if not all(sorted(self._member) == self._member):
+            raise ValueError("'_member' should be sorted")
+
+        self.sizes = np.zeros(2 * self.n_cluster)
+        self.sizes[: self.n_cluster] = [
+            sum(self._member == i) for i in range(self.n_cluster)
+        ]  # calculate initial cluster sizes
+
+        # array of within distances
+        grouped = X.copy().set_index(self._member).groupby(level=0)
+        within = grouped.apply(lambda x: get_distance(x, x, self.alpha))
+
+        # array of between-within distances
+        self.distances = np.empty((2 * self.n_cluster, 2 * self.n_cluster))
+
+        for i, xi in grouped:
+            self.distances[: self.n_cluster, i] = (
+                2 * grouped.apply(lambda xj: get_distance(xi, xj, self.alpha))  # noqa
+                - within[i]
+                - within
+            )
+
+        np.fill_diagonal(self.distances, 0)
+
+        # set up left and right neighbors
+        # special case for clusters 0 and n_cluster-1 to allow for cyclic merging
+        self.left = np.zeros(2 * self.n_cluster - 1, dtype=int)
+        self.left[: self.n_cluster] = [
+            i - 1 if i >= 1 else self.n_cluster - 1 for i in range(self.n_cluster)
+        ]
+
+        self.right = np.zeros(2 * self.n_cluster - 1, dtype=int)
+        self.right[: self.n_cluster] = [
+            i + 1 if i + 1 < self.n_cluster else 0 for i in range(self.n_cluster)
+        ]
+
+        # True means that a cluster has not been merged_
+        self.open = np.ones(2 * self.n_cluster - 1, dtype=bool)
+
+        # which clusters were merged_ at each step
+        self.merged_ = np.empty((self.n_cluster - 1, 2))
+
+        # set initial gof_ value
+        self.gof_ = np.array(
+            [
+                sum(
+                    self.distances[i, self.left[i]] + self.distances[i, self.right[i]]
+                    for i in range(self.n_cluster)
+                )
+            ]
+        )
+
+        # change point progression
+        self.progression = np.empty((self.n_cluster, self.n_cluster + 1))
+        self.progression[0, :] = [
+            sum(self.sizes[:i]) if i > 0 else 0 for i in range(self.n_cluster + 1)
+        ]  # N + 1 for cyclic mergers
+
+        # array to specify the starting point of a cluster
+        self.lm = np.zeros(2 * self.n_cluster - 1, dtype=int)
+        self.lm[: self.n_cluster] = range(self.n_cluster)
+
+    def _gof_update(self, i: int) -> float:
+        """Compute the updated goodness-of-fit statistic, left cluster given by i."""
+        fit = self.gof_[-1]
+        j = self.right[i]
+
+        # get new left and right clusters
+        rr = self.right[j]
+        ll = self.left[i]
+
+        # remove unneeded values in the gof_
+        fit -= 2 * (
+            self.distances[i, j] + self.distances[i, ll] + self.distances[j, rr]
+        )
+
+        # get cluster sizes
+        n1 = self.sizes[i]
+        n2 = self.sizes[j]
+
+        # add distance to new left cluster
+        n3 = self.sizes[ll]
+        k = (
+            (n1 + n3) * self.distances[i, ll]
+            + (n2 + n3) * self.distances[j, ll]
+            - n3 * self.distances[i, j]
+        ) / (n1 + n2 + n3)
+        fit += 2 * k
+
+        # add distance to new right
+        n3 = self.sizes[rr]
+        k = (
+            (n1 + n3) * self.distances[i, rr]
+            + (n2 + n3) * self.distances[j, rr]
+            - n3 * self.distances[i, j]
+        ) / (n1 + n2 + n3)
+        fit += 2 * k
+
+        return fit
+
+    def _find_closest(self, K: int) -> Tuple[int, int]:
+        """Determine which clusters will be merged_, for K clusters.
+
+        Greedily optimize the goodness-of-fit statistic by merging the pair of adjacent
+        clusters that results in the largest increase of the statistic's value.
+
+        Parameters
+        ----------
+        K: int
+            Number of clusters
+
+        Returns
+        -------
+        result : Tuple[int, int]
+            Tuple of left cluster and right cluster index values
+        """
+        best_fit = -1e10
+        result = (0, 0)
+
+        # iterate through each cluster to see how the gof_ value changes if merged_
+        for i in range(K + 1):
+            if self.open[i]:
+                gof_ = self._gof_update(i)
+                if gof_ > best_fit:
+                    best_fit = gof_
+                    result = (i, self.right[i])
+
+        self.gof_ = np.append(self.gof_, best_fit)
+        return result
+
+    def _update_distances(self, i: int, j: int, K: int) -> None:
+        """Update distance from new cluster to other clusters, store to self."""
+        # which clusters were merged_, info only
+        self.merged_[K - self.n_cluster + 1, 0] = (
+            -i if i <= self.n_cluster else i - self.n_cluster
+        )
+        self.merged_[K - self.n_cluster + 1, 1] = (
+            -j if j <= self.n_cluster else j - self.n_cluster
+        )
+
+        # update left and right neighbors
+        ll = self.left[i]
+        rr = self.right[j]
+        self.left[K + 1] = ll
+        self.right[K + 1] = rr
+        self.right[ll] = K + 1
+        self.left[rr] = K + 1
+
+        # update information about which clusters have been merged_
+        self.open[i] = False
+        self.open[j] = False
+
+        # assign size to newly created cluster
+        n1 = self.sizes[i]
+        n2 = self.sizes[j]
+        self.sizes[K + 1] = n1 + n2
+
+        # update set of change points
+        self.progression[K - self.n_cluster + 2, :] = self.progression[
+            K - self.n_cluster + 1,
+        ]
+        self.progression[K - self.n_cluster + 2, self.lm[j]] = np.nan
+        self.lm[K + 1] = self.lm[i]
+
+        # update distances
+        for k in range(K + 1):
+            if self.open[k]:
+                n3 = self.sizes[k]
+                n = n1 + n2 + n3
+                val = (
+                    (n - n2) * self.distances[i, k]
+                    + (n - n1) * self.distances[j, k]
+                    - n3 * self.distances[i, j]
+                ) / n
+                self.distances[K + 1, k] = val
+                self.distances[k, K + 1] = val
+
+    def _get_penalty_func(self) -> Callable:  # sourcery skip: raise-specific-error
+        """Define penalty function given (possibly string) input."""
+        PENALTIES = {"len_penalty": len_penalty, "mean_diff_penalty": mean_diff_penalty}
+
+        if callable(self.penalty):
+            return self.penalty
+
+        elif isinstance(self.penalty, str):
+            if self.penalty in PENALTIES:
+                return PENALTIES[self.penalty]
+
+        raise Exception(
+            f"'penalty' must be callable or {PENALTIES.keys()}, got {self.penalty}"
+        )
+
+    @classmethod
+    def get_test_params(cls) -> List[Dict]:
+        """Test parameters."""
+        return [
+            {"alpha": 1.0, "penalty": None},
+            {"alpha": 2.0, "penalty": "len_penalty"},
+        ]
+
+
+def get_distance(X: pd.DataFrame, Y: pd.DataFrame, alpha: float) -> float:
+    """Calculate within/between cluster distance."""
+    return np.power(cdist(X, Y, "euclidean"), alpha).mean()
+
+
+def len_penalty(x: pd.DataFrame) -> int:
+    """Penalize goodness-of-fit statistic for number of change points."""
+    return -len(x)
+
+
+def mean_diff_penalty(x: pd.DataFrame) -> float:
+    """Penalize goodness-of-fit statistic.
+
+    Favors segmentations with larger sizes, while taking into consideration
+    the size of the new segments.
+    """
+    return np.mean(np.diff(np.sort(x)))


Test patch:
diff --git a/sktime/annotation/tests/test_eagglo.py b/sktime/annotation/tests/test_eagglo.py
new file mode 100644
--- /dev/null
+++ b/sktime/annotation/tests/test_eagglo.py
@@ -0,0 +1,128 @@
+# -*- coding: utf-8 -*-
+"""Tests for E-Agglo (agglomerative clustering algorithm)."""
+
+__author__ = ["KatieBuc"]
+
+import numpy as np
+import pandas as pd
+
+from sktime.annotation.eagglo import EAgglo
+
+
+def test_fit_default_params_univariate():
+    """Test univariate data and defualt parameters.
+
+    These numbers are generated from the original implemention
+    in R, with the following code:
+
+    set.seed(1234)
+    X <- c(rnorm(15, mean = -6), 0, rnorm(16, mean = 6))
+    X <- as.matrix(X[c(1,2,17,18)])
+    ret = e.agglo(X)
+    """
+    X = pd.DataFrame([-7.207066, -5.722571, 5.889715, 5.488990])
+
+    cluster_expected = [0, 0, 1, 1]
+    fit_expected = [104.77424, 134.51387, 186.92586, -31.15431]
+
+    model = EAgglo()
+    fitted_model = model._fit(X)
+
+    cluster_actual = fitted_model.cluster_
+    fit_actual = fitted_model.gof_
+
+    assert np.allclose(cluster_actual, cluster_expected)
+    assert np.allclose(fit_actual, fit_expected)
+
+
+def test_fit_other_params_univariate():
+    """Test univariate data with alternative starting clusters."""
+    X = pd.DataFrame([-7.207066, -5.722571, 5.889715, 5.488990])
+
+    cluster_expected = [0, 0, 1, 1]
+    fit_expected = [1182.754, 1772.526, -295.421]
+
+    model = EAgglo(member=np.array([0, 0, 1, 2]), alpha=2)
+    fitted_model = model._fit(X)
+
+    cluster_actual = fitted_model.cluster_
+    fit_actual = fitted_model.gof_
+
+    assert np.allclose(cluster_actual, cluster_expected)
+    assert np.allclose(fit_actual, fit_expected)
+
+
+def test_fit_default_params_multivariate():
+    """Test multivariate data with defauly parameters.
+
+    These numbers are generated from the original implemention
+    in R, with the following code:
+
+    set.seed(1234)
+    X <- c(rnorm(15, mean = -6), 0, rnorm(16, mean = 6))
+    X <- as.matrix(cbind(X[c(1,2,17,18,19)],X[c(3,4,5,6,7)]))
+    ret = e.agglo(X)
+    """
+    X = pd.DataFrame(
+        [
+            [-7.207, -4.916],
+            [-5.723, -8.346],
+            [5.890, -5.571],
+            [5.489, -5.494],
+            [5.089, -6.575],
+        ]
+    )
+
+    cluster_expected = [0, 0, 1, 1, 1]
+    fit_expected = [118.58235, 132.67919, 156.54531, 208.61512, -33.52743]
+
+    model = EAgglo()
+    fitted_model = model._fit(X)
+
+    cluster_actual = fitted_model.cluster_
+    fit_actual = fitted_model.gof_
+
+    assert np.allclose(cluster_actual, cluster_expected)
+    assert np.allclose(fit_actual, fit_expected)
+
+
+def test_len_penalty():
+    """Test multivariate data with penalty function as string input."""
+    X = pd.DataFrame(
+        [
+            [-7.207, -4.916],
+            [-5.723, -8.346],
+            [5.890, -5.571],
+            [5.489, -5.494],
+            [5.089, -6.575],
+        ]
+    )
+
+    cluster_expected = [0, 0, 1, 1, 1]
+    fit_expected = [112.58235, 127.67919, 152.54531, 205.61512, -35.52743]
+
+    model = EAgglo(penalty="len_penalty")
+    fitted_model = model._fit(X)
+
+    cluster_actual = fitted_model.cluster_
+    fit_actual = fitted_model.gof_
+
+    assert np.allclose(cluster_actual, cluster_expected)
+    assert np.allclose(fit_actual, fit_expected)
+
+
+def test_custom_penalty():
+    """Test multivariate data with functional input as penalty."""
+    X = pd.DataFrame([-7.207066, -5.722571, 5.889715, 5.488990])
+
+    cluster_expected = [0, 0, 1, 1]
+    fit_expected = [105.77424, 135.84720, 188.92586, -29.15431]
+
+    model = EAgglo(penalty=lambda x: np.mean(np.diff(np.sort(x))))
+    fitted_model = model._fit(X)
+
+    cluster_actual = fitted_model.cluster_
+    fit_actual = fitted_model.gof_
+
+    assert np.allclose(cluster_actual, cluster_expected)
+    assert np.allclose(fit_actual, fit_expected)