Repository: jameschapman19/cca_zoo (language: python)

Issue:
Format read me
make the landing page fitting markdown syntax


Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index db39fb7a..557e9bb6 100644
--- a/README.md
+++ b/README.md
@@ -5,22 +5,35 @@
 [![version](https://img.shields.io/pypi/v/cca-zoo)](https://pypi.org/project/cca-zoo/)
 [![downloads](https://img.shields.io/pypi/dm/cca-zoo)](https://pypi.org/project/cca-zoo/)
 
-# Installation
-Note: for standard install use: 
-pip install cca-zoo
+# CCA zoo
+
+`cca-zoo` is a collection of linear, kernel, and deep methods for canonical correlation analysis of multiview data. 
+Where possible I have followed the `scikit-learn`/`mvlearn` APIs and models therefore have `fit`/`transform`/`fit_transform` methods as standard.
+
+## Installation
+
+Dependency of some implemented algorithms are heavy, such as `pytorch` and `numpyro`. 
+We provide several options to accomandate the user's needs.
+For full details of algoritms inclused, please refere to section [Implemented Methods](#implemented-methods)
+
+Standard installation: 
 
+```
+pip install cca-zoo
+```
 For deep learning elements use:
+```
 pip install cca-zoo[deep]
+```
 
 For probabilistic elements use:
+```
 pip install cca-zoo[probabilistic]
-
-This means that there is no need to install the large pytorch package or numpyro to run cca-zoo unless you wish to use deep learning
-
-# Documentation
+```
+## Documentation
 Available at https://cca-zoo.readthedocs.io/en/latest/
   
-# Citation:
+## Citation:
 If this repository was helpful to you please do give a star.
 
 In case this work is used as part of research I attach a DOI bibtex entry:
@@ -38,86 +51,73 @@ In case this work is used as part of research I attach a DOI bibtex entry:
 }
 ```
 
-# Implemented Methods
-## Standard Install
-### CCA (Canonical Correlation Analysis)
-Solutions based on either alternating least squares or as the solution to genrralized eigenvalue problem
-### PLS (Partial Least Squares)
-
-### rCCA (Ridge Regularized Canonical Correlation Analysis)
-https://www.sciencedirect.com/science/article/abs/pii/0304407676900105?via%3Dihub
-### GCCA (Generalized CCA)  :
-https://academic.oup.com/biomet/article-abstract/58/3/433/233349?redirectedFrom=fulltext
-### MCCA (Multiset CCA) : 
-### K(M)CCA (kernel Multiset CCA) : 
-### TCCA (Tensor CCA) :
-https://arxiv.org/pdf/1502.02330.pdf
-### KTCCA (kernel Tensor CCA) :
-https://arxiv.org/pdf/1502.02330.pdf
-### SCCA (Sparse CCA) :
-https://onlinelibrary.wiley.com/doi/abs/10.1111/biom.13043
-### SPLS (Sparse PLS/Penalized Matrix Decomposition)  :
-https://web.stanford.edu/~hastie/Papers/PMD_Witten.pdf
-### ElasticCCA (Penalized CCA) :
-https://pubmed.ncbi.nlm.nih.gov/19689958/
-### SWCCA (Sparse Weighted CCA) : 
-https://arxiv.org/abs/1710.04792v1#:~:text=However%2C%20classical%20and%20sparse%20CCA%20models%20consider%20the,where%20weights%20are%20used%20for%20regularizing%20different%20samples.
-### SpanCCA
-http://akyrillidis.github.io/pubs/Conferences/cca.pdf
-
-## Deep Install
-### DCCA (Deep CCA) : 
-https://ttic.uchicago.edu/~klivescu/papers/andrew_icml2013.pdf
-https://arxiv.org/pdf/1510.02054v1.pdf
-Using either Andrew's original Tracenorm Objective or Wang's alternating least squares solution
-### DGCCA (Deep Generalized CCA)  :
-https://www.aclweb.org/anthology/W19-4301.pdf
-An alternative objective based on the linear GCCA solution. Can be extended to more than 2 views
-### DMCCA (Deep Multiset CCA) :
-https://arxiv.org/abs/1904.01775
-An alternative objective based on the linear MCCA solution. Can be extended to more than 2 views
-### DTCCA (Deep Tensor CCA) :
-https://arxiv.org/pdf/2005.11914.pdf
-### DCCAE (Deep Canonically Correlated Autoencoders) :
-http://proceedings.mlr.press/v37/wangb15.pdf
-### DVCCA/DVCCA Private (Deep variational CCA):
-https://arxiv.org/pdf/1610.03454.pdf
-## Probabilistic Install
-### Variational Bayes CCA
-https://ieeexplore.ieee.org/document/4182407
-
-# Contributions
+## Implemented Methods
+
+### Standard Install
+- CCA (Canonical Correlation Analysis): Solutions based on either alternating least squares or as the solution to genrralized eigenvalue problem
+- PLS (Partial Least Squares)
+- [rCCA (Ridge Regularized Canonical Correlation Analysis)](https://www.sciencedirect.com/science/article/abs/pii/0304407676900105?via%3Dihub)
+- [GCCA (Generalized CCA)](https://academic.oup.com/biomet/article-abstract/58/3/433/233349?redirectedFrom=fulltext)
+- MCCA (Multiset CCA)
+- K(M)CCA (kernel Multiset CCA)
+- [TCCA (Tensor CCA)](https://arxiv.org/pdf/1502.02330.pdf)
+- [KTCCA (kernel Tensor CCA)](https://arxiv.org/pdf/1502.02330.pdf)
+- [SCCA (Sparse CCA)](https://onlinelibrary.wiley.com/doi/abs/10.1111/biom.13043)
+- [SPLS (Sparse PLS/Penalized Matrix Decomposition](https://web.stanford.edu/~hastie/Papers/PMD_Witten.pdf)
+- [ElasticCCA (Penalized CCA)](https://pubmed.ncbi.nlm.nih.gov/19689958/)
+- [SWCCA (Sparse Weighted CCA)](https://arxiv.org/abs/1710.04792v1#:~:text=However%2C%20classical%20and%20sparse%20CCA%20models%20consider%20the,where%20weights%20are%20used%20for%20regularizing%20different%20samples)
+- [SpanCCA](http://akyrillidis.github.io/pubs/Conferences/cca.pdf)
+
+### `[deep]` Install
+- DCCA (Deep CCA)
+
+  Using either Andrew's original [Tracenorm Objective](https://ttic.uchicago.edu/~klivescu/papers/andrew_icml2013.pdf) or Wang's [alternating least squares solution](https://arxiv.org/pdf/1510.02054v1.pdf)
+  
+- [DGCCA (Deep Generalized CCA)](https://www.aclweb.org/anthology/W19-4301.pdf)
+
+  An alternative objective based on the linear GCCA solution. Can be extended to more than 2 views
+ 
+- [DMCCA (Deep Multiset CCA)](https://arxiv.org/abs/1904.01775)
+
+  An alternative objective based on the linear MCCA solution. Can be extended to more than 2 views
+  
+- [DTCCA (Deep Tensor CCA)](https://arxiv.org/pdf/2005.11914.pdf)
+- [DCCAE (Deep Canonically Correlated Autoencoders)](http://proceedings.mlr.press/v37/wangb15.pdf)
+- [DVCCA/DVCCA Private (Deep variational CCA)](https://arxiv.org/pdf/1610.03454.pdf)
+
+### `[probabilistic]` Install
+- [Variational Bayes CCA](https://ieeexplore.ieee.org/document/4182407)
+
+## Contributions
 A guide to contributions is available at https://cca-zoo.readthedocs.io/en/latest/developer_info/contribute.html
 
-# Sources
+## Sources
 
 I've added this section to give due credit to the repositories that helped me in addition to their copyright notices in
 the code where relevant.
 
-Models can be tested on data from MNIST datasets provided by the torch package (https://pytorch.org/) and the UCI
-dataset provided by mvlearn package (https://mvlearn.github.io/)
+### Other Implementations of (regularised)CCA/PLS
 
-## Other Implementations of (regularised)CCA/PLS:
+[MATLAB implementation](https://github.com/anaston/PLS_CCA_framework)
 
-MATLAB implementation https://github.com/anaston/PLS_CCA_framework
+### Implementation of Sparse PLS
 
-## Implementation of Sparse PLS:
+MATLAB implementation of SPLS by [@jmmonteiro](https://github.com/jmmonteiro/spls)
 
-MATLAB implementation of SPLS by @jmmonteiro (https://github.com/jmmonteiro/spls)
+### Other Implementations of DCCA/DCCAE
 
-## Other Implementations of DCCA/DCCAE:
+Keras implementation of DCCA from [@VahidooX's github page](https://github.com/VahidooX)
 
-Keras implementation of DCCA from @VahidooX's github page(https://github.com/VahidooX)
 The following are the other implementations of DCCA in MATLAB and C++. These codes are written by the authors of the original paper:
 
-Torch implementation of DCCA from @MichaelVll & @Arminarj: https://github.com/Michaelvll/DeepCCA
+[Torch implementation](https://github.com/Michaelvll/DeepCCA) of DCCA from @MichaelVll & @Arminarj
 
-C++ implementation of DCCA from Galen Andrew's website (https://homes.cs.washington.edu/~galen/)
+C++ implementation of DCCA from Galen Andrew's [website](https://homes.cs.washington.edu/~galen/)
 
-MATLAB implementation of DCCA/DCCAE from Weiran Wang's website (http://ttic.uchicago.edu/~wwang5/dccae.html)
+MATLAB implementation of DCCA/DCCAE from Weiran Wang's [website](http://ttic.uchicago.edu/~wwang5/dccae.html)
 
-MATLAB implementation of TCCA from https://github.com/rciszek/mdr_tcca
+MATLAB implementation of [TCCA](https://github.com/rciszek/mdr_tcca)
 
-## Implementation of VAE:
+### Implementation of VAE
 
-Torch implementation of VAE (https://github.com/pytorch/examples/tree/master/vae)
+[Torch implementation of VAE](https://github.com/pytorch/examples/tree/master/vae)
diff --git a/cca_zoo/__init__.py b/cca_zoo/__init__.py
index 64fba86c..ca19c40e 100644
--- a/cca_zoo/__init__.py
+++ b/cca_zoo/__init__.py
@@ -1,6 +1,6 @@
 import cca_zoo.data
 import cca_zoo.models
-
+import cca_zoo.model_selection
 try:
     import cca_zoo.deepmodels
 except ModuleNotFoundError:
diff --git a/cca_zoo/data/simulated.py b/cca_zoo/data/simulated.py
index 12c27e76..ba2b1850 100644
--- a/cca_zoo/data/simulated.py
+++ b/cca_zoo/data/simulated.py
@@ -10,16 +10,16 @@
 
 
 def generate_covariance_data(
-        n: int,
-        view_features: List[int],
-        latent_dims: int = 1,
-        view_sparsity: List[Union[int, float]] = None,
-        correlation: Union[List[float], float] = 1,
-        structure: Union[str, List[str]] = None,
-        sigma: Union[List[float], float] = None,
-        decay: float = 0.5,
-        positive=None,
-        random_state: Union[int, np.random.RandomState] = None,
+    n: int,
+    view_features: List[int],
+    latent_dims: int = 1,
+    view_sparsity: List[Union[int, float]] = None,
+    correlation: Union[List[float], float] = 1,
+    structure: Union[str, List[str]] = None,
+    sigma: Union[List[float], float] = None,
+    decay: float = 0.5,
+    positive=None,
+    random_state: Union[int, np.random.RandomState] = None,
 ):
     """
     Function to generate CCA dataset with defined population correlations
@@ -59,7 +59,7 @@ def generate_covariance_data(
             covs = []
             true_features = []
             for view_p, sparsity, view_structure, view_positive, view_sigma in zip(
-                    view_features, view_sparsity, structure, positive, sigma
+                view_features, view_sparsity, structure, positive, sigma
             ):
                 # Covariance Bit
                 if view_structure == "identity":
@@ -111,12 +111,12 @@ def generate_covariance_data(
                     # Cross Bit
                     cross += covs[i] @ A @ covs[j]
                 cov[
-                splits[i]: splits[i] + view_features[i],
-                splits[j]: splits[j] + view_features[j],
+                    splits[i] : splits[i] + view_features[i],
+                    splits[j] : splits[j] + view_features[j],
                 ] = cross
                 cov[
-                splits[j]: splits[j] + view_features[j],
-                splits[i]: splits[i] + view_features[i],
+                    splits[j] : splits[j] + view_features[j],
+                    splits[i] : splits[i] + view_features[i],
                 ] = cross.T
 
             X = np.zeros((n, sum(view_features)))
@@ -129,17 +129,19 @@ def generate_covariance_data(
             completed = False
             attempts += 1
     if not completed:
-        raise ValueError('Could not generate positive definite covariance matrix for supplied parameters.')
+        raise ValueError(
+            "Could not generate positive definite covariance matrix for supplied parameters."
+        )
     return views, true_features
 
 
 def generate_simple_data(
-        n: int,
-        view_features: List[int],
-        view_sparsity: List[Union[int, float]] = None,
-        eps: float = 0,
-        transform=False,
-        random_state=None,
+    n: int,
+    view_features: List[int],
+    view_sparsity: List[Union[int, float]] = None,
+    eps: float = 0,
+    transform=False,
+    random_state=None,
 ):
     """
     Simple latent variable model to generate data with one latent factor
@@ -200,9 +202,9 @@ def _gaussian(x, mu, sig, dn):
     :param dn:
     """
     return (
-            np.exp(-np.power(x - mu, 2.0) / (2 * np.power(sig, 2.0)))
-            * dn
-            / (np.sqrt(2 * np.pi) * sig)
+        np.exp(-np.power(x - mu, 2.0) / (2 * np.power(sig, 2.0)))
+        * dn
+        / (np.sqrt(2 * np.pi) * sig)
     )
 
 
diff --git a/cca_zoo/deepmodels/dcca.py b/cca_zoo/deepmodels/dcca.py
index c61cc86d..64cc9f6b 100644
--- a/cca_zoo/deepmodels/dcca.py
+++ b/cca_zoo/deepmodels/dcca.py
@@ -22,7 +22,7 @@ def __init__(
         objective=objectives.MCCA,
         encoders=None,
         r: float = 0,
-        eps: float = 1e-3,
+        eps: float = 1e-5,
     ):
         """
         Constructor class for DCCA
diff --git a/cca_zoo/deepmodels/dcca_barlow_twins.py b/cca_zoo/deepmodels/dcca_barlow_twins.py
index d1e16036..e05dd2dc 100644
--- a/cca_zoo/deepmodels/dcca_barlow_twins.py
+++ b/cca_zoo/deepmodels/dcca_barlow_twins.py
@@ -44,8 +44,8 @@ def forward(self, *args):
     def loss(self, *args):
         z = self(*args)
         cross_cov = z[0].T @ z[1] / (z[0].shape[0] - 1)
-        invariance = torch.mean(torch.pow(1 - torch.diag(cross_cov), 2))
-        covariance = torch.mean(
+        invariance = torch.sum(torch.pow(1 - torch.diag(cross_cov), 2))
+        covariance = torch.sum(
             torch.triu(torch.pow(cross_cov, 2), diagonal=1)
-        ) + torch.mean(torch.tril(torch.pow(cross_cov, 2), diagonal=-1))
+        ) + torch.sum(torch.tril(torch.pow(cross_cov, 2), diagonal=-1))
         return invariance + covariance
diff --git a/cca_zoo/deepmodels/dcca_noi.py b/cca_zoo/deepmodels/dcca_noi.py
index 9b8ac318..b2c6968e 100644
--- a/cca_zoo/deepmodels/dcca_noi.py
+++ b/cca_zoo/deepmodels/dcca_noi.py
@@ -1,7 +1,7 @@
 import torch
 
-from cca_zoo.deepmodels import objectives
 from cca_zoo.deepmodels.dcca import DCCA
+from cca_zoo.deepmodels.objectives import mat_pow
 
 
 class DCCA_NOI(DCCA):
@@ -68,7 +68,7 @@ def loss(self, *args):
         z_copy = [z_.detach().clone() for z_ in z]
         self._update_covariances(*z_copy)
         covariance_inv = [
-            torch.linalg.inv(objectives.MatrixSquareRoot.apply(cov))
+            mat_pow(cov,-0.5,self.eps)
             for cov in self.covs
         ]
         preds = [z_ @ covariance_inv[i] for i, z_ in enumerate(z_copy)]
diff --git a/cca_zoo/deepmodels/dcca_sdl.py b/cca_zoo/deepmodels/dcca_sdl.py
index de5676ee..d2a1fdc6 100644
--- a/cca_zoo/deepmodels/dcca_sdl.py
+++ b/cca_zoo/deepmodels/dcca_sdl.py
@@ -21,7 +21,7 @@ def __init__(
         encoders=None,
         r: float = 0,
         rho: float = 0.2,
-        eps: float = 1e-3,
+        eps: float = 1e-5,
         shared_target: bool = False,
         lam=0.5,
     ):
diff --git a/cca_zoo/deepmodels/dccae.py b/cca_zoo/deepmodels/dccae.py
index 054465a7..ab17dc3b 100644
--- a/cca_zoo/deepmodels/dccae.py
+++ b/cca_zoo/deepmodels/dccae.py
@@ -23,7 +23,7 @@ def __init__(
         encoders=None,
         decoders=None,
         r: float = 0,
-        eps: float = 1e-3,
+        eps: float = 1e-5,
         lam=0.5,
     ):
         """
diff --git a/cca_zoo/deepmodels/dtcca.py b/cca_zoo/deepmodels/dtcca.py
index 419d5760..b42bb445 100644
--- a/cca_zoo/deepmodels/dtcca.py
+++ b/cca_zoo/deepmodels/dtcca.py
@@ -20,7 +20,7 @@ class DTCCA(DCCA):
     """
 
     def __init__(
-        self, latent_dims: int, encoders=None, r: float = 0, eps: float = 1e-3
+        self, latent_dims: int, encoders=None, r: float = 0, eps: float = 1e-5
     ):
         """
 
diff --git a/cca_zoo/deepmodels/objectives.py b/cca_zoo/deepmodels/objectives.py
index 0fb08caf..be1b7734 100644
--- a/cca_zoo/deepmodels/objectives.py
+++ b/cca_zoo/deepmodels/objectives.py
@@ -4,44 +4,15 @@
 import torch
 from tensorly.cp_tensor import cp_to_tensor
 from tensorly.decomposition import parafac
-from torch.autograd import Function
+from torch import diag
 
 
-class MatrixSquareRoot(Function):
-    """Square root of a positive definite matrix.
-    NOTE: matrix square root is not differentiable for matrices with
-          zero eigenvalues.
-    """
-
-    @staticmethod
-    def forward(ctx, input):
-        m = input.detach().cpu().numpy().astype(np.float_)
-        sqrtm = torch.from_numpy(scipy.linalg.sqrtm(m).real).to(input)
-        ctx.save_for_backward(sqrtm)
-        return sqrtm
-
-    @staticmethod
-    def backward(ctx, grad_output):
-        grad_input = None
-        if ctx.needs_input_grad[0]:
-            (sqrtm,) = ctx.saved_tensors
-            sqrtm = sqrtm.data.cpu().numpy().astype(np.float_)
-            gm = grad_output.data.cpu().numpy().astype(np.float_)
-
-            # Given a positive semi-definite matrix X,
-            # since X = X^{1/2}X^{1/2}, we can compute the gradient of the
-            # matrix square root dX^{1/2} by solving the Sylvester equation:
-            # dX = (d(X^{1/2})X^{1/2} + X^{1/2}(dX^{1/2}).
-            grad_sqrtm = scipy.linalg.solve_sylvester(sqrtm, sqrtm, gm)
-
-            grad_input = torch.from_numpy(grad_sqrtm).to(grad_output)
-        return grad_input
-
-
-def _minimal_regularisation(M, eps):
-    M_smallest_eig = torch.relu(-torch.min(torch.linalg.eigvalsh(M))) + eps
-    M = M + M_smallest_eig * torch.eye(M.shape[0], device=M.device)
-    return M
+def mat_pow(mat, pow_, epsilon):
+    # Computing matrix to the power of pow (pow can be negative as well)
+    [D, V] = torch.linalg.eigh(mat)
+    mat_pow = V @ diag((D + epsilon).pow(pow_)) @ V.T
+    mat_pow[mat_pow != mat_pow] = epsilon  # For stability
+    return mat_pow
 
 
 def _demean(*views):
@@ -87,11 +58,7 @@ def loss(self, *views):
 
         C = C - torch.block_diag(*[view.T @ view for view in views]) + D
 
-        D = _minimal_regularisation(D, self.eps)
-
-        R = torch.linalg.inv(
-            MatrixSquareRoot.apply(_minimal_regularisation(D, self.eps))
-        )
+        R = mat_pow(D, -0.5, self.eps)
 
         # In MCCA our eigenvalue problem Cv = lambda Dv
         C_whitened = R @ C @ R.T
@@ -139,7 +106,7 @@ def loss(self, *views):
 
         eigen_views = [
             view
-            @ torch.inverse(_minimal_regularisation(view.T @ view, self.eps))
+            @ mat_pow(view.T @ view, -1, self.eps)
             @ view.T
             for view in views
         ]
@@ -199,18 +166,14 @@ def loss(self, H1, H2):
 
         SigmaHat12 = (1.0 / (n - 1)) * H1bar.T @ H2bar
         SigmaHat11 = (1 - self.r) * (
-            1.0 / (n - 1)
+                1.0 / (n - 1)
         ) * H1bar.T @ H1bar + self.r * torch.eye(o1, device=H1.device)
         SigmaHat22 = (1 - self.r) * (
-            1.0 / (n - 1)
+                1.0 / (n - 1)
         ) * H2bar.T @ H2bar + self.r * torch.eye(o2, device=H2.device)
 
-        SigmaHat11RootInv = torch.linalg.inv(
-            MatrixSquareRoot.apply(_minimal_regularisation(SigmaHat11, self.eps))
-        )
-        SigmaHat22RootInv = torch.linalg.inv(
-            MatrixSquareRoot.apply(_minimal_regularisation(SigmaHat22, self.eps))
-        )
+        SigmaHat11RootInv = mat_pow(SigmaHat11, -0.5, self.eps)
+        SigmaHat22RootInv = mat_pow(SigmaHat22, -0.5, self.eps)
 
         Tval = SigmaHat11RootInv @ SigmaHat12 @ SigmaHat22RootInv
         trace_TT = Tval.T @ Tval
@@ -250,13 +213,7 @@ def loss(self, *z):
             + self.r * torch.eye(z_.size(1), device=z_.device)
             for z_ in z
         ]
-        whitened_z = [
-            z_
-            @ torch.linalg.inv(
-                MatrixSquareRoot.apply(_minimal_regularisation(cov, self.eps))
-            )
-            for z_, cov in zip(z, covs)
-        ]
+        whitened_z = [z_ @ mat_pow(cov, -0.5, self.eps) for z_, cov in zip(z, covs)]
         # The idea here is to form a matrix with M dimensions one for each view where at index
         # M[p_i,p_j,p_k...] we have the sum over n samples of the product of the pth feature of the
         # ith, jth, kth view etc.
diff --git a/cca_zoo/deepmodels/trainers.py b/cca_zoo/deepmodels/trainers.py
index 328e1cfb..934dae4e 100644
--- a/cca_zoo/deepmodels/trainers.py
+++ b/cca_zoo/deepmodels/trainers.py
@@ -27,13 +27,10 @@ def __init__(
         self.model = model
         self.sanity_check = True
 
-    def forward(self, *args):
-        z = self.encode(*args)
+    def forward(self, *args, **kwargs):
+        z = self.model(*args, **kwargs)
         return z
 
-    def loss(self, *args, **kwargs):
-        return self.model.loss(*args, **kwargs)
-
     def configure_optimizers(self):
         if isinstance(self.hparams.optimizer, torch.optim.Optimizer):
             optimizer = self.hparams.optimizer
@@ -52,16 +49,19 @@ def configure_optimizers(self):
     def training_step(self, batch, batch_idx):
         data, label = batch
         loss = self.model.loss(*data)
+        self.log("train loss", loss)
         return loss
 
     def validation_step(self, batch, batch_idx):
         data, label = batch
         loss = self.model.loss(*data)
+        self.log("val loss", loss)
         return loss
 
     def test_step(self, batch, batch_idx):
         data, label = batch
         loss = self.model.loss(*data)
+        self.log("test loss", loss)
         return loss
 
     def on_train_epoch_end(self, unused: Optional = None) -> None:
diff --git a/cca_zoo/models/_cca_base.py b/cca_zoo/models/_cca_base.py
index 4c3437d6..a7328690 100644
--- a/cca_zoo/models/_cca_base.py
+++ b/cca_zoo/models/_cca_base.py
@@ -84,7 +84,7 @@ def fit_transform(self, views: Iterable[np.ndarray], y=None, **kwargs):
         :param views: list/tuple of numpy arrays or array likes with the same number of rows (samples)
         :param kwargs: any additional keyword arguments required by the given model
         """
-        return self.fit(views, **kwargs).transform(views)
+        return self.fit(views, **kwargs).transform(views, **kwargs)
 
     def get_loadings(self, views: Iterable[np.ndarray], y=None, **kwargs):
         """
diff --git a/cca_zoo/models/_innerloop.py b/cca_zoo/models/_innerloop.py
index 1ceec0f0..99c86866 100644
--- a/cca_zoo/models/_innerloop.py
+++ b/cca_zoo/models/_innerloop.py
@@ -4,10 +4,7 @@
 
 import numpy as np
 from sklearn.exceptions import ConvergenceWarning
-from sklearn.linear_model import (
-    ElasticNet,
-    SGDRegressor,
-)
+from sklearn.linear_model import ElasticNet, SGDRegressor, LinearRegression, Ridge
 from sklearn.utils._testing import ignore_warnings
 from sklearn.utils.validation import check_random_state
 
@@ -276,6 +273,14 @@ def _check_params(self):
                         random_state=self.random_state,
                     )
                 )
+            elif alpha == 0:
+                self.regressors.append(
+                    Ridge(
+                        alpha=self.tol,
+                        fit_intercept=False,
+                        positive=positive,
+                    )
+                )
             else:
                 self.regressors.append(
                     ElasticNet(
diff --git a/cca_zoo/models/iterative.py b/cca_zoo/models/iterative.py
index 321885ed..367c607f 100644
--- a/cca_zoo/models/iterative.py
+++ b/cca_zoo/models/iterative.py
@@ -336,11 +336,13 @@ def _set_loop_params(self):
 
 class CCA_ALS(ElasticCCA):
     r"""
-    Fits a CCA model with CCA deflation by NIPALS algorithm. Implemented by ElasticCCA with 0 regularisation
+    Fits a CCA model with CCA deflation by NIPALS algorithm. Implemented by ElasticCCA with no regularisation
+
+    :Maths:
 
     .. math::
 
-        w_{opt}=\underset{w}{\mathrm{argmax}}\{\sum_i\sum_{j\neq i} \|X_iw_i-X_jw_j\|^2 }\\
+        w_{opt}=\underset{w}{\mathrm{argmax}}\{\sum_i\sum_{j\neq i} \|X_iw_i-X_jw_j\|^2\}\\
 
         \text{subject to:}
 
@@ -403,7 +405,6 @@ def __init__(
             positive=positive,
             random_state=random_state,
             deflation=deflation,
-            c=1e-5,
             maxvar=False,
         )
 
@@ -412,6 +413,8 @@ class SCCA(ElasticCCA):
     r"""
     Fits a sparse CCA model by iterative rescaled lasso regression. Implemented by ElasticCCA with l1 ratio=1
 
+    :Maths:
+
     .. math::
 
         w_{opt}=\underset{w}{\mathrm{argmax}}\{\sum_i\sum_{j\neq i} \|X_iw_i-X_jw_j\|^2 + \text{l1_ratio}\|w_i\|_1\}\\
@@ -490,6 +493,8 @@ class PMD(_Iterative):
     r"""
     Fits a Sparse CCA (Penalized Matrix Decomposition) model.
 
+    :Maths:
+
     .. math::
 
         w_{opt}=\underset{w}{\mathrm{argmax}}\{ w_1^TX_1^TX_2w_2  \}\\
diff --git a/cca_zoo/models/partialcca.py b/cca_zoo/models/partialcca.py
index 851164be..173768ed 100644
--- a/cca_zoo/models/partialcca.py
+++ b/cca_zoo/models/partialcca.py
@@ -9,7 +9,24 @@
 
 
 class PartialCCA(MCCA):
-    r""" """
+    r"""
+    A class used to fit a partial cca model. The key difference between this and a vanilla CCA or MCCA is that
+    the canonical score vectors must be orthogonal to the supplied confounding variables.
+
+    :Citation:
+
+    Rao, B. Raja. "Partial canonical correlations." Trabajos de estadistica y de investigación operativa 20.2-3 (1969): 211-219.
+
+    :Example:
+    >>> from cca_zoo.models import PartialCCA
+    >>> X1 = np.random.rand(10,5)
+    >>> X2 = np.random.rand(10,5)
+    >>> partials = np.random.rand(10,3)
+    >>> model = PartialCCA()
+    >>> model.fit((X1,X2),partials=partials).score((X1,X2),partials=partials)
+    array([0.99993046])
+
+    """
 
     def __init__(
         self,
@@ -42,15 +59,15 @@ def __init__(
         self.c = c
         self.eps = eps
 
-    def _setup_evp(self, views: Iterable[np.ndarray], confounds=None):
-        if confounds is None:
+    def _setup_evp(self, views: Iterable[np.ndarray], partials=None):
+        if partials is None:
             raise ValueError(
-                f"confounds is {confounds}. Require matching confounds to transform with"
+                f"partials is {partials}. Require matching partials to transform with"
                 f"partial CCA."
             )
-        self.confound_betas = [np.linalg.pinv(confounds) @ view for view in views]
+        self.confound_betas = [np.linalg.pinv(partials) @ view for view in views]
         views = [
-            view - confounds @ np.linalg.pinv(confounds) @ view
+            view - partials @ np.linalg.pinv(partials) @ view
             for view, confound_beta in zip(views, self.confound_betas)
         ]
         all_views = np.concatenate(views, axis=1)
@@ -69,15 +86,15 @@ def _setup_evp(self, views: Iterable[np.ndarray], confounds=None):
         return views, C, D
 
     # TODO TRANSFORM
-    def transform(self, views: Iterable[np.ndarray], y=None, confounds=None):
+    def transform(self, views: Iterable[np.ndarray], y=None, partials=None):
         """
         Transforms data given a fit model
 
         :param views: numpy arrays with the same number of rows (samples) separated by commas
         """
-        if confounds is None:
+        if partials is None:
             raise ValueError(
-                f"confounds is {confounds}. Require matching confounds to transform with"
+                f"partials is {partials}. Require matching partials to transform with"
                 f"partial CCA."
             )
         check_is_fitted(self, attributes=["weights"])
@@ -88,43 +105,7 @@ def transform(self, views: Iterable[np.ndarray], y=None, confounds=None):
         transformed_views = []
         for i, (view) in enumerate(views):
             transformed_view = (
-                view - confounds @ self.confound_betas[i]
+                view - partials @ self.confound_betas[i]
             ) @ self.weights[i]
             transformed_views.append(transformed_view)
-        return transformed_views
-
-
-def main():
-    import numpy as np
-
-    n = 100
-    p = 10
-    q = 10
-    Z = np.random.rand(100, 1)
-    Z -= Z.mean(axis=0)
-    C = np.random.rand(100, 1)
-    C -= C.mean(axis=0)
-    Wx = np.random.rand(1, p)
-    Wxc = np.random.rand(1, p)
-    Wy = np.random.rand(1, q)
-    Wyc = np.random.rand(1, q)
-    X = Z @ Wx + C @ Wxc + 2 * np.random.rand(n, p)
-    Y = Z @ Wy + C @ Wyc + 2 * np.random.rand(n, q)
-    pcca = PartialCCA(centre=False, scale=False).fit((X, Y), confounds=C)
-    a = pcca.score((X, Y), confounds=C)
-    from cca_zoo.models import MCCA
-
-    cca = MCCA(centre=False, scale=False).fit((X, Y), confounds=C)
-    b = cca.score((X, Y))
-    bb = np.corrcoef(cca.transform((X, Y))[0], C, rowvar=False)
-    X_deconf = (np.eye(n) - C @ np.linalg.inv(C.T @ C) @ C.T) @ X
-    Y_deconf = (np.eye(n) - C @ np.linalg.inv(C.T @ C) @ C.T) @ Y
-    m = MCCA(centre=False, scale=False).fit((X_deconf, Y_deconf))
-    c = m.score((X_deconf, Y_deconf))
-    cc = np.corrcoef(m.transform((X_deconf, Y_deconf))[0], C, rowvar=False)
-    aa = np.corrcoef(pcca.transform((X, Y), confounds=C)[0], C, rowvar=False)
-    print()
-
-
-if __name__ == "__main__":
-    main()
+        return transformed_views
\ No newline at end of file
diff --git a/docs/source/api/models.rst b/docs/source/api/models.rst
index 350803c2..ba4b444b 100644
--- a/docs/source/api/models.rst
+++ b/docs/source/api/models.rst
@@ -129,10 +129,22 @@ Sparse CCA by ADMM
     :exclude-members: get_params, set_params
 
 Miscellaneous
+-----------------------------------------------------
+
+Nonparametric CCA
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. autoclass:: cca_zoo.models.NCCA
+    :inherited-members:
+    :exclude-members: get_params, set_params
+
+Partial CCA
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. autoclass:: cca_zoo.models.PartialCCA
+    :inherited-members:
+    :exclude-members: get_params, set_params
 
 Sparse Weighted CCA
-"""""""""""""""""""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 .. autoclass:: cca_zoo.models.SWCCA
     :inherited-members:
     :exclude-members: get_params, set_params
diff --git a/requirements/basic.txt b/requirements/basic.txt
index 4927c91c..768e4e35 100644
--- a/requirements/basic.txt
+++ b/requirements/basic.txt
@@ -1,5 +1,5 @@
 numpy
-scikit-learn>=0.23
+scikit-learn>=1.0
 mvlearn
 scipy>=1.7
 matplotlib


Test patch:
diff --git a/cca_zoo/test/test_deepmodels.py b/cca_zoo/test/test_deepmodels.py
index e7ea258a..9c30b812 100644
--- a/cca_zoo/test/test_deepmodels.py
+++ b/cca_zoo/test/test_deepmodels.py
@@ -46,7 +46,7 @@ def test_DCCA_methods():
     encoder_1 = architectures.Encoder(latent_dims=latent_dims, feature_size=10)
     encoder_2 = architectures.Encoder(latent_dims=latent_dims, feature_size=12)
     dcca_noi = DCCA_NOI(latent_dims, N, encoders=[encoder_1, encoder_2], rho=0)
-    optimizer = optim.Adam(dcca_noi.parameters(), lr=1e-3)
+    optimizer = optim.Adam(dcca_noi.parameters(), lr=1e-2)
     dcca_noi = CCALightning(dcca_noi, optimizer=optimizer)
     trainer = pl.Trainer(
         max_epochs=epochs, log_every_n_steps=10, enable_checkpointing=False
@@ -80,7 +80,7 @@ def test_DCCA_methods():
         encoders=[encoder_1, encoder_2],
         objective=objectives.CCA,
     )
-    optimizer = optim.SGD(dcca.parameters(), lr=1e-2)
+    optimizer = optim.SGD(dcca.parameters(), lr=1e-1)
     dcca = CCALightning(dcca, optimizer=optimizer)
     trainer = pl.Trainer(
         max_epochs=epochs, log_every_n_steps=10, enable_checkpointing=False
diff --git a/cca_zoo/test/test_models.py b/cca_zoo/test/test_models.py
index 8defef34..d8261860 100644
--- a/cca_zoo/test/test_models.py
+++ b/cca_zoo/test/test_models.py
@@ -43,8 +43,9 @@ def test_unregularized_methods():
     # Tests unregularized CCA methods. The idea is that all of these should give the same result.
     latent_dims = 2
     cca = CCA(latent_dims=latent_dims).fit([X, Y])
-    iter = CCA_ALS(latent_dims=latent_dims, tol=1e-9, stochastic=False).fit([X, Y])
-    iter_pls = PLS_ALS(latent_dims=latent_dims, tol=1e-9, centre=False).fit([X, Y])
+    iter = CCA_ALS(
+        latent_dims=latent_dims, tol=1e-9, stochastic=False, random_state=rng
+    ).fit([X, Y])
     gcca = GCCA(latent_dims=latent_dims).fit([X, Y])
     mcca = MCCA(latent_dims=latent_dims, eps=1e-9).fit([X, Y])
     kcca = KCCA(latent_dims=latent_dims).fit([X, Y])
@@ -57,7 +58,6 @@ def test_unregularized_methods():
     corr_kcca = kcca.score((X, Y))
     corr_kgcca = kgcca.score((X, Y))
     corr_tcca = tcca.score((X, Y))
-    # Check the correlations from each unregularized method are the same
     assert np.testing.assert_array_almost_equal(corr_cca, corr_iter, decimal=1) is None
     assert np.testing.assert_array_almost_equal(corr_cca, corr_mcca, decimal=1) is None
     assert np.testing.assert_array_almost_equal(corr_cca, corr_gcca, decimal=1) is None
@@ -122,7 +122,11 @@ def test_sparse_input():
     latent_dims = 2
     cca = CCA(latent_dims=latent_dims, centre=False).fit((X_sp, Y_sp))
     iter = CCA_ALS(
-        latent_dims=latent_dims, tol=1e-9, stochastic=False, centre=False
+        latent_dims=latent_dims,
+        tol=1e-9,
+        stochastic=False,
+        centre=False,
+        random_state=rng,
     ).fit((X_sp, Y_sp))
     iter_pls = PLS_ALS(latent_dims=latent_dims, tol=1e-9, centre=False).fit(
         (X_sp, Y_sp)
@@ -146,7 +150,9 @@ def test_unregularized_multi():
     # Tests unregularized CCA methods for more than 2 views. The idea is that all of these should give the same result.
     latent_dims = 2
     cca = rCCA(latent_dims=latent_dims).fit((X, Y, Z))
-    iter = CCA_ALS(latent_dims=latent_dims, stochastic=False, tol=1e-12).fit((X, Y, Z))
+    iter = CCA_ALS(
+        latent_dims=latent_dims, stochastic=False, tol=1e-12, random_state=rng
+    ).fit((X, Y, Z))
     gcca = GCCA(latent_dims=latent_dims).fit((X, Y, Z))
     mcca = MCCA(latent_dims=latent_dims).fit((X, Y, Z))
     kcca = KCCA(latent_dims=latent_dims).fit((X, Y, Z))
@@ -193,7 +199,7 @@ def test_non_negative_methods():
         tol=1e-9,
         positive=True,
         c=[1e-6, 1e-6],
-        random_state=0,
+        random_state=rng,
         initialization="random",
     ).fit((X, Y))
     nnelastic = ElasticCCA(
@@ -202,12 +208,12 @@ def test_non_negative_methods():
         positive=True,
         l1_ratio=[0.5, 0.5],
         c=[1e-5, 1e-5],
-        random_state=0,
+        random_state=rng,
         initialization="random",
-    ).fit([X, Y])
+    ).fit((X, Y))
     nnals = CCA_ALS(
-        latent_dims=latent_dims, tol=1e-9, positive=True, random_state=0
-    ).fit([X, Y])
+        latent_dims=latent_dims, tol=1e-9, positive=True, random_state=rng, initialization='pls'
+    ).fit((X, Y))
     assert np.testing.assert_array_less(-1e-9, nnelastic.weights[0]) is None
     assert np.testing.assert_array_less(-1e-9, nnelastic.weights[1]) is None
     assert np.testing.assert_array_less(-1e-9, nnals.weights[0]) is None
@@ -296,8 +302,12 @@ def test_NCCA():
 
 
 def test_l0():
-    span_cca = SpanCCA(latent_dims=1, regularisation="l0", c=[2, 2]).fit([X, Y])
-    swcca = SWCCA(latent_dims=1, c=[2, 2], sample_support=5).fit([X, Y])
+    span_cca = SpanCCA(
+        latent_dims=1, regularisation="l0", c=[2, 2], random_state=rng
+    ).fit([X, Y])
+    swcca = SWCCA(latent_dims=1, c=[2, 2], sample_support=5, random_state=rng).fit(
+        [X, Y]
+    )
     assert (np.abs(span_cca.weights[0]) > 1e-5).sum() == 2
     assert (np.abs(span_cca.weights[1]) > 1e-5).sum() == 2
     assert (np.abs(swcca.weights[0]) > 1e-5).sum() == 2
@@ -306,11 +316,11 @@ def test_l0():
 
 
 def test_partialcca():
-    # Tests that partial CCA scores are not correlated with confounds.
+    # Tests that partial CCA scores are not correlated with partials
     pcca = PartialCCA(latent_dims=3)
-    pcca.fit((X, Y), confounds=Z)
+    pcca.fit((X, Y), partials=Z)
     assert np.allclose(
-        np.corrcoef(pcca.transform((X, Y), confounds=Z)[0], Z, rowvar=False)[:3, :3]
+        np.corrcoef(pcca.transform((X, Y), partials=Z)[0], Z, rowvar=False)[:3, :3]
         - np.eye(3),
         0,
         atol=0.001,
@@ -330,17 +340,22 @@ def test_VCCA():
     numpyro = pytest.importorskip("numpyro")
     from cca_zoo.probabilisticmodels import VariationalCCA
     from cca_zoo.data import generate_simple_data
+
     np.random.seed(0)
     # Tests tensor CCA methods
     (X, Y), (_) = generate_covariance_data(20, [5, 5])
     latent_dims = 1
     cca = CCA(latent_dims=latent_dims).fit([X, Y])
-    vcca = VariationalCCA(latent_dims=latent_dims, num_warmup=1000, num_samples=1000).fit([X, Y])
+    vcca = VariationalCCA(
+        latent_dims=latent_dims, num_warmup=1000, num_samples=1000
+    ).fit([X, Y])
     # Test that vanilla CCA and VCCA produce roughly similar latent space ie they are correlated
     assert (
-        np.abs(np.corrcoef(
-            cca.transform([X, Y])[1].T,
-            vcca.posterior_samples["z"].mean(axis=0)[:, 0],
-        )[0, 1])
+        np.abs(
+            np.corrcoef(
+                cca.transform([X, Y])[1].T,
+                vcca.posterior_samples["z"].mean(axis=0)[:, 0],
+            )[0, 1]
+        )
         > 0.9
     )