Repository: scverse/scanpy

Issue:
Adding MAGIC to Scanpy
Hi,

We spoke a while ago about adding [PHATE](https://github.com/KrishnaswamyLab/PHATE) and eventually [MAGIC](https://github.com/KrishnaswamyLab/MAGIC) to Scanpy. MAGIC has just been submitted to CRAN and is in a stable state. 

How should a tool such as MAGIC interact with Scanpy? Do you currently have any imputation methods included in the package that I can use to model the API?

Thanks,
Scott


Discussion:
Hi Scott,

sure, I remember! :smile: For some reason, I forgot to mention you personally in the [release notes](http://scanpy.readthedocs.io/en/latest/#version-1-1-may-31-2018), is now fixed. Sorry about that! 

You could add MAGIC as a preprocessing similar to DCA in the imputation section: http://scanpy.readthedocs.io/en/latest/api/index.html#preprocessing-pp.

In terms of code, I would also adapt the conventions of DCA: https://github.com/theislab/scanpy/blob/master/scanpy/preprocessing/dca.py

We had some discussions on how to do this best: https://github.com/theislab/scanpy/issues/142 and https://github.com/theislab/scanpy/pull/186.

If you think you have better conventions, happy to adopt these. DCA is also not yet released...

Best,
  Alex
@falexwolf 
MAGIC uses root square transformation, not the frequently used log transformation, which causes the incompatibility with batch correction methods, such as CCA and MNN. Is DCA compatible with MNN and CCA ?
@wangjiawen2013 we recommend using square root transform with MAGIC but it's certainly not incompatible. So long as the inputs have been library size normalized and transformed with any of log, sqrt, arcsinh or some other sublinear transformation, MAGIC will work just fine.
Also I'm surprised to see I never left a note on your message @falexwolf : thanks! I'm working on the API now, will send in a PR when it's done or leave a note here if I think the DCA API could do with some modification.
I find some negative values in the imputation data, how did them generate ?
@wangjiawen2013 if you have any questions about MAGIC, I recommend you post them in [the MAGIC repo](https://github.com/KrishnaswamyLab/MAGIC). The negative values are an artifact of the imputation process, but the absolute values of expression are not really important, since normalized scRNAseq data is only really a measure of relative expression anyway.

Resolving patch:
diff --git a/scanpy/preprocessing/magic.py b/scanpy/preprocessing/magic.py
--- a/scanpy/preprocessing/magic.py
+++ b/scanpy/preprocessing/magic.py
@@ -18,7 +18,7 @@ def magic(adata,
           random_state=None,
           n_jobs=None,
           verbose=False,
-          copy=False,
+          copy=None,
           **kwargs):
     """Markov Affinity-based Graph Imputation of Cells (MAGIC) API [vanDijk18]_.
 
@@ -64,13 +64,16 @@ def magic(adata,
     verbose : `bool`, `int` or `None`, optional (default: `sc.settings.verbosity`)
         If `True` or an integer `>= 2`, print status messages.
         If `None`, `sc.settings.verbosity` is used.
-    copy : `bool`, optional. Default: `False`.
-        If true, a copy of anndata is returned.
+    copy : `bool` or `None`, optional. Default: `None`.
+        If true, a copy of anndata is returned. If `None`, `copy` is True if
+        `genes` is not `'all_genes'` or `'pca_only'`. `copy` may only be False
+        if `genes` is `'all_genes'` or `'pca_only'`, as the resultant data
+        will otherwise have different column names from the input data.
     kwargs : additional arguments to `magic.MAGIC`
 
     Returns
     -------
-    If `copy` is true, AnnData object is returned.
+    If `copy` is True, AnnData object is returned.
 
     If `subset_genes` is not `all_genes`, PCA on MAGIC values of cells are stored in
     `adata.obsm['X_magic']` and `adata.X` is not modified.
@@ -84,8 +87,8 @@ def magic(adata,
     >>> adata = sc.datasets.paul15()
     >>> sc.pp.normalize_per_cell(adata)
     >>> sc.pp.sqrt(adata)  # or sc.pp.log1p(adata)
-    >>> sc.pp.magic(adata, name_list=['Mpo', 'Klf1', 'Ifitm1'], k=5)
-    >>> adata.obsm['X_magic'].shape
+    >>> adata_magic = sc.pp.magic(adata, name_list=['Mpo', 'Klf1', 'Ifitm1'], k=5)
+    >>> adata_magic.shape
     (2730, 3)
     >>> sc.pp.magic(adata, name_list='pca_only', k=5)
     >>> adata.obsm['X_magic'].shape
@@ -103,6 +106,16 @@ def magic(adata,
             'git+git://github.com/KrishnaswamyLab/MAGIC.git#subdirectory=python`')
 
     logg.info('computing PHATE', r=True)
+    needs_copy = not (name_list is None or
+                      (isinstance(name_list, str) and
+                       name_list in ["all_genes", "pca_only"]))
+    if copy is None:
+        copy = needs_copy
+    elif needs_copy and not copy:
+        raise ValueError(
+            "Can only perform MAGIC in-place with `name_list=='all_genes' or "
+            "`name_list=='pca_only'` (got {}). Consider setting "
+            "`copy=True`".format(name_list))
     adata = adata.copy() if copy else adata
     verbose = settings.verbosity if verbose is None else verbose
     if isinstance(verbose, int):
@@ -122,16 +135,19 @@ def magic(adata,
     logg.info('    finished', time=True,
               end=' ' if settings.verbosity > 2 else '\n')
     # update AnnData instance
-    adata.raw = adata
-    if X_magic.shape[1] != adata.X.shape[1]:
-        adata.obsm["X_magic"] = X_magic
-        if name_list == "pca_only":
-            logg.hint('added\n'
-                      '    \'X_magic\', PCA on MAGIC coordinates (adata.obsm)')
-        else:
-            logg.hint('added\n'
-                      '    \'X_magic\', MAGIC smoothed gene values (adata.obsm)')
+    if name_list == "pca_only":
+        # special case - update adata.obsm with smoothed values
+        adata.obsm["X_magic"] = X_magic.X
+        logg.hint('added\n'
+                  '    \'X_magic\', PCA on MAGIC coordinates (adata.obsm)')
+    elif copy:
+        # just return X_magic
+        X_magic.raw = adata
+        adata = X_magic
     else:
-        adata.X = X_magic
+        # replace data with smoothed data
+        adata.raw = adata
+        adata.X = X_magic.X
+
     if copy:
         return adata


Test patch: