Repository: scverse/scanpy

Issue:
sc.pp.neighbors n_pcs doesn't subset if use_rep isn't X_pca
Smaller reproducible example below: https://github.com/theislab/scanpy/issues/1846#issuecomment-863823148 –@ivirshup

--------------------------

- [x] I have checked that this issue has not already been reported.
- [ ] I have confirmed this bug exists on the latest version of scanpy.
- [ ] (optional) I have confirmed this bug exists on the master branch of scanpy.

---

Hi, 

not sure whether this is the right place to post this but i'm having a slight issue after running harmony from scanpy. It's producing very different results and i'm not sure why. It didn't used to be a problem when i was using scanpy 1.6x

### Minimal code sample (that we can copy&paste without having any data)

without running harmony:
```python
sc.tl.pca(adata, svd_solver='arpack')
sc.pp.neighbors(adata, n_pcs = 20)
sc.tl.umap(adata)
sc.pl.umap(adata, color = 'sample')
```
![image](https://user-images.githubusercontent.com/26215587/118844728-1aa3ad80-b8c3-11eb-82d8-57f3b6346c44.png)

with harmony
```python
adata1 = adata.copy()
sc.external.pp.harmony_integrate(adata1, key = 'sample')
sc.pp.neighbors(adata1, n_pcs = 20, use_rep = 'X_pca_harmony')
sc.tl.umap(adata1)
sc.pl.umap(adata1, color = 'sample')
```
![image](https://user-images.githubusercontent.com/26215587/118845227-7ec67180-b8c3-11eb-9bce-a877ef26b47a.png)

If i don't use harmony and just ran with `n_pcs = 50`, the result is similar.
```python
adatax = adata.copy()
sc.pp.neighbors(adatax, n_pcs = 50)
sc.tl.umap(adatax, min_dist = .3)
sc.pl.umap(adatax, color = 'sample')
```
![image](https://user-images.githubusercontent.com/26215587/118845811-01e7c780-b8c4-11eb-8ca9-3ea639d97684.png)

What i don't get is, why does the following work then?
```python
# my current workaround
adata2 = adata.copy()
sc.external.pp.harmony_integrate(adata2, key = 'sample', adjusted_basis='X_pca')
sc.pp.neighbors(adata2, n_pcs = 20)
sc.tl.umap(adata2)
sc.pl.umap(adata2, color = 'sample')
```
![image](https://user-images.githubusercontent.com/26215587/118843486-04492200-b8c2-11eb-8e2b-6aaf59564721.png)


#### Versions

<details>

-----
anndata     0.7.5
scanpy      1.7.1
sinfo       0.3.1
-----
Bio                 1.78
PIL                 8.2.0
adjustText          NA
anndata             0.7.5
annoy               NA
backcall            0.2.0
brotli              NA
cachecontrol        0.12.6
cairo               1.19.1
certifi             2020.06.20
cffi                1.14.5
changeo             1.0.2
chardet             4.0.0
cloudpickle         1.6.0
cycler              0.10.0
cython_runtime      NA
dandelion           0.1.2
dask                2021.03.0
dateutil            2.8.1
decorator           5.0.6
descartes           NA
distance            NA
get_version         2.1
google              NA
h5py                2.10.0
harmonypy           NA
hdmedians           NA
idna                2.10
igraph              0.8.3
ipykernel           5.3.4
ipython_genutils    0.2.0
ipywidgets          7.6.3
jedi                0.17.0
joblib              1.0.1
kiwisolver          1.3.1
legacy_api_wrap     1.2
leidenalg           0.8.3
llvmlite            0.34.0
matplotlib          3.3.4
mizani              0.7.2
mpl_toolkits        NA
msgpack             1.0.2
natsort             7.1.1
networkx            2.5
numba               0.51.2
numexpr             2.7.1
numpy               1.20.2
packaging           20.9
palettable          3.3.0
pandas              1.2.4
parso               0.8.2
patsy               0.5.1
pexpect             4.8.0
pickleshare         0.7.5
pkg_resources       NA
plotnine            0.7.1
polyleven           NA
presto              0.6.2
prompt_toolkit      3.0.17
ptyprocess          0.7.0
pycparser           2.20
pygments            2.8.1
pynndescent         0.5.2
pyparsing           2.4.7
pytoml              NA
pytz                2021.1
pywt                1.1.1
requests            2.25.1
scanpy              1.7.1
scipy               1.6.3
scrublet            NA
seaborn             0.11.1
setuptools_scm      NA
sinfo               0.3.1
six                 1.15.0
skbio               0.5.6
skimage             0.18.1
sklearn             0.23.2
socks               1.7.1
sparse              0.11.2
sphinxcontrib       NA
statsmodels         0.12.1
storemagic          NA
tables              3.6.1
texttable           1.6.3
tlz                 0.11.1
toolz               0.11.1
tornado             6.1
tqdm                4.59.0
traitlets           5.0.5
typing_extensions   NA
umap                0.5.1
urllib3             1.26.4
wcwidth             0.2.5
yaml                5.4.1
zmq                 20.0.0
-----
IPython             7.22.0
jupyter_client      6.1.12
jupyter_core        4.7.1
notebook            6.3.0
-----
Python 3.8.8 (default, Feb 24 2021, 21:46:12) [GCC 7.3.0]
Linux-4.15.0-142-generic-x86_64-with-glibc2.10
64 logical CPU cores, x86_64
-----
Session information updated at 2021-05-19 16:44

</details>



Discussion:
@zktuong, since you're able to see the same small groups from both harmony and pca, I would suspect that it's actually a feature of your data. I've recently seen something similar in [this preprint](https://www.biorxiv.org/content/10.1101/2020.12.01.405886v2) (see figure 4). Maybe you are seeing similar spurious expression effects?
hi @ivirshup 

I think it might be to do with `n_pcs` not being used in `_choose_representation` if `use_rep` is not `None` or `X_pca`:
https://github.com/theislab/scanpy/blob/c488909a54e9ab1462186cca35b537426e4630db/scanpy/tools/_utils.py

```python
def _choose_representation(adata, use_rep=None, n_pcs=None, silent=False):
    verbosity = settings.verbosity
    if silent and settings.verbosity > 1:
        settings.verbosity = 1
    if use_rep is None and n_pcs == 0:  # backwards compat for specifying `.X`
        use_rep = 'X'
    if use_rep is None:
        # rest of code in lines 30 - 48
    else:
        if use_rep in adata.obsm.keys():
            X = adata.obsm[use_rep]
            if use_rep == 'X_pca' and n_pcs is not None:
                X = adata.obsm[use_rep][:, :n_pcs]
        # and so on
```

I'm guessing it should have the additional bits like so:

```python
def _choose_representation(adata, use_rep=None, n_pcs=None, silent=False):
    verbosity = settings.verbosity
    if silent and settings.verbosity > 1:
        settings.verbosity = 1
    if use_rep is None and n_pcs == 0:  # backwards compat for specifying `.X`
        use_rep = 'X'
    if use_rep is None:
        # rest of code in lines 30 - 48
    else:
        if use_rep in adata.obsm.keys():
            if n_pcs is not None:
                X = adata.obsm[use_rep][:, :n_pcs]
            else:
                X = adata.obsm[use_rep]
            if use_rep == 'X_pca' and n_pcs is not None:
                X = adata.obsm[use_rep][:, :n_pcs]
        # and so on
```
I think this is why it 'worked' in my last example; I just made harmony overwrite the `X_pca` slot and `_choose_representation` was able to run it as if it was `X_pca`.
Ah, I see what you're saying now, I was confused about what the `adjusted_basis` argument was doing.

When the array distance's are being calculated on isn't called `"X_pca"`, `n_pcs` doesn't have an effect.

Smaller reproducible example:

```python
import scanpy as sc

pbmc = sc.datasets.pbmc3k_processed().raw.to_adata()
sc.pp.pca(pbmc)
pbmc.obsm["pca_copy"] = pbmc.obsm["X_pca"].copy()

a = pbmc.copy()
b = pbmc.copy()

sc.pp.neighbors(a, use_rep="X_pca", n_pcs=2)
sc.pp.neighbors(b, use_rep="pca_copy", n_pcs=2)

sc.tl.umap(a)
sc.tl.umap(b)

sc.pl.umap(a)
```

![tmpe9g8ufns](https://user-images.githubusercontent.com/8238804/122524253-45685980-d05b-11eb-8b70-e64e6bd7588b.PNG)


```python
sc.pl.umap(b)
```

![tmp11zkjcms](https://user-images.githubusercontent.com/8238804/122524311-51ecb200-d05b-11eb-84a6-23aebf32a020.PNG)


Resolving patch:
diff --git a/scanpy/tools/_utils.py b/scanpy/tools/_utils.py
--- a/scanpy/tools/_utils.py
+++ b/scanpy/tools/_utils.py
@@ -47,10 +47,16 @@ def _choose_representation(adata, use_rep=None, n_pcs=None, silent=False):
             logg.info('    using data matrix X directly')
             X = adata.X
     else:
-        if use_rep in adata.obsm.keys():
+        if use_rep in adata.obsm.keys() and n_pcs is not None:
+            if n_pcs > adata.obsm[use_rep].shape[1]:
+                raise ValueError(
+                    f'{use_rep} does not have enough Dimensions. Provide a '
+                    'Representation with equal or more dimensions than'
+                    '`n_pcs` or lower `n_pcs` '
+                )
+            X = adata.obsm[use_rep][:, :n_pcs]
+        elif use_rep in adata.obsm.keys() and n_pcs is None:
             X = adata.obsm[use_rep]
-            if use_rep == 'X_pca' and n_pcs is not None:
-                X = adata.obsm[use_rep][:, :n_pcs]
         elif use_rep == 'X':
             X = adata.X
         else:


Test patch:
diff --git a/scanpy/tests/test_pca.py b/scanpy/tests/test_pca.py
--- a/scanpy/tests/test_pca.py
+++ b/scanpy/tests/test_pca.py
@@ -136,3 +136,20 @@ def test_pca_chunked(pbmc3k_normalized):
         np.abs(chunked.uns["pca"]["variance_ratio"]),
         np.abs(default.uns["pca"]["variance_ratio"]),
     )
+
+
+def test_pca_n_pcs(pbmc3k_normalized):
+    """
+    Tests that the n_pcs parameter also works for
+    representations not called "X_pca"
+    """
+    pbmc = pbmc3k_normalized
+    sc.pp.pca(pbmc, dtype=np.float64)
+    pbmc.obsm["X_pca_test"] = pbmc.obsm["X_pca"]
+    original = sc.pp.neighbors(pbmc, n_pcs=5, use_rep="X_pca", copy=True)
+    renamed = sc.pp.neighbors(pbmc, n_pcs=5, use_rep="X_pca_test", copy=True)
+
+    assert np.allclose(original.obsm["X_pca"], renamed.obsm["X_pca_test"])
+    assert np.allclose(
+        original.obsp["distances"].toarray(), renamed.obsp["distances"].toarray()
+    )