Repository: scverse/scanpy

Issue:
normalize_total throws ValueError when passed a large view
When I run sc.pp.normalize_total(adata, target_sum=1e4),I got the error:ValueError: could not convert integer scalar,and How can I fixed this issue?


Discussion:
I  encounter the same error as well when i tried sc.pp.normalize_total(adata, target_sum=5e4).  

Environment:
scanpy==1.4.6 anndata==0.7.1 umap==0.4.1 numpy==1.18.1 scipy==1.4.1 pandas==1.0.3 scikit-learn==0.22.2.post1 statsmodels==0.11.1 python-igraph==0.8.0

[conda list]
https://github.com/phamidko/codesnippets/blob/master/scanpy-conda-list.txt
[ipynb]
https://github.com/phamidko/codesnippets/blob/master/Tissue-Tcell-activation.ipynb


@phamidko, could you export that conda environment with `conda list --export`? I'd like to see if I can recreate with your environment.
Hi Issac,
Thank you for looking into this. I tried downgrading to python 3.7 but the
error still persists.

Best,
Philip

On Tue, Apr 28, 2020 at 12:16 AM Isaac Virshup <notifications@github.com>
wrote:

> @phamidko <https://github.com/phamidko>, could you export that conda
> environment with conda list --export? I'd like to see if I can recreate
> with your environment.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/theislab/scanpy/issues/1183#issuecomment-620426227>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AJ7QRZ6CCZW74XVYGDEPRFDROZ7CRANCNFSM4MRFHJHQ>
> .
>

<envlist moved to next post by @ivirshup>
@phamidko I've modified your comment, and placed the output in the collapsed section below so I don't have to scroll so much on this issue. Unfortunately replies sent from email don't support markdown :(

<details>
<summary> phamidko's environment </summary>

```
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
_libgcc_mutex=0.1=main
anndata=0.7.1=pypi_0
attrs=19.3.0=py_0
backcall=0.1.0=py_0
bleach=3.1.4=pyh9f0ad1d_0
brotlipy=0.7.0=py37h8f50634_1000
ca-certificates=2020.4.5.1=hecc5488_0
cairo=1.16.0=hcf35c78_1003
certifi=2020.4.5.1=py37hc8dfbb8_0
cffi=1.14.0=py37hd463f26_0
chardet=3.0.4=py37hc8dfbb8_1006
cryptography=2.9.2=py37hb09aad4_0
cycler=0.10.0=pypi_0
decorator=4.4.2=py_0
defusedxml=0.6.0=py_0
entrypoints=0.3=py37hc8dfbb8_1001
fontconfig=2.13.1=h86ecdb6_1001
freetype=2.10.1=he06d7ca_0
get-version=2.1=pypi_0
gettext=0.19.8.1=hc5be6a0_1002
glib=2.64.2=h6f030ca_0
gmp=6.2.0=he1b5a44_2
h5py=2.10.0=pypi_0
icu=64.2=he1b5a44_1
idna=2.9=py_1
importlib-metadata=1.6.0=py37hc8dfbb8_0
importlib_metadata=1.6.0=0
ipykernel=5.2.1=py37h43977f1_0
ipython=7.13.0=py37hc8dfbb8_2
ipython_genutils=0.2.0=py_1
jedi=0.17.0=py37hc8dfbb8_0
jinja2=2.11.2=pyh9f0ad1d_0
joblib=0.14.1=pypi_0
json5=0.9.0=py_0
jsonschema=3.2.0=py37hc8dfbb8_1
jupyter_client=6.1.3=py_0
jupyter_contrib_core=0.3.3=py_2
jupyter_contrib_nbextensions=0.5.1=py37_0
jupyter_core=4.6.3=py37hc8dfbb8_1
jupyter_highlight_selected_word=0.2.0=py37_1000
jupyter_latex_envs=1.4.6=py37_1000
jupyter_nbextensions_configurator=0.4.1=py37_0
jupyterlab=2.1.1=py_0
jupyterlab_server=1.1.1=py_0
kiwisolver=1.2.0=pypi_0
ld_impl_linux-64=2.33.1=h53a641e_7
legacy-api-wrap=1.2=pypi_0
leidenalg=0.8.0=py37h43df1e8_0
libedit=3.1.20181209=hc058e9b_0
libffi=3.2.1=hd88cf55_4
libgcc-ng=9.1.0=hdf63c60_0
libgfortran-ng=7.3.0=hdf63c60_5
libiconv=1.15=h516909a_1006
libpng=1.6.37=hed695b0_1
libsodium=1.0.17=h516909a_0
libstdcxx-ng=9.1.0=hdf63c60_0
libuuid=2.32.1=h14c3975_1000
libxcb=1.13=h14c3975_1002
libxml2=2.9.10=hee79883_0
libxslt=1.1.33=h31b3aaa_0
llvmlite=0.32.0=pypi_0
lxml=4.5.0=py37he3881c9_1
markupsafe=1.1.1=py37h8f50634_1
matplotlib=3.2.1=pypi_0
mistune=0.8.4=py37h8f50634_1001
natsort=7.0.1=pypi_0
nbconvert=5.6.1=py37hc8dfbb8_1
nbformat=5.0.6=py_0
ncurses=6.2=he6710b0_0
networkx=2.4=pypi_0
notebook=6.0.3=py37_0
numba=0.49.0=pypi_0
numexpr=2.7.1=pypi_0
numpy=1.18.3=pypi_0
openssl=1.1.1g=h516909a_0
packaging=20.3=pypi_0
pandas=1.0.3=pypi_0
pandoc=2.9.2.1=0
pandocfilters=1.4.2=py_1
parso=0.7.0=pyh9f0ad1d_0
patsy=0.5.1=pypi_0
pcre=8.44=he1b5a44_0
pexpect=4.8.0=py37hc8dfbb8_1
pickleshare=0.7.5=py37hc8dfbb8_1001
pip=20.0.2=py37_1
pixman=0.38.0=h516909a_1003
prometheus_client=0.7.1=py_0
prompt-toolkit=3.0.5=py_0
psutil=5.7.0=py37h8f50634_1
pthread-stubs=0.4=h14c3975_1001
ptyprocess=0.6.0=py_1001
pycairo=1.19.1=py37h01af8b0_3
pycparser=2.20=py_0
pygments=2.6.1=py_0
pyopenssl=19.1.0=py_1
pyparsing=2.4.7=pypi_0
pyrsistent=0.16.0=py37h8f50634_0
pysocks=1.7.1=py37hc8dfbb8_1
python=3.7.7=hcf32534_0_cpython
python-dateutil=2.8.1=py_0
python-igraph=0.8.1=pypi_0
python_abi=3.7=1_cp37m
pytz=2019.3=pypi_0
pyyaml=5.3.1=py37h8f50634_0
pyzmq=19.0.0=py37hac76be4_1
readline=8.0=h7b6447c_0
requests=2.23.0=pyh8c360ce_2
scanpy=1.4.6=pypi_0
scikit-learn=0.22.2.post1=pypi_0
scipy=1.4.1=pypi_0
seaborn=0.10.1=pypi_0
send2trash=1.5.0=py_0
setuptools=46.1.3=py37_0
setuptools-scm=3.5.0=pypi_0
six=1.14.0=py_1
sqlite=3.31.1=h62c20be_1
statsmodels=0.11.1=pypi_0
tables=3.6.1=pypi_0
tbb=2020.0.133=pypi_0
terminado=0.8.3=py37hc8dfbb8_1
testpath=0.4.4=py_0
texttable=1.6.2=py_0
tk=8.6.8=hbc83047_0
tornado=6.0.4=py37h8f50634_1
tqdm=4.45.0=pypi_0
traitlets=4.3.3=py37hc8dfbb8_1
umap-learn=0.4.1=pypi_0
urllib3=1.25.9=py_0
wcwidth=0.1.9=pyh9f0ad1d_0
webencodings=0.5.1=py_1
wheel=0.34.2=py37_0
xorg-kbproto=1.0.7=h14c3975_1002
xorg-libice=1.0.10=h516909a_0
xorg-libsm=1.2.3=h84519dc_1000
xorg-libx11=1.6.9=h516909a_0
xorg-libxau=1.0.9=h14c3975_0
xorg-libxdmcp=1.1.3=h516909a_0
xorg-libxext=1.3.4=h516909a_0
xorg-libxrender=0.9.10=h516909a_1002
xorg-renderproto=0.11.1=h14c3975_1002
xorg-xextproto=7.3.0=h14c3975_1002
xorg-xproto=7.0.31=h14c3975_1007
xz=5.2.5=h7b6447c_0
yaml=0.2.4=h516909a_0
zeromq=4.3.2=he1b5a44_2
zipp=3.1.0=py_0
zlib=1.2.11=h7b6447c_3
```

</details>

I've recreated your environment, but cannot reproduce this error.

Here's how I created the environment:

```bash
# Where the output you pasted above is in scanpy_1183_env.txt
$ grep -v pypi_0 scanpy_1183_env.txt > scanpy_1183_env_nopip.txt
$ grep pypi_0 scanpy_1183_env.txt | sed 's/=pypi_0//' | sed 's/=/==/' > scanpy_1183_pip.txt
$ conda create -y --name scanpy1183 --file scanpy_1183_env_nopip.txt
$ conda activate scanpy1183
$ pip install -r scanpy_1183_pip.txt
```

Then I tested this using:

```python
import scanpy as sc

adata = sc.datasets.pbmc3k()
sc.pp.normalize_total(adata, target_sum=1e4)
```

But did not get an error. ~Could you send a snippet that reproduces the error for you?~ Oops, forgot that you already did this in the notebook. Taking a look at that now.
Hmm. I'm not able to replicate the exact error, but I get a different one. Could you try running:

```python
adata = adata.copy()
```

right before `normalize_total` and see if that works?

----------------------

Update: tried on a different machine and could replicate your error. The issue is that `normalize_total` doesn't make sure the anndata object isn't a view before assigning to it.

As a work-around, you can just run `adata = adata.copy()` before `sc.pp.normalize_total(adata)`.

Resolving patch:
diff --git a/scanpy/preprocessing/_normalization.py b/scanpy/preprocessing/_normalization.py
--- a/scanpy/preprocessing/_normalization.py
+++ b/scanpy/preprocessing/_normalization.py
@@ -7,6 +7,7 @@
 
 from .. import logging as logg
 from .._compat import Literal
+from .._utils import view_to_actual
 
 
 def _normalize_data(X, counts, after=None, copy=False):
@@ -134,6 +135,8 @@ def normalize_total(
             f"`layers` needs to be a list of strings or 'all', not {layers!r}"
         )
 
+    view_to_actual(adata)
+
     gene_subset = None
     msg = 'normalizing counts per cell'
     if exclude_highly_expressed:


Test patch:
diff --git a/scanpy/tests/test_normalization.py b/scanpy/tests/test_normalization.py
--- a/scanpy/tests/test_normalization.py
+++ b/scanpy/tests/test_normalization.py
@@ -4,6 +4,7 @@
 from scipy.sparse import csr_matrix
 
 import scanpy as sc
+from anndata.tests.helpers import assert_equal
 
 X_total = [[1, 0], [3, 0], [5, 6]]
 X_frac = [[1, 0, 1], [3, 0, 1], [5, 6, 1]]
@@ -30,3 +31,16 @@ def test_normalize_total_layers(typ, dtype):
     adata.layers["layer"] = adata.X.copy()
     sc.pp.normalize_total(adata, layers=["layer"])
     assert np.allclose(adata.layers["layer"].sum(axis=1), [3.0, 3.0, 3.0])
+
+
+@pytest.mark.parametrize('typ', [np.array, csr_matrix], ids=lambda x: x.__name__)
+@pytest.mark.parametrize('dtype', ['float32', 'int64'])
+def test_normalize_total_view(typ, dtype):
+    adata = AnnData(typ(X_total), dtype=dtype)
+    v = adata[:, :]
+
+    sc.pp.normalize_total(v)
+    sc.pp.normalize_total(adata)
+
+    assert not v.is_view
+    assert_equal(adata, v)