Repository: scverse/scanpy

Issue:
Memory issue running zheng17 on ~1M cells
Running https://github.com/theislab/scanpy_usage/blob/master/170522_visualizing_one_million_cells/cluster.py on the latest released Scanpy (1.4.4.post1) gives a memory error:

```
reading 1M_neurons_filtered_gene_bc_matrices_h5.h5
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
 (0:01:39)
running recipe zheng17
filtered out 3983 genes that are detectedin less than 1 counts
Killed
```

This is running with 60GB of memory (n1-standard-16), but also occurs with 104GB (n1-highmem-16). It looks like there has been a regression somewhere since this used to run OK. I think the error may be happening in anndata.
memory requirements
Hi,
I am trying to run the full 1.3M 10X mouse cell dataset (using the 1M_neurons_filtered_gene_bc_matrices_h5.h5 file from 10X website).
I have 126GB RAM and Intel® Xeon(R) W-2123 CPU @ 3.60GHz × 8 which is above the requirements you mention needed to run the full cluster.py method without subsampling. (https://github.com/theislab/scanpy_usage/tree/master/170522_visualizing_one_million_cells)
I get a memory error at the normalization and filter_genes_dispersion stage, should i modify the code in anyway? (without subsampling)
Thanks,Shobi


Discussion:
Hm, seems the same as https://github.com/theislab/scanpy/issues/511
So i'm not sure it was ok before.
It should definitely run through with these resources, and I and many other people ran it already. @Koncopd, could you check whether everything behaves still normally? I don't know how we can test this, but I also can't see who we might have broken it.
hi Alex, I'm not sure how to test this either. I tried to run in a new clean conda environment but cant use more than 50% of the cells in pre-processing (otherwise the process is killed at even the normalization stage, "pp.normalize_ per_cell" and uses over 120GB RAM)
The file from 10X i'm using is the link to "1M_neurons aggr - Gene / cell matrix HDF5 (filtered)" from https://support.10xgenomics.com/single-cell-gene-expression/datasets/1.3.0/ 
This is what i type into the terminal
$ python cluster_mouse_brain.py 1M_neurons_filtered_gene_bc_matrices_h5.h5
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
reading 1M_neurons_filtered_gene_bc_matrices_h5.h5 (0:01:19.78)
running recipe zheng17
filtered out 3983 genes that are detected in less than 1 counts
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
Killed

@falexwolf 
Yes, i will run through it and check the memory usage and so on.
I also just tried to run on a Windows machine (to make sure it was not just something to do with the memory management in my Linux machine which has 128GB RAM) with 64GB RAM and 6 cores @3.6GHz and also get Memory error at the normalization step when trying to process the 1.3M neuron file.
Sergei (@Koncopd) tested it out and will get back to you. He also found a peak memory usage of 121 GB. I have to admit that I never made checks with that degree of detail and I fear that for now, I'll simply update the documentation stating that peak memory usage can go up to ~120 GB. I'm still puzzled by that, and maybe some efficiency found it's way into the code which wasn't there (simple guess: is everything in `float32`?). But we'll need some time to work it out.
Just updated the readme https://github.com/theislab/scanpy_usage/blob/master/170522_visualizing_one_million_cells/.
Hi, @ShobiStassen.
As @falexwolf said, i tested the memory usage in the step where you have the problem.
https://github.com/Koncopd/anndata-scanpy-benchmarks/blob/master/cluster_1m_neurons.ipynb
The peak usage is around 121 GB, but it should still be possible to run with 126 GB RAM. Maybe another process took some serious amount of memory?
hi Koncopd, ive tried running your exact lines of code and my memory increment is the same as yours for reading the file and filtering the genes by count. Once i hit the normalization function, my memory shoots above 120 and crashes. in your case, your peak memory usage during normalization is around 92GB and only during filter_genes_variable do you hit 121GB
I dont have any other processes that are being triggered so it's very puzzling why my usage is different from yours... hmmm
Ok, so the first problem is normalization for some reason.
@falexwolf , it seems we should just use new normalize_total when the pull request is accepted, it is more memory efficient.
![image](https://user-images.githubusercontent.com/3065736/54003760-9afeed80-4153-11e9-9187-8e474a2fa03e.png)
Cool, @Koncopd! I'll rewrite a recipe for this with the new functions, which should be a lot more memory effective. 

Resolving patch:
diff --git a/scanpy/preprocessing/_recipes.py b/scanpy/preprocessing/_recipes.py
--- a/scanpy/preprocessing/_recipes.py
+++ b/scanpy/preprocessing/_recipes.py
@@ -4,6 +4,7 @@
 
 from . import _simple as pp
 from ._deprecated.highly_variable_genes import filter_genes_dispersion, filter_genes_cv_deprecated
+from ._normalization import normalize_total
 from .. import logging as logg
 
 
@@ -59,7 +60,7 @@ def recipe_seurat(adata, log=True, plot=False, copy=False):
     if copy: adata = adata.copy()
     pp.filter_cells(adata, min_genes=200)
     pp.filter_genes(adata, min_cells=3)
-    pp.normalize_per_cell(adata, counts_per_cell_after=1e4)
+    normalize_total(adata, target_sum=1e4)
     filter_result = filter_genes_dispersion(
         adata.X, min_mean=0.0125, max_mean=3, min_disp=0.5, log=not log)
     if plot:
@@ -114,8 +115,8 @@ def recipe_zheng17(adata, n_top_genes=1000, log=True, plot=False, copy=False):
     start = logg.info('running recipe zheng17')
     if copy: adata = adata.copy()
     pp.filter_genes(adata, min_counts=1)  # only consider genes with more than 1 count
-    pp.normalize_per_cell(adata,  # normalize with total UMI count per cell
-                          key_n_counts='n_counts_all')
+    normalize_total(adata,  # normalize with total UMI count per cell
+                    key_added='n_counts_all')
     filter_result = filter_genes_dispersion(
         adata.X, flavor='cell_ranger', n_top_genes=n_top_genes, log=False)
     if plot:
@@ -124,7 +125,7 @@ def recipe_zheng17(adata, n_top_genes=1000, log=True, plot=False, copy=False):
     # actually filter the genes, the following is the inplace version of
     #     adata = adata[:, filter_result.gene_subset]
     adata._inplace_subset_var(filter_result.gene_subset)  # filter genes
-    pp.normalize_per_cell(adata)  # renormalize after filtering
+    normalize_total(adata)  # renormalize after filtering
     if log: pp.log1p(adata)  # log transform: X = log(X + 1)
     pp.scale(adata)
     logg.info('    finished', time=start)


Test patch: