Repository: scverse/scanpy

Issue:
sc.pp.highly_variable does not always handle `subset` and `inplace` arguments consistently
Hey,
while writing tests for #1715 I noted the following behavior:

`output = sc.pp.highly_variable(adata,inplace=True,subset=False,n_top_genes=100)`
--> Returns nothing :heavy_check_mark: 
--> `adata.var` fields are updated but shape stays the same :heavy_check_mark: 

`output = sc.pp.highly_variable(adata,inplace=True,subset=True,n_top_genes=100)`
--> Returns nothing  :heavy_check_mark: 
--> `adata.var` fields are updated and shape is subsetted correctly :heavy_check_mark: 

`output = sc.pp.highly_variable(adata,inplace=False,subset=False,n_top_genes=100)`
--> output is a dataframe with the original number of genes as rows :heavy_check_mark: 
--> `adata` is unchanged :heavy_check_mark: 

`output = sc.pp.highly_variable(adata,inplace=False,subset=True,n_top_genes=100)`
--> Returns nothing :x: 
--> `adata` shape is changed an `var` fields are updated :x: 

I think the last case is unexpected, assuming that `inplace=False` should protect `adata` from changes and results in an output. I think these parts of code are the cause (depending on the flavor):


https://github.com/theislab/scanpy/blob/4dd8de9e355ce8d59009c15522670a3d0970462c/scanpy/preprocessing/_highly_variable_genes.py#L148-L174

https://github.com/theislab/scanpy/blob/4dd8de9e355ce8d59009c15522670a3d0970462c/scanpy/preprocessing/_highly_variable_genes.py#L528-L553


To fix this, one could change `if subset or inplace:` to `if inplace:`. Then one would need to add another `if subet:` in the `else` block, like so (half-pseudocode):

```
    if inplace:        
        
       #update adata
        
        if batch_key is not None:
           #drop batch related keys
        if subset:
            adata._inplace_subset_var(df['highly_variable'].values)
    else:
        if batch_key is None:
            #drop batch related keys
        if subset:            
            df=df.iloc[df.highly_variable.values,:]
            
        return df
```

I can make a quick PR if this is the intended behavior. :slightly_smiling_face: 
best, jan.


Discussion:
PS: Minimal example:

```
adata = sc.datasets.pbmc3k()
sc.pp.filter_genes(adata,min_cells=1)
print(adata.shape)
output = sc.pp.highly_variable_genes(adata,flavor='cell_ranger',inplace=False,subset=True)
print(adata.shape)
print(output)
```


Resolving patch:
diff --git a/scanpy/preprocessing/_highly_variable_genes.py b/scanpy/preprocessing/_highly_variable_genes.py
--- a/scanpy/preprocessing/_highly_variable_genes.py
+++ b/scanpy/preprocessing/_highly_variable_genes.py
@@ -146,7 +146,7 @@ def _highly_variable_genes_seurat_v3(
     df["highly_variable"] = False
     df.loc[sorted_index[: int(n_top_genes)], "highly_variable"] = True
 
-    if inplace or subset:
+    if inplace:
         adata.uns["hvg"] = {"flavor": "seurat_v3"}
         logg.hint(
             "added\n"
@@ -172,6 +172,9 @@ def _highly_variable_genes_seurat_v3(
     else:
         if batch_key is None:
             df = df.drop(["highly_variable_nbatches"], axis=1)
+        if subset:
+            df = df.iloc[df.highly_variable.values, :]
+
         return df
 
 
@@ -544,7 +547,7 @@ def highly_variable_genes(
 
     logg.info("    finished", time=start)
 
-    if inplace or subset:
+    if inplace:
         adata.uns["hvg"] = {"flavor": flavor}
         logg.hint(
             "added\n"
@@ -559,6 +562,7 @@ def highly_variable_genes(
         adata.var["dispersions_norm"] = df["dispersions_norm"].values.astype(
             "float32", copy=False
         )
+
         if batch_key is not None:
             adata.var["highly_variable_nbatches"] = df[
                 "highly_variable_nbatches"
@@ -568,5 +572,9 @@ def highly_variable_genes(
             ].values
         if subset:
             adata._inplace_subset_var(df["highly_variable"].values)
+
     else:
+        if subset:
+            df = df.iloc[df.highly_variable.values, :]
+
         return df


Test patch:
diff --git a/scanpy/tests/test_highly_variable_genes.py b/scanpy/tests/test_highly_variable_genes.py
--- a/scanpy/tests/test_highly_variable_genes.py
+++ b/scanpy/tests/test_highly_variable_genes.py
@@ -516,3 +516,38 @@ def test_cellranger_n_top_genes_warning():
         match="`n_top_genes` > number of normalized dispersions, returning all genes with normalized dispersions.",
     ):
         sc.pp.highly_variable_genes(adata, n_top_genes=1000, flavor="cell_ranger")
+
+
+@pytest.mark.parametrize("flavor", ["seurat", "cell_ranger"])
+@pytest.mark.parametrize("subset", [True, False])
+@pytest.mark.parametrize("inplace", [True, False])
+def test_highly_variable_genes_subset_inplace_consistency(
+    flavor,
+    subset,
+    inplace,
+):
+    adata = sc.datasets.blobs(n_observations=20, n_variables=80, random_state=0)
+    adata.X = np.abs(adata.X).astype(int)
+
+    if flavor == "seurat" or flavor == "cell_ranger":
+        sc.pp.normalize_total(adata, target_sum=1e4)
+        sc.pp.log1p(adata)
+
+    elif flavor == "seurat_v3":
+        pass
+
+    else:
+        raise ValueError(f"Unknown flavor {flavor}")
+
+    n_genes = adata.shape[1]
+
+    output_df = sc.pp.highly_variable_genes(
+        adata,
+        flavor=flavor,
+        n_top_genes=15,
+        subset=subset,
+        inplace=inplace,
+    )
+
+    assert (output_df is None) == inplace
+    assert len(adata.var if inplace else output_df) == (15 if subset else n_genes)