Repository: scverse/scanpy

Issue:
sc.get.aggregate looses values
### Please make sure these conditions are met

- [X] I have checked that this issue has not already been reported.
- [X] I have confirmed this bug exists on the latest version of scanpy.
- [ ] (optional) I have confirmed this bug exists on the main branch of scanpy.

### What happened?

I'm working with the `sc.get.aggregate` function from the latest release candidate. In certain combinations of `groupby` variables, some valuese get lost. 

I wasn't able to make a minimal reproducible example, but I obfuscated the `obs` table of my real data and can share it here: 
https://www.dropbox.com/scl/fi/jsbrb2ulki7mmih2242kc/adata_aggregate_bug.h5ad?rlkey=qczuaf2v5vlwb00zyuxmzjkix&dl=1




### Minimal code sample

```python
>>> test_adata = sc.read_h5ad("adata_aggregate_bug.h5ad")

>>> test_adata.obs["patient_id"].nunique()
69

>>> test_adata.obs.isnull().sum()
patient_id           0
timepoint            0
external_batch_id    0
dtype: int64

>>> pb = sc.get.aggregate(
      test_adata,
      by=[
          "patient_id",
          "timepoint",
          "external_batch_id",
      ],
      func="mean",
  )
  pb.obs["patient_id"].nunique()

15

>>> pb = sc.get.aggregate(
      test_adata,
      by=[
          "patient_id",
          "external_batch_id",
      ],
      func="mean",
  )
  pb.obs["patient_id"].nunique()

69

>>> pb = sc.get.aggregate(
      test_adata,
      by=[
          "patient_id",
          "timepoint",
      ],
      func="mean",
  )
  pb.obs["patient_id"].nunique()

69
```


### Error output

```pytb
So only if using all three variables, some patient IDs are lost. I don't see why this would be happening.
```


### Versions

<details>

```
Package                   Version         Editable project location
------------------------- --------------- -------------------------------------------------------------------------------------------------------------------------
aiohttp                   3.9.3
aiosignal                 1.3.1
anndata                   0.10.5.post1
anyio                     4.3.0
appdirs                   1.4.4
argon2-cffi               23.1.0
argon2-cffi-bindings      21.2.0
array_api_compat          1.5
arrow                     1.3.0
asciitree                 0.3.3
asttokens                 2.4.1
async-lru                 2.0.4
async-timeout             4.0.3
attrs                     23.2.0
Babel                     2.14.0
beautifulsoup4            4.12.3
bleach                    6.1.0
bokeh                     3.3.4
branca                    0.7.1
Brotli                    1.1.0
cached-property           1.5.2
cachetools                5.3.3
certifi                   2024.2.2
cffi                      1.16.0
charset-normalizer        3.3.2
click                     8.1.7
click-plugins             1.1.1
cligj                     0.7.2
cloudpickle               3.0.0
colorama                  0.4.6
colorcet                  3.1.0
comm                      0.2.1
confluent-kafka           1.9.2
contourpy                 1.2.0
cubinlinker               0.3.0
cucim                     24.2.0
cuda-python               11.8.3
cudf                      24.2.2
cudf_kafka                24.2.2
cugraph                   24.2.0
cuml                      24.2.0
cuproj                    24.2.0
cupy                      12.2.0
cuspatial                 24.2.0
custreamz                 24.2.2
cuxfilter                 24.2.0
cycler                    0.12.1
cytoolz                   0.12.3
dask                      2024.1.1
dask-cuda                 24.2.0
dask-cudf                 24.2.2
datashader                0.16.0
debugpy                   1.8.1
decorator                 5.1.1
decoupler                 1.6.0
defusedxml                0.7.1
distributed               2024.1.1
docrep                    0.3.2
entrypoints               0.4
et-xmlfile                1.1.0
exceptiongroup            1.2.0
executing                 2.0.1
fa2                       0.3.5
fasteners                 0.19
fastjsonschema            2.19.1
fastrlock                 0.8.2
fcsparser                 0.2.8
filelock                  3.13.1
fiona                     1.9.5
folium                    0.16.0
fonttools                 4.49.0
fqdn                      1.5.1
frozenlist                1.4.1
fsspec                    2024.2.0
GDAL                      3.8.1
gdown                     5.1.0
geopandas                 0.14.3
h11                       0.14.0
h2                        4.1.0
h5py                      3.10.0
harmonypy                 0.0.9
holoviews                 1.18.3
hpack                     4.0.0
httpcore                  1.0.4
httpx                     0.27.0
hyperframe                6.0.1
idna                      3.6
igraph                    0.11.4
imagecodecs               2024.1.1
imageio                   2.34.0
importlib_metadata        7.0.2
importlib_resources       6.1.3
inflect                   7.0.0
ipykernel                 6.29.3
ipylab                    1.0.0
ipython                   8.22.2
ipywidgets                8.1.2
isoduration               20.11.0
jedi                      0.19.1
Jinja2                    3.1.3
joblib                    1.3.2
json5                     0.9.22
jsonpointer               2.4
jsonschema                4.21.1
jsonschema-specifications 2023.12.1
jupyter_client            8.6.0
jupyter_core              5.7.1
jupyter-events            0.9.0
jupyter-lsp               2.2.4
jupyter_server            2.13.0
jupyter_server_proxy      4.1.0
jupyter_server_terminals  0.5.2
jupyterlab                4.1.4
jupyterlab_pygments       0.3.0
jupyterlab_server         2.25.3
jupyterlab_widgets        3.0.10
kiwisolver                1.4.5
lamin_utils               0.13.0
lazy_loader               0.3
legacy-api-wrap           1.4
leidenalg                 0.10.2
linkify-it-py             2.0.3
llvmlite                  0.42.0
locket                    1.0.0
louvain                   0.8.1
lz4                       4.3.3
mapclassify               2.6.1
Markdown                  3.5.2
markdown-it-py            3.0.0
MarkupSafe                2.1.5
matplotlib                3.8.3
matplotlib-inline         0.1.6
mdit-py-plugins           0.4.0
mdurl                     0.1.2
mistune                   3.0.2
msgpack                   1.0.7
mudata                    0.2.3
multidict                 6.0.5
multipledispatch          0.6.0
munkres                   1.1.4
muon                      0.1.5
natsort                   8.4.0
nbclient                  0.8.0
nbconvert                 7.16.2
nbformat                  5.9.2
nbproject                 0.10.1
nest_asyncio              1.6.0
networkx                  3.2.1
notebook                  7.1.1
notebook_shim             0.2.4
numba                     0.59.0
numcodecs                 0.12.1
numpy                     1.24.4
nvtx                      0.2.10
omnipath                  1.0.8
openpyxl                  3.1.2
orjson                    3.9.15
overrides                 7.7.0
packaging                 24.0
pandas                    1.5.3
pandocfilters             1.5.0
panel                     1.3.8
param                     2.0.2
parso                     0.8.3
partd                     1.4.1
patsy                     0.5.6
pexpect                   4.9.0
pickleshare               0.7.5
pillow                    10.2.0
pip                       24.0
pkgutil_resolve_name      1.3.10
platformdirs              4.2.0
pooch                     1.8.1
prometheus_client         0.20.0
prompt-toolkit            3.0.42
protobuf                  4.25.3
psutil                    5.9.8
ptxcompiler               0.8.1
ptyprocess                0.7.0
pure-eval                 0.2.2
pyarrow                   14.0.2
pyarrow-hotfix            0.6
pycparser                 2.21
pyct                      0.5.0
pydantic                  1.10.14
pyee                      8.1.0
Pygments                  2.17.2
pylibcugraph              24.2.0
pylibraft                 24.2.0
pynndescent               0.5.11
pynvml                    11.4.1
pyparsing                 3.1.2
pyppeteer                 1.0.2
pyproj                    3.6.1
PySocks                   1.7.1
python-dateutil           2.9.0
python-json-logger        2.0.7
pytometry                 0.1.4
pytz                      2024.1
pyviz_comms               3.0.1
PyWavelets                1.4.1
PyYAML                    6.0.1
pyzmq                     25.1.2
raft-dask                 24.2.0
rapids_singlecell         0.9.6
readfcs                   1.1.7
referencing               0.33.0
requests                  2.31.0
rfc3339-validator         0.1.4
rfc3986-validator         0.1.1
rich                      13.7.1
rmm                       24.2.0
rpds-py                   0.18.0
Rtree                     1.2.0
scanpy                    1.10.0rc2
scikit-image              0.22.0
scikit-learn              1.4.1.post1
scikit-misc               0.3.1
scipy                     1.12.0
seaborn                   0.13.2
Send2Trash                1.8.2
session-info              1.0.0
setuptools                69.1.1
shapely                   2.0.3
simpervisor               1.0.0
single_cell_helper        0.0.1          
six                       1.16.0
sniffio                   1.3.1
sortedcontainers          2.4.0
soupsieve                 2.5
stack-data                0.6.2
statsmodels               0.14.1
stdlib-list               0.10.0
streamz                   0.6.4
tblib                     3.0.0
terminado                 0.18.0
texttable                 1.7.0
threadpoolctl             3.3.0
tifffile                  2024.2.12
tinycss2                  1.2.1
tomli                     2.0.1
toolz                     0.12.1
tornado                   6.4
tqdm                      4.66.2
traitlets                 5.14.1
treelite                  4.0.0
types-python-dateutil     2.8.19.20240311
typing_extensions         4.10.0
typing-utils              0.1.0
uc-micro-py               1.0.3
ucx-py                    0.36.0
umap-learn                0.5.5
unicodedata2              15.1.0
uri-template              1.3.0
urllib3                   1.26.18
wcwidth                   0.2.13
webcolors                 1.13
webencodings              0.5.1
websocket-client          1.7.0
websockets                10.4
wget                      3.2
wheel                     0.42.0
widgetsnbextension        4.0.10
wrapt                     1.16.0
xarray                    2024.2.0
xgboost                   2.0.3
xyzservices               2023.10.1
yarl                      1.9.4
zarr                      2.17.1
zict                      3.0.0
zipp                      3.17.0
```

</details>



Discussion:
Thanks for the report, I can confirm. Here's a more minimal case:

```python
import pandas as pd
from scanpy.get._aggregated import _combine_categories

_combine_categories(
    pd.DataFrame({
        "a": [1, 1, 1, 1, 2],
        "b": [1, 1, 2, 2, 3],
        "c": [1, 2, 1, 2, 1],
    }),
    list("abc")
)
```

```
(['1_1_1', '1_1_2', '1_2_1', '1_2_2', '2_2_2']
 Categories (5, object): ['1_1_1', '1_1_2', '1_2_1', '1_2_2', '2_2_2'],
        a  b  c
 1_1_1  1  1  1
 1_1_2  1  1  2
 1_2_1  1  2  1
 1_2_2  1  2  2
 2_2_2  2  2  2)
```

There are wrong factors being calculated here. They should be larger. I'm still a little unclear on what the condition is where they are wrong.
 https://github.com/scverse/scanpy/blob/65f567e6905afbbf673ecf715ad0f796794ce9cb/scanpy/get/_aggregated.py#L379-L381
It's meant to be cumprod, not cumsum
FYI @Intron7 

Resolving patch:
diff --git a/scanpy/get/_aggregated.py b/scanpy/get/_aggregated.py
--- a/scanpy/get/_aggregated.py
+++ b/scanpy/get/_aggregated.py
@@ -377,7 +377,7 @@ def _combine_categories(
 
     # Calculating result codes
     factors = np.ones(len(cols) + 1, dtype=np.int32)  # First factor needs to be 1
-    np.cumsum(n_categories[::-1], out=factors[1:])
+    np.cumprod(n_categories[::-1], out=factors[1:])
     factors = factors[:-1][::-1]
 
     code_array = np.zeros((len(cols), df.shape[0]), dtype=np.int32)


Test patch:
diff --git a/scanpy/tests/test_aggregated.py b/scanpy/tests/test_aggregated.py
--- a/scanpy/tests/test_aggregated.py
+++ b/scanpy/tests/test_aggregated.py
@@ -460,3 +460,19 @@ def test_dispatch_not_implemented():
     adata = sc.datasets.blobs()
     with pytest.raises(NotImplementedError):
         sc.get.aggregate(adata.X, adata.obs["blobs"], "sum")
+
+
+def test_factors():
+    from itertools import product
+
+    obs = pd.DataFrame(
+        product(range(5), range(5), range(5), range(5)), columns=list("abcd")
+    )
+    obs.index = [f"cell_{i:04d}" for i in range(obs.shape[0])]
+    adata = ad.AnnData(
+        X=np.arange(obs.shape[0]).reshape(-1, 1),
+        obs=obs,
+    )
+
+    res = sc.get.aggregate(adata, by=["a", "b", "c", "d"], func="sum")
+    np.testing.assert_equal(res.layers["sum"], adata.X)