Repository: scverse/scanpy

Issue:
combat processing errors
<!-- Please give a clear and concise description of what the bug is: -->
...when I did combat batch correction, it outputs errors

<!-- Put a minimal reproducible example that reproduces the bug in the code block below: -->
```python
# ComBat batch correction
sc.pp.combat(adata, key='sample')
```

<!-- Put your Error output in this code block (if applicable, else delete the block): -->
```pytb
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-350690ae55dc> in <module>
      1 # ComBat batch correction
----> 2 sc.pp.combat(adata, key='sample')

~/anaconda2/envs/scanpy/lib/python3.6/site-packages/scanpy/preprocessing/_combat.py in combat(adata, key, covariates, inplace)
    266         denom =  np.dot(dsq, np.ones((1, n_batches[j])))
    267         numer = np.array(bayesdata[batch_idxs] - np.dot(batch_design.loc[batch_idxs], gamma_star).T)
--> 268         bayesdata[batch_idxs] = numer / denom
    269 
    270     vpsq = np.sqrt(var_pooled).reshape((len(var_pooled), 1))

ValueError: operands could not be broadcast together with shapes (23259,18243) (23259,15479) 
```

#### Versions:
<!-- Output of scanpy.logging.print_versions() -->
> scanpy==1.4.6 anndata==0.7.1 umap==0.3.10 numpy==1.17.3 scipy==1.3.1 pandas==0.25.2 scikit-learn==0.21.3 statsmodels==0.11.1 python-igraph==0.8.0



Discussion:
Not sure what could cause this without a bit more info. Could you make an example that reproduces this bug that I could run on my machine? Something like [this](https://mrocklin.github.com/blog/work/2018/02/28/minimal-bug-reports).
> 不知道是什么原因导致的，而没有更多的信息。您能否举一个例子，重现我可以在计算机上运行的错误？像[这样](https://mrocklin.github.com/blog/work/2018/02/28/minimal-bug-reports)。

sorry, I cannot open the URL https://mrocklin.github.com/blog/work/2018/02/28/minimal-bug-reports
Can I send a tiny data and my code to you?
Huh, that's strange.

> Can I send a tiny data and my code to you?

Sounds good!
> 嗯，这很奇怪。
> 
> > 我可以发送少量数据和代码给您吗？
> 
> 听起来不错！

Can you give me your contact information so that I can send the data and code?
The best thing to do would just be to put a link to the data here, and paste the code.

It's actually easier to debug the more you can simplify the data that replicates the bug. Ideally, you could just send code that generates the data to replicate the bug.

If that isn't working out, you could send me a DM on the discourse forum?
@Zifeng1995, @ivirshup 

I received the same error caused by wrong shape of  "numer" in line 268 in _combat.py. 
In my case, I could resolved this problem by generating unique cell names ( i.e. adata.obs_names) as following. 

**Workaround** 
Before combat execution, in concatenate process of batch data, I specified index_unique='-' . 
e.g.) adata1.concatenate(adata2, adata3, ..., index_unique='-')

When index_unique=None, the error was occurred. However index_unique='-' was fine in my case.
I'm afraid that couldn't trace root cause of this problem due to busy day, but I hope this helps you. 

Sincerely. 
 
Thanks @rika-N! That's very helpful in figuring this out. I can now reproduce this with:

```python
import scanpy as sc
import numpy as np
import pandas as pd

a = sc.AnnData(
    np.random.random((5, 5)),
    obs=pd.DataFrame({"batch": list("aaabb")}, index=list("11234"))
)
sc.pp.combat(a, "batch")
```

<details>
<summary> Traceback </summary>

```pytb
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-16-e35e04bc067c> in <module>
      7     obs=pd.DataFrame({"batch": list("aaabb")}, index=list("11234"))
      8 )
----> 9 sc.pp.combat(a, "batch")

~/github/scanpy/scanpy/preprocessing/_combat.py in combat(adata, key, covariates, inplace)
    266         denom =  np.dot(dsq, np.ones((1, n_batches[j])))
    267         numer = np.array(bayesdata[batch_idxs] - np.dot(batch_design.loc[batch_idxs], gamma_star).T)
--> 268         bayesdata[batch_idxs] = numer / denom
    269 
    270     vpsq = np.sqrt(var_pooled).reshape((len(var_pooled), 1))

ValueError: operands could not be broadcast together with shapes (5,5) (5,3) 
```
</details>

The issue is that the `combat` function currently does not work when the observation names are not unique.

Resolving patch:
diff --git a/scanpy/preprocessing/_combat.py b/scanpy/preprocessing/_combat.py
--- a/scanpy/preprocessing/_combat.py
+++ b/scanpy/preprocessing/_combat.py
@@ -11,9 +11,7 @@
 
 
 def _design_matrix(
-        model: pd.DataFrame,
-        batch_key: str,
-        batch_levels: Collection[str],
+    model: pd.DataFrame, batch_key: str, batch_levels: Collection[str],
 ) -> pd.DataFrame:
     """\
     Computes a simple design matrix.
@@ -46,9 +44,9 @@ def _design_matrix(
 
     if other_cols:
         col_repr = " + ".join("Q('{}')".format(x) for x in other_cols)
-        factor_matrix = patsy.dmatrix("~ 0 + {}".format(col_repr),
-                                      model[other_cols],
-                                      return_type="dataframe")
+        factor_matrix = patsy.dmatrix(
+            "~ 0 + {}".format(col_repr), model[other_cols], return_type="dataframe"
+        )
 
         design = pd.concat((design, factor_matrix), axis=1)
         logg.info(f"Found {len(other_cols)} categorical variables:")
@@ -65,9 +63,7 @@ def _design_matrix(
 
 
 def _standardize_data(
-    model: pd.DataFrame,
-    data: pd.DataFrame,
-    batch_key: str,
+    model: pd.DataFrame, data: pd.DataFrame, batch_key: str,
 ) -> Tuple[pd.DataFrame, pd.DataFrame, np.ndarray, np.ndarray]:
     """\
     Standardizes the data per gene.
@@ -107,26 +103,26 @@ def _standardize_data(
     # compute pooled variance estimator
     B_hat = np.dot(np.dot(la.inv(np.dot(design.T, design)), design.T), data.T)
     grand_mean = np.dot((n_batches / n_array).T, B_hat[:n_batch, :])
-    var_pooled = (data - np.dot(design, B_hat).T)**2
+    var_pooled = (data - np.dot(design, B_hat).T) ** 2
     var_pooled = np.dot(var_pooled, np.ones((int(n_array), 1)) / int(n_array))
 
     # Compute the means
     if np.sum(var_pooled == 0) > 0:
-        print(
-            'Found {} genes with zero variance.'
-            .format(np.sum(var_pooled == 0))
-        )
-    stand_mean = np.dot(grand_mean.T.reshape((len(grand_mean), 1)), np.ones((1, int(n_array))))
+        print(f'Found {np.sum(var_pooled == 0)} genes with zero variance.')
+    stand_mean = np.dot(
+        grand_mean.T.reshape((len(grand_mean), 1)), np.ones((1, int(n_array)))
+    )
     tmp = np.array(design.copy())
     tmp[:, :n_batch] = 0
     stand_mean += np.dot(tmp, B_hat).T
 
     # need to be a bit careful with the zero variance genes
     # just set the zero variance genes to zero in the standardized data
-    s_data = np.where(var_pooled == 0, 0, (
-        (data - stand_mean) /
-        np.dot(np.sqrt(var_pooled), np.ones((1, int(n_array))))
-    ))
+    s_data = np.where(
+        var_pooled == 0,
+        0,
+        ((data - stand_mean) / np.dot(np.sqrt(var_pooled), np.ones((1, int(n_array))))),
+    )
     s_data = pd.DataFrame(s_data, index=data.index, columns=data.columns)
 
     return s_data, design, var_pooled, stand_mean
@@ -179,7 +175,9 @@ def combat(
         cov_exist = np.isin(covariates, adata.obs_keys())
         if np.any(~cov_exist):
             missing_cov = np.array(covariates)[~cov_exist].tolist()
-            raise ValueError('Could not find the covariate(s) {!r} in adata.obs'.format(missing_cov))
+            raise ValueError(
+                'Could not find the covariate(s) {!r} in adata.obs'.format(missing_cov)
+            )
 
         if key in covariates:
             raise ValueError('Batch key and covariates cannot overlap')
@@ -192,17 +190,13 @@ def combat(
         X = adata.X.A.T
     else:
         X = adata.X.T
-    data = pd.DataFrame(
-        data=X,
-        index=adata.var_names,
-        columns=adata.obs_names,
-    )
+    data = pd.DataFrame(data=X, index=adata.var_names, columns=adata.obs_names,)
 
     sanitize_anndata(adata)
 
     # construct a pandas series of the batch annotation
     model = adata.obs[[key] + (covariates if covariates else [])]
-    batch_info = model.groupby(key).groups.values()
+    batch_info = model.groupby(key).indices.values()
     n_batch = len(batch_info)
     n_batches = np.array([len(v) for v in batch_info])
     n_array = float(sum(n_batches))
@@ -215,12 +209,14 @@ def combat(
     logg.info("Fitting L/S model and finding priors\n")
     batch_design = design[design.columns[:n_batch]]
     # first estimate of the additive batch effect
-    gamma_hat = np.dot(np.dot(la.inv(np.dot(batch_design.T, batch_design)), batch_design.T), s_data.T)
+    gamma_hat = (
+        la.inv(batch_design.T @ batch_design) @ batch_design.T @ s_data.T
+    ).values
     delta_hat = []
 
     # first estimate for the multiplicative batch effect
     for i, batch_idxs in enumerate(batch_info):
-        delta_hat.append(s_data[batch_idxs].var(axis=1))
+        delta_hat.append(s_data.iloc[:, batch_idxs].var(axis=1))
 
     # empirically fix the prior hyperparameters
     gamma_bar = gamma_hat.mean(axis=1)
@@ -238,7 +234,7 @@ def combat(
         # temp[0] is the additive batch effect
         # temp[1] is the multiplicative batch effect
         gamma, delta = _it_sol(
-            s_data[batch_idxs].values,
+            s_data.iloc[:, batch_idxs].values,
             gamma_hat[i],
             delta_hat[i].values,
             gamma_bar[i],
@@ -261,11 +257,14 @@ def combat(
         # we basically substract the additive batch effect, rescale by the ratio
         # of multiplicative batch effect to pooled variance and add the overall gene
         # wise mean
-        dsq = np.sqrt(delta_star[j,:])
+        dsq = np.sqrt(delta_star[j, :])
         dsq = dsq.reshape((len(dsq), 1))
-        denom =  np.dot(dsq, np.ones((1, n_batches[j])))
-        numer = np.array(bayesdata[batch_idxs] - np.dot(batch_design.loc[batch_idxs], gamma_star).T)
-        bayesdata[batch_idxs] = numer / denom
+        denom = np.dot(dsq, np.ones((1, n_batches[j])))
+        numer = np.array(
+            bayesdata.iloc[:, batch_idxs]
+            - np.dot(batch_design.iloc[batch_idxs], gamma_star).T
+        )
+        bayesdata.iloc[:, batch_idxs] = numer / denom
 
     vpsq = np.sqrt(var_pooled).reshape((len(var_pooled), 1))
     bayesdata = bayesdata * np.dot(vpsq, np.ones((1, int(n_array)))) + stand_mean
@@ -329,13 +328,17 @@ def _it_sol(
     # we place a normally distributed prior on gamma and and inverse gamma prior on delta
     # in the loop, gamma and delta are updated together. they depend on each other. we iterate until convergence.
     while change > conv:
-        g_new = (t2*n*g_hat + d_old*g_bar) / (t2*n + d_old)
-        sum2 = s_data - g_new.reshape((g_new.shape[0], 1)) @ np.ones((1, s_data.shape[1]))
+        g_new = (t2 * n * g_hat + d_old * g_bar) / (t2 * n + d_old)
+        sum2 = s_data - g_new.reshape((g_new.shape[0], 1)) @ np.ones(
+            (1, s_data.shape[1])
+        )
         sum2 = sum2 ** 2
         sum2 = sum2.sum(axis=1)
-        d_new = (0.5*sum2 + b) / (n/2.0 + a-1.0)
+        d_new = (0.5 * sum2 + b) / (n / 2.0 + a - 1.0)
 
-        change = max((abs(g_new - g_old) / g_old).max(), (abs(d_new - d_old) / d_old).max())
+        change = max(
+            (abs(g_new - g_old) / g_old).max(), (abs(d_new - d_old) / d_old).max()
+        )
         g_old = g_new  # .copy()
         d_old = d_new  # .copy()
         count = count + 1
@@ -346,10 +349,10 @@ def _it_sol(
 def _aprior(delta_hat):
     m = delta_hat.mean()
     s2 = delta_hat.var()
-    return (2*s2 + m**2) / s2
+    return (2 * s2 + m ** 2) / s2
 
 
 def _bprior(delta_hat):
     m = delta_hat.mean()
     s2 = delta_hat.var()
-    return (m*s2 + m**3) / s2
+    return (m * s2 + m ** 3) / s2


Test patch:
diff --git a/scanpy/tests/test_combat.py b/scanpy/tests/test_combat.py
--- a/scanpy/tests/test_combat.py
+++ b/scanpy/tests/test_combat.py
@@ -2,6 +2,8 @@
 import pandas as pd
 from sklearn.metrics import silhouette_score
 
+from anndata.tests.helpers import assert_equal
+
 import scanpy as sc
 from scanpy.preprocessing._combat import _standardize_data, _design_matrix
 
@@ -12,17 +14,16 @@ def test_norm():
     # load in data
     adata = sc.datasets.blobs()
     key = 'blobs'
-    data = pd.DataFrame(data=adata.X.T, index=adata.var_names,
-                        columns=adata.obs_names)
+    data = pd.DataFrame(data=adata.X.T, index=adata.var_names, columns=adata.obs_names)
 
     # construct a pandas series of the batch annotation
     batch = pd.Series(adata.obs[key])
     model = pd.DataFrame({'batch': batch})
 
     # standardize the data
-    s_data, design, var_pooled,  stand_mean = _standardize_data(model, data, 'batch')
+    s_data, design, var_pooled, stand_mean = _standardize_data(model, data, 'batch')
 
-    assert np.allclose(s_data.mean(axis = 1), np.zeros(s_data.shape[0]))
+    assert np.allclose(s_data.mean(axis=1), np.zeros(s_data.shape[0]))
 
 
 def test_covariates():
@@ -36,7 +37,9 @@ def test_covariates():
     adata.obs['cat2'] = np.random.binomial(2, 0.1, size=(adata.n_obs))
     adata.obs['num1'] = np.random.normal(size=(adata.n_obs))
 
-    X2 = sc.pp.combat(adata, key=key, covariates=['cat1', 'cat2', 'num1'], inplace=False)
+    X2 = sc.pp.combat(
+        adata, key=key, covariates=['cat1', 'cat2', 'num1'], inplace=False
+    )
     sc.pp.combat(adata, key=key, covariates=['cat1', 'cat2', 'num1'], inplace=True)
 
     assert X1.shape == X2.shape
@@ -48,6 +51,26 @@ def test_covariates():
     assert len(design.columns) == 4 + len(batch_cats) - 1
 
 
+def test_combat_obs_names():
+    # Test for fix to #1170
+    X = np.random.random((200, 100))
+    obs = pd.DataFrame(
+        {"batch": pd.Categorical(np.random.randint(0, 2, 200))},
+        index=np.repeat(np.arange(100), 2).astype(str),  # Non-unique index
+    )
+    a = sc.AnnData(X, obs)
+    b = a.copy()
+    b.obs_names_make_unique()
+
+    sc.pp.combat(a, "batch")
+    sc.pp.combat(b, "batch")
+
+    assert_equal(a.X, b.X)
+
+    a.obs_names_make_unique()
+    assert_equal(a, b)
+
+
 def test_silhouette():
     # this test checks wether combat can align data from several gaussians
     # it checks this by computing the silhouette coefficient in a pca embedding
@@ -63,6 +86,6 @@ def test_silhouette():
     X_pca = adata.obsm['X_pca']
 
     # compute silhouette coefficient in pca
-    sh = silhouette_score(X_pca[:, :2], adata.obs['blobs'].values )
+    sh = silhouette_score(X_pca[:, :2], adata.obs['blobs'].values)
 
     assert sh < 0.1