Repository: graspologic-org/graspologic

Issue:
[BUG] SBMEstimator is not right; it is systematically underbiased for loopless graphs
## Expected Behavior

SBMEstimator does not seem to properly account for loops.

## Actual Behavior

I cannot find anywhere in the codebase to support that the loops are properly removed before estimating block probabilities. From what I gather, the SBMEstimator will first obtain block indices for each vertex https://github.com/microsoft/graspologic/blob/ff34382d1ffa0b7ea5f0e005525b7364f977e86f/graspologic/models/sbm_estimators.py#L208, and then move in the next line to calculate the block probability matrix https://github.com/microsoft/graspologic/blob/ff34382d1ffa0b7ea5f0e005525b7364f977e86f/graspologic/models/sbm_estimators.py#L212 but does not take an input for whether or not the network is directed, which feels like it should be critical. This probability is then computed https://github.com/microsoft/graspologic/blob/ff34382d1ffa0b7ea5f0e005525b7364f977e86f/graspologic/models/sbm_estimators.py#L482, which is then passed along to https://github.com/microsoft/graspologic/blob/ff34382d1ffa0b7ea5f0e005525b7364f977e86f/graspologic/models/base.py#L16, all of which are also performed ignorant whether or not there are loops, so I cannot possibly see how loops could be being properly accounted for. Seems unnecessarily tedious to trickle through with a debugger line-by-line, as I can just prove that this mishandle occurs using code instead.

## Example Code

The below code simulates/fits an ER(10, 0.5) graph, and uses the fact that the adjustment factor for the probability is going to be n^2 in the computation (not right) instead of n*(n-1) (which is right). If this adjustment is handled wrong, we should be able to just account for it when producing our probability estimate, which is what I do below.

```
import graspologic as gp
import numpy as np

n = 10
A = gp.simulations.er_np(n, 0.5, directed=False, loops=False)
# fit it using graspologic
fit_mod = gp.models.EREstimator(directed=False, loops=False).fit(A)

print("Estimate: {:.4f}".format(fit_mod.p_))
# if loops were improperly accounted for (e.g., ignored)
# we should get the same answer for the next two...
# the first one assumes that loops result in an overcount by a factor of n^2/(2*binom(n, 2)) based on how
# graspologic's sbm handles directed networks (ignores)
print("Estimate (adjusted with assumption of loops being mishandled): {:.4f}".format(fit_mod.p_ * (n**2/(n*(n-1)))))

inds = np.triu_indices(n, k=1)
print("Correct answer: {:.4f}".format(np.mean(A[inds])))
```

Which produces:

```
Estimate: 0.5400
Estimate (adjusted with assumption of loops being mishandled): 0.6000
Correct answer: 0.6000
```

The first estimate is underbiased in all my attempts, whereas the second two are both *always* correct, which tells me that the diagonal is being ignored, as if the diagonal were not being ignored, these would otherwise not be giving the same answer.

## Your Environment
* Python version: 3.8.4
* graspologic version: whatever is on master as of 3/10/2022; looks like 1.0.1

## Additional Details
Any other contextual information you might feel is important.



Discussion:

Resolving patch:
diff --git a/graspologic/models/sbm_estimators.py b/graspologic/models/sbm_estimators.py
--- a/graspologic/models/sbm_estimators.py
+++ b/graspologic/models/sbm_estimators.py
@@ -209,7 +209,9 @@ def fit(
 
         if not self.loops:
             graph = remove_loops(graph)
-        block_p = _calculate_block_p(graph, block_inds, block_vert_inds)
+        block_p = _calculate_block_p(
+            graph, block_inds, block_vert_inds, loops=self.loops
+        )
 
         if not self.directed:
             block_p = symmetrize(block_p)
@@ -409,7 +411,9 @@ def fit(
 
         if not self.loops:
             graph = graph - np.diag(np.diag(graph))
-        block_p = _calculate_block_p(graph, block_inds, block_vert_inds)
+        block_p = _calculate_block_p(
+            graph, block_inds, block_vert_inds, loops=self.loops
+        )
 
         out_degree = np.count_nonzero(graph, axis=1).astype(float)
         in_degree = np.count_nonzero(graph, axis=0).astype(float)
@@ -484,12 +488,14 @@ def _calculate_block_p(
     block_inds: Collection[int],
     block_vert_inds: List[int],
     return_counts: bool = False,
+    loops: bool = False,
 ) -> np.ndarray:
     """
     graph : input n x n graph
     block_inds : list of length n_communities
     block_vert_inds : list of list, for each block index, gives every node in that block
     return_counts : whether to calculate counts rather than proportions
+    loops : whether self loops are possible in the graph
     """
 
     n_blocks = len(block_inds)
@@ -502,6 +508,10 @@ def _calculate_block_p(
         from_inds = block_vert_inds[from_block]
         to_inds = block_vert_inds[to_block]
         block = graph[from_inds, :][:, to_inds]
+        # if a block is from a community to itself, self loops are possible, so remove
+        # them from the computation to avoid underbias
+        if from_block == to_block and not loops:
+            block = block[~np.eye(block.shape[0], dtype=bool)]
         if return_counts:
             p = np.count_nonzero(block)
         else:


Test patch: