Repository: sbi-dev/sbi

Issue:
SBC tutorial
we need a detailed tutorial about SBC and including help for understanding the SBC rank plots. 

E.g., it would be great to build SBC intuition by showing how SBC detects bias and dispersion in a simple linear Gaussian simulator example, like here: 

<img width="863" alt="image" src="https://user-images.githubusercontent.com/11921527/150772043-c8512bbf-3eaa-4bce-ba7b-2b2825e7879b.png">



Discussion:
I am happy to help with this. But it would be cool to know where the plots came from and how they were generated.
I generated those plots using a simple Gaussian example and calculating SBC and coverage for different amounts of biases and dispersions.
Ok, to me ... this is wonderful but hard to digest. I think such plots can be the finish of this tutorial. But let's not get ahead of ourselves. I'd first take one of the corner cases you outlined and illustrate the rank plots, then go to the rank plots summary statistics (either the posterior ranks or the KS pvalues). Then add everything up and compile such a facetgrid or matrix of plots.

Cool, question would be how to go about this? 
Should this be a standalone jupyter notebook or something more like [this markdown](https://github.com/mackelab/sbi/blob/main/docs/docs/tutorial/03_embedding_net.md) (which likely was generated from a notebook)?
yes, totally agree. I just posted this here for reference because I had this plot lying around in some notebook. 

Yes, let's plan come up with an outline of the tutorial here and one of us makes a first draft in a notebook and then we improve from there? 

How about we draft an outline together in Notion and then post is as reference here? 
https://www.notion.so/SBC-Tutorial-e36c4b0a0e7d4715806e6845951c36cf
Sounds like a plan.

Resolving patch:
diff --git a/tutorials/utils_13_diagnosis_sbc.py b/tutorials/utils_13_diagnosis_sbc.py
new file mode 100644
--- /dev/null
+++ b/tutorials/utils_13_diagnosis_sbc.py
@@ -0,0 +1,79 @@
+import math
+from typing import Callable, Optional
+
+import torch
+
+from sbi.types import Shape
+
+
+# Mocking a neural posterior
+class BiasedPosterior:
+    def __init__(self, posterior: Callable, shift: float = 5.0):
+        """give me a prior/posterior and I'll shift it by a scalar `shift`. All calls to `sample` are wrapped."""
+
+        self.shift = shift
+        self.posterior = posterior
+
+    def set_default_x(self, x):
+
+        self.posterior.set_default_x(x)
+
+    def sample(
+        self,
+        sample_shape: Shape = torch.Size(),
+        x: Optional[torch.Tensor] = None,
+        show_progress_bars: bool = False,
+    ):
+        return (
+            self.posterior.sample(
+                sample_shape, x, show_progress_bars=show_progress_bars
+            )
+            + self.shift
+        )
+
+
+class DispersedPosterior:
+    def __init__(self, posterior: Callable, dispersion: float = 1.05):
+        """give me a posterior and I'll disperse it. All calls to `sample` are wrapped.
+        This class exploits: Var(aX) = a**2 * Var(X) for any random variable X
+        while retaining the expectation value E[X] of all samples.
+
+        Parameters:
+            posterior: posterior distribution modelled like NeuralPosterior
+            dispersion: choose values <1. to make the variance smaller,
+                choose values >1. to make the variance larger (distribution more wide)
+        """
+
+        self.dispersion = math.sqrt(dispersion)
+        self.posterior = posterior
+
+    def set_default_x(self, x):
+
+        self.posterior.set_default_x(x)
+
+    def sample(
+        self,
+        sample_shape: Shape = torch.Size(),
+        x: Optional[torch.Tensor] = None,
+        show_progress_bars: bool = False,
+    ):
+
+        value = self.posterior.sample(
+            sample_shape, x, show_progress_bars=show_progress_bars
+        )
+
+        # obtain the median of all samples before applying
+        # the dispersion to them (use median for more robust estimate)
+        median = torch.median(value, dim=0)  # dim 0 is the batch dimension
+
+        # disperse the samples
+        dispersed = value * self.dispersion
+
+        # obtain the new median after the dispersion
+        median_ = torch.median(dispersed, dim=0)
+
+        # shift to obtain the original expectation values
+        # (we only want to disperse the samples, not offset)
+        shift = median.values - median_.values
+
+        return dispersed + shift


Test patch: