Repository: sbi-dev/sbi

Issue:
Z-scoring can strongly affect results when outliers are present
When the training data is distributed very unevenly, e.g., count data with many very small counts and some very large counts, and the data we care about is in the small range, then z-scoring can prevent SNPE (or any other method) from obtaining an accurate posterior.

## Data
Poisson GLM count data: 

![Screenshot from 2020-09-24 16-13-31](https://user-images.githubusercontent.com/11921527/94156976-fa7d3500-fe80-11ea-9d3a-57fb0b4d6ee7.png)

## Problem
With `SNPE` and the default inference setting (MAF, z-scoring) this results in a `C2ST>0.95` (this problem is tractable). However, with `z_score_x=False` and the same settings otherwise, it results in `C2ST=0.51`. 

## Reason
Because of the huge differences in the counts and the resulting huge standard deviations, the relevant data points, i.e., the counts between 0 and, say 10, all map to the same data points after z-scoring. Thus, the density estimator will have a very hard time to learn the posterior in that region.
![Screenshot from 2020-09-24 16-18-12](https://user-images.githubusercontent.com/11921527/94157476-93ac4b80-fe81-11ea-891f-ac7f716536b7.png)

## Possible solution 
We could catch this problem with a `UserWarning` by
1) checking the number of unique training samples before and after the z-scoring?
2) checking for very large standard deviations in the data, or other measures of outliers?
3) ...?


Discussion:
+1 for checking the number of unique training samples before and after the z-scoring and possibly raising warnings

Resolving patch:
diff --git a/sbi/inference/base.py b/sbi/inference/base.py
--- a/sbi/inference/base.py
+++ b/sbi/inference/base.py
@@ -21,6 +21,7 @@
     handle_invalid_x,
     warn_on_invalid_x,
     warn_on_invalid_x_for_snpec_leakage,
+    warn_if_zscoring_changes_data,
 )
 from sbi.utils.sbiutils import get_simulations_since_round
 from sbi.utils.torchutils import process_device
@@ -247,6 +248,8 @@ def get_simulations(
 
         # Check for NaNs in simulations.
         is_valid_x, num_nans, num_infs = handle_invalid_x(x, exclude_invalid_x)
+        # Check for problematic z-scoring
+        warn_if_zscoring_changes_data(x)
         if warn_on_invalid:
             warn_on_invalid_x(num_nans, num_infs, exclude_invalid_x)
             warn_on_invalid_x_for_snpec_leakage(
diff --git a/sbi/utils/__init__.py b/sbi/utils/__init__.py
--- a/sbi/utils/__init__.py
+++ b/sbi/utils/__init__.py
@@ -29,6 +29,7 @@
     standardizing_transform,
     warn_on_invalid_x,
     warn_on_invalid_x_for_snpec_leakage,
+    warn_if_zscoring_changes_data,
     x_shape_from_simulation,
 )
 from sbi.utils.torchutils import (
diff --git a/sbi/utils/sbiutils.py b/sbi/utils/sbiutils.py
--- a/sbi/utils/sbiutils.py
+++ b/sbi/utils/sbiutils.py
@@ -10,10 +10,39 @@
 from torch import nn as nn
 from torch import ones, zeros
 from tqdm.auto import tqdm
+import warnings
 
 from sbi.utils.torchutils import BoxUniform
 
 
+def warn_if_zscoring_changes_data(x: Tensor, duplicate_tolerance: float = 0.1) -> None:
+    """Raise warning if z-scoring would create duplicate data points.
+    
+    Args:
+        x: data
+        duplicate_tolerance: tolerated proportion of duplicates after z-scoring.
+    """
+
+    # Count unique xs.
+    num_unique = torch.unique(x, dim=0).numel()
+
+    # z-score.
+    zx = (x - x.mean(0)) / x.std(0)
+
+    # Count again and warn on too many new duplicates.
+    num_unique_z = torch.unique(zx, dim=0).numel()
+
+    if (1 - duplicate_tolerance) * num_unique_z < num_unique:
+        warnings.warn(
+            """Z-scoring these data will result in many data points being mapped on the
+            same z-scored value. This can occur due to numerical inaccuracies when the
+            data covers a large range of values. Consider setting z_score_x=False if
+            you still want to run the inference and beware that large variance in the
+            data can be problematic for the underlying NN.""",
+            UserWarning,
+        )
+
+
 def x_shape_from_simulation(batch_x: Tensor) -> torch.Size:
     ndims = batch_x.ndim
     assert ndims >= 2, "Simulated data must be a batch with at least two dimensions."


Test patch:
diff --git a/tests/inference_with_NaN_simulator_test.py b/tests/inference_with_NaN_simulator_test.py
--- a/tests/inference_with_NaN_simulator_test.py
+++ b/tests/inference_with_NaN_simulator_test.py
@@ -138,3 +138,19 @@ def linear_gaussian_nan(
 
     # Compute the c2st and assert it is near chance level of 0.5.
     check_c2st(samples, target_samples, alg=f"{SNPE_C}")
+
+
+def test_z_scoring_warning():
+
+    # Create data with large variance.
+    num_dim = 2
+    theta = torch.ones(100, num_dim)
+    x = torch.rand(100, num_dim)
+    x[:50] += 1e7
+
+    # Make sure a warning is raised because z-scoring will map these data to duplicate
+    # data points.
+    with pytest.warns(UserWarning, match="Z-scoring these data will"):
+        SNPE_C(utils.BoxUniform(zeros(num_dim), ones(num_dim))).append_simulations(
+            theta, x
+        ).train(max_num_epochs=1)