Repository: statsmodels/statsmodels

Issue:
use loc/scale in ProbPlot if provided by the user
This resolves most of the issue in #3254.
The code now allows to pass custom loc/scale parameters to the gofplots ProbPlot.

However, `self.fit_params` is still called even if `fit` is `False`, since it may be used in `ProbPlot.sample_quantiles`.


Discussion:

[![Coverage Status](https://coveralls.io/builds/10302328/badge)](https://coveralls.io/builds/10302328)

Coverage remained the same at 90.306% when pulling **9e4ff5f3e3279dca606a6b3e2221c03533336c7a on paberr:master** into **898ddfc483c45bb0f8e5156dd8506abda84c9b63 on statsmodels:master**.

Need a test to make sure that this has an effect.
Tests have been added to check whether args are correctly set and then passed to the distribution.
They fail on the current master, but they do pass with my changes.

[![Coverage Status](https://coveralls.io/builds/10311914/badge)](https://coveralls.io/builds/10311914)

Coverage increased (+0.003%) to 90.308% when pulling **375f25e23bed79ba4ed42bbfbbe34170ae06afdb on paberr:master** into **898ddfc483c45bb0f8e5156dd8506abda84c9b63 on statsmodels:master**.

@josef-pkt LGTM.
Unfortunately I cannot tell whether this is the correct fix or not.

Probability plots are a bit messy, there are too many possible variations to display them and it mixes computational and display issues, and there are no good reference packages that I know of. I have a avoided this issue for years. The specific problem here is whether loc and scale are supposed to be distribution parameters or handled generically as plot element, related to the line based on loc scale.
I put it on my todo list, but it might take quite a bit of time to understand this again.
see also #2183

As work around if you have all distribution parameters, then providing a frozen distribution should work.

to clarify: I agree with the intention of allowing users to specify loc and scale, but I cannot tell right now how this is supposed to work. Maybe it is a straightforward bug.

and to be more specific (what I just remembered)

 #2183 item `ii`: We have to distinguish whether the plot refers to the null hypothesis that the data comes from a loc-scale family (e.g. is it normal, t(5) with some mean and variance) or does it come from a specific distribution (with all parameters including loc and scale specified).

I don't know if we have the necessary options in prob plot to distinguish the two cases.
The fix here looks correct, but is only part of the problem.
Trying to think may way at least partially through this again, AFAICS, the use of fit_params in `sample_quantiles` is wrong, and prevents from seeing a lack of fit in loc and scale in the plots.
https://github.com/statsmodels/statsmodels/issues/3981
that is it always standardizes by loc-scale estimated from the data instead of using the provided loc and scale.
Or, more likely we need more options
using loc and scale in `dist`, changes the distribution on the x-axis, i.e. the null distribution.
What I was thinking is that it should changed the standardization on the y axis, i.e. we should standardize the data under the null.

The related but separate option that we need is whether x-axis (null) or y-axis (sample) should show loc-scale standardized distributions, or values based on the given or estimated loc-scale (i.e. non-standardized)
AFAICS, the unstandardized plot is in `probplot`, but then we need different `self.dist` for standardized qqplot and ppplot on one hand and unstandardized probplot on the other hand.

Also, some of the plots in other packages, mix the two, show standardized/standard null distribution on x-axis, and non-standardized sample values on the y-axis. In that case, the scatter plot should be close to a line but not necessarily the 45degree line.

Resolving patch:
diff --git a/statsmodels/graphics/gofplots.py b/statsmodels/graphics/gofplots.py
--- a/statsmodels/graphics/gofplots.py
+++ b/statsmodels/graphics/gofplots.py
@@ -28,13 +28,15 @@ class ProbPlot(object):
         scipy.stats.distributions.norm (a standard normal).
     distargs : tuple
         A tuple of arguments passed to dist to specify it fully
-        so dist.ppf may be called.
-    loc : float
-        Location parameter for dist
+        so dist.ppf may be called. distargs must not contain loc
+        or scale. These values must be passed using the loc or
+        scale inputs.
     a : float
         Offset for the plotting position of an expected order
         statistic, for example. The plotting positions are given
         by (i - a)/(nobs - 2*a + 1) for i in range(0,nobs+1)
+    loc : float
+        Location parameter for dist
     scale : float
         Scale parameter for dist
     fit : boolean
@@ -177,11 +179,20 @@ def __init__(self, data, dist=stats.norm, fit=False,
             self.scale = self.fit_params[-1]
             if len(self.fit_params) > 2:
                 self.dist = dist(*self.fit_params[:-2],
-                                 **dict(loc = 0, scale = 1))
+                                 **dict(loc=0, scale=1))
             else:
                 self.dist = dist(loc=0, scale=1)
-        elif distargs or loc == 0 or scale == 1:
-            self.dist = dist(*distargs, **dict(loc=loc, scale=scale))
+        elif distargs or loc != 0 or scale != 1:
+            try:
+                self.dist = dist(*distargs, **dict(loc=loc, scale=scale))
+            except Exception:
+                distargs = ', '.join([str(da) for da in distargs])
+                cmd = 'dist({distargs}, loc={loc}, scale={scale})'
+                cmd = cmd.format(distargs=distargs, loc=loc, scale=scale)
+                raise TypeError('Initializing the distribution failed.  This '
+                                'can occur if distargs contains loc or scale. '
+                                'The distribution initialization command '
+                                'is:\n{cmd}'.format(cmd=cmd))
             self.loc = loc
             self.scale = scale
         else:


Test patch:
diff --git a/statsmodels/graphics/tests/test_gofplots.py b/statsmodels/graphics/tests/test_gofplots.py
--- a/statsmodels/graphics/tests/test_gofplots.py
+++ b/statsmodels/graphics/tests/test_gofplots.py
@@ -137,6 +137,12 @@ def setup(self):
         self.line = '45'
         super(TestProbPlotRandomNormalLocScale, self).setup()
 
+    def test_loc_set(self):
+        assert self.prbplt.loc == 8.25
+
+    def test_scale_set(self):
+        assert self.prbplt.scale == 3.25
+
 
 class TestCompareSamplesDifferentSize(object):
     def setup(self):
@@ -158,6 +164,28 @@ def test_ppplot(self, close_figures):
         self.data2.ppplot(other=self.data1)
 
 
+class TestProbPlotRandomNormalLocScaleDist(BaseProbplotMixin):
+
+    def setup(self):
+        np.random.seed(5)
+        self.data = np.random.normal(loc=8.25, scale=3.25, size=37)
+        self.prbplt = sm.ProbPlot(self.data, loc=8, scale=3)
+        self.line = '45'
+        super(TestProbPlotRandomNormalLocScaleDist, self).setup()
+
+    def test_loc_set(self):
+        assert self.prbplt.loc == 8
+
+    def test_scale_set(self):
+        assert self.prbplt.scale == 3
+
+    def test_loc_set_in_dist(self):
+        assert self.prbplt.dist.mean() == 8.
+
+    def test_scale_set_in_dist(self):
+        assert self.prbplt.dist.var() == 9.
+
+
 class TestTopLevel(object):
     def setup(self):
         self.data = sm.datasets.longley.load(as_pandas=False)
@@ -193,3 +221,13 @@ def test_qqplot_2samples_arrays(self, close_figures):
         for line in ['r', 'q', '45', 's']:
             # test with arrays
             sm.qqplot_2samples(self.res, self.other_array, line=line)
+
+
+def test_invalid_dist_config(close_figures):
+    # GH 4226
+    np.random.seed(5)
+    data = sm.datasets.longley.load(as_pandas=False)
+    data.exog = sm.add_constant(data.exog, prepend=False)
+    mod_fit = sm.OLS(data.endog, data.exog).fit()
+    with pytest.raises(TypeError, match=r'dist\(0, 1, 4, loc=0, scale=1\)'):
+        sm.ProbPlot(mod_fit.resid, stats.t, distargs=(0, 1, 4))