Repository: statsmodels/statsmodels

Issue:
BUG: mutable model.df_model in discrete_model
Motivated by #3727:

> The k_extra attribute is for some generic, inherited methods, I don't remember which.

`k_extra` modifies `FooResults.df_model` and `df_resid`, but handling `k_extra` in a consistent manner requires a bit of kludge-work.  In the process of writing out _why_ it requires kludge-work, I found that a pet peeve was actually a bug.

Each of the `L1FooResults` classes alter `model.df_model` and `df_resid` within their `__init__` methods.  I think this should be avoided, as `model.df_foo` should be essentially immutable.  The problem caused by mutability is this:

```
>>> data = sm.datasets.randhie.load()
>>> model = sm.NegativeBinomial(data.endog, data.exog, loglike_method='nb2')
>>> res1 = model.fit()
>>> res2 = model.fit_regularized(alpha=100)
>>> res3 = model.fit()
>>> res1.df_model
8.0
>>> res3.df_model
6
>>> (res1.aic, res3.aic)
(87507.710855394151, 87503.710855394151)
```

`res1` and `res3` should be identical, but are not because the L1FooResults instance is modifying model.df_bar.  

In my local "screw backward compatibility" branch, the closest thing I've found to a solution is to define the Results-level `k_extra`, `df_model`, and `df_resid` as properties/cache_readonlys that get mixed in to DiscreteResults:

```
class DiscreteResultsFreedomMixin(object):

	@cache_readonly
	def k_extra(self):
		model = self.model
		k_extra = model.k_extra
		if getattr(model, 'loglike_method', '').startswith('nb'): ## FIXME: kludge
			k_extra -= 1
		return k_extra

	@cache_readonly
	def df_model(self):
		k_constr = self.k_constr
		k_extra = self.k_extra

		ret = self.model.df_model - (k_constr + k_extra)
		return ret

	@cache_readonly
	def df_resid(self):
		k_constr = self.k_constr
		k_extra = self.k_extra

		ret = self.model.df_resid + (k_constr + k_extra)
		return ret
```

This avoids the need to alter model.df_foo (which gets set only in `DiscreteModel.initialize` and `MultinomialModel.initialize`), but I'm not wild about the kludge that sets `k_extra`, in part because I'm not sure it is generally-correct as opposed to doesn't-cause-test-failures-correct.



Discussion:
> Each of the L1FooResults classes alter model.df_model and df_resid within their __init__ methods. I think this should be avoided, as model.df_foo should be essentially immutable.

(I had to partially rewrite this because initially I misread your points.)

about Results.df_xxx:

If the penalization parameter is set in `fit_regularized`, then the df_xxx are only determined in `fit` and the obvious location is to put it into the Results. The effective degrees of freedom depend on the amount of penalization and is usually defined as the trace of the projection matrix.

If constraints are specified in `fit_constrained`, then the df_xxx also depend on the number of constraints.

cov_type='cluster' is adjusting df for inference only, so it sets `df_resid_inference`. Other adjustments are done through small sample options, but leave the original exog based df_resid, df_model unchanged.

about Model.df_xxx
You are right, those should not be changed by `fit` and is a bug. 

specific:
what is your k_extra? I introduced k_extra exactly to avoid hardcoding models in the generic results methods, i.e. for NegativeBinomial, and GeneralizedPoisson and similar in future.

I still don't think that cached properties give us enough flexibility to override df_xxx in the results instance model or case specific.


I think we're in agreement on the main point regarding the bug: setting `res.df_foo` should not be altering `model.df_foo`.  This can be fixed in a pretty non-invasive way inside discrete_model.  I'll make a PR after wrapping up some day-job responsibilities.

In cases where `df_resid_inference` is used, does that affect what the "correct" values for `df_model` and `df_resid` are?

> If constraints are specified in fit_constrained, then the df_xxx also depend on the number of constraints.

In the interest of brevity I omitted the mixin class that defines `df_model` and `df_resid` for the `L1BazResults` classes.

> specific:
what is your k_extra? I introduced k_extra exactly to avoid hardcoding models in the generic results methods, i.e. for NegativeBinomial, and GeneralizedPoisson and similar in future.

Out of personal preference I initialize `k_extra` to 0 in `Model.__init__` and avoid `getattr`.  So `model.k_extra` here is equivalent to `getattr(model, 'k_extra', 0)` in master.  That could have been clearer; my mistake.

Discussion of `k_extra` and more generally `df_model` is a big topic on which we already have a few threads going.  My main contentions are a) defining these in fewer places is better than more, b) modifying them in place is undesirable, c) being explicit about the reasoning behind cross-model definitions would be helpful.

> I still don't think that cached properties give us enough flexibility to override df_xxx in the results instance model or case specific.

That's fair.  Would you be more comfortable with a `OneTimeProperty`?  Or the `self._df_model` approach used in #3725?
to Results.df_resid or Results.df_model:
We need to be able to set it in `Results.__init__`. All descriptors like the cached properties are a pain to override, so I think they are out for df_xxx.
If we use a explicit property, then `._df_resid` `_df_model` can be created in `Result.__init__` and that should be ok. Although, I'm not completely convinced we need a property if we set it in `Result.__init__` by default/inheritance anyway. (maybe using a property is a simpler way to set the default.)
BTW: page search doesn't show any `_df_resid` or `_df_resid` in #3725 

`df_resid_inference` is essentially like `use_t` and only affects Wald inference if use_t is true, it will not affect AIC/BIC, LR tests and similar that use df_resid. ("Never say never" I have no idea about or did not run into df correction for those. We have AICc but I think the theoretical idea behind it is different.)
It's currently only used for the cluster robust cov_type, but is more general. There are various cases where t/F distribution with an adjusted degrees of freedom provides a better small sample approximation to the distribution of a wald test statistic. (example in two sample t-test with unequal variance the df is adjusted)
Those df correction are data and application specific, so currently almost everywhere we rely by default on asymptotic normality. 
This reminds me of another case: With imputation of missing values, e.g. with MICE, we don't have the same number of observations for each exog. In some cases, the R package used different nobs/df_resid for different exog. I think the simple way is just to use the average nobs. (I don't really know the details, and asymptotic normality is the simplifying assumption.)


> BTW: page search doesn't show any _df_resid or _df_resid in #3725

You're right; I was confusing this with an earlier conversation in #1664.

> We need to be able to set it in `Results.__init__`. All descriptors like the cached properties are a pain to override, so I think they are out for df_xxx.

For the purposes of the actual bug under discussion and the upcoming PR, I'm planning to get rid of the lines that modify model.df_xxx, but otherwise leave the design unchanged.


to add to df_resid_inference: this could also be data and variable specific (`*`), but it will be all special case code, and any extensions there will be part of small sample inference improvements and separate from generic df_resid, df_model usage.
(`*`) the idea behind panel or cluster df correction is that we might have only little variation in the data, e.g. if a variable is fixed over time, then we have only cross-cluster or cross-panel variation and the effective nobs for **between** variation is the number of clusters or panel units and not the total number of observations.
>  I'm planning to get rid of the lines that modify model.df_xxx, but otherwise leave the design unchanged.

It should assign the adjusted values for the df_xxx directly to the results instance without changing the model.
`Poisson.fit_constrained` is directly adjusting the created results instance. (There is no special results class for constrained fit.)
That sounds very much like what I was trying to communicate, but apparently failed.  This should get clarified shortly when the PR hits.

Resolving patch:
diff --git a/statsmodels/discrete/discrete_model.py b/statsmodels/discrete/discrete_model.py
--- a/statsmodels/discrete/discrete_model.py
+++ b/statsmodels/discrete/discrete_model.py
@@ -3646,16 +3646,14 @@ def __init__(self, model, cntfit):
         # entry in params has been set zero'd out.
         self.trimmed = cntfit.mle_retvals['trimmed']
         self.nnz_params = (self.trimmed == False).sum()
-        # update degrees of freedom
-        self.model.df_model = self.nnz_params - 1
-        self.model.df_resid = float(self.model.endog.shape[0] - self.nnz_params)
+
+        # Set degrees of freedom.  In doing so,
         # adjust for extra parameter in NegativeBinomial nb1 and nb2
         # extra parameter is not included in df_model
         k_extra = getattr(self.model, 'k_extra', 0)
-        self.model.df_model -= k_extra
-        self.model.df_resid += k_extra
-        self.df_model = self.model.df_model
-        self.df_resid = self.model.df_resid
+
+        self.df_model = self.nnz_params - 1 - k_extra
+        self.df_resid = float(self.model.endog.shape[0] - self.nnz_params) + k_extra
 
 class PoissonResults(CountResults):
     def predict_prob(self, n=None, exog=None, exposure=None, offset=None,
@@ -3884,10 +3882,8 @@ def __init__(self, model, bnryfit):
         # entry in params has been set zero'd out.
         self.trimmed = bnryfit.mle_retvals['trimmed']
         self.nnz_params = (self.trimmed == False).sum()
-        self.model.df_model = self.nnz_params - 1
-        self.model.df_resid = float(self.model.endog.shape[0] - self.nnz_params)
-        self.df_model = self.model.df_model
-        self.df_resid = self.model.df_resid
+        self.df_model = self.nnz_params - 1
+        self.df_resid = float(self.model.endog.shape[0] - self.nnz_params)
 
 
 class MultinomialResults(DiscreteResults):
@@ -4034,11 +4030,9 @@ def __init__(self, model, mlefit):
         self.trimmed = mlefit.mle_retvals['trimmed']
         self.nnz_params = (self.trimmed == False).sum()
 
-        #Note: J-1 constants
-        self.model.df_model = self.nnz_params - (self.model.J - 1)
-        self.model.df_resid = float(self.model.endog.shape[0] - self.nnz_params)
-        self.df_model = self.model.df_model
-        self.df_resid = self.model.df_resid
+        # Note: J-1 constants
+        self.df_model = self.nnz_params - (self.model.J - 1)
+        self.df_resid = float(self.model.endog.shape[0] - self.nnz_params)
 
 
 #### Results Wrappers ####


Test patch:
diff --git a/statsmodels/discrete/tests/test_discrete.py b/statsmodels/discrete/tests/test_discrete.py
--- a/statsmodels/discrete/tests/test_discrete.py
+++ b/statsmodels/discrete/tests/test_discrete.py
@@ -2354,6 +2354,31 @@ def test_optim_kwds_prelim():
     # rough check that convergence makes sense
     assert_allclose(res.predict().mean(), y.mean(), rtol=0.1)
 
+
+def test_unchanging_degrees_of_freedom():
+    # see GH3734
+    data = sm.datasets.randhie.load()
+    model = sm.NegativeBinomial(data.endog, data.exog, loglike_method='nb2')
+    params = np.array([-0.05654134, -0.21213734,  0.08783102, -0.02991825,
+                       0.22902315,  0.06210253,  0.06799444,  0.08406794,
+                       0.18530092,  1.36645186])
+
+    res1 = model.fit(start_params=params)
+    assert_equal(res1.df_model, 8)
+
+    reg_params = np.array([-0.04854   , -0.15019404,  0.08363671, -0.03032834,  0.17592454,
+        0.06440753,  0.01584555,  0.        ,  0.        ,  1.36984628])
+
+    res2 = model.fit_regularized(alpha=100, start_params=reg_params)
+    assert_(res2.df_model != 8)
+    # If res2.df_model == res1.df_model, then this test is invalid.
+
+    res3 = model.fit()
+    # Test that the call to `fit_regularized` didn't modify model.df_model inplace.
+    assert_equal(res3.df_model, res1.df_model)
+    assert_equal(res3.df_resid, res1.df_resid)
+
+
 if __name__ == "__main__":
     import pytest
     pytest.main([__file__, '-vvs', '-x', '--pdb'])