Repository: statsmodels/statsmodels

Issue:
BUG : MICEData creates new values after imputation
This imputation model is supposed to impute values from other(pre-existing) observations, as I know.

It does, however, create a value that did never exist in other observations. 

I used this imputation after applying LabelEnconder in Scikit-Learn. 

In this process, NaN values were replaced with 0(if NaN exists) and after they were encoded by LabelEncoder, I again replaced 0 with NaN values. And then I did imputation as uploaded here.

Any suggestions about this problem?


Discussion:
I'm not sure what might be going on here.  Can you send us a reproducible,
simple example?

On Mon, Sep 24, 2018 at 12:03 PM swab2000 <notifications@github.com> wrote:

> This imputation model is supposed to impute values from
> other(pre-existing) observations, as I know.
>
> It does, however, create a value that did never exist in other
> observations.
>
> I used this imputation after applying LabelEnconder in Scikit-Learn.
>
> In this process, NaN values were replaced with 0(if NaN exists) and after
> they were encoded by LabelEncoder, I again replaced 0 with NaN values. And
> then I did imputation as uploaded here.
>
> Any suggestions about this problem?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/5254>, or mute the
> thread
> <https://github.com/notifications/unsubscribe-auth/ACiww7K11_XatznnlgYv_IedkT8xjnCXks5ueQJmgaJpZM4W291G>
> .
>

MICE imputation code that you built in python is based on Predictive Mean Matching(PMM), as I see. The problem is PMM process only creates imputation that already exists in observed data. For example,

X :

x1 | x2 | x3
-- | -- | --
1 | 0.6 | A
4 | NA | B
NA | 1.4 | C
2 | 3.8 | B
3 | 5.2 | NA

if the dataset has 3 NA values as I mentioned above, and for x1 case, the MICE algorithm based on PMM should suggest imputation for NA value in 1, 4, 2, 3. Likewise, for x3 case, NA must be imputed within A, B, C. However, MICEData(X) creates somewhat strange values that never exists in observed data, like 0.404994. Moreover, if there are several NA values in one column(variable), all the NA values are imputed by the same, not-existing, strange values.

As I mentioned before, I used LabelEncoder in scikit-learn packages for categorical variables, in the example, x3(A B C B --> 1 2 3 1). I did this because MICEData does not take 'string' or 'object' type as input. I know this is a bad idea for categorical variables and also there is few reports imputing categorical variables via Multiple Imputation. Anyway, my total framework for imputation process is as follows.

Numerical Variables    -------------------------------------------------> MICE
Categorical Variables  ---------------->  LabelEncoder -------------->  MICE

I am not sure if the problem takes place in LabelEncoder, but this problem happens in both numerical and categorical cases.

I appreciate your prompt response and I hope you could capture the problem I explained. I cannot go further because of this problem.


Not an imputation expert, but in an effort to help triage, looking at the code for MICEData here is what I can tell where those strange values are coming from.  If I am reading the doc string for `_initial_imputation` it may be working not as intended.

Looking at `_initial_imputation` inside `MICEData` the docstring:
![image](https://user-images.githubusercontent.com/31365056/46484565-9fe3fd80-c7bf-11e8-8f89-02ff558b5387.png)

The way I interpret that statement is that the missing values in a given column will initially be set to the observed value in the column closest to the mean.  But in reality it is setting it to the absolute distance from the mean, which in the sample case is 0.5 for x1.

```Python
from statsmodels.imputation import mice
import numpy as np
import pandas as pd
d = {'x1':[1,4,np.nan,2,3], 'x2':[0.6,np.nan, 1.4, 3.8, 5.2], 'x3':[1,2,3,4,np.nan]}
df = pd.DataFrame(d)
df
```
![image](https://user-images.githubusercontent.com/31365056/46484858-387a7d80-c7c0-11e8-88f3-3871796e143b.png)
```Python
imp = mice.MICEData(df)
imp.data
```
![image](https://user-images.githubusercontent.com/31365056/46484937-5ea01d80-c7c0-11e8-8815-467fef09eb11.png)

Where as the actual values that are closest should be something as follows:
```Python
for x,y in np.abs(df - df.mean()).idxmin().iteritems():
    print('Observed Closest to Mean:',str(x), df[x].loc[y])
```
![image](https://user-images.githubusercontent.com/31365056/46485335-18978980-c7c1-11e8-8bd8-ed6f73805c63.png)

Not sure if MICEData was intended to be used in that manner of just taking the initial imputation, or if was intended to be run through multiple iterations as that would bring the values in range.  Running just one MICE iteration would provide values in the range of the observed data.

```Python
imp.update_all()
imp.data
```
![image](https://user-images.githubusercontent.com/31365056/46485735-ea667980-c7c1-11e8-9749-e35567306efb.png)

Again not an imputation expert so I cannot say how much effect it the initial value will have on one imputation iteration if it is indeed incorrect.  My guess would be the effect depends on the actual numbers in the applicable dataset.



looks like a bug to me, 
the effect could last for many iterations if the neighborhood is not densely populated.

Resolving patch:
diff --git a/statsmodels/imputation/mice.py b/statsmodels/imputation/mice.py
--- a/statsmodels/imputation/mice.py
+++ b/statsmodels/imputation/mice.py
@@ -293,7 +293,7 @@ def _initial_imputation(self):
             di = self.data[col] - self.data[col].mean()
             di = np.abs(di)
             ix = di.idxmin()
-            imp = di.loc[ix]
+            imp = self.data[col].loc[ix]
             self.data[col].fillna(imp, inplace=True)
 
     def _split_indices(self, vec):


Test patch:
diff --git a/statsmodels/imputation/tests/test_mice.py b/statsmodels/imputation/tests/test_mice.py
--- a/statsmodels/imputation/tests/test_mice.py
+++ b/statsmodels/imputation/tests/test_mice.py
@@ -81,12 +81,16 @@ def test_default(self):
                         np.concatenate((np.arange(10),
                                         np.arange(11, 30, 2),
                                         np.arange(30, 200))))
+        assert_equal([set(imp_data.data[col]) for col in imp_data.data],
+                     [set(df[col].dropna()) for col in df])
 
         for k in range(3):
             imp_data.update_all()
             assert_equal(imp_data.data.shape[0], nrow)
             assert_equal(imp_data.data.shape[1], ncol)
             assert_allclose(orig[mx], imp_data.data[mx])
+            assert_equal([set(imp_data.data[col]) for col in imp_data.data],
+                         [set(df[col].dropna()) for col in df])
 
         fml = 'x1 ~ x2 + x3 + x4 + x5 + y'
         assert_equal(imp_data.conditional_formula['x1'], fml)
@@ -341,13 +345,13 @@ def test_combine(self):
         mi = mice.MICE("y ~ x1 + x2", sm.OLS, idata, n_skip=20)
         result = mi.fit(10, 20)
 
-        fmi = np.asarray([ 0.1920533 ,  0.1587287 ,  0.33174032])
+        fmi = np.asarray([0.1778143, 0.11057262, 0.29626521])
         assert_allclose(result.frac_miss_info, fmi, atol=1e-5)
 
-        params = np.asarray([-0.05397474,  0.97273307,  1.01652293])
+        params = np.asarray([-0.03486102, 0.96236808, 0.9970371])
         assert_allclose(result.params, params, atol=1e-5)
 
-        tvalues = np.asarray([ -0.84781698,  15.10491582,  13.59998039])
+        tvalues = np.asarray([-0.54674776, 15.28091069, 13.61359403])
         assert_allclose(result.tvalues, tvalues, atol=1e-5)