Pivot table drops column/index names=nan when dropna=false
State: closed

- [x] closes #14072
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Input:

```
import pandas as pd
M = pd.DataFrame([
[1, 'a', 'A'], 
[1, 'b', 'B'], 
[1, 'c', None]], 
columns=['x', 'y', 'z'])
P = M.pivot_table(values='x', index='y', columns='z', aggfunc='sum',
                             fill_value=0, margins=True, dropna=False)
print P
```

Output **was**:

| z | A | B | All |
| --- | --- | --- | --- |
| y |  |  |  |
| a | 1.0 | 0.0 | 1.0 |
| b | 0.0 | 1.0 | 1.0 |
| All | 1.0 | 1.0 | 3.0 |

Output after fix:

| z | NaN | A | B | All |
| --- | --- | --- | --- | --- |
| y |  |  |  |  |
| a | 0.0 | 0.0 | 1.0 | 1.0 |
| b | 0.0 | 1.0 | 0.0 | 1.0 |
| c | 1.0 | 0.0 | 0.0 | 1.0 |
| All | 1.0 | 1.0 | 1.0 | 3.0 |

I think the rearrangement of columns is due to the hash function problem similar as the one in #12679 and needs extra work to solve.

@jreback I realized that this issue has a lot overlap with Issue #3729 and PR #12607. 
The problem is that `dropna` is not passed at all from `pivot_table` and the `algorithm.factorize` just set the `dropna=True`. 
I don't see other options than passing the param on and adding a param to classes seems to be the easiest solution. 
I did the nosetests at local and it is failing on 2 out of 10000+ with weird type errors. I think these can be fixed.

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14246?src=pr) is 84.77% (diff: 100%)
> Merging [#14246](https://codecov.io/gh/pandas-dev/pandas/pull/14246?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

```diff
@@             master     #14246   diff @@
==========================================
  Files           145        145          
  Lines         51129      51154    +25   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43343      43368    +25   
  Misses         7786       7786          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [74e20a0...c27d3d3](https://codecov.io/gh/pandas-dev/pandas/compare/74e20a0c4146919a82312c1027dcb57b48b613e3...c27d3d369079d8d31927a42b9a8662ff620d6f08?src=pr)
@jreback All tests pass now. I can move `dropna` out of  `__init__` if you think the modification is too aggressive, but it's gonna be quite verbose. [**EDIT**: I was thinking wrong. I need to think about how to restrain the use of `dropna` in `groupby.py`.]
Also the tests of `pivot_table` are a little bit messy..I will clean them up and add the test for `dropna` if you think we could continue working on this PR. 

Sorry I was thinking wrong. Will figure out how to limit the use of `dropna` in `groupby.py`

can you rebase
@jreback It seems that the appveyor exceeded time limit. 

Also I did a `pep8radius master --diff` and it suggested changes to 13 files I haven't touched. Not sure whether I should update those as well.
can you rebase and we can see where this is
@jreback I made some major changes and now the changes are restrained to `pivot.py`. Also some tests with `dropna=False` were not accurate.
Update: There is a conflict between the merge recently (#14944) and the fix I am trying to do. I need the `check_null` [here](https://github.com/pandas-dev/pandas/blob/master/pandas/src/hashtable_class_helper.pxi.in#L340) to be `dropna`. I am still trying to solve the conflict.
@OXPHOS can you rebase / update
I started over and now I'm half way through. I'll open a new pr once I cleaned up.