BUG: Don't ignore na_rep in DataFrame.to_html
State: closed

- [x] closes #13828
- [x] closes #9046
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: IO LaTeX
description: to_latex

Comments:
Off the top of my head I recall a discussion where this may not be a bug. There may be a duplicate issue.

I've not checked the code. but if a user supplies a formatter, they are responsible for the na values too. Is that what is happening here?
> Off the top of my head I recall a discussion where this may not be a bug. There may be a duplicate issue.
> 
> I've not checked the code. but if a user supplies a formatter, they are responsible for the na values too. Is that what is happening here?

There was an old PR similar to this that was almost merged but ended up being closed as stale, so I assume this is considered a real bug.

I could see how it might also be seen as ambiguous since NaN is technically a float, so that the user is asking for two different formats for these values (but in that case maybe the docs should explicitly clarify what happens in this situation).
does this also close #9046? and presumably to_string does the same?
I've not yet found the issue but from

> float_format: one-parameter function, optional, default None
Formatter function to apply to columns’ elements if they are floats. The result of this function must be a unicode string.

so the changes here break existing code (this is a long standing behaviour even if considered a bug)

```
def my_formatter(x):
    if np.isnan(x):
        return "ted"
    else:
        return str(x)


df = pd.DataFrame(
    [
        ["A", 1.2225],
        [
            "A",
        ],
    ],
    columns=["Group", "Data"],
)
df.to_html(float_format=my_formatter)
```

1.1.2
![image](https://user-images.githubusercontent.com/13159005/94610028-8cc57480-0297-11eb-90aa-a59eac9a8632.png)

this PR
![image](https://user-images.githubusercontent.com/13159005/94610092-aa92d980-0297-11eb-85bc-f91a1199affb.png)

> does this also close #9046? and presumably to_string does the same?

Yes, tests added for those cases
> so the changes here break existing code (this is a long standing behaviour even if considered a bug)

Ah I was forgetting "NaN" is actually the default and not None. Made a change which fixes this but leaves a different "bug" (albiet a more obscure one, it only ignores the specific string "NaN"; I don't think it's possible to respect both arguments in absolutely all cases):
```python
[ins] In [1]: import numpy as np
         ...: import pandas as pd
         ...:
         ...: def my_formatter(x):
         ...:     if np.isnan(x):
         ...:         return "ted"
         ...:     else:
         ...:         return str(x)
         ...:
         ...:
         ...: df = pd.DataFrame(
         ...:     [
         ...:         ["A", 1.2225],
         ...:         [
         ...:             "A",
         ...:         ],
         ...:     ],
         ...:     columns=["Group", "Data"],
         ...: )
         ...: print(df.to_html(na_rep="NaN", float_format=my_formatter))
         ...:
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Group</th>
      <th>Data</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>A</td>
      <td>1.2225</td>
    </tr>
    <tr>
      <th>1</th>
      <td>A</td>
      <td>ted</td>
    </tr>
  </tbody>
</table>
```
>K Can add a release note once the doc is started for 1.1.4

generally we won't do older bug fixes on a point release; only regressions. the reason is sometimes bug fixes can introduce a further bug :-> want to minimize the risk.
so returning to issue #13828, the fact that "{:.2f}".format(np.nan)" is "nan", I think the output is correct. and this is not a bug https://github.com/pandas-dev/pandas/pull/36690#issuecomment-700915940

but raising if `na_rep` is specified when `float_format` is specified maybe a better solution.

currently, in this PR

![image](https://user-images.githubusercontent.com/13159005/94916578-d89a3880-04a6-11eb-84a1-74aad1051f52.png)

This is inconsistent.
> but raising if `na_rep` is specified when `float_format` is specified maybe a better solution.

Raising when both are specified makes sense to me (or rather when float_format is specified and na_rep != "NaN", unless the default were also changed), since there is in fact no way to apply this consistently
despite my previous comments, I think we could only raise if not a mixed frame. otherwise may cause another regression.

I looked at sorting out some of the formatting in the past, and it's very easy to break things. (needs many more tests with lots of parameterisation to know for sure)

It is maybe that we do need to break things, and maybe the case in https://github.com/pandas-dev/pandas/pull/36690#issuecomment-700957174 is one of those.

> Thanks @dsaxton lgtm.
> 
> regarding the corner case where a user custom function handles missing values which is now broken, any idea on how we should communicate that to the users?

Maybe warrants a note in user_guide/io.rst?
> Maybe warrants a note in user_guide/io.rst?

how about just a versionchanged tag for float_format with a one-liner along the lines "If a function is passed, only non-missing values are passed to the function and na_rep is used for missing values"
> how about just a versionchanged tag for float_format with a one-liner along the lines "If a function is passed, only non-missing values are passed to the function and na_rep is used for missing values"

Added a note and versionchanged to the common docstring
@jreback I think this is good to go, CI failure is unrelated
thanks @dsaxton