BUG: Respect na_rep in DataFrame.to_latex() when used with formatters (#9046)
State: closed

- [x] closes #9046
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

A very old issue so low priority! 

I'd be tempted to remove the shortcut in `_format_strings()` since I'd guess in most cases it doesn't really save much time. 

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: IO LaTeX
description: to_latex

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=h1) Report
> Merging [#25799](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/37d04a3d3df82c3abb80afc2ef6fcfa0b7d5df44?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25799/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #25799   +/-   ##
=======================================
  Coverage   41.77%   41.77%           
=======================================
  Files         173      173           
  Lines       53002    53002           
=======================================
  Hits        22141    22141           
  Misses      30861    30861
```

| Flag | Coverage Δ | |
|---|---|---|
| #single | `41.77% <0%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/formats/format.py](https://codecov.io/gh/pandas-dev/pandas/pull/25799/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvZm9ybWF0LnB5) | `50.82% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=footer). Last update [37d04a3...bb2549e](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=h1) Report
> Merging [#25799](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/b727c6b328c3c586ddab54baae36680a0992e961?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `20%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25799/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25799      +/-   ##
==========================================
- Coverage   91.82%    91.8%   -0.02%     
==========================================
  Files         175      175              
  Lines       52580    52583       +3     
==========================================
- Hits        48279    48275       -4     
- Misses       4301     4308       +7
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.36% <20%> (-0.01%)` | :arrow_down: |
| #single | `41.91% <0%> (-0.05%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/formats/format.py](https://codecov.io/gh/pandas-dev/pandas/pull/25799/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvZm9ybWF0LnB5) | `97.54% <20%> (-0.35%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/25799/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `75% <0%> (-12.5%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/25799/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.79% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=footer). Last update [b727c6b...a47b305](https://codecov.io/gh/pandas-dev/pandas/pull/25799?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

i'm -1 on this PR since i thought the idea of the custom formatters was to override the behavior with a user defined function, hence the shortcut.

if you want to define a custom formatter for a particular column, but have the standard na_rep for the remaining columns, doesn't this change prevent that?
@simonjayhawkins this is a good point that I hadn't considered. 

I'm happy for this to be closed. 
It would be good to know what needs to be done to close #9046?
Are doc updates needed?
> Are doc updates needed?

the `formatters` parameter is in the common docstring and common across to_html, to_latex and to_string

```
        formatters : list or dict of one-param. functions, optional
            Formatter functions to apply to columns' elements by position or
            name.
            The result of each function must be a unicode string.
            List must be of length equal to the number of columns.
```
which is out-of-sync with the only reference to `formatters` in the IO user guide https://pandas-docs.github.io/pandas-docs-travis/user_guide/io.html#writing-a-formatted-string

```
formatters default None, a dictionary (by column) of functions each of which takes a single argument and returns a formatted string
```
so any improvements here would be welcome, but I don't think it is strictly necessary in order to close the issue.


Thinking about it, this could work if we change the docstring. At the moment the custom function must return a string. we could allow nan, None whatever to be returned from the custom function. Then apply the `na_rep` to only those. This would allow the user defined function to choose whether to handle the missing values.
There's been a issue raised, #26278, regarding using string formatters. This make sense to be compatible with the .style api. I think that it would also make sense, that if string formatters were accepted in to_latex, to_html etc then the na_rep should be respected.

so although I was -1 on this PR originally, I think that, in combination with adding string formatters, and keeping the behavior unchanged for callables, then this PR would close the original issue.
Ok- I think that it is probably still best to close this for now, as I think we'd still want to keep the current behaviour if a list of callables is given if I'm reading your suggestion correctly, e.g.


```python
df.to_latex(formatters=[i.format for i in ('{:d}', '{:1.1f}', '{:1.3f}')])
```

Then the `na_rep` behaviour can be dealt with if changes are made so you can do

```python
df.to_latex(formatters=['%1i', '%1.1f', '%1.3f'])
```

It might be worth considering how this change would (or maybe wouldn't) affect the `float_format` argument too, as one might naively assume that 
```python
df.to_latex(float_format=['%1i', '%1.1f', '%1.3f'])
```
could be a viable choice for this feature too.

Anyway, If there's no objection I'll close this PR tomorrow. 
> Anyway, If there's no objection I'll close this PR tomorrow.

No problem. Thanks @tomneep for looking into this.