Repository: pandas-dev/pandas
Pull request: to_html formatter not called for float values in a mixed-type column
- [x] closes #13021
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

can't shortcut completely without changing existing tested behaviour in a couple of tests.

for instance, the expected output for `test_to_string_with_formatters` in `pandas\tests\io\formats\test_format.py` has two spaces between float and object column, whereas shortcuting would only have one (the same as between the int and float columns)

```
  int  float    object
0 0x1 [ 1.0]  -(1, 2)-
1 0x2 [ 2.0]    -True-
2 0x3 [ 3.0]   -False-
```
if it's OK to change a couple of tests, we can remove this subtle leading_space bug in this PR.


Problem statement:
to_html formatter not called for float values in a mixed-type column
#### When column x has float64 type, the custom formatter is applied to every value

```
t = pd.DataFrame([[0], [10.4], [3]], columns=['x'])
display(HTML(t.to_html(formatters=dict(x='xxx'.format))))

   x
------
0  xxx
1  xxx
2  xxx
```
#### But when column x is mixed type, the formatter isn't called for (only) floating point values

```
t = pd.DataFrame([['a'], [0], [10.4], [3]], columns=['x'])
display(HTML(t.to_html(formatters=dict(x='xxx'.format))))

   x
------
0  xxx
1  xxx
2  10.4
3  xxx
```
#### output of `pd.show_versions()`
## INSTALLED VERSIONS

commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Darwin
OS-release: 14.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.0
nose: None
pip: 1.5.4
setuptools: 20.7.0
Cython: None
numpy: 1.11.0
scipy: 0.17.0
statsmodels: None
xarray: None
IPython: 4.1.2
sphinx: None
patsy: None
dateutil: 2.4.2
pytz: 2016.3
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.5.2
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.12
pymysql: None
psycopg2: None
jinja2: None
boto: None

Hints from issue discussion:
what exactly are you expecting here?

I'm expecting it to call my format function for each value in the column.  It doesn't seem to call the formatter at all.  Obviously `'xxx'.format` is a stupid example - but it illustrates that it's not applying the function to the 10.4 value when it occurs in a mixed-type column.  A more realistic formatter would be something like `lambda x: x if type(x) is str else '${:,.0f}'.format(x)`  which displays:

```
    x
0   a
1   $0
2   10.4
3   $3
```

when I'd expect $10 for the 3rd row.

how so? show an expected result (with and w/o object columns)

```
def f(x):
    return x if type(x) is str else '${:,.0f}'.format(x)

t = pd.DataFrame([['a'], [0], [10.4], [3]], columns=['x'])
display(HTML(t.to_html(formatters=dict(x=f))))
```

produces

```
    x
0   a
1   $0
2   10.4
3   $3
```

instead of what I expected:

```
    x
0   a
1   $0
2   $10
3   $3
```

or am I missing something?  (the 'xxx'.format in my example was just to illustrate that it doesn't call my function at all for the 10.4 value)

If instead you create the data frame without the 'a' row, so the column type is float64, you get the expected result:

```
t = pd.DataFrame([[0], [10.4], [3]], columns=['x'])
display(HTML(t.to_html(formatters=dict(x=f))))

    x
0   $0
1   $10
2   $3
```


This is still an issue in 0.21.1

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.10.0-42-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.1
pytest: 3.0.7
pip: 9.0.1
setuptools: 36.0.1
Cython: None
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.1.0
sphinx: 1.6.3
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 1.5.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.1.4
pymysql: 0.7.11.None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
@jobevers hence the open status
Also having problems with formatters

Commit message: to_html formatter not called for float values in a mixed-type column

Patch to review:
diff --git a/doc/source/whatsnew/v0.25.0.rst b/doc/source/whatsnew/v0.25.0.rst
index 1ef05ae5f9c6b..e01185b54f49d 100644
--- a/doc/source/whatsnew/v0.25.0.rst
+++ b/doc/source/whatsnew/v0.25.0.rst
@@ -344,6 +344,7 @@ I/O
 ^^^
 
 - Bug in :func:`DataFrame.to_html()` where values were truncated using display options instead of outputting the full content (:issue:`17004`)
+- Bug in :meth:`DataFrame.to_html` that would ignore ``formatters`` argument for float values in a column with ``dtype=object`` (:issue:`13021`)
 - Fixed bug in missing text when using :meth:`to_clipboard` if copying utf-16 characters in Python 3 on Windows (:issue:`25040`)
 - Bug in :func:`read_json` for ``orient='table'`` when it tries to infer dtypes by default, which is not applicable as dtypes are already defined in the JSON schema (:issue:`21345`)
 - Bug in :func:`read_json` for ``orient='table'`` and float index, as it infers index dtype by default, which is not applicable because index dtype is already defined in the JSON schema (:issue:`25433`)
diff --git a/pandas/io/formats/format.py b/pandas/io/formats/format.py
index b658c8a53dc8b..e2f01e03e3cef 100644
--- a/pandas/io/formats/format.py
+++ b/pandas/io/formats/format.py
@@ -980,6 +980,17 @@ def _format(x):
         if leading_space is None:
             leading_space = is_float_type.any()
 
+        if leading_space is False:
+            # False specifically, so that the default is
+            # to include a space if we get here.
+            tpl = '{v}'
+        else:
+            tpl = ' {v}'
+
+        # shortcut
+        if self.formatter is not None:
+            return [tpl.format(v=self.formatter(x)) for x in self.values]
+
         fmt_values = []
         for i, v in enumerate(vals):
             if not is_float_type[i] and leading_space:
@@ -987,12 +998,6 @@ def _format(x):
             elif is_float_type[i]:
                 fmt_values.append(float_format(v))
             else:
-                if leading_space is False:
-                    # False specifically, so that the default is
-                    # to include a space if we get here.
-                    tpl = '{v}'
-                else:
-                    tpl = ' {v}'
                 fmt_values.append(tpl.format(v=_format(v)))
 
         return fmt_values
diff --git a/pandas/tests/io/formats/test_to_html.py b/pandas/tests/io/formats/test_to_html.py
index d146e9c16e114..436dcff0aed60 100644
--- a/pandas/tests/io/formats/test_to_html.py
+++ b/pandas/tests/io/formats/test_to_html.py
@@ -633,3 +633,13 @@ def test_to_html_invalid_classes_type(classes):
 
     with pytest.raises(TypeError, match=msg):
         df.to_html(classes=classes)
+
+
+def test_to_html_formatters_object_type():
+    # GH 13021
+    def f(x):
+        return x if type(x) is str else '${:,.0f}'.format(x)
+
+    df = pd.DataFrame([['a'], [0], [10.4], [3]], columns=['x'])
+    result = df.to_html(formatters=dict(x=f))
+    assert '$10' in result


Code review comments:
diff_hunk: @@ -633,3 +633,13 @@ def test_to_html_invalid_classes_type(classes):
 
     with pytest.raises(TypeError, match=msg):
         df.to_html(classes=classes)
+
+
+def test_to_html_formatters_object_type():
+    # GH 13021
+    def f(x):
+        return x if type(x) is str else '${:,.0f}'.format(x)
path: pandas/tests/io/formats/test_to_html.py
text: @user1:
Use isinstance instead of type

diff_hunk: @@ -633,3 +633,13 @@ def test_to_html_invalid_classes_type(classes):
 
     with pytest.raises(TypeError, match=msg):
         df.to_html(classes=classes)
+
+
+def test_to_html_formatters_object_type():
+    # GH 13021
+    def f(x):
+        return x if type(x) is str else '${:,.0f}'.format(x)
+
+    df = pd.DataFrame([['a'], [0], [10.4], [3]], columns=['x'])
+    result = df.to_html(formatters=dict(x=f))
+    assert '$10' in result
path: pandas/tests/io/formats/test_to_html.py
text: @user1:
Can we make a better assertion here? I understand the original issue pertained to floats only but probably doesn't hurt to make an assertion around the other values here as well