Repository: pandas-dev/pandas
Pull request: BUG: Display precision doesn't affect complex float numbers #25514
- [x] closes #25514
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
Display precision doesn't affect complex float numbers
#### Code Sample

```python
import numpy as np
import pandas as pd

pd.set_option("display.precision", 3)

A = np.random.rand(3,3)
df_A = pd.DataFrame(A)

B = np.random.rand(3,3)+1j*np.random.rand(3,3)
df_B = pd.DataFrame(B)

print(df_A)
print(df_B)

```

**Output:**

```
    0      1      2
0  0.665  0.824  0.953
1  0.341  0.078  0.408
2  0.945  0.411  0.078
                                            0  \
0    (0.2553045397887609+0.8464631779709604j)   
1  (0.5724405361700162+0.047300152111596105j)   
2     (0.5824766758024993+0.974755917539835j)   

                                           1  \
0  (0.3533797280121552+0.20515824034905894j)   
1    (0.928312255311234+0.3909443194373995j)   
2   (0.2700966337410404+0.9137483492310767j)   

                                           2  
0  (0.21748538025578568+0.7730350615682122j)  
1  (0.2435709287109903+0.46366472945887327j)  
2   (0.6531622218102532+0.5102462678094016j)
```



#### Problem description

Display precision (`pd.set_option("display.precision", 3)`) doesn't affect complex float numbers like shown in the example.

#### Expected Output

```
    0      1      2
0  0.665  0.824  0.953
1  0.341  0.078  0.408
2  0.945  0.411  0.078

               0               1               2
0   0.976+0.935j    0.739+0.851j    0.436+0.734j
1   0.998+0.867j    0.774+0.849j    0.553+0.749j
2   0.405+0.049j    0.965+0.912j    0.292+0.958j

```
#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 142 Stepping 9, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.23.4
pytest: None
pip: 18.1
setuptools: 39.0.1
Cython: 0.29
numpy: 1.15.4
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.1.1
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.9
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: None
feather: None
matplotlib: 3.0.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:
Agreed. Investigations and PR's welcome!
I'm relatively new to the project but here is what I found:
Currently `format.py` doesn't handle complex values formatting properly. For integers there is an `IntArrayFormatter`, for floats a `FloatArrayFormatter` and so on. The complex numbers are `complex128` data types and are fed into a `GenericArrayFormatter` which just outputs the values however Numpy generates them. We need something like a `ComplexArrayFormatter`.

This `ComplexArrayFormatter` needs to distinguish between a complex float array and a complex int array, and for this, an `is_complex_float` type checker is needed in the util library. It would then need to apply the display.precision for complex floats (as is done for floats) and the return the proper formatted output.
Maybe we can just dispatch the complex formatting to the `FloatArrayFormatter`? pandas only (somewhat) supports Numpy's complex dtypes which are all float based AFAICT.  
Yes, the `FloatArrayFormatter` will format to the `display.precision` properly, but after that all complex numbers will have the format of `1.0+2.0j` instead of `(1+2j)`. Is this something we might be okay with?
I think this would be perfectly fine, after all the complex numbers is still a float (in this case) and when you have the dot zero format you immediately know that it is a float and not an integer complex number. Its the same reason why you have the dot zero format for plain float numbers.

Commit message: BUG: Display precision doesn't affect complex float numbers #25514

Patch to review:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
index 6b4d95055d06d..e2d23f9041759 100644
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -2498,12 +2498,12 @@ def memory_usage(self, index=True, deep=False):
         ...              for t in dtypes])
         >>> df = pd.DataFrame(data)
         >>> df.head()
-           int64  float64  complex128 object  bool
-        0      1      1.0      (1+0j)      1  True
-        1      1      1.0      (1+0j)      1  True
-        2      1      1.0      (1+0j)      1  True
-        3      1      1.0      (1+0j)      1  True
-        4      1      1.0      (1+0j)      1  True
+           int64  float64              complex128 object  bool
+        0      1      1.0      1.000000+0.000000j      1  True
+        1      1      1.0      1.000000+0.000000j      1  True
+        2      1      1.0      1.000000+0.000000j      1  True
+        3      1      1.0      1.000000+0.000000j      1  True
+        4      1      1.0      1.000000+0.000000j      1  True
 
         >>> df.memory_usage()
         Index            80
diff --git a/pandas/io/formats/format.py b/pandas/io/formats/format.py
index f68ef2cc39006..9a99947c70982 100644
--- a/pandas/io/formats/format.py
+++ b/pandas/io/formats/format.py
@@ -16,9 +16,9 @@
 from pandas.compat import StringIO, lzip, map, u, zip
 
 from pandas.core.dtypes.common import (
-    is_categorical_dtype, is_datetime64_dtype, is_datetime64tz_dtype,
-    is_extension_array_dtype, is_float, is_float_dtype, is_integer,
-    is_integer_dtype, is_list_like, is_numeric_dtype, is_scalar,
+    is_categorical_dtype, is_complex_dtype, is_datetime64_dtype,
+    is_datetime64tz_dtype, is_extension_array_dtype, is_float, is_float_dtype,
+    is_integer, is_integer_dtype, is_list_like, is_numeric_dtype, is_scalar,
     is_timedelta64_dtype)
 from pandas.core.dtypes.generic import (
     ABCIndexClass, ABCMultiIndex, ABCSeries, ABCSparseArray)
@@ -885,6 +885,8 @@ def format_array(values, formatter, float_format=None, na_rep='NaN',
         fmt_klass = ExtensionArrayFormatter
     elif is_float_dtype(values.dtype):
         fmt_klass = FloatArrayFormatter
+    elif is_complex_dtype(values.dtype):
+        fmt_klass = FloatArrayFormatter
     elif is_integer_dtype(values.dtype):
         fmt_klass = IntArrayFormatter
     else:
diff --git a/pandas/tests/io/formats/test_format.py b/pandas/tests/io/formats/test_format.py
index b0cf5a2f17609..1384f7788ada9 100644
--- a/pandas/tests/io/formats/test_format.py
+++ b/pandas/tests/io/formats/test_format.py
@@ -1417,6 +1417,18 @@ def test_to_string_float_index(self):
                     '5.0  4')
         assert result == expected
 
+    def test_to_string_complex_float_formatting(self):
+        fmt.set_option('display.precision', 5)
+        df = DataFrame({'x': [
+            (0.4467846931321966 + 0.0715185102060818j),
+            (0.2739442392974528 + 0.23515228785438969j),
+            (0.26974928742135185 + 0.3250604054898979j)]})
+        result = df.to_string()
+        expected = ('                  x\n0  0.44678+0.07152j\n'
+                    '1  0.27394+0.23515j\n'
+                    '2  0.26975+0.32506j')
+        assert result == expected
+
     def test_to_string_ascii_error(self):
         data = [('0  ', u('                        .gitignore '), u('     5 '),
                  ' \xe2\x80\xa2\xe2\x80\xa2\xe2\x80'


Code review comments:
diff_hunk: @@ -1417,6 +1417,18 @@ def test_to_string_float_index(self):
                     '5.0  4')
         assert result == expected
 
+    def test_to_string_complex_float_formatting(self):
+        fmt.set_option('display.precision', 5)
path: pandas/tests/io/formats/test_format.py
text: @user1:
do we need to use the context manger here?

@user2:
yep

@user2:
can you use the context manager here

diff_hunk: @@ -2498,12 +2498,12 @@ def memory_usage(self, index=True, deep=False):
         ...              for t in dtypes])
         >>> df = pd.DataFrame(data)
         >>> df.head()
-           int64  float64  complex128 object  bool
-        0      1      1.0      (1+0j)      1  True
-        1      1      1.0      (1+0j)      1  True
-        2      1      1.0      (1+0j)      1  True
-        3      1      1.0      (1+0j)      1  True
-        4      1      1.0      (1+0j)      1  True
path: pandas/core/frame.py
text: @user1:
Is this considered an improvement though?  Look at all of those zero's...

@author:
There is no `display.precision` set for this, and displaying six digits after the decimal point for a float is default behavior, apparently. I'm not sure why the float64 doesn't appear as `1.000000` rather that `1.0` when generated in that way. When I generate a random float using Numpy `random`, or `random.uniform` I get the default six digit values.

@user1:
@user2 : Thoughts on this?

@user2:
you can step thru the code and see why the float displays as is. I *believe* this is correct. This is a pretty complex interaction of options.

@user2:
did you have a chance to figure this out?

@author:
A little, yeah. Basically floats' zeros are being removed by the `_trim_zeros` method, where for every digit starting from the right hand side are being removed until we get a decimal or a non-zero number. We would need to modify this function to accept complex numbers as well.

@user2:
I think it might be better to do this for complex as well.

@author:
Updated my PR with the fixes. Please review.