Repository: pandas-dev/pandas

Issue:
Displaying DataFrame with values from range [1e-7, 5e-7] shows 0 in some situations.
Hi,

For illustration purpose, I'll assume the following dataset:

``` python

In [1]: d=pd.DataFrame({'col1':[9.999e-8, 1e-7, 1.0001e-7, 2e-7, 4.999e-7, 5e-7, 5.0001e-7, 6e-7]})

In [2]: d
Out[2]:
           col1
0  9.999000e-08
1  1.000000e-07
2  1.000100e-07
3  2.000000e-07
4  4.999000e-07
5  5.000000e-07
6  5.000100e-07
7  6.000000e-07
```

I've noticed the following behavior (in pandas 0.16 and 0.15.2):

1) When values from range [1e-7, 5e-7] are displayed along with values that are less than 1e-7, the output is OK:

``` python

In [3]: d[0:6]
Out[3]:
           col1
0  9.999000e-08
1  1.000000e-07
2  1.000100e-07
3  2.000000e-07
4  4.999000e-07
5  5.000000e-07
```

2) When values exclusively from that range are displayed, the output is 0:

``` python

In [4]: d[1:6]
Out[4]:
   col1
1     0
2     0
3     0
4     0
5     0
```

3) When values from that range are displayed along with values greater than 5e-7, the output for them is 0.000000:

``` python

In [5]: d[1:8]
Out[5]:
       col1
1  0.000000
2  0.000000
3  0.000000
4  0.000000
5  0.000000
6  0.000001
7  0.000001
```

I'm assuming it's a rounding / output formatting error.

Also:

``` python

In [6]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.2.final.0
python-bits: 64
OS: Linux
OS-release: 2.6.32-431.el6.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.16.0
nose: 1.3.4
Cython: 0.21.2
numpy: 1.9.2
scipy: 0.14.0
statsmodels: 0.6.1
IPython: 2.3.1
sphinx: None
patsy: 0.3.0
dateutil: 2.4.1
pytz: 2015.2
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.4.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
```



Discussion:
could be a bug or just an odd setting of options: http://pandas.pydata.org/pandas-docs/stable/options.html#list-of-options

want to see what is affecting the output here?

I've looked at my options and all that are related to formatting floats are set to defaults (display.chop_threshold is None, display.float_format is None, display.precision is 7).

It seems to be an issue in pandas/core/format.py - working on a fix.


Resolving patch:
diff --git a/pandas/core/format.py b/pandas/core/format.py
--- a/pandas/core/format.py
+++ b/pandas/core/format.py
@@ -1996,7 +1996,7 @@ def _format_strings(self):
 
             # this is pretty arbitrary for now
             has_large_values = (abs_vals > 1e8).any()
-            has_small_values = ((abs_vals < 10 ** (-self.digits)) &
+            has_small_values = ((abs_vals < 10 ** (-self.digits+1)) &
                                 (abs_vals > 0)).any()
 
             if too_long and has_large_values:


Test patch:
diff --git a/pandas/tests/test_format.py b/pandas/tests/test_format.py
--- a/pandas/tests/test_format.py
+++ b/pandas/tests/test_format.py
@@ -2986,6 +2986,25 @@ def test_format(self):
         self.assertEqual(result[0], " 12")
         self.assertEqual(result[1], "  0")
 
+    def test_output_significant_digits(self):
+        # Issue #9764
+        
+        # In case default display precision changes:
+        with pd.option_context('display.precision', 7):
+            # DataFrame example from issue #9764
+            d=pd.DataFrame({'col1':[9.999e-8, 1e-7, 1.0001e-7, 2e-7, 4.999e-7, 5e-7, 5.0001e-7, 6e-7, 9.999e-7, 1e-6, 1.0001e-6, 2e-6, 4.999e-6, 5e-6, 5.0001e-6, 6e-6]})
+            
+            expected_output={
+                (0,6):'           col1\n0  9.999000e-08\n1  1.000000e-07\n2  1.000100e-07\n3  2.000000e-07\n4  4.999000e-07\n5  5.000000e-07',
+                (1,6):'           col1\n1  1.000000e-07\n2  1.000100e-07\n3  2.000000e-07\n4  4.999000e-07\n5  5.000000e-07',
+                (1,8):'           col1\n1  1.000000e-07\n2  1.000100e-07\n3  2.000000e-07\n4  4.999000e-07\n5  5.000000e-07\n6  5.000100e-07\n7  6.000000e-07',
+                (8,16):'            col1\n8   9.999000e-07\n9   1.000000e-06\n10  1.000100e-06\n11  2.000000e-06\n12  4.999000e-06\n13  5.000000e-06\n14  5.000100e-06\n15  6.000000e-06',
+                (9,16):'        col1\n9   0.000001\n10  0.000001\n11  0.000002\n12  0.000005\n13  0.000005\n14  0.000005\n15  0.000006'
+            }
+
+            for (start, stop), v in expected_output.items():
+                self.assertEqual(str(d[start:stop]), v)
+
 
 class TestRepr_timedelta64(tm.TestCase):