BUG: DataFrame.to_string with formatters, header and index False
State: closed

closes #13032 
- [x] tests added / passed - added test specific to format bug
- [x] passes `pep8radius master --diff`
- [x] whatsnew entry - not needed

Found this bug experimenting with formatters. First pull request to pandas, but I believe guidelines are quite clear. I can explain what was happening in more detail if that is necessary.


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
is this related to #13032 ?

can you show an example of before / after

Turns out is exactly related to #13032. I did see this issue, but didn't make the connection initially. In my testing, the lines should be removed not modified. To be honest, I didn't understand what these lines accomplished when I was reviewing the code originally.

Minimal code to reproduce (stripped down from #13032, easier to see with `formatters`):

```
>>> import pandas as pd
>>> frame = pd.DataFrame(data={0: 0, 1: 0}, index=[0])
>>> formatter = lambda x: '{:10.3f}'.format(x)
>>> print(frame.to_string(index=False, header=False))
```

Before (adding slashes to make space counting easier):
`\ \ \ \ 0.000\ \ \ \ \ \ 0.000`
After:
`\ \ \ \ \ 0.000\ \ \ \ \ \ 0.000`

Output from PR for #13032 (without formatter):

```
>>> print(df.to_string(index=False))
      one       two     three
 1.722364  0.846757  0.094394
-0.578834  0.836656  0.665414
 0.345460  1.782786  1.760175
```

Output from PR for #13032 (with formatter `10.3f`, slashes added once for column count):

```
>>> print(df.to_string(index=False,formatters=[formatter,formatter,formatter]))
       one        two      three
\ \ \ \ \ 1.722      0.847      0.094
    -0.579      0.837      0.665
     0.345      1.783      1.760
```

I will add the PR and Issues numbers plus make corrections shortly.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13350?src=pr) is 84.23% (diff: 100%)

> Merging [#13350](https://codecov.io/gh/pydata/pandas/pull/13350?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13350   diff @@
==========================================
  Files           138        138          
  Lines         50713      50721     +8   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42715      42723     +8   
  Misses         7998       7998          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [ce56542...73d7b7e](https://codecov.io/gh/pydata/pandas/compare/ce56542d1226adf8b3439c51f0c34b49dd53bb28...73d7b7edd24060e4f59ff2b11a156be36b497d75?src=pr)

Additionally, the failed Travis CI checks don't appear to be related to my subtrations/additions. Lastly, the unit test must use `assert` because of the location in the code. Should it be moved?

@barrymoo can you also read thru the comments on that issue (and the referenced one) and see if covering bases.

@jreback Of course,

From comment https://github.com/pydata/pandas/issues/13032#issuecomment-222835024:
Before:

```
Name    Value
                                  Short        1
                                 Longer  9374518
Much Longer name to the Max -----------    32432
```

After (desired output, correct?):

```
                                    Name    Value
                                   Short        1
                                  Longer  9374518
 Much Longer name to the Max -----------    32432
```

I think the others comments would be unaffected. To be fair though, I am very new to pandas.

I see what you mean now. From #11833:

Code:

```
>>> import pandas as pd
>>> df = pd.DataFrame({'a':range(5)})
>>> df.to_string(index=False)
u' a\n 0\n 1\n 2\n 3\n 4'
>>> formatter = lambda x: '{:1d}'.format(x)
>>> df.to_string(formatters=[formatter], index=False)
u'a\n0\n1\n2\n3\n4'
```

I should be able to fix this issue quickly.

A different perspective: where are those spaces coming from in the first place? (I will try to track this down)

Code:

```
#!/usr/bin/env python
import pandas as pd
import numpy as np

# Test 1
frame = pd.DataFrame(data={0: 0, 1: 0}, index=[0])
formatter = lambda x: '{:10.3f}'.format(x)
string = frame.to_string(index=False, header=False)
print('--> Begin Test 1 <--')
print(string)
print('-->  End Test 1  <--')

# Test 2
df = pd.DataFrame(np.random.randn(3, 3), columns=['one', 'two', 'three'])
string = df.to_string(index=False)
print('--> Begin Test 2 <--')
print(string)
print('-->  End Test 2  <--')

# Test 3
df = pd.DataFrame({'a':range(5)})
string = df.to_string(index=False)
print('--> Begin Test 3 <--')
print(string)
print('-->  End Test 3  <--')

# Test 4
NAMES = ['Short', 'Longer', 'Much Longer name to the Max -----------']
VALUES = [1, 9374518, 32432]
d = pd.DataFrame({'Name': NAMES, 'Value': VALUES})
string = d.to_string(index=False)
print('--> Begin Test 4 <--')
print(string)
print('-->  End Test 4  <--')
```

Produces:

```
$ python quick-test.py
--> Begin Test 1 <--
0 0
-->  End Test 1  <--
--> Begin Test 2 <--
      one       two     three
-0.117275 -0.410192 -2.170441
 0.194766  0.521318  0.936951
-0.923841 -1.829388  1.078478
-->  End Test 2  <--
--> Begin Test 3 <--
a
0
1
2
3
4
-->  End Test 3  <--
--> Begin Test 4 <--
                                   Name   Value
                                  Short       1
                                 Longer 9374518
Much Longer name to the Max -----------   32432
-->  End Test 4  <--
```

I believe this is the desired output. Unfortunately, this code fails about 20 tests. Hopefully, it is because the spacing in the expected output has changed slightly.

About 30 failed checks from `nosetests pandas/tests/formats/test_format.py`. I will checking the expected outputs this weekend. Note, I edited https://github.com/pydata/pandas/pull/13350#issuecomment-223483638 to include the new code edits.

I have created [barrymoo/pandas-pr-13350-supplement](https://github.com/barrymoo/pandas-pr-13350-supplement) to document the test failures (and eventually generate new `expected` strings). This is a work-in-progress.

I have added some more tests to the supplement. There is one test which I am having some difficulty with, starts: https://github.com/barrymoo/pandas-pr-13350-supplement/blob/master/tests.py#L474. I think the frame is supposed to overcome the terminal size, but it doesn't. 

Hey @jreback I finished my supplement but I need some opinions about output formatting especially concerning the `test_*_east_asian*` tests (I don't know what these should look like). Is there anyone else we could pull in to review this? That way I can fix the rest of the formatting concerns and fix everything with one PR.

For the supplement, clone it, activate the dev environment (tested with 2 & 3, but have not examined diffs of the output), run `python tests.py`, and review formatting. Submit issues to the other repo with concerns of specific tests.

@barrymoo not sure what you mean by supplement. simply update this PR, comments can just be done here.
cc @sinhrks 

@barrymoo I live in Japan and am willing to check the output format amd provide test cases:)

@jreback I ripped out the failing tests so one can easily print the results out on the command line. That way I can get some input from the community on how people want things formatted and make additional changes.

Here's a great example for why I need the supplement. For `test_datetimelike_frame`, my changes lead to the following output

```
                          dt  x
0  2011-01-01 00:00:00-05:00  1
1  2011-01-01 00:00:00-05:00  2
..                       ... ..
8                        NaT  9
9                        NaT 10
```

But, do you like:

```
                          dt  x
 0 2011-01-01 00:00:00-05:00  1
 1 2011-01-01 00:00:00-05:00  2
..                       ... ..
 8                       NaT  9
 9                       NaT 10
```

or...

```
                         dt  x
0 2011-01-01 00:00:00-05:00  1
1 2011-01-01 00:00:00-05:00  2
.                       ... ..
8                       NaT  9
9                       NaT 10
```

I can easily generate all of these outputs. Or, would you rather I pick what I like and get all of the tests working.

I also worked on this; getting all of the tests to pass afterward is a nightmare. It looks like this change removes the leading space on integers but leaves it on floats. Is that true?

@evanpw it's very time consuming. I am still working through all of the tests, but I don't have a ton of free time. If you're looking at all positive numbers there is an extra space for the nonexistent "-" sign. There is an example in one of the above comments.

After this change there won't be an extra leading space for a column of positive integers, but there will still be one for a column of positive floats, right?

That's correct, but I can likely fix that too. Again the majority of this work is fixing the tests.

can you rebase / update?

I didn't do this correct, sorry have not mastered this bit of git yet. I will submit a different pull request