BUG: on .to_string(index=False)
State: closed

- [ ] closes #24980
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
Hello @charlesdong1991! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-06-12 19:48:17 UTC
Thanks. Can you add a release note to 0.24.1 under fixed regressions?
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=h1) Report
> Merging [#25000](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/6449bc15758325bd753bbb13160ee825197023c3?src=pr&el=desc) will **decrease** coverage by `49.49%`.
> The diff coverage is `66.66%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25000/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25000      +/-   ##
==========================================
- Coverage   92.38%   42.88%   -49.5%     
==========================================
  Files         166      166              
  Lines       52400    52402       +2     
==========================================
- Hits        48409    22473   -25936     
- Misses       3991    29929   +25938
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `42.88% <66.66%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/formats/format.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvZm9ybWF0LnB5) | `50.76% <66.66%> (-47.23%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.35%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.17%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.15%)` | :arrow_down: |
| ... and [124 more](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=footer). Last update [6449bc1...9871550](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=h1) Report
> Merging [#25000](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/634577eb9ae53da6c16a54773f9f9f8016e2a1ed?src=pr&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `94.73%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25000/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25000      +/-   ##
==========================================
+ Coverage   91.86%   91.89%   +0.02%     
==========================================
  Files         179      175       -4     
  Lines       50707    52508    +1801     
==========================================
+ Hits        46583    48250    +1667     
- Misses       4124     4258     +134
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.44% <94.73%> (-0.01%)` | :arrow_down: |
| #single | `40.72% <15.78%> (-0.47%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `93.54% <ø> (-0.66%)` | :arrow_down: |
| [pandas/io/formats/format.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvZm9ybWF0LnB5) | `97.81% <94.73%> (-0.1%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `75% <0%> (-25%)` | :arrow_down: |
| [pandas/plotting/\_misc.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19taXNjLnB5) | `38.46% <0%> (-21.04%)` | :arrow_down: |
| [pandas/io/gcs.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2djcy5weQ==) | `80% <0%> (-20%)` | :arrow_down: |
| [pandas/io/clipboard/clipboards.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC9jbGlwYm9hcmRzLnB5) | `18.51% <0%> (-16.27%)` | :arrow_down: |
| [pandas/io/clipboard/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC9fX2luaXRfXy5weQ==) | `39.21% <0%> (-15.14%)` | :arrow_down: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `77.5% <0%> (-14.5%)` | :arrow_down: |
| [pandas/io/s3.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3MzLnB5) | `89.47% <0%> (-10.53%)` | :arrow_down: |
| [pandas/io/excel/\_xlrd.py](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsL194bHJkLnB5) | `93.93% <0%> (-6.07%)` | :arrow_down: |
| ... and [121 more](https://codecov.io/gh/pandas-dev/pandas/pull/25000/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=footer). Last update [634577e...5a6b867](https://codecov.io/gh/pandas-dev/pandas/pull/25000?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

added!
hmm... i am doubting if this solution is correct... I just find out that for numeric value, this is still incorrect:
`pd.Series(1).to_string(index=False)`, the output still contain leading space
but if it's a mixed array, then leading space goes away, e.g. `pd.Series([1, "a"]).to_string(index=False)`.
The same behaviour remains in DataFrame, only works if column value is not numeric...
e.g. `pd.DataFrame({"A": ["a", "b"], "B": [1, "dd"]}).to_string(index=False)` works fine
but `pd.DataFrame({"A": ["a", "b"], "B": [1, 2]}).to_string(index=False)` has leading space...
any thoughts? @TomAugspurger 
Indeed, the changes here are for GenericArrayFormater. We would maybe need something similar for the other formatters...

I'm re-reading https://github.com/pandas-dev/pandas/pull/22505 now...
@gshiba can you confirm that `Series.to_string` is supposed to have a leading space, even when `index=False`?
This was apparently to fix an alignment bug with a mixture of positive / negative values

```python
# 0.23.4
In [3]: print(pd.Series([0, -1]).to_string(index=False))
0
-1
```


```python
# 0.24.0
In [2]: print(pd.Series([0, -1]).to_string(index=False))
 0
-1
```

I suppose that's fine, but the change to non-negative series surprises

```python
In [3]: print(pd.Series([0, 1]).to_string(index=False))
 0
 1
```

Can we apply the leading space just when necessary (any values less than 0?)
@TomAugspurger btw, i tried to fix it while keeping the original as much as possible, probably it's not very efficient, but looks working... look forward to your feedback and suggestion!
FYI, i haven't updated whatsnew since i am not sure if my solution is correct way to go, will update later once this is settled.
now, the output looks like:
```python
# 0.24.0
In [1]: print(pd.Series([0, -1]).to_string(index=False))
 0
-1

In [2]: print(pd.Series([0, 1]).to_string(index=False))
0
1
```
It seems there has been many PRs and Issues for this that it's hard to summarize succinctly, but here goes my understanding.

(My investigation only concerned numerical columns -- integers specifically; I did not look at string columns)

### Old history

The leading space originates from the `% d` in `IntArrayFormatter.get_result` (106fe99; this was in year 2012. it is now in `IntArrayFormatter._format_strings`).

So as far as I can tell, the following behavior is intended:
* there *is* a leading space for positive values in Int columns
* there is *no* leading space (but a negative sign instead) for negative values in Int columns

And thus, when
* an Int column is the first column in a DataFrame (or we have a Integer Series) *and*
* it only contains zero or positive values *and*
* `index=False`,

it *looks like* there's a redundant white space.

### More recent history

In an attempt to fix the redundant white space, PR #11942 was merged, which in turn broke alignment with column headers, depending on the max width of values vs width of column name, etc (as described in Issue #13032)

I had two PRs going:

* PR #22437 attempted to ~~revert~~ change `% d` to `%d` as suggested here: https://github.com/pandas-dev/pandas/issues/13032#issue-151973347 That turned out to affect a lot of tests, and was rejected in favor of PR #22505.

* PR #22505 reverted parts of PR #11942 and embraced the leading space even when `index=False`. With this PR, `df.to_string(index=False)` will print the leading space when the first column is positive only, as well as fix the Issue #13032 by preserving leading/trailing spaces on first/last lines.

PR #22505 was merged in 30b942a (and released in 0.24.0), and *alignment* looks good now, but the leading white space *is* still there.

### Going forward

I understand leading white spaces are unexpected when only positive values are in a Series.

If this is going to be fixed, care should be taken not to break alignment among the values, and also with the column header which could be wider/shorter than the values for DataFrames.

(I hope this doesn't add to the confusion!)
very detailed and thorough explanation, awesome!!! @gshiba 
Though that edge case is there in 0.23.4...

```python
>>> print(pd.Series([' a', ' b']).to_string(index=False))
a
 b
```

We could exclude that erroneous strip in the case of all strings starting with a ' ' with `self.series.str.startswith(' ').all()`, with suitable type-checking before doing the `self.series.str` so that doesn't raise.
Applying `strip` may have unintended consequences.

Current behavior

```
In [1]: import pandas as pd

In [2]: print_ = lambda s: print('\n'.join([f'^{l}$' for l in s.split('\n')]))

In [3]: print_(pd.Series(['a', ' ']).to_string(index=False))
^ a$
^  $

In [4]: pd.__version__
Out[4]: '0.24.0'
```

If `strip` is used, I think we'd end up with something like:
```
In [X]: print_(pd.Series(['a', ' ']).to_string(index=False))
^a$
```
Yeah, strip is not a good solution here. 

> On Jan 31, 2019, at 17:40, Gosuke Shibahara <notifications@github.com> wrote:
> 
> Applying strip may have unintended consequences.
> 
> Current behavior
> 
> In [1]: import pandas as pd
> 
> In [2]: print_ = lambda s: print('\n'.join([f'^{l}$' for l in s.split('\n')]))
> 
> In [3]: print_(pd.Series(['a', ' ']).to_string(index=False))
> ^ a$
> ^  $
> 
> In [4]: pd.__version__
> Out[4]: '0.24.0'
> If strip is used, I think we'd end up with something like:
> 
> In [X]: print_(pd.Series(['a', ' ']).to_string(index=False))
> ^a$
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

I don't see a straightforward solution to this, that isn't going to regress in other ways.

It seems to me like we need to push down some of the *column* formatting to the *value* formatting. When deciding the values in the array

```
[10, -10]
```

we need to know that the array contains negative values, to decide how the first value, `10`, should be formatted.

Going to push to 0.24.2.

@charlesdong1991 feel free to keep working on this if you want, and let us know if you get stuck (though I won't be able to take a look for a week or two).
@TomAugspurger  I just added more edge cases in pytest as you mentioned above with my current fix, result looks as expected... e.g.
```python
>>> print(pd.Series([' a', ' b']).to_string(index=False))
 a
 b
```
```python
>>> print(pd.Series(['10', '-10']).to_string(index=False))
 10
-10
```
```python
>>> print(pd.Series(['1', '.1']).to_string(index=False))
 1
.1
```
have to say this fix is not very elegant... but if you agree on this fix, I could make leading_space more explicit as @jreback suggested.

Hi, sorry to bother you @TomAugspurger @jreback master seemed broken last a couple days, so the test didn't pass... today when i am trying to merge the master to get latest changes, and resubmit the PR, i still get an error which seems unrelated to my PR, could you give any hint where I should change in my PR to fix this issue? thank you very much!
```python
Traceback (most recent call last):
  File "C:\Miniconda\lib\site-packages\pkg_resources\__init__.py", line 357, in get_provider
    module = sys.modules[moduleOrReq]
KeyError: 'numpy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 738, in <module>
    ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
  File "setup.py", line 481, in maybe_cythonize
    numpy_incl = pkg_resources.resource_filename('numpy', 'core/include')
  File "C:\Miniconda\lib\site-packages\pkg_resources\__init__.py", line 1142, in resource_filename
    return get_provider(package_or_requirement).get_resource_filename(
  File "C:\Miniconda\lib\site-packages\pkg_resources\__init__.py", line 359, in get_provider
    __import__(moduleOrReq)
ModuleNotFoundError: No module named 'numpy'
Obtaining file:///D:/a/1/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "C:\Miniconda\lib\site-packages\pkg_resources\__init__.py", line 357, in get_provider
        module = sys.modules[moduleOrReq]
    KeyError: 'numpy'
```
I changed the default value for `leading_space` to `'compat'`as you suggested. Not sure if it's what you mean... feel free to leave your comments and advice. thanks @jreback 
if you can merge master i can have a look
@jreback sorry for my late reply. I have been with other things recently. I just fixed conflicts and merge the latest master. pls take a look if you have time
Hi, @jreback i fixed the conflicts and merged to master, pls take a look if you get some time, thanks very much!
can you merge master and will look again.
Hi, @jreback i merged to master and found this type validation failure, do you have idea why/how it is caused?
Hmm most likely caused by the merging of #26802 - thought it was green when I merged but apparently not. Let me take a look
@charlesdong1991 merging master again should resolve prior issue
@charlesdong1991 I think this is stale so closing for now but ping if you want to pick back up