Make DataFrame.to_string output full content by default
State: closed

I modeled this off of https://github.com/pandas-dev/pandas/pull/24841. Some alternatives I considered:

* Instead of setting the option_context here, we could wind the param into the depths of the formatter. I tried this, actually, and started finding a number of edge cases and bugs. I realized that the issue only occurs in a pretty narrow case - if the user is explicitly calling to_string - because most of the time, when representing a DataFrame, the user *will* want long strings truncated for readability. So I think the safest way is to do it at the top level without interfering with lower-level formatters.
* Series.to_string() could arguably benefit from the same treatment, although that wasn't mentioned in the original issue (and I have never found the need to use it personally) so I didn't bring that in.

Here's an example on a real dataset showing long columns preserved in a text file produced by to_string():

![Screen Shot 2019-08-20 at 10 21 06 PM](https://user-images.githubusercontent.com/59923/63400279-d3f4ec80-c398-11e9-85cf-380a8981d726.png)

Additional manual testing:

* Main use case- by default, no limits and ignores the display options, but can still override:

```
>>> print(df.to_string())
        A       B
0     NaN     NaN
1 -1.0000     foo
2 -2.1234   foooo
3  3.0000  fooooo
4  4.0000     bar
>>> with option_context('display.max_colwidth', 5):
...     print(df.to_string())
... 
        A       B
0     NaN     NaN
1 -1.0000     foo
2 -2.1234   foooo
3  3.0000  fooooo
4  4.0000     bar
>>> print(df.to_string(max_colwidth=5))
      A     B
0   NaN   NaN
1 -1...   foo
2 -2...  f...
3  3...  f...
4  4...   bar
```

* The string representation of DataFrame **does** still use the display options (so it's only the explicit ```to_string``` that doesn't:

```
>>> with option_context('display.max_colwidth', 5):
...     print(str(df))
... 
      A     B
0   NaN   NaN
1 -1...   foo
2 -2...  f...
3  3...  f...
4  4...   bar
```

* The new parameter validates for None and positive ints, but rejects anything else:

```
>>> print(df.to_string(max_colwidth=-5))
    ...
    raise ValueError(msg)
ValueError: Value must be a nonnegative integer or None
```

- [ ] closes #9784 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: API Design

Comments:
Hello @lshepard! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-09-14 14:09:02 UTC
> I think we'll need to deprecate the current behavior, rather than just changing the default. 
> Users will need to be explicit about the max_colwidth for now I think.

To make sure I understand, what should happen when the user calls ```df.to_string()``` without parameters?

* Limits columns to the ```display.max_colwidth``` config option (the current default)
* No limit to column width (this PR)
* Make max_colwidth a required parameter (so vanilla ```to_string()``` no longer works)
* One of the above, but with a deprecation warning

Sounds like you're suggesting that we continue the current behavior, but with a deprecation warning? Should we only sound the deprecation warning if the data frame contains columns that would have otherwise been truncated? Seems like in most cases, the truncation won't be a difference in behavior and I would hate to make vanilla ```df.to_string()``` not work...

To update: I chose to punt the justify question as I think that's a separate issue that pre-existed and I may not understand the use case that well anyway.

I changed the behavior of max_colwidth so that it will use ```None``` to mean unlimited instead of the sentinel of 999999. I also changed the other two places from where I had copied the style (in the clipboard formatter & the html formatter).

I have not yet added a deprecation warning  - happy to do so if you think desired behavior is that the ```max_colwidth``` param should be required in the future. That doesn't make as much sense to me.
> To make sure I understand, what should happen when the user calls df.to_string() without parameters?

That needs to be decided.  It seems that in https://github.com/pandas-dev/pandas/pull/28052 we decided that to_html not outputting the entire dataframe was a bug. Presumably we would say the same about this, but I'm not sure.

cc @simonjayhawkins 
> > To make sure I understand, what should happen when the user calls df.to_string() without parameters?
> 
> That needs to be decided. It seems that in #28052 we decided that to_html not outputting the entire dataframe was a bug. Presumably we would say the same about this, but I'm not sure.
> 
> cc @simonjayhawkins

we should probably maintain consistency. although long html would probably wrap within a cell and that does not apply to to_string.

I agree- the original issue came from someone calling vanilla to_string and being surprised:

> I am calling to_string() without any 
> parameters and it beautifully fixed-
> formatted my dataframe apart from my 
> very wide filename column, that is being 
> truncated with "...". How can I avoid that?

I think no truncation by default is the most intuitive approach, and matches to_html behavior. Someone who is surprised can find the parameter to truncate pretty easily in the docs, and it’s unlikely to be a surprise the same way that truncation might be.


Thanks for the feedback! I added the whatsnew notice, swapped to the correct validator, and did some more manual testing to ensure it worked as expected.
Ready for review, thanks!
@lshepard merge conflict in the release notes. Could you merge master & repush?
Huh, failed with “worker 'gw0' crashed while running 'pandas/tests/test_sorting.py::TestSafeSort::test_labels_out_of_bound[-1]'”. I don’t  think that’s related but I’ll take a look.
Could be a random failure. I'd start by repushing an empty commit.
Thanks for all the detailed comments on the review!
@lshepard can you fix the merge conflict on the whatsnew? @TomAugspurger mind taking a look at this one?
Nice PR - thanks a lot @lshepard 
Excited to try this out and see it resolved.  I hope proper left justification can be resolved soon too.  Thanks.