ENH: IO header formatting
State: closed

 - [x] closes #15536 
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [x] whatsnew entry

In this branch I changed ``pd.formats.format.DataFrameFormatter._to_str_columns`` which
is used by ``to_latex`` and ``to_string`` to format output nicely.
They support now optional headers for the column titles.
See the appended [example jupyter notebook](https://github.com/pandas-dev/pandas/files/813954/Examples_IO_formatting_issue15536.txt).

``to_html`` does not use ``pd.formats.format.DataFrameFormatter._to_str_columns`` and remains an issue.

Test cases were created.

Labels:
name: Enhancement
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO LaTeX
description: to_latex

Comments:
so this has a bit of overlap with code in ``CSVFormatter._save_header``. anything that you can combine would be helpful (e.g. maybe make a member function that can be called from both for example).
fyi your travis job was cancled. master is breaking, so just wait a min while I fix (then rebase).
@mcocdawc about combining code. I merely want you to make a helper method or function (maybe ``_header_from_columns``, which can be called from *both* ``_save_header`` and ``_str_columns`` as these *seem* to be doing a lot of the same things (in part)
This function ``_header_from_columns`` is then neither a method of the ``formatCSVFormatter`` nor of the ``format.DataFrameFormatter`` class alone.

So should it just reside as function in the ``format`` namespace and then I make a construction like this:
```python
def f(self, x):
    return self.value * x

class example1:
    def __init__(self):
        self.value = 5
    def f(self, x):
        return f(self, x)

class example2:
    def __init__(self):
        self.value = 10
    def f(self, x):
        return f(self, x)
```
Or what is the preferred structure?
@mcocdawc you can call it as a function directly, no? It might be more convenient as a method (e.g. in ``DataFrameFormatter``, because then its easy to call and has access to state (e.g. ``.header``).

FYI you can rebase on master now and push
Ok, but if I define it as method in ``DataFrameFormatter`` that relies on the state of a ``DataFrameFormatter`` object. How could it be reused by a ``CSVFormatter`` object.

Should it be similar to this construct?:
```python
class example3:
    def __init__(self, value):
        self.value = value
    def g(self):
        return self.value * 10

class example4:
    def __init__(self, value):
        self.g = example3(value).g
```
yeah this is a dumb problem. ``CSVFormatter`` should *really* inherit from ``DataFrameFormatter``. So what you can do instead is aimply add a Mixin

e.g.

```
class FormatterMixin(object):
     def _my_shared_method(.....):
          .....

class DataFrameFormatter(...., FormatterMixin):
       ....

class CSVFormatter(....., FormatterMixin):
      ....

```

*maybe* they could be combined more. I would actually do this as a separate PR (merged before this one). To make this clean.

1. Of course, a Mixin Class really looks like a clean solution implementationwise.
2. Still there is the question of why ``CSVFormatter`` and ``DataFrameFormatter`` differ so much.
Is this just legacy because there was only ``CSVFormatter`` in the beginning and ``DataFrameFormatter`` came later? Or is there a real performance reason why ``CSVFormatter`` writes directly. Because if "directly writing" is not required, one could easily include ``CSV`` into ``DataFrameFormatter``.
3. I am really interested into improving this code part, but I am a student in the middle of my exam period. So I cannot start doing such a project before the middle of April.
4. The PR does not introduce any structural changes, but it is still better than the current state. So for the time being I would include it; but that is your decision.
5. If you see flaws in code style..., I am happy to learn.
> 2. Still there is the question of why ``CSVFormatter`` and ``DataFrameFormatter`` differ so much.
Is this just legacy because there was only ``CSVFormatter`` in the beginning and ``DataFrameFormatter`` came later? Or is there a real performance reason why ``CSVFormatter`` writes directly. Because if "directly writing" is not required, one could easily include ``CSV`` into ``DataFrameFormatter``.

I think just historical (should be fixed)

> 3. I am really interested into improving this code part, but I am a student in the middle of my exam period. So I cannot start doing such a project before the middle of April.

no problem. please come back to help on it! (you can make an issue to describe what needs to be done so won't be lost)

you need to rebase on master and force push, you have picked up commits which are already there

something like
```
git fetch origin/master
git checkout thisbranch
git rebase -i origin/master
git push yourfork yourbranch -f
```
Sorry I don't completely understand the problem and the output of your commands. 
My local branch for doing stuff is ``IO_formatting``.
I typed:
```
git fetch origin master 
git checkout IO_formatting
git rebase -i origin/master
```
The output of the last command was:
```
The previous cherry-pick is now empty, possibly due to conflict resolution.
If you wish to commit it anyway, use:

    git commit --allow-empty

Otherwise, please use 'git reset'
interactive rebase in progress; onto 3f91d5a
Last commands done (10 commands done):
   pick f329179 renamed test_to_string_specified_header
   pick 522dcdd ENH: _to_str_columns accepts now optional header Hereby also to_latex and to_string accept optional headers
Next commands to do (3 remaining commands):
   pick 5df79b5 Added tests for to_string and to_latex
   pick 23f420f Added tests for to_latex
You are currently rebasing branch 'IO_formatting' on '3f91d5a'.

nothing to commit, working directory clean
Could not apply 522dcddbf470aaf9458d4429f6a60bfdb00950df... ENH: _to_str_columns accepts now optional header Hereby also to_latex and to_string accept optional headers
```
Do I have to cheryypick something or can I proceed with ``git push IO_formatting -f`` (it defaults to my origin which is my fork, doesn't it?)
hmm, maybe just try with ``git rebase origin/master``. it should just fast-forward.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15548?src=pr&el=h1) Report
> Merging [#15548](https://codecov.io/gh/pandas-dev/pandas/pull/15548?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/0bf45320440e0d477f6b31ac8825f34e8212f152?src=pr&el=desc) will **decrease** coverage by `-0.03%`.
> The diff coverage is `100%`.


```diff
@@            Coverage Diff             @@
##           master   #15548      +/-   ##
==========================================
- Coverage   91.06%   91.04%   -0.03%     
==========================================
  Files         137      137              
  Lines       49307    49314       +7     
==========================================
- Hits        44900    44896       -4     
- Misses       4407     4418      +11
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15548?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/compare/0bf45320440e0d477f6b31ac8825f34e8212f152...2be1fb187907c88d71d368308a003a63cf7168fb?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.83% <100%> (-0.1%)` | :x: |
| [pandas/formats/format.py](https://codecov.io/gh/pandas-dev/pandas/compare/0bf45320440e0d477f6b31ac8825f34e8212f152...2be1fb187907c88d71d368308a003a63cf7168fb?src=pr&el=tree#diff-cGFuZGFzL2Zvcm1hdHMvZm9ybWF0LnB5) | `95.35% <100%> (+0.01%)` | :white_check_mark: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/compare/0bf45320440e0d477f6b31ac8825f34e8212f152...2be1fb187907c88d71d368308a003a63cf7168fb?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :x: |
| [pandas/tools/merge.py](https://codecov.io/gh/pandas-dev/pandas/compare/0bf45320440e0d477f6b31ac8825f34e8212f152...2be1fb187907c88d71d368308a003a63cf7168fb?src=pr&el=tree#diff-cGFuZGFzL3Rvb2xzL21lcmdlLnB5) | `91.78% <0%> (-0.35%)` | :x: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15548?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15548?src=pr&el=footer). Last update [0bf4532...2be1fb1](https://codecov.io/gh/pandas-dev/pandas/compare/0bf45320440e0d477f6b31ac8825f34e8212f152...2be1fb187907c88d71d368308a003a63cf7168fb?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
can you add a whatsnew note. And update the ``.to_latex`` doc-string as needed.
1. The header parameter for ``.to_latex`` comes from ``fmt.common_docstring``.
It is also used for ``.to_html`` and ``.to_string``.
Since ``.to_html`` is implemented in a way that it makes no use of ``._to_str_columns``
I will create a wrong documentation for ``.to_html`` if I change the ``fmt.common_docstring``.
I could create a ``fmt.common_docstring_html`` which takes care of the special case of html.

2. Where do I create the whatsnew notes correctly.

3. I will create an Issue for the refactoring of the format module.
> 1. The header parameter for ``.to_latex`` comes from ``fmt.common_docstring``.
It is also used for ``.to_html`` and ``.to_string``.
Since ``.to_html`` is implemented in a way that it makes no use of ``._to_str_columns``
I will create a wrong documentation for ``.to_html`` if I change the ``fmt.common_docstring``.
I could create a ``fmt.common_docstring_html`` which takes care of the special case of html.

you can use replaceable parameters in the doc-string with Substitution

> 2. Where do I create the whatsnew notes correctly.

doc/source/whatsnew/v0.20.0.txt in Enhancements


Ok, I tried to make a clean rebase operation and it still does not work.
The bad thing is that I created an awfully cluttered history of snapshots.
My proposed solution is to create a new branch which is rebased on upstream/master from the beginning on.
I will stop this pull request then and make a new one for the new branch.

To summarise the results of your review:
- ``format.py``
     - rewrite ``_to_str_columns`` to use provided aliases in the header parameter
     - uses ``is_list_like`` for the conditional
     - provides substitution for the header in ``common_docstring``
- ``frame.py``
    - make ``@Substitutions`` for the header parameter in the docstrings of ``to_latex``, ``to_string`` and ``to_html``
- ``tests/formats``
    - ``test_format.py``: Make function`` test_to_string_specified_header``
    - ``test_to_latex.py``: Make function ``test_to_latex_specified_header``
- ``whatsnew/v0.20.0.txt``
    - summarise changes
@mcocdawc it doesn't matter to make this super clean. The merge will squash it. Though I generally squash my own PR's as they are simpler (to rebase) and understand. Lots of commits are fine, people do things differently.
the key thing is to rebase on origin/master; that will put all of your commits on the top
@mcocdawc I fixed your rebase. if any comments left by @jorisvandenbossche 
Thank you very much. Do I have to do now anything anymore for this commit?
@mcocdawc Thanks!