DOC: Make explicit in pandas IO doc the imports and options
State: closed

- [x] works on #28038 
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] Removes code header from IO doc page.
- [x] Uses explicit imports in the IO doc page of user guide on first occurrence of requirement.

Reference @datapythonista - [149](https://github.com/python-sprints/pandas-mentoring/issues/149)

Labels:
name: Docs

Comments:
Thanks for the work on this @TanyaaCJain, looks great. Did you check if the pandas and numpy options are being used? It'd probably be useful to generate a diff of the current (master) html generated for that page, and the one with this changes, and without the options. So we know for sure what's being affected by this change.

@TomAugspurger @jorisvandenbossche @jreback this is the option that makes more sense to me. I think having a block with the imports and the options at the top of every file doesn't really look great. Thought?
This (suppressing the imports) seems fine for rendered HTML on the majority of the pages. I think we intentionally don't suppress them on the first few of the getting started.

For the binder version, presumably these will be visible / executed? Or are we not thinking that far ahead yet?
The main motivation of this change is not having hidden code, so the integration with Binder is easier and there is no magic for the users.

For the API pages I guess it will be more controversial. But in my opinion the changes to the user guide are very small and surely worth.
> The main motivation of this change is not having hidden code,

Note that in the current diff of this PR, it is still "hidden" to the user for the online html docs
The new commit does this:
- [x] imported `pandas` and added its options in the first visible code block
- [x] moved the suppressed or invisible code block using `clipdf` just before the code block using the `clipdf` variable
- [x] imported `BytesIO` in its first occurrence of requirement unlike previously imported along with `StringIO`
- [x] imported `os` again for adding it on its first occurrence of requirement in a visible code block and is earlier imported in a suppressed code block.

I need help with:
- `np.random.seed(123456)` is currently (in this PR) being called just before the first occurrence of `np.random.randn()` which is in a suppressed block. What is to be done?
   - Add it again in a visible code block with first occurrence of `np.random.randn()` as done with `import os`?
   - Let it only be in the suppressed code block?
- I am not able to get any diff while comparing the current (master) html with the one in this PR. Can you help me with the code. Probably, not writing the right one.

Please suggest if there are any other changes as well.
> np.random.seed(123456)

In general, do we care much about setting this? 
> * I am not able to get any diff while comparing the current (master) html with the one in this PR. Can you help me with the code. Probably, not writing the right one.

Also, @datapythonista can you help me with this?

> Also, @datapythonista can you help me with this?

My idea was to go to an updated master branch `git checkout master && git fetch upstream && git merge upstream/master`, then generate the documentation (just this page if you want to save time) `./doc/make.py --single=user_guide/io.rst` and then save somewhere the resulting html file `doc/build/html/user_guide/io.html`.

The, go to your branch, generate that page again and do a `diff -u io_from_master.html io_without_header.html`.

That should highlight everything that will be different for the user. Better to do it wihout removing the seed, otherwise all the data will be different.

Does that make sense?
This is the [diff.txt](https://github.com/pandas-dev/pandas/files/3535388/diff.txt) generated, can be viewed as html if the extension is changed. The file compared to original did not have the `clipdf` assignment along, but the numpy print option and seed. I'll commit the new changes after this.
didn't think about that, but the changes make the number of the `In [x]` and `Out [x]` of jupyter change, and the diff is huge, not so easy to see what changed. But thank you for it.
Answering this comment from @WillAyd on the issue: https://github.com/pandas-dev/pandas/issues/28038#issuecomment-528432981

This is still open to discussion, but personally I much like this approach than the header. There are some advantages:
- Simpler
- Runnable code with no magic for users
- If we make the code runnable with binder, not having headers/hidden code will probably help
- Sphinx will report the correct line on errors
- If what we did in this PR can extrapolate to the rest, there is not so much needed in the header anyway

I agree with your point that in a huge documentation page, having the import just at the beginning doesn't necessarily make sense, the document will often not be accessed sequentially. But discussed with @jorisvandenbossche, and seems reasonable to split very long pages. And with binder in mind, probably worth having the imports repeated at the beginning of main sections, rather than hidden.

So, while this PR may not be a perfect solution yet, I think it's a step forward in the right direction.
Agreed with Marc.

On Tue, Sep 10, 2019 at 10:41 AM Marc Garcia <notifications@github.com>
wrote:

> Answering this comment from @WillAyd <https://github.com/WillAyd> on the
> issue: #28038 (comment)
> <https://github.com/pandas-dev/pandas/issues/28038#issuecomment-528432981>
>
> This is still open to discussion, but personally I much like this approach
> than the header. There are some advantages:
>
>    - Simpler
>    - Runnable code with no magic for users
>    - If we make the code runnable with binder, not having headers/hidden
>    code will probably help
>    - Sphinx will report the correct line on errors
>    - If what we did in this PR can extrapolate to the rest, there is not
>    so much needed in the header anyway
>
> I agree with your point that in a huge documentation page, having the
> import just at the beginning doesn't necessarily make sense, the document
> will often not be accessed sequentially. But discussed with
> @jorisvandenbossche <https://github.com/jorisvandenbossche>, and seems
> reasonable to split very long pages. And with binder in mind, probably
> worth having the imports repeated at the beginning of main sections, rather
> than hidden.
>
> So, while this PR may not be a perfect solution yet, I think it's a step
> forward in the right direction.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28089?email_source=notifications&email_token=AAKAOIXHWWUQ3ALT6M46GLLQI65YZA5CNFSM4IOU7FTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6LRWUQ#issuecomment-529996626>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIQIAFX4ZXEVNEXKWOTQI65YZANCNFSM4IOU7FTA>
> .
>

Would you all prefer to split up this file into smaller sections too or just want it for way more lengthier ones? Though I think splitting doc into sections would help to understand difference in file much easily, if preferred.


> didn't think about that, but the changes make the number of the `In [x]` and `Out [x]` of jupyter change, and the diff is huge, not so easy to see what changed. But thank you for it.

I tried for a few other ways that could possibly generate a difference ignoring the ipython lines but none generated the desired output. I’ll update if I do find one.

If runnable code with binder is the main draw here (that's what I am inferencing at least, perhaps incorrectly) is there any reason why we don't defer moving things around like this until something like binder would actually get implemented?
Not sure where Marc is at on that, but I stopped my prototype at
https://github.com/TomAugspurger/pandas-binder when I noticed that the `{{
header }}` stuff wasn't working. So from that attempt's POV, this is a
blocker.

On Tue, Sep 10, 2019 at 2:58 PM William Ayd <notifications@github.com>
wrote:

> If runnable code with binder is the main draw here (that's what I am
> inferencing at least, perhaps incorrectly) is there any reason why we don't
> defer moving things around like this until something like binder would
> actually get implemented?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28089?email_source=notifications&email_token=AAKAOIT3J3HUHTBE6H4BZYDQI735VA5CNFSM4IOU7FTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6MJYLQ#issuecomment-530095150>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIRBFLK5GSV6QPQWRCLQI735VANCNFSM4IOU7FTA>
> .
>

Sounds good. I'll defer to you all then as I think you have a better big picture view of this and I don't think anything here is a showstopper. Just wanted to offer an alternative view - thanks!
@WillAyd to come back to your concern (as it was not fully addressed IMO):

> I'm not sure I am on board with the concept. Associating each import with its first reference in the rst feels a little arbitrary and I think is just confusing. Seems like it could be somewhat of a nuisance when moving things around; nothing major but in any case a step backwards from just knowing standard imports are done in the header

Currently, all standard imports (pandas, numpy) are at the beginning of each rst file, but hidden for the user. Now we still do them at the top of the file, but visible. So a net improvement, and not arbitrary.

Or at least, that's the case for the pandas import. But you are right that if the `from io import BytesIO` import is somewhere in the middle of the file, and then re-used without import near the end of the file in a totally different section, this can be confusing (not necessarily more confusing than not showing the import at all as it is now, though):

- for writing docs, I don't think this is a problem, we have CI checking for this now to prevent errors
- for really common imports (I think mainly `numpy`), it's maybe worth to still put this at the top of the file with the pandas import instead at the location of first usage? (in the end, that is also what I do in my notebook workflow)
- for less common imports that we want to do at the place of usage, I think we can actually repeat this import multiple times if they happen in a single file but in unrelated sections. 

And as Marc said, going to smaller files will also help with this.
> Would you all prefer to split up this file into smaller sections too

@TanyaaCJain I think we want to work towards that, but that's certainly for another PR (for the IO page we have https://github.com/pandas-dev/pandas/issues/10446 about this)
status on this PR @jorisvandenbossche @datapythonista 
If nobody objects to remove the header in the docs, the seeds, and make the imports explicit, this is ready to be merged, and we can start working on the rest of the docs.
I am fully behind removing the header and moving the imports in actual code-blocks.

One thing that might be good to decide on before starting other PRs on the rest of the docs, is where to put those imports.  
In the current PR, it is "the first code block where it is used" for everything. 

I think that is fine for less common imports, but it could be an option to still put eg numpy and pandas (and matplotlib.pyplot for those rst files that use it) in a visible code block in the beginning of each file. 

(but also fine to first try out with the approach as in this PR, and evaluate later about that question)
@datapythonista @jorisvandenbossche still want to merge?
Yeah, I think so.

Another point, maybe already mentioned, is that explicit imports will fix
the line-number reported by sphinx in errors.

On Thu, Nov 7, 2019 at 3:11 PM William Ayd <notifications@github.com> wrote:

> @datapythonista <https://github.com/datapythonista> @jorisvandenbossche
> <https://github.com/jorisvandenbossche> still want to merge?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28089?email_source=notifications&email_token=AAKAOIV7TW5CQC5DA6CRXS3QSSAABA5CNFSM4IOU7FTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDN2M4Y#issuecomment-551265907>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIXHK67JQH2H4M4JQT3QSSAABANCNFSM4IOU7FTA>
> .
>

Thanks @TanyaaCJain