DOC-#57585: Add `Use Modin` section on `Scaling to large datasets` page
State: closed

- [x] closes #57585  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
pre-commit.ci autofix
@rhshadrach, could you shed some light on why `Doc Build` is failing and what we should do to fix it?
> @rhshadrach, could you shed some light on why `Doc Build` is failing and what we should do to fix it?

L474, documentation is spelled incorrectly.
> > @rhshadrach, could you shed some light on why `Doc Build` is failing and what we should do to fix it?
> 
> L474, documentation is spelled incorrectly.

Oh, I see, thanks! FIxed.
I see the Dask section was added 5 years ago, where there were probably no other alternatives. I think it's surely valuable for users to know about other tools to distribute work with a pandas API, but I don't think our user guide is the place to have and have to maintain a short tutorial of Dask. Modin, PySpark/Koalas and any other system that provides a distributed pandas-like API.

What do you think if you have in each project documentation a page for pandas users that we can simply link here? @phofl @YarShev @HyukjinKwon
> I see the Dask section was added 5 years ago, where there were probably no other alternatives. I think it's surely valuable for users to know about other tools to distribute work with a pandas API, but I don't think our user guide is the place to have and have to maintain a short tutorial of Dask. Modin, PySpark/Koalas and any other system that provides a distributed pandas-like API.
> 
> What do you think if you have in each project documentation a page for pandas users that we can simply link here? @phofl @YarShev @HyukjinKwon

While other projects can have a documentation page about their tool and pandas as to which library to use and when, I think mentioning other alternatives to pandas in pandas documentation itself is valuable to users since pandas is much more popular than the alternatives, and many more people will be able to see concrete options for efficient data processing.
I don't disagree with you, and I think it's valuable for users to know the other packages in the ecosystem to scale what pandas can't handle.

My point is that in my opinion the pandas codebase is not where so much information from Dask or Modin should exist. I think we should have a paragraph so users know about the tools and what to expect about them, and all the details could be in a landing page for pandas users that you maintain in your docs, keep updated, and that we simply have a link here next to the summary of what each tool does.

Just my personal opinion, but this should be as good for the users, but easier to maintain and keep updated.
I tried to make the changes as concise as possible giving some pros and cons of Modin and when it should be used. I don't see anything in the changes that could be hard to maintain in future. If you have any improvements we could introduce in the paragraph, please let me know. It would be also great to hear opinions from other reviewers. cc @rhshadrach, @jbrockmendel 
Building on what @datapythonista said - I don't see much advantage in having the documentation as it is in this PR vs just the intro paragraph ending with a line like `See more information in [modin's documentation](https://modin.readthedocs.io/en/stable/getting_started/quickstart.html#example-instant-scalability-with-no-extra-effort).` (or a better landing spot if there is one).

> I don't see anything in the changes that could be hard to maintain in future.

Changes to any of the third party packages currently listed would necessitate a change to the pandas documentation and is therefore quite likely to go stale. In addition each new tool that comes along we have to review another PR. It is all around less maintenance for us to have a summary paragraph and a link to the projects documentation where they compare to pandas.

What is the disadvantage by having a link rather than the documentation live in pandas itself?
@datapythonista what about pandas here points out the tutorial of individual project?

Basically I support:

> he intro paragraph ending with a line like See more information in [modin's documentation](https://modin.readthedocs.io/en/stable/getting_started/quickstart.html#example-instant-scalability-with-no-extra-effort). (or a better landing spot if there is one).

We won't have to duplicate the tutorials in pandas page, and individual project page.

Apache Spark is sort of doing in the same way: https://spark.apache.org/third-party-projects.html

FWIW, Koalas has been migrated to PySpark as pandas API on Spark component so I have to update it in any event.
> @datapythonista what about pandas here points out the tutorial of individual project?

Sounds great. My idea is that users reaching our page about scaling should know about the relevant options, but without pandas having lots of details in our docs when we can simply link to each projects docs. The exact page you link to I don't know, maybe a dedicated landing page for users coming from pandas, maybe the home of the Koalas documentation, no strong preference from my side.

Also worth mentioning that we already do it in the Ecosystem page in our website. Maybe for now I can just copy what it's there in a separate PR, and you all can review it or follow up with any change if needed. What do you think?
@datapythonista, @HyukjinKwon, @rhshadrach, all your thoughts make sense to me now. We should probably put the detailed information about Modin and its joint usage with pandas in a single flow to Modin's documentation. I left three small paragraphs about Modin in this PR. How does it look like to you now?
Ready for the next round of review. Hope it is fine now 😄 
@datapythonista, @HyukjinKwon, just a friendly reminder for review.
@datapythonista, sounds good! Could you please tag me in the future PRs where you would reduce the Dask section and and PySpark/Koalas? I would like Modin to be in line with those libraries sections.
Thanks @YarShev. Agreed that eventually this should just link to the ecosystem docs, but this is good as is.