Localization of scikit-learn website content.

Hi,

I work for Quansight Labs and am helping set things up for translation of main project website content for core projects in the Scientific Python ecosystem. This work is supported by the [Scientific Python Community & Communications Infrastructure grant](https://scientific-python.org/doc/scientific-python-community-and-communications-infrastructure-2022.pdf) from CZI. I created a Github discussion for this here: https://github.com/scikit-learn/scikit-learn/discussions/28105, but @glemaitre pointed out that an issue would be more appropriate.

A deliverable for this grant is to have the brochure websites of at least 8 of the 10 Scientific Python core projects translated into at least 3 commonly used languages. You may have seen the language drop-down selector at https://numpy.org/.  The goal is for a cross functional team of Quansight staff and volunteers to handle the bulk of the work, taking the burden off of project maintainers.

The localization management platform [Crowdin](https://crowdin.com/) is offering a free supported enterprise organization for Scientific Python translations. This is the platform we used for numpy.org. At this moment you would not need to decide what if anything you would do with translated content. *However, to get things started, I'd like to ask permission to fill out the [Crowdin Open Source Project Setup Request ](https://crowdin.com/page/open-source-project-setup-request) form on your behalf, to allow me to add a project to the Scientific Python Crowdin organization for Scikit-Learn.* 

For those interested. Here are the specifics of what I'd do with this.

- Create a GitHub repository which mirrors the content from the Scikit-Learn brochure website.
- Setup a cron github action which polls for updates to the Scikit-Learn website content and helps keep the mirror up to date.
- Sync this repository to the Scikit-Learn Crowdin project. Translate can then translate the content, and Crowdin will automatically push commits to a PR against the repository mirroring the website content.

The end product will be a repository with parallel versions of the website content in a variety of languages which is kept up to date. If you want to host the translations on your webpage like NumPy does, myself and/or a colleague from Quansight would be supported by the grant to help make that happen.

Please let me know if you have any questions.

COMMENT:
Thanks for providing context @Charlie-XIAO. I knew I would miss something. Like you pointed out, the rationale for limiting the scope of translations to only core website content is laid out in NEP 28. 

*We start with an assertion: maintaining translations of all documentation, or even the whole user guide, as part of the NumPy project is not feasible. One simply has to look at the volume of our documentation and the frequency with which we change it to realize that that’s the case. Perhaps it will be feasible though to translate just the top-level pages of the website. Those do not change very often, and it will be a limited amount of content (order of magnitude 5-10 pages of text).*

I think up-to-date translations of API documentation, tutorials, examples etc. would be incredibly valuable, but also very difficult, and its outside the scope of the deliverables for the grant I'm working under. 

To answer more questions:

> One thing I don't understand from the original comment is which part of the website you plan to translate (API docs, examples, user guide, ..?) and how the translations would appear in the scikit-learn website (how would a user discover that translations exist)? What would the process be to keep things up to date across releases but also within a release?

For the question of how the translations would appear: At https:://numpy.org, there is a dropdown in the top right corner for selecting the language. For other projects like scikit-learn, I think the decision on how translations appear can be deferred for later. What I'm trying to do now is get all of the infrastructure set up for translation management for the projects. The plan is that in around one month this project will be taken over by someone at Quansight with more relevant experience, and my job is to have all of the groundwork in place for whoever that ends up being when they get started. 

@Charlie-XIAO has already helped answer the question about the scope of content to translate. Having a limited scope will simplify the process of keeping things up to date, since core website content shouldn't change as much. For NumPy, so far changes have been small when they are made, and we just let stuff remain out of date until the new translations come in.

> But I'm also concerned about the second part because I'm not sure how the Crowdin part of the workflow works. If it cannot be fully automated (e.g. need people to review) then the update likely cannot be done at the granularity of commits. We would also need native speakers of the language (or at least people who are familiar with the language) to maintain that part.

The plan is to have a common community of volunteer and perhaps sometimes paid translators across different projects in the Scientific Python ecosystem. Maintainers from individual projects wouldn't need to find or work directly with translators. For the Crowdin workflow: Crowdin creates a branch in the synched repo that it pushes individual commits to for each update made to the translations in Crowdin. The plan is that these would be made to a repo that mirrors (and stays up to date with) the content from the Scikit-Learn website. The goal would be to maintain parallel versions of the content in this mirror repo in different languages. I think how these are used in Scikit-Learn's website can be settled down later; but we hope minimize the burden on maintainers of the core projects.

> Another of my major concern is that, I think (guess) the most viewed part of the scikit-learn documentation is exactly its technical part (user guide, examples, API) and those are the largest parts. How much value is left if we leave all those behind, or how much effort we need to pay to take those parts into account?

For now, this grant supported project only covers core project websites, but this could potentially be just the beginning steps. Perhaps if a strong enough community of translators is organized, it may become more feasible to start maintaining translations of technical documentation. I agree that translated technical documentation would be more valuable, but if we're going to get there, small steps need to be taken first.

For now, what I'm asingk is for permission to set up a repo for mirroring content from the Scikit-Learn website and to fill out the Crowdin Open Source Request form I linked to above on your behalf, so I can create a Crowdin project for Scikit-Learn within the Scientific Python Crowdin organization and sync it to this repo. Where things go from there can be settled down later I hope. Even in the case where you ultimately decide not to host the translations on your web page, I think it would still be valuable for them to be created and maintained.

Labels: Needs Decision