Localization of scikit-learn website content.

Hi,

I work for Quansight Labs and am helping set things up for translation of main project website content for core projects in the Scientific Python ecosystem. This work is supported by the [Scientific Python Community & Communications Infrastructure grant](https://scientific-python.org/doc/scientific-python-community-and-communications-infrastructure-2022.pdf) from CZI. I created a Github discussion for this here: https://github.com/scikit-learn/scikit-learn/discussions/28105, but @glemaitre pointed out that an issue would be more appropriate.

A deliverable for this grant is to have the brochure websites of at least 8 of the 10 Scientific Python core projects translated into at least 3 commonly used languages. You may have seen the language drop-down selector at https://numpy.org/.  The goal is for a cross functional team of Quansight staff and volunteers to handle the bulk of the work, taking the burden off of project maintainers.

The localization management platform [Crowdin](https://crowdin.com/) is offering a free supported enterprise organization for Scientific Python translations. This is the platform we used for numpy.org. At this moment you would not need to decide what if anything you would do with translated content. *However, to get things started, I'd like to ask permission to fill out the [Crowdin Open Source Project Setup Request ](https://crowdin.com/page/open-source-project-setup-request) form on your behalf, to allow me to add a project to the Scientific Python Crowdin organization for Scikit-Learn.* 

For those interested. Here are the specifics of what I'd do with this.

- Create a GitHub repository which mirrors the content from the Scikit-Learn brochure website.
- Setup a cron github action which polls for updates to the Scikit-Learn website content and helps keep the mirror up to date.
- Sync this repository to the Scikit-Learn Crowdin project. Translate can then translate the content, and Crowdin will automatically push commits to a PR against the repository mirroring the website content.

The end product will be a repository with parallel versions of the website content in a variety of languages which is kept up to date. If you want to host the translations on your webpage like NumPy does, myself and/or a colleague from Quansight would be supported by the grant to help make that happen.

Please let me know if you have any questions.

COMMENT:
> I think setting up the account on crowdin makes sense then. The danger here that we're gonna have translated docs, which people will read and then complain about, and we have no idea what those translations say. I wonder how much the translations from crowdin is going to be better than a auto google translate button on our website.

These are good concerns @adrinjalali . These topics came up in discussion with the Pandas core developers too, https://github.com/pandas-dev/pandas/issues/56301#issuecomment-1883943726. To summarize, based on translator's experience working on https://numpy.org, machine translation is best used as a starting point, and appears not to be advanced enough that it can be trusted to accurately translate the technical and jargon-filled language of scientific computing. It's valuable to have humans who understand the subject matter in the loop who can review and fix up the machine translations. I think @betatim observed this in his example too. 

For the danger of having translated docs people read and complain about; I think it would need to be very clear that if someone wants to complain, core project maintainers are not the people they should complain too. When someone tries to create an issue, there should be an option for issues with translations which links to the new issue page on the associated repo for the translated content. There should be clearly visible notes in the translated content pointing out where issues should be submitted too.

> I think it makes sense to ask some questions and discuss some of the details of how to get the translations back into the website before setting out on the translation effort. This is not just "community projects being slow and slowing things down with endless discussions". For me it is a counter balance to my first impression of this effort which is "someone somewhere applied for some grant to have someone else do something for someone on the internet", which makes me worry that we are driving full throttle in some random direction.

I think that's reasonable. At the moment I just want to get as many of the projects set up with repos mirroring the content synced to Crowdin to lay the ground work and basically just tick off a box.  I was explicitly told not to try to reach out to translators and push for translating content. I can let the existing translators who've worked on numpy.org not to work on Scikit-Learn translations yet. Basically, if I budget time to setting projects up, due to overhead it doesn't make a big difference in total effort whether I set up 1, 2, 3 or 10 projects. But if I need to context switch between holding discussions with maintainers, setting up projects, working on interactive documentation for SciPy, and my primary focus as a mathematical/statistical software developer it's going to make it difficult for me to keep up with everything. It may well be that the exact deliverables specified in the grant are misguided, and if so, this will come out in discussions with project maintainers, and I think the funders will only care that a good faith effort was made. 

> Worried... What I would want to know is: what's the mechanism for updating (hum, there is a business opportunity here to provide a system for automated translation, with fixes, that learns from the fixes and tries to apply them to new versions). I would really want to have a written plan for updating the translation, and one that have a realistic approach.

When it comes time to move forward with more than the groundwork, the idea is to work on a SPEC which contains such a written plan. This is the route things first went with when discussing things with Pandas maintainers. You might want to look at the PDEP I wrote and the discussion around it https://github.com/pandas-dev/pandas/pull/57204. The earliest draft in the commit history was much more detailed than what was there when the issue was closed.  A core goal of this project is to try to create a community of translators who can help keep the content up to date. In around one month someone skilled in these kinds of community building efforts will take over this project at Quansight, and my role will be to help maintain the infrastructure.

Labels: Needs Decision