ENH: Implement translations infrastructure
State: open

Hello team! 

This PR is a proposal for adding the translations infrastructure to the pandas web page.

Following the discussion in https://github.com/pandas-dev/pandas/issues/56301, we (a group of folks working on the Scientific Python grant) have been working to set up infrastructure and translate the contents of the pandas web site. As of this moment, we have 100% translations for the pandas website into Spanish and Brazilian Portuguese, with other languages available for translation (depending on volunteer translators).

To build, the command remains the same:

```bash
python pandas_web.py pandas/content --target-path build
```

If you want to check out other related work, please take a look at https://github.com/scipy/scipy.org/pull/617

You an read more about how the translation process works at https://scientific-python-translations.github.io/docs/

## What this PR does?

- Download and extract the latest available translations (over 90% completion) from https://github.com/Scientific-Python-Translations/pandas-translations. The setting can be changed [here](https://github.com/Scientific-Python-Translations/pandas-translations/blob/main/.github/workflows/sync_translations.yml#L21)
- Adds a Language switcher (Thanks @melissawm ❤️ 🚀 ).
- Added a new section to the config to store additional translations information.
- Handles site generation for each language.
- Left everything in the same script.

Supersedes https://github.com/pandas-dev/pandas/pull/61220

## Demo

![pandas](https://github.com/user-attachments/assets/fdb07a75-8661-4dc1-8239-29e792bac7ee)

---

cc @mroeschke @datapythonista 

Labels:
name: Docs
name: Stale

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/61380/
Thanks @goanpeca for this. Do you mind adding some more context here? I can't see in https://github.com/Scientific-Python-Translations/pandas-translations much information, like what languages are available, or how to fix a bad translation, which would be useful to know.

Also, in the docs generated from this PR I can see any language dropdown or anything different from our current docs. What are we expecting?
Hi @datapythonista  - this is a follow up to #61220, a proof-of-concept CI job to build the website with translations that don't live in this repo. This PR and #61220 are meant to work together and I'm happy to incorporate one into the other once we agree on the general direction and workflow for this.

Let us know if we can answer any other questions. Unfortunately I'm not sure how to get the preview for the other PR, I relied on building locally to test that things were working.
Sorry, I missed https://github.com/pandas-dev/pandas/pull/61220 and the issue discussion.

I don't fully understand what you're doing here, but I describe next how to add translations without adding too much complexity in this repo, which I don't think any core dev would be onboard with.

1. You decide on how to generate translations and manage it independently from this repo, and end up with a structure like this with the translated documents:

```
+ es/
  - index.md
  + about/
    - team.md
    - ...
  - ...
+ pt/
  - index.md
  + about/
    - team.md
    - ...
  - ...
```
2. In our CI, before calling `pandas_web.py` you download this directory structure to the `web/` directory. No other changes needed, this will create all translated pages.
3. We add a dropdown with the languages to the website (you can add the language list to `web/pandas/config.yml`)

I think this makes everyone's life easy, and we get the expected result.
Thanks @datapythonista !

Can you clarify what is missing from #61220 to match your description? That is pretty much what is done in that PR. Maybe this is confusing because we chose to do it in two parts exactly because we wanted to decouple the reorganization of the repo + switcher (in #61220) from the actual translations (this PR). 

Happy to follow up with any feedback in the other PR as well. Cheers!
In #61220 you are moving all the current website pages, that should be undone. You are adding the translated pages to this repo, we don't want it. You are making changes to pandas_web.py, this is not needed based on what I described above.

Only changes in a PR to this pandas repo should be addi g a CI step as per step 2, editing the wevsite template with the language dropdown as per step 3.
I see! I will rework what I have there to match your proposal. Thanks!
>Great improvement to the PR, this makes a lot of sense to me.

Thanks for the review @datapythonista.

>First, if I understand correctly, you download the translations of the web, and the percentage of the translated content, amd then you check for each language if it's translated enough to be published. Personally, I think you should better take care of this logic in your repo when generating the tar, not here. First to simplify the code here, and second to avoid downloading translations that are not going to be used.

Fixed!

>Just as a suggestion, I wouldn't use this approach, even in your repo. Imagine you publish translations that are at least 90%, and we have Spanish at 100%. Then I add new content that is 11% of the website. And automatically the Spanish translations that are already indexed by search engines, in user bookmarks, in links in blog posts... are deleted from our website. Not great in my opinion, much better to simply get the new content in English and hope it will eventually be translated.

This is also fixed!

>Another thing I would do is to extract the tar file as it is downloaded. So the tar file is passed to gzip/tarfile in memory, with the io module, not as a path in disk. With this you can get all the code here in a single short function. Or we ciuld even create a github action in your repo with this, as it's generic, and just use it here. So only the CI step would live in this PR.

Did not follow this one as I think things are now simpler and in a single script.

>Finally, we already have a configuration file for the website. We could save the url of the translations tar there. Also good in the script, just a question of preference. Or if you go for the github action approach, it could simply be a parameter in the CI step. Then you would need another one for the target dir in this repo.

Added the information the to the config file as requested and updated the scripts to handle site generation for languages. Moved all logic to the existing script.

>In any case, the approach here is also very reasonable, all above are suggestions that personally I think would make things simpler.

Please let me know what do you think about the current changes.

---

This PR now supersedes https://github.com/pandas-dev/pandas/pull/61220

Thanks @melissawm!
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/61380/
Thanks @goanpeca, a couple of comments.

Feels like we are repeating in this PR our logic that we already have implemented. If a markdown file is added to the web directory before the rendering of the website, it will be rendered in the same way as tge existing pages. There is no need to make changes to pandas_web.py for the translations, just copy the content of the translations tar in the web directory, and all the translated content will be in the rendered web.

In the preview links to translations are broken. This is likely to be caused by assuming the website is always going to be hosted at the root of the domain, not a subdirectory. This can't be assumed.

Finally, I think it'd be good to handle the downloading amd uncompressing of the translation file in a github action. Or less neat, in the step on the CI config, but out of pandas_web.py. This will keep things simple in pandas_web.py, and it'll be easier to maintain.
Hi again, thanks for the comments @datapythonista !

>There is no need to make changes to pandas_web.py for the translations, just copy the content of the translations tar in the web directory, and all the translated content will be in the rendered web.

1. Yes, **there is need** as the navbar is different for each language and this comes from the config file which contains translatable content. So each language has its own config and each language is treated as a separate site/build.

>In the preview links to translations are broken. This is likely to be caused by assuming the website is always going to be hosted at the root of the domain, not a subdirectory. This can't be assumed.

2. Fixed!

>Finally, I think it'd be good to handle the downloading amd uncompressing of the translation file in a github action. Or less neat, in the step on the CI config, but out of pandas_web.py. This will keep things simple in pandas_web.py, and it'll be easier to maintain.

3. AFAICT It is not possible (with the current way the website is build), to make the translations work without changes to the `pandas_web`script. I tried my best to keep the changes to the minimum on that file, and having the downloads there "is needed" to know the languages that are available and process things accordingly as pointed in 1. 

If this request stems from not wanting to have this happen locally, I could add a flag to process translations, and only handle english if not.

```
# Download and extract and copy translations
python pandas_web.py pandas/content --target-path build --translations

# Will just process english, with no translations download, extract or copy
python pandas_web.py pandas/content --target-path build

```

Thanks for the clarification @goanpeca, I understand now.

The idea is that pandas maintainers shouldn't have to spend time maintaining the script to build the website, so I want to keep it as simple as possible. Ideally we'd like to use Hugo or another static site generator. The problem is that the pandas website has a decent amount of content that is dynamically fetched from different sources. So, what I did is to do an almost single function extremely simple static site generation that simply renders markdown files into html using a template and leaving them in the same structure as they are found. And it uses a context which is the yaml config file parsed as a python object. There are a couple of tiny helper functions to this, and also a Preprocessors class which enriches the context with the external sources.

My concern here is that the 120 lines of code that our static site generator needs (excluding the preprocessors, which are independent small functions) grow singnificantly, in size and complexity.

I still think what I said about creating a github action in the repo managing the translation is the way to go. 90% of what it's needed is to download a tar file and uncompress it in a directory. If we don't translate the navbar, I guess this is fine and super simple, and the only thing needed in pandas_web.py is adding to the context the language of the page being translated. Does this make sense?

If we get to this point, then only the translation of the navigation bar would be missing while keeping things very simple. And for those, we could just add a json file to each language directory, fetch a different file with all the translations from the server in the navbar preprocessor, or something very simple like that.
Hi @datapythonista, thanks for the comments.

>The idea is that pandas maintainers shouldn't have to spend time maintaining the script to build the website, so I want to keep it as simple as possible. Ideally we'd like to use Hugo or another static site generator. The problem is that the pandas website has a decent amount of content that is dynamically fetched from different sources. So, what I did is to do an almost single function extremely simple static site generation that simply renders markdown files into html using a template and leaving them in the same structure as they are found. And it uses a context which is the yaml config file parsed as a python object. There are a couple of tiny helper functions to this, and also a Preprocessors class which enriches the context with the external sources.

I moved all the translations logic to a separate file `pandas_translations.py`.

>My concern here is that the 120 lines of code that our static site generator needs (excluding the preprocessors, which are independent small functions) grow singnificantly, in size and complexity.

I kept the changes to the bare minimum in the `pandas_web.py` script to be able to handle translations (~40 LOC).
- The context is generated once
- Created a separate `navbar.yml` file so that it can be translated separately and removed this section from the `config.yml`
- Added a utility function to update the context for the navbar only per language
- Maintained the `for` to iterate over languages, but is simpler now. Having it process all translation folders as normal content, and try to inject the context there just makes things more difficult to reason about.
- Added some typing that was missing
- Added an extra flag to process translations so:

```bash
# This will only build english as it normally did, it still downloads translations
# but does not include them in the process
python web/pandas_web.py web/pandas --target-path=web/build

# This will build all languages, download and process translations
python web/pandas_web.py web/pandas --target-path=web/build --translations
```

>I still think what I said about creating a github action in the repo managing the translation is the way to go. 90% of what it's needed is to download a tar file and uncompress it in a directory. If we don't translate the navbar, I guess this is fine and super simple, and the only thing needed in pandas_web.py is adding to the context the language of the page being translated. Does this make sense?

A GitHub action doesn't seem too helpful, because it isn't applicable anywhere else than this project. This is why I separated the logic into `pandas_translations.py` and imported this in `pandas_web.py` to be able to process language information

>If we get to this point, then only the translation of the navigation bar would be missing while keeping things very simple. And for those, we could just add a json file to each language directory, fetch a different file with all the translations from the server in the navbar preprocessor, or something very simple like that.

This is now solved by splitting the navbar content to a separate `navbar.yml` file. I wholeheartedly disagree with not having the navbar items translated as that would hamper accessibility of the translated website.

>In the preview links to translations are broken. This is likely to be caused by assuming the website is always going to be hosted at the root of the domain, not a subdirectory. This can't be assumed.

This has been fixed.

>Ideally we'd like to use Hugo or another static site generator. 

Great! I believe that is definitely a move in the right direction 😄 

In the meantime I believe this the best I can come up with to keep things simple, but functional.

Cheers!

Thanks @goanpeca, this looks better.

I think we can still make things significantly simpler. In the pandas_web script we iterate over every file in the `web/pandas/` directory, and we render it with the context: https://github.com/pandas-dev/pandas/blob/main/web/pandas_web.py#L477

My point was that if you copy the translated markdowns there, inside their language directory (e.g. `web/pandas/es/`) the files will be automatically rendered without doing anything else. And to render the navbar translated, you'll just need the navbar translated in the context (same as now, but for example wrapped in a dict `{"en": ..., "es":...}`. And then you'll need to add to the context the language of the file being rendered. You can simply consider all directories with two characters to be a translation directory.
Hi @datapythonista, thanks for the explanation.

I will try it.
@datapythonista, implemented the suggestions. 
>Thanks for the updates. Added some more comments, but looking better.

Thanks for the comments. I just replied to all of them and will post some updates later today. 
Hi @datapythonista, I implemented the suggestions.
Hi @datapythonista, I implemented most of the additional suggestions. Please see comments.
> I'd split the preprocessors in a slightly different way as suggested in the comments, but I think the way the code is now is very simple and easy to understand and maintain. Thanks a lot for all the updates here.

Made some new changes based on your suggestions @datapythonista.
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/61380/
The sponsor logos in the home page don't render correctly. I guess the problem is not in this PR, but in the translation of the html file, no?
>The sponsor logos in the home page don't render correctly. I guess the problem is not in this PR, but in the translation of the html file, no?

Would it be ok to use absolute URLs? since the english pages live in the root but the translated pages live in `es/something`. Would not work on preview though.

I could use `/static/img...` instead of `../static` at is currently used. Frameworks rely on filters like `relative_url / absolute_url` to handle this cases and append the appropriate base_folder or base_url to the link. 

Either that, or copying assets folder into each language.

The images of the books should be implemented in the same exact way, and those seem to be working fine in the translated pages. Doesn't seem like we need to change the links, feels more like a problem in the translated content for those images, no?
>The images of the books should be implemented in the same exact way, and those seem to be working fine in the translated pages. Doesn't seem like we need to change the links, feels more like a problem in the translated content for those images, no?

I will look into the content.
>I guess the problem is not in this PR, but in the translation of the html file, no?

Correct. This is an issue in the translations. (Working on those fixes)

¿Besides that is there anything else you consider needs a revision?
/preview