Improve Wikipedia Loading Script
State: closed | Author: geohci (CONTRIBUTOR)

* More structured approach to detecting redirects
* Remove redundant template filter code (covered by strip_code)
* Add language-specific lists of additional media namespace aliases for filtering
* Add language-specific lists of category namespace aliases for new link text cleaning step
* Remove magic words (parser directions like __TOC__ that occasionally occur in text)

Fix #3400

With support from @albertvillanova 
CC @yjernite

Comments: I wanted to flag a change from since we discussed this: I initially wrote a function for using the Wikimedia APIs to collect namespace aliases, but decided that adding in more http requests to the script wasn't a great idea so instead used that code to build a static list that I just added directly to the code.

Also, an FYI that python library dependencies weren't working on my local end so I wasn't able to directly test the code. I tested a copy with the problematic elements stripped (beam etc.) that worked fine, but someone with a working local copy may want to test just to make sure I didn't accidentally break anything.

Also, while I would argue more strongly for some of the changes in this code, they are five distinct changes so not so hard to remove one or two if other folks think they aren't worth the overhead etc.

I also add a comment by @geohci in the Issue page:
> See https://public.paws.wmcloud.org/User:Isaac_(WMF)/HuggingFace%20Wikipedia%20Processing.ipynb for more implementation details / some data around the overhead induced by adding the extra preprocessing steps (stripping link prefixes and magic words)

Hi ! Thanks a lot, this is very cool ! Note that unfortunately if we change the processing right now, users won't be able to load the "big" languages like english anymore, because it requires an Apache Beam runtime to process them. Some Wikipedia dumps have been processed by Hugging Face so that users don't need to run Apache Beam stuff.

Therefore, we can merge this change after we have processed dumps using this new processing, and host them on the Hugging Face google storage.

I think we can take care of this and let you know once this is ready ? What do you think @albertvillanova ?

This is also an opportunity to have the latest dumps ready, the current ones are from 2020

Related PR on updating to the latest dates: https://github.com/huggingface/datasets/pull/3612

@lhoestq if the additional processing steps are validated, we could go on generating the processed datasets for the big languages.

The only thing before doing that is that we should also validate other change (so that we include it also in the processed datasets):
- #3398 

> @lhoestq if the additional processing steps are validated, we could go on generating the processed datasets for the big languages.

Cool ! Looking forward to it :)

> The only thing before doing that is that we should also validate other change (so that we include it also in the processed datasets):
> 
> https://github.com/huggingface/datasets/issues/3398

Sounds good ! We can definitely add the URL as asked by the Wikipedia to provide credits to the authors.

@geohci I do not have push rights to this PR. See: [Enabling repository maintainer permissions on existing pull requests](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork#enabling-repository-maintainer-permissions-on-existing-pull-requests).

I would like to merge the master branch so that all tests pass. Once done, I will be able approve this PR.

> @geohci I do not have push rights to this PR. See: [Enabling repository maintainer permissions on existing pull requests](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork#enabling-repository-maintainer-permissions-on-existing-pull-requests).
> 
> I would like to merge the master branch so that all tests pass. Once done, I will be able approve this PR.

@albertvillanova the `Allow edits by maintainers` box was already checked (what your instructions indicated) and indicates `If checked, users with write access to huggingface/datasets can add new commits to your wikipedia-updates branch. You can always change this setting later.` so you should have permissions already. If there's something else I'm missing or can do, please let me know. If it's not easy to resolve, I am plenty comfortable with you creating a new PR with these changes under your account too.