Wikipedia download_and_prepare for GCS
Opened by adrianfagerland (closed)

### Describe the bug

I am unable to download the wikipedia dataset onto GCS.

When I run the script provided the memory firstly gets eaten up, then it crashes. 
I tried running this on a VM with 128GB RAM and all I got was a two empty files: _data_builder.lock_, _data.incomplete/beam-temp-wikipedia-train-1ab2039acf3611ed87a9893475de0093_

I have troubleshot this for two straight days now, but I am just unable to get the dataset into storage.

### Steps to reproduce the bug

Run this and insert a path:
```
import datasets

builder = datasets.load_dataset_builder(
    "wikipedia", language="en", date="20230320", beam_runner="DirectRunner")

builder.download_and_prepare({path}, file_format="parquet")
```
This is where the problem of it eating RAM occurs.



I have also tried several versions of this, based on the docs:
```
import gcsfs
import datasets

storage_options = {"project": "tdt4310", "token": "cloud"}
fs = gcsfs.GCSFileSystem(**storage_options)

output_dir = "gcs://wikipediadata/"
builder = datasets.load_dataset_builder(
    "wikipedia", date="20230320", language="en",  beam_runner="DirectRunner")
builder.download_and_prepare(
    output_dir, storage_options=storage_options, file_format="parquet")
```
The error message that is received here is:

> ValueError: Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: gcs://wikipediadata/wikipedia-train [while running 'train/Save to parquet/Write/WriteImpl/InitializeWrite']

I have ran `pip install apache-beam[gcp]`

### Expected behavior

The wikipedia data loaded into GCS

Everything worked when testing with a smaller demo dataset found somewhere in the docs

### Environment info

Newest published version of datasets. Python 3.9. Also tested with Python 3.7. 128GB RAM Google Cloud VM instance.

Comments:
Hi @adrianfagerland, thanks for reporting.

Please note that "wikipedia" is a special dataset, with an Apache Beam builder: https://beam.apache.org/
You can find more info about Beam datasets in our docs: https://huggingface.co/docs/datasets/beam

It was implemented to be run in parallel processing, using one of the distributed back-ends supported by Apache Beam: https://beam.apache.org/get-started/beam-overview/#apache-beam-pipeline-runners

That is, you are trying to process the source wikipedia data on your machine (not distributed) when passing `beam_runner="DirectRunner"`.

As documented in the wikipedia dataset page (https://huggingface.co/datasets/wikipedia):

    Some subsets of Wikipedia have already been processed by HuggingFace, and you can load them just with:
    
        from datasets import load_dataset
    
        load_dataset("wikipedia", "20220301.en")

    The list of pre-processed subsets is:
    - "20220301.de"
    - "20220301.en"
    - "20220301.fr"
    - "20220301.frr"
    - "20220301.it"
    - "20220301.simple"

To download the available processed data (in Arrow format):
```python
builder = datasets.load_dataset_builder("wikipedia", "20220301.en")
builder.download_and_prepare(your_path)
```

When running this using :
```
import datasets
from apache_beam.options.pipeline_options import PipelineOptions
from gcsfs import GCSFileSystem

storage_options = {"project":"tdt4310", "token":"cloud"}
fs = GCSFileSystem(**storage_options)

output_dir = "gcs://quiz_transformer/"
beam_options = PipelineOptions(
        region="europe-west4",
        project="tdt4310",
        temp_location=output_dir+"tmp/")


builder = datasets.load_dataset_builder("wikipedia", "20220301.en", beam_runner="dataflow", beam_options=beam_options)
builder.download_and_prepare(
        output_dir, storage_options=storage_options, file_format="parquet")
```
I now get this error:
```
raise FileNotFoundError(f"Couldn't find file at {url}")
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/enwiki/20220301/dumpstatus.json
Downloading data files:   0%|                                                            | 0/1 [00:00<?, ?it/s]
```

I get the same error for this:
```
import datasets
from gcsfs import GCSFileSystem

storage_options = {"project":"tdt4310", "token":"cloud"}
fs = GCSFileSystem(**storage_options)

output_dir = "gcs://quiz_transformer/"
builder = datasets.load_dataset_builder("wikipedia", "20220301.en")
builder.download_and_prepare(
        output_dir, storage_options=storage_options, file_format="parquet")
```





`wikipedia` is no longer a Beam dataset, so the above code should work now.

PS: You can use [these files](https://huggingface.co/datasets/wikipedia/tree/main/data/20220301.en) (or a newer dump at https://huggingface.co/datasets/wikimedia/wikipedia/tree/main/20231101.en) instead of generating the Parquet version yourself