Dataset joelito/mc4_legal does not work with multiple files
Opened by JoelNiklaus (closed)

### Describe the bug

The dataset https://huggingface.co/datasets/joelito/mc4_legal works for languages like bg with a single data file, but not for languages with multiple files like de. It shows zero rows for the de dataset.


joelniklaus@Joels-MacBook-Pro ~/N/P/C/L/p/m/mc4_legal (main) [1]> python test_mc4_legal.py                                                                                                                      (debug) 
Found cached dataset mc4_legal (/Users/joelniklaus/.cache/huggingface/datasets/mc4_legal/de/0.0.0/fb6952a097180f8c936e2a7605525ff670354a344fc1a2c70107684d3f7cb02f)
Dataset({
    features: ['index', 'url', 'timestamp', 'matches', 'text'],
    num_rows: 0
})
joelniklaus@Joels-MacBook-Pro ~/N/P/C/L/p/m/mc4_legal (main)> python test_mc4_legal.py                                                                                                                          (debug) 
Downloading and preparing dataset mc4_legal/bg to /Users/joelniklaus/.cache/huggingface/datasets/mc4_legal/bg/0.0.0/fb6952a097180f8c936e2a7605525ff670354a344fc1a2c70107684d3f7cb02f...
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1240.55it/s]
Dataset mc4_legal downloaded and prepared to /Users/joelniklaus/.cache/huggingface/datasets/mc4_legal/bg/0.0.0/fb6952a097180f8c936e2a7605525ff670354a344fc1a2c70107684d3f7cb02f. Subsequent calls will reuse this data.
Dataset({
    features: ['index', 'url', 'timestamp', 'matches', 'text'],
    num_rows: 204
})

### Steps to reproduce the bug

import datasets
from datasets import load_dataset, get_dataset_config_names

language = "bg"

test = load_dataset("joelito/mc4_legal", language, split='train')

### Expected behavior

It should display the correct number of rows for the de dataset which should be a large number (thousands or more).

### Environment info

Package                  Version
------------------------ --------------
absl-py                  1.3.0
aiohttp                  3.8.1
aiosignal                1.2.0
astunparse               1.6.3
async-timeout            4.0.2
attrs                    22.1.0
beautifulsoup4           4.11.1
blinker                  1.4
blis                     0.7.8
Bottleneck               1.3.4
brotlipy                 0.7.0
cachetools               5.2.0
catalogue                2.0.7
certifi                  2022.5.18.1
cffi                     1.15.1
chardet                  4.0.0
charset-normalizer       2.1.0
click                    8.0.4
conllu                   4.5.2
cryptography             38.0.1
cymem                    2.0.6
datasets                 2.6.1
dill                     0.3.5.1
docker-pycreds           0.4.0
fasttext                 0.9.2
fasttext-langdetect      1.0.3
filelock                 3.0.12
flatbuffers              20210226132247
frozenlist               1.3.0
fsspec                   2022.5.0
gast                     0.4.0
gcloud                   0.18.3
gitdb                    4.0.9
GitPython                3.1.27
google-auth              2.9.0
google-auth-oauthlib     0.4.6
google-pasta             0.2.0
googleapis-common-protos 1.57.0
grpcio                   1.47.0
h5py                     3.7.0
httplib2                 0.21.0
huggingface-hub          0.8.1
idna                     3.4
importlib-metadata       4.12.0
Jinja2                   3.1.2
joblib                   1.0.1
keras                    2.9.0
Keras-Preprocessing      1.1.2
langcodes                3.3.0
lxml                     4.9.1
Markdown                 3.3.7
MarkupSafe               2.1.1
mkl-fft                  1.3.1
mkl-random               1.2.2
mkl-service              2.4.0
multidict                6.0.2
multiprocess             0.70.13
murmurhash               1.0.7
numexpr                  2.8.1
numpy                    1.22.3
oauth2client             4.1.3
oauthlib                 3.2.1
opt-einsum               3.3.0
packaging                21.3
pandas                   1.4.2
pathtools                0.1.2
pathy                    0.6.1
pip                      21.1.2
preshed                  3.0.6
promise                  2.3
protobuf                 4.21.9
psutil                   5.9.1
pyarrow                  8.0.0
pyasn1                   0.4.8
pyasn1-modules           0.2.8
pybind11                 2.9.2
pycountry                22.3.5
pycparser                2.21
pydantic                 1.8.2
PyJWT                    2.4.0
pylzma                   0.5.0
pyOpenSSL                22.0.0
pyparsing                3.0.4
PySocks                  1.7.1
python-dateutil          2.8.2
pytz                     2021.3
PyYAML                   6.0
regex                    2021.4.4
requests                 2.28.1
requests-oauthlib        1.3.1
responses                0.18.0
rsa                      4.8
sacremoses               0.0.45
scikit-learn             1.1.1
scipy                    1.8.1
sentencepiece            0.1.96
sentry-sdk               1.6.0
setproctitle             1.2.3
setuptools               65.5.0
shortuuid                1.0.9
six                      1.16.0
smart-open               5.2.1
smmap                    5.0.0
soupsieve                2.3.2.post1
spacy                    3.3.1
spacy-legacy             3.0.9
spacy-loggers            1.0.2
srsly                    2.4.3
tabulate                 0.8.9
tensorboard              2.9.1
tensorboard-data-server  0.6.1
tensorboard-plugin-wit   1.8.1
tensorflow               2.9.1
tensorflow-estimator     2.9.0
termcolor                2.1.0
thinc                    8.0.17
threadpoolctl            3.1.0
tokenizers               0.12.1
torch                    1.13.0
tqdm                     4.64.0
transformers             4.20.1
typer                    0.4.1
typing-extensions        4.3.0
Unidecode                1.3.6
urllib3                  1.26.12
wandb                    0.12.20
wasabi                   0.9.1
web-anno-tsv             0.0.1
Werkzeug                 2.1.2
wget                     3.2
wheel                    0.35.1
wrapt                    1.14.1
xxhash                   3.0.0
yarl                     1.8.1
zipp                     3.8.0

Python 3.8.10


Comments:
Thanks for reporting @JoelNiklaus.

Please note that since we moved all dataset loading scripts to the Hub, the issues and pull requests relative to specific datasets are directly handled on the Hub, in their Community tab. I'm transferring this issue there: https://huggingface.co/datasets/joelito/mc4_legal/discussions

I am also having a look at the bug in your script.

Issue transferred to: https://huggingface.co/datasets/joelito/mc4_legal/discussions/1