GitHub issue #5305: Dataset joelito/mc4_legal does not work with multiple files
State: closed | Created: 2022-11-28T00:16:16Z

### Describe the bug

The dataset https://huggingface.co/datasets/joelito/mc4_legal works for languages like bg with a single data file, but not for languages with multiple files like de. It shows zero rows for the de dataset.


joelniklaus@Joels-MacBook-Pro ~/N/P/C/L/p/m/mc4_legal (main) [1]> python test_mc4_legal.py                                                                                                                      (debug) 
Found cached dataset mc4_legal (/Users/joelniklaus/.cache/huggingface/datasets/mc4_legal/de/0.0.0/fb6952a097180f8c936e2a7605525ff670354a344fc1a2c70107684d3f7cb02f)
Dataset({
    features: ['index', 'url', 'timestamp', 'matches', 'text'],
    num_rows: 0
})
joelniklaus@Joels-MacBook-Pro ~/N/P/C/L/p/m/mc4_legal (main)> python test_mc4_legal.py                                                                                                                          (debug) 
Downloading and preparing dataset mc4_legal/bg to /Users/joelniklaus/.cache/huggingface/datasets/mc4_legal/bg/0.0.0/fb6952a097180f8c936e2a7605525ff670354a344fc1a2c70107684d3f7cb02f...
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1240.55it/s]
Dataset mc4_legal downloaded and prepared to /Users/joelniklaus/.cache/huggingface/datasets/mc4_legal/bg/0.0.0/fb6952a097180f8c936e2a7605525ff670354a344fc1a2c70107684d3f7cb02f. Subsequent calls will reuse this data.
Dataset({
    features: ['index', 'url', 'timestamp', 'matches', 'text'],
    num_rows: 204
})

### Steps to reproduce the bug

import datasets
from datasets import load_dataset, get_dataset_config_names

language = "bg"

test = load_dataset("joelito/mc4_legal", language, split='train')

### Expected behavior

It should display the correct number of rows for the de dataset which should be a large number (thousands or more).

### Environment info

Package                  Version
------------------------ --------------
absl-py                  1.3.0
aiohttp                  3.8.1
aiosignal                1.2.0
astunparse               1.6.3
async-timeout            4.0.2
attrs                    22.1.0
beautifulsoup4           4.11.1
blinker                  1.4
blis                     0.7.8
Bottleneck               1.3.4
brotlipy                 0.7.0
cachetools               5.2.0
catalogue                2.0.7
certifi                  2022.5.18.1
cffi                     1.15.1
chardet                  4.0.0
charset-normalizer       2.1.0
click                    8.0.4
conllu                   4.5.2
cryptography             38.0.1
cymem                    2.0.6
datasets                 2.6.1
dill                     0.3.5.1
docker-pycreds           0.4.0
fasttext                 0.9.2
fasttext-langdetect      1.0.3
filelock                 3.0.12
flatbuffers              20210226132247
frozenlist               1.3.0
fsspec                   2022.5.0
gast                     0.4.0
gcloud                   0.18.3
gitdb                    4.0.9
GitPython                3.1.27
google-auth              2.9.0
google-auth-oauthlib     0.4.6
google-pasta             0.2.0
googleapis-common-protos 1.57.0
grpcio                   1.47.0
h5py                     3.7.0
httplib2                 0.21.0
huggingface-hub          0.8.1
idna                     3.4
importlib-metadata       4.12.0
Jinja2                   3.1.2
joblib                   1.0.1
keras                    2.9.0
Keras-Preprocessing      1.1.2
langcodes                3.3.0
lxml                     4.9.1
Markdown                 3.3.7
MarkupSafe               2.1.1
mkl-fft                  1.3.1
mkl-random               1.2.2
mkl-service              2.4.0
multidict                6.0.2
multiprocess             0.70.13
murmurhash               1.0.7
numexpr                  2.8.1
numpy                    1.22.3
oauth2client             4.1.3
oauthlib                 3.2.1
opt-einsum               3.3.0
packaging                21.3
pandas                   1.4.2
pathtools                0.1.2
pathy                    0.6.1
pip                      21.1.2
preshed                  3.0.6
promise                  2.3
protobuf                 4.21.9
psutil                   5.9.1
pyarrow                  8.0.0
pyasn1                   0.4.8
pyasn1-modules           0.2.8
pybind11                 2.9.2
pycountry                22.3.5
pycparser                2.21
pydantic                 1.8.2
PyJWT                    2.4.0
pylzma                   0.5.0
pyOpenSSL                22.0.0
pyparsing                3.0.4
PySocks                  1.7.1
python-dateutil          2.8.2
pytz                     2021.3
PyYAML                   6.0
regex                    2021.4.4
requests                 2.28.1
requests-oauthlib        1.3.1
responses                0.18.0
rsa                      4.8
sacremoses               0.0.45
scikit-learn             1.1.1
scipy                    1.8.1
sentencepiece            0.1.96
sentry-sdk               1.6.0
setproctitle             1.2.3
setuptools               65.5.0
shortuuid                1.0.9
six                      1.16.0
smart-open               5.2.1
smmap                    5.0.0
soupsieve                2.3.2.post1
spacy                    3.3.1
spacy-legacy             3.0.9
spacy-loggers            1.0.2
srsly                    2.4.3
tabulate                 0.8.9
tensorboard              2.9.1
tensorboard-data-server  0.6.1
tensorboard-plugin-wit   1.8.1
tensorflow               2.9.1
tensorflow-estimator     2.9.0
termcolor                2.1.0
thinc                    8.0.17
threadpoolctl            3.1.0
tokenizers               0.12.1
torch                    1.13.0
tqdm                     4.64.0
transformers             4.20.1
typer                    0.4.1
typing-extensions        4.3.0
Unidecode                1.3.6
urllib3                  1.26.12
wandb                    0.12.20
wasabi                   0.9.1
web-anno-tsv             0.0.1
Werkzeug                 2.1.2
wget                     3.2
wheel                    0.35.1
wrapt                    1.14.1
xxhash                   3.0.0
yarl                     1.8.1
zipp                     3.8.0

Python 3.8.10