Cannot load the blog_authorship_corpus due to codec errors
State: closed | Author: izaskr (NONE)
Labels: name: bug

## Describe the bug
A codec error is raised while loading the blog_authorship_corpus. 

## Steps to reproduce the bug
```
from datasets import load_dataset
raw_datasets = load_dataset("blog_authorship_corpus")
```


## Expected results
Loading the dataset without errors.

## Actual results
An error similar to the one below was raised for (what seems like) every XML file.
/home/izaskr/.cache/huggingface/datasets/downloads/extracted/7cf52524f6517e168604b41c6719292e8f97abbe8f731e638b13423f4212359a/blogs/788358.male.24.Arts.Libra.xml cannot be loaded. Error message: 'utf-8' codec can't decode byte 0xe7 in position 7551: invalid continuation byte

Traceback (most recent call last):         
  File "<stdin>", line 1, in <module>
  File "/home/izaskr/anaconda3/envs/local_vae_older/lib/python3.8/site-packages/datasets/load.py", line 856, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/izaskr/anaconda3/envs/local_vae_older/lib/python3.8/site-packages/datasets/builder.py", line 583, in download_and_prepare
    self._download_and_prepare(
  File "/home/izaskr/anaconda3/envs/local_vae_older/lib/python3.8/site-packages/datasets/builder.py", line 671, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/home/izaskr/anaconda3/envs/local_vae_older/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=610252351, num_examples=532812, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='train', num_bytes=614706451, num_examples=535568, dataset_name='blog_authorship_corpus')}, {'expected': SplitInfo(name='validation', num_bytes=37500394, num_examples=31277, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='validation', num_bytes=32553710, num_examples=28521, dataset_name='blog_authorship_corpus')}]


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Linux-4.15.0-132-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyArrow version: 4.0.1



Comments: Hi @izaskr, thanks for reporting.

However the traceback you joined does not correspond to the codec error message: it is about other error `NonMatchingSplitsSizesError`. Maybe you missed some important part of your traceback...

I'm going to have a look at the dataset anyway...

Hi @izaskr, thanks again for having reported this issue.

After investigation, I have created a Pull Request (#2685) to fix several issues with this dataset:
- the `NonMatchingSplitsSizesError`
- the `UnicodeDecodeError`

Once the Pull Request merged into master, you will be able to load this dataset if you install `datasets` from our GitHub repository master branch. Otherwise, you will be able to use it after our next release, by updating `datasets`: `pip install -U datasets`.

@albertvillanova 
Can you shed light on how this fix works?

We're experiencing a similar issue. 

If we run several runs (eg in a Wandb sweep) the first run "works" but then we get `NonMatchingSplitsSizesError`

| run num | actual train examples # | expected example # | recorded example # |
| ------- | -------------- | ----------------- | -------- |
| 1       | 100            | 100               | 100      |
| 2       | 102            | 100               | 102      |
| 3       | 100            | 100               | 202      | 
| 4       | 40             | 100               | 40       |
| 5       | 40             | 100               | 40       |
| 6       | 40             | 100               | 40       | 


The second through the nth all crash with 

```
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=19980970, num_examples=100, dataset_name='cies'), 'recorded': SplitInfo(name='train', num_bytes=40163811, num_examples=202, dataset_name='cies')}]

```