Incorrect Russian filenames encoding after extraction by datasets.DownloadManager.download_and_extract()
Opened by iluvvatar (closed)

## Dataset viewer issue for 'MalakhovIlya/RuREBus'

**Link:** https://huggingface.co/datasets/MalakhovIlya/RuREBus

**Description**
Using os.walk(topdown=False) in DatasetBuilder causes following error:
Status code:   400
Exception:     TypeError
Message:       xwalk() got an unexpected keyword argument 'topdown'
Couldn't find where "xwalk" come from. How can I fix this?

Am I the one who added this dataset ? Yes


Comments:
To reproduce:

```python
>>> import datasets
>>> datasets.get_dataset_split_names('MalakhovIlya/RuREBus', config_name='raw_txt')
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 280, in get_dataset_config_info
    for split_generator in builder._split_generators(
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/MalakhovIlya--RuREBus/21046f5f1a0cf91187d68c30918d78d934ec7113ec435e146776d4f28f12c4ed/RuREBus.py", line 101, in _split_generators
    decode_file_names(folder)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/MalakhovIlya--RuREBus/21046f5f1a0cf91187d68c30918d78d934ec7113ec435e146776d4f28f12c4ed/RuREBus.py", line 26, in decode_file_names
    for root, dirs, files in os.walk(folder, topdown=False):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/streaming.py", line 66, in wrapper
    return function(*args, use_auth_token=use_auth_token, **kwargs)
TypeError: xwalk() got an unexpected keyword argument 'topdown'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 323, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 285, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

It's not related to the dataset viewer. Maybe @albertvillanova or @lhoestq could help more on this issue.

Hi! This issue stems from the fact that `xwalk`, which is a streamable version of `os.walk`, doesn't support the `topdown` param due to `fsspec`'s `walk` also not supporting it, so fixing this issue could be tricky.  

@MalakhovIlyaPavlovich You can avoid the error by tweaking your data processing and not using this param. (and `Path.rename`, which also cannot be streamed) 

@mariosasko thank you for your reply. I couldn't reproduce error showed by @severo either on Ubuntu 20.04.3 LTS, Windows 10 and Google Colab environments. But trying to avoid using os.walk(topdown=False) and Path.rename(), In _split_generators I replaced
```
def decode_file_names(folder):
    for root, dirs, files in os.walk(folder, topdown=False):
        root = Path(root)
        for file in files:
            old_name = root / Path(file)
            new_name = root / Path(
                file.encode('cp437').decode('cp866'))
            old_name.rename(new_name)
        for dir in dirs:
            old_name = root / Path(dir)
            new_name = root / Path(dir.encode('cp437').decode('cp866'))
            old_name.rename(new_name)

folder = dl_manager.download_and_extract(self._RAW_TXT_URLS)['raw_txt']
decode_file_names(folder)
```
by
```
def extract(zip_file_path):
    p = Path(zip_file_path)
    dest_dir = str(p.parent / 'extracted' / p.stem)
    os.makedirs(dest_dir, exist_ok=True)
    with zipfile.ZipFile(zip_file_path) as archive:
        for file_info in tqdm(archive.infolist(), desc='Extracting'):
            filename = file_info.filename.encode('cp437').decode('cp866')
            target = os.path.join(dest_dir, *filename.split('/'))
            os.makedirs(os.path.dirname(target), exist_ok=True)
            if not file_info.is_dir():
                with archive.open(file_info) as source, open(target, 'wb') as dest:
                    shutil.copyfileobj(source, dest)
    return dest_dir

zip_file = dl_manager.download(self._RAW_TXT_URLS)['raw_txt']
if not is_url(zip_file):
    folder = extract(zip_file)
else:
    folder = None
```
and now everything works well except data viewer for "raw_txt" subset: dataset preview on hub shows "No data.". As far as I understand dl_manager.download returns original URL when we are calling datasets.get_dataset_split_names and my suspicions are that dataset viewer can do smth similar. I couldn't find information about how it works. I would be very grateful, if you could tell me how to fix this)

This is what I get when I try to stream the `raw_txt` subset:
```python
>>> dset = load_dataset("MalakhovIlya/RuREBus", "raw_txt", split="raw_txt", streaming=True)
>>> next(iter(dset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
```
So there is a bug in your script.

streaming=True helped me to find solution. I fixed
```
def extract(zip_file_path):
    p = Path(zip_file_path)
    dest_dir = str(p.parent / 'extracted' / p.stem)
    os.makedirs(dest_dir, exist_ok=True)
    with zipfile.ZipFile(zip_file_path) as archive:
        for file_info in tqdm(archive.infolist(), desc='Extracting'):
            filename = file_info.filename.encode('cp437').decode('cp866')
            target = os.path.join(dest_dir, *filename.split('/'))
            os.makedirs(os.path.dirname(target), exist_ok=True)
            if not file_info.is_dir():
                with archive.open(file_info) as source, open(target, 'wb') as dest:
                    shutil.copyfileobj(source, dest)
    return dest_dir

zip_file = dl_manager.download(self._RAW_TXT_URLS)['raw_txt']
folder = extract(zip_file)
```
by 
```
folder = dl_manager.download_and_extract(self._RAW_TXT_URLS)['raw_txt']
path = os.path.join(folder, 'MED_txt/unparsed_txt')
for root, dirs, files in os.walk(path):
    decoded_root_name = Path(root).name.encode('cp437').decode('cp866')
```
@mariosasko thank you for your help :)