`.shuffle` throwing error `ValueError: Protocol not known: parent`
Opened by prabhakar267 (open)

### Describe the bug

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In [16], line 1
----> 1 train_dataset = train_dataset.shuffle()

File /opt/conda/envs/pytorch/lib/python3.9/site-packages/datasets/arrow_dataset.py:551, in transmit_format.<locals>.wrapper(*args, **kwargs)
    544 self_format = {
    545     "type": self._format_type,
    546     "format_kwargs": self._format_kwargs,
    547     "columns": self._format_columns,
    548     "output_all_columns": self._output_all_columns,
    549 }
    550 # apply actual function
--> 551 out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    552 datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    553 # re-apply format to the output

File /opt/conda/envs/pytorch/lib/python3.9/site-packages/datasets/fingerprint.py:480, in fingerprint_transform.<locals>._fingerprint.<locals>.wrapper(*args, **kwargs)
    476             validate_fingerprint(kwargs[fingerprint_name])
    478 # Call actual function
--> 480 out = func(self, *args, **kwargs)
    482 # Update fingerprint of in-place transforms + update in-place history of transforms
    484 if inplace:  # update after calling func so that the fingerprint doesn't change if the function fails

File /opt/conda/envs/pytorch/lib/python3.9/site-packages/datasets/arrow_dataset.py:3616, in Dataset.shuffle(self, seed, generator, keep_in_memory, load_from_cache_file, indices_cache_file_name, writer_batch_size, new_fingerprint)
   3610         return self._new_dataset_with_indices(
   3611             fingerprint=new_fingerprint, indices_cache_file_name=indices_cache_file_name
   3612         )
   3614 permutation = generator.permutation(len(self))
-> 3616 return self.select(
   3617     indices=permutation,
   3618     keep_in_memory=keep_in_memory,
   3619     indices_cache_file_name=indices_cache_file_name if not keep_in_memory else None,
   3620     writer_batch_size=writer_batch_size,
   3621     new_fingerprint=new_fingerprint,
   3622 )

File /opt/conda/envs/pytorch/lib/python3.9/site-packages/datasets/arrow_dataset.py:551, in transmit_format.<locals>.wrapper(*args, **kwargs)
    544 self_format = {
    545     "type": self._format_type,
    546     "format_kwargs": self._format_kwargs,
    547     "columns": self._format_columns,
    548     "output_all_columns": self._output_all_columns,
    549 }
    550 # apply actual function
--> 551 out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    552 datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    553 # re-apply format to the output

File /opt/conda/envs/pytorch/lib/python3.9/site-packages/datasets/fingerprint.py:480, in fingerprint_transform.<locals>._fingerprint.<locals>.wrapper(*args, **kwargs)
    476             validate_fingerprint(kwargs[fingerprint_name])
    478 # Call actual function
--> 480 out = func(self, *args, **kwargs)
    482 # Update fingerprint of in-place transforms + update in-place history of transforms
    484 if inplace:  # update after calling func so that the fingerprint doesn't change if the function fails

File /opt/conda/envs/pytorch/lib/python3.9/site-packages/datasets/arrow_dataset.py:3266, in Dataset.select(self, indices, keep_in_memory, indices_cache_file_name, writer_batch_size, new_fingerprint)
   3263             return self._select_contiguous(start, length, new_fingerprint=new_fingerprint)
   3265 # If not contiguous, we need to create a new indices mapping
-> 3266 return self._select_with_indices_mapping(
   3267     indices,
   3268     keep_in_memory=keep_in_memory,
   3269     indices_cache_file_name=indices_cache_file_name,
   3270     writer_batch_size=writer_batch_size,
   3271     new_fingerprint=new_fingerprint,
   3272 )

File /opt/conda/envs/pytorch/lib/python3.9/site-packages/datasets/arrow_dataset.py:551, in transmit_format.<locals>.wrapper(*args, **kwargs)
    544 self_format = {
    545     "type": self._format_type,
    546     "format_kwargs": self._format_kwargs,
    547     "columns": self._format_columns,
    548     "output_all_columns": self._output_all_columns,
    549 }
    550 # apply actual function
--> 551 out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    552 datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    553 # re-apply format to the output

File /opt/conda/envs/pytorch/lib/python3.9/site-packages/datasets/fingerprint.py:480, in fingerprint_transform.<locals>._fingerprint.<locals>.wrapper(*args, **kwargs)
    476             validate_fingerprint(kwargs[fingerprint_name])
    478 # Call actual function
--> 480 out = func(self, *args, **kwargs)
    482 # Update fingerprint of in-place transforms + update in-place history of transforms
    484 if inplace:  # update after calling func so that the fingerprint doesn't change if the function fails

File /opt/conda/envs/pytorch/lib/python3.9/site-packages/datasets/arrow_dataset.py:3389, in Dataset._select_with_indices_mapping(self, indices, keep_in_memory, indices_cache_file_name, writer_batch_size, new_fingerprint)
   3387     logger.info(f"Caching indices mapping at {indices_cache_file_name}")
   3388     tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(indices_cache_file_name), delete=False)
-> 3389     writer = ArrowWriter(
   3390         path=tmp_file.name, writer_batch_size=writer_batch_size, fingerprint=new_fingerprint, unit="indices"
   3391     )
   3393 indices = indices if isinstance(indices, list) else list(indices)
   3395 size = len(self)

File /opt/conda/envs/pytorch/lib/python3.9/site-packages/datasets/arrow_writer.py:315, in ArrowWriter.__init__(self, schema, features, path, stream, fingerprint, writer_batch_size, hash_salt, check_duplicates, disable_nullable, update_features, with_metadata, unit, embed_local_files, storage_options)
    312 self._disable_nullable = disable_nullable
    314 if stream is None:
--> 315     fs_token_paths = fsspec.get_fs_token_paths(path, storage_options=storage_options)
    316     self._fs: fsspec.AbstractFileSystem = fs_token_paths[0]
    317     self._path = (
    318         fs_token_paths[2][0]
    319         if not is_remote_filesystem(self._fs)
    320         else self._fs.unstrip_protocol(fs_token_paths[2][0])
    321     )

File /opt/conda/envs/pytorch/lib/python3.9/site-packages/fsspec/core.py:593, in get_fs_token_paths(urlpath, mode, num, name_function, storage_options, protocol, expand)
    591 else:
    592     urlpath = stringify_path(urlpath)
--> 593 chain = _un_chain(urlpath, storage_options or {})
    594 if len(chain) > 1:
    595     inkwargs = {}

File /opt/conda/envs/pytorch/lib/python3.9/site-packages/fsspec/core.py:330, in _un_chain(path, kwargs)
    328 for bit in reversed(bits):
    329     protocol = split_protocol(bit)[0] or "file"
--> 330     cls = get_filesystem_class(protocol)
    331     extra_kwargs = cls._get_kwargs_from_urls(bit)
    332     kws = kwargs.get(protocol, {})

File /opt/conda/envs/pytorch/lib/python3.9/site-packages/fsspec/registry.py:240, in get_filesystem_class(protocol)
    238 if protocol not in registry:
    239     if protocol not in known_implementations:
--> 240         raise ValueError("Protocol not known: %s" % protocol)
    241     bit = known_implementations[protocol]
    242     try:

ValueError: Protocol not known: parent

```

This is what the `train_dataset` object looks like
```
Dataset({
    features: ['label', 'input_ids', 'attention_mask'],
    num_rows: 364166
})
```

### Steps to reproduce the bug

The `train_dataset` obj is created by concatenating two datasets
And then shuffle is called, but it throws the mentioned error.

### Expected behavior

Should shuffle the dataset properly.

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-1022-aws-x86_64-with-glibc2.31
- Python version: 3.9.13
- PyArrow version: 10.0.0
- Pandas version: 1.4.4


Comments:
Hi ! The indices mapping is written in the same cachedirectory as your dataset.

Can you run this to show your current cache directory ?
```python
print(train_dataset.cache_files)
```

```
[{'filename': '.../train/dataset.arrow'}, {'filename': '.../train/dataset.arrow'}]
```

These are the actual paths where `.hf` files are stored. 

I'm not aware of any `.hf` file ? What are you referring to ?

Also the error says "Protocol unknown: parent". Is there a chance you may have ended up with a path that contains this string `parent://` ?

I figured out why the issue was occuring but don't know the long-term fix.
The dataset I was trying to shuffle was loaded from a saved file which had `::` delimiter in filename. When I try with the exact same file without `::` in filename, it works as expected.
Quick fix is to not use colons in filename. But if this is expected behaviour, this should be clearly stated in the documentation.
Thanks for help @lhoestq