Some question about raw dataset download info in the project .
State: closed | Author: svjack (NONE)

Hi , i review the code in 
https://github.com/huggingface/datasets/blob/master/datasets/conll2003/conll2003.py
in the _split_generators function is the truly logic of download raw datasets with dl_manager
and use Conll2003 cls by use import_main_class in load_dataset function
My question is that , with this logic it seems that i can not have the raw dataset download location
in variable in downloaded_files in _split_generators.
If someone also want use huggingface datasets as raw dataset downloader,
how can he retrieve the raw dataset download path from attributes in 
datasets.dataset_dict.DatasetDict ?

Comments: Hi ! The `dl_manager` is a `DownloadManager` object and is responsible for downloading the raw data files.
It is used by dataset builders in their `_split_generators` method to download the raw data files that are necessary to build the datasets splits.

The `Conll2003` class is a dataset builder, and so you can download all the raw data files by calling `_split_generators` with a download manager:
```python
from datasets import DownloadManager
from datasets.load import import_main_class

conll2003_builder = import_main_class(...)

dl_manager = DownloadManager()
splis_generators = conll2003_builder._split_generators(dl_manager)
```

Then you can see what files have been downloaded with
```python
dl_manager.get_recorded_sizes_checksums()
```
It returns a dictionary with the format {url: {num_bytes: int, checksum: str}}

Then you can get the actual location of the downloaded files with
```python
from datasets import cached_path

local_path_to_downloaded_file = cached_path(url)
```

------------------

Note that you can also get the urls from the Dataset object:
```python
from datasets import load_dataset

conll2003 = load_dataset("conll2003")
print(conll2003["train"].download_checksums)
```
It returns the same dictionary with the format {url: {num_bytes: int, checksum: str}}

I am afraid that there is not a very straightforward way to get that location.

Another option, from _split_generators would be to use:
- `dl_manager._download_config.cache_dir` to get the directory where all the raw downloaded files are:
  ```python
  download_dir = dl_manager._download_config.cache_dir
  ```
- the function `datasets.utils.file_utils.hash_url_to_filename` to get the filenames of the raw downloaded files:
  ```python
  filenames = [hash_url_to_filename(url) for url in urls_to_download.values()]
  ```
Therefore the complete path to the raw downloaded files would be the join of both:
```python
downloaded_paths = [os.path.join(download_dir, filename) for filename in filenames]
```

Maybe it would be interesting to make these paths accessible more easily. I could work on this. What do you think, @lhoestq ?

Sure it would be nice to have an easier access to these paths !
The dataset builder could have a method to return those, what do you think ?
Feel free to work on this @albertvillanova , it would be a nice addition :) 

Your suggestion does work as well @albertvillanova if you complete it by specifying `etag=` to `hash_url_to_filename`.

The ETag is obtained by a HEAD request and is used to know if the file on the remote host has changed. Therefore if a file is updated on the remote host, then the hash returned by `hash_url_to_filename` is different.

Once #1846 will be merged, the paths to the raw downloaded files will be accessible as:
```python
builder_instance.dl_manager.downloaded_paths
```