Update SciFact URL
State: closed | Author: dwadden (CONTRIBUTOR)

Hi,

I'm following up this [issue](https://github.com/huggingface/datasets/issues/1717). I'm the SciFact dataset creator, and I'm trying to update the SciFact data url in your repo. Thanks again for adding the dataset!

Basically, I'd just like to change the `_URL` to `"https://scifact.s3-us-west-2.amazonaws.com/release/latest/data.tar.gz"`. I changed `scifact.py` appropriately and tried running

```
python datasets-cli test datasets/scifact --save_infos --all_configs
```
which I was hoping would update the `dataset_infos.json` for SciFact. But for some reason the code still seems to be looking for the old version of the dataset. Full stack trace below. I've tried to clear all my Huggingface-related caches, and I've `git grep`'d to make sure that the old path to the dataset isn't floating around somewhere. So I'm not sure why this is happening?

Can you help me switch the download URL?

```
(datasets) $ python datasets-cli test datasets/scifact --save_infos --all_configs
Checking datasets/scifact/scifact.py for additional imports.
Found main folder for dataset datasets/scifact/scifact.py at /Users/dwadden/.cache/huggingface/modules/datasets_modules/datasets/scifact
Found specific version folder for dataset datasets/scifact/scifact.py at /Users/dwadden/.cache/huggingface/modules/datasets_modules/datasets/scifact/2b43b4e125ce3369da7d6353961d9d315e6593f24cc7bbe9ede5e5c911d11534
Found script file from datasets/scifact/scifact.py to /Users/dwadden/.cache/huggingface/modules/datasets_modules/datasets/scifact/2b43b4e125ce3369da7d6353961d9d315e6593f24cc7bbe9ede5e5c911d11534/scifact.py
Found dataset infos file from datasets/scifact/dataset_infos.json to /Users/dwadden/.cache/huggingface/modules/datasets_modules/datasets/scifact/2b43b4e125ce3369da7d6353961d9d315e6593f24cc7bbe9ede5e5c911d11534/dataset_infos.json
Found metadata file for dataset datasets/scifact/scifact.py at /Users/dwadden/.cache/huggingface/modules/datasets_modules/datasets/scifact/2b43b4e125ce3369da7d6353961d9d315e6593f24cc7bbe9ede5e5c911d11534/scifact.json
Loading Dataset Infos from /Users/dwadden/.cache/huggingface/modules/datasets_modules/datasets/scifact/2b43b4e125ce3369da7d6353961d9d315e6593f24cc7bbe9ede5e5c911d11534
Testing builder 'corpus' (1/2)
Generating dataset scifact (/Users/dwadden/.cache/huggingface/datasets/scifact/corpus/1.0.0/2b43b4e125ce3369da7d6353961d9d315e6593f24cc7bbe9ede5e5c911d11534)
Downloading and preparing dataset scifact/corpus (download: 2.72 MiB, generated: 7.63 MiB, post-processed: Unknown size, total: 10.35 MiB) to /Users/dwadden/.cache/huggingface/datasets/scifact/corpus/1.0.0/2b43b4e125ce3369da7d6353961d9d315e6593f24cc7bbe9ede5e5c911d11534...
Downloading took 0.0 min
Checksum Computation took 0.0 min
Traceback (most recent call last):
  File "/Users/dwadden/proj/datasets/datasets-cli", line 36, in <module>
    service.run()
  File "/Users/dwadden/proj/datasets/src/datasets/commands/test.py", line 139, in run
    builder.download_and_prepare(
  File "/Users/dwadden/proj/datasets/src/datasets/builder.py", line 562, in download_and_prepare
    self._download_and_prepare(
  File "/Users/dwadden/proj/datasets/src/datasets/builder.py", line 622, in _download_and_prepare
    verify_checksums(
  File "/Users/dwadden/proj/datasets/src/datasets/utils/info_utils.py", line 32, in verify_checksums
    raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
datasets.utils.info_utils.ExpectedMoreDownloadedFiles: {'https://ai2-s2-scifact.s3-us-west-2.amazonaws.com/release/2020-05-01/data.tar.gz'}
```

Comments: Hi ! The error you get is the result of some verifications the library is doing when loading a dataset that already has some metadata in the dataset_infos.json. You can ignore the verifications with 
```
python datasets-cli test datasets/scifact --save_infos --all_configs --ignore_verifications
```
This will update the dataset_infos.json :)

Nice, I ran that command and `dataset_infos` seems to have been updated appropriately; I added this to the PR. But when I try to load the dataset it still seems like it's getting a path to the old URL somehow. I `pip install -e`'d my fork of the repo, so I'm not sure why `load_dataset` is still looking for the old version of the file. Stack trace below.

```
In [1]: import datasets

In [2]: ds = datasets.load_dataset("scifact", "claims")
Downloading: 7.34kB [00:00, 2.58MB/s]
Downloading: 3.38kB [00:00, 1.36MB/s]
Downloading and preparing dataset scifact/claims (download: 2.72 MiB, generated: 258.64 KiB, post-processed: Unknown size, total: 2.97 MiB) to /Users/dwadden/.cache/huggingface/datasets/scifact/claims/1.0.0/2bb675b2003716a061a4d8ce27fab32ab7f6d010016bab08ffaccea3c14ec6e7...
---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
<ipython-input-2-9a50b954d89a> in <module>
----> 1 ds = datasets.load_dataset("scifact", "claims")

~/proj/datasets/src/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, script_version, **config_kwargs)
    672
    673     # Download and prepare data
--> 674     builder_instance.download_and_prepare(
    675         download_config=download_config,
    676         download_mode=download_mode,

~/proj/datasets/src/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    560                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    561                     if not downloaded_from_gcs:
--> 562                         self._download_and_prepare(
    563                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    564                         )

~/proj/datasets/src/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    616         split_dict = SplitDict(dataset_name=self.name)
    617         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 618         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    619
    620         # Checksums verification

~/.cache/huggingface/modules/datasets_modules/datasets/scifact/2bb675b2003716a061a4d8ce27fab32ab7f6d010016bab08ffaccea3c14ec6e7/scifact.py in _split_generators(self, dl_manager)
     92         # dl_manager is a datasets.download.DownloadManager that can be used to
     93         # download and extract URLs
---> 94         dl_dir = dl_manager.download_and_extract(_URL)
     95
     96         if self.config.name == "corpus":

~/proj/datasets/src/datasets/utils/download_manager.py in download_and_extract(self, url_or_urls)
    256             extracted_path(s): `str`, extracted paths of given URL(s).
    257         """
--> 258         return self.extract(self.download(url_or_urls))
    259
    260     def get_recorded_sizes_checksums(self):

~/proj/datasets/src/datasets/utils/download_manager.py in download(self, url_or_urls)
    177
    178         start_time = datetime.now()
--> 179         downloaded_path_or_paths = map_nested(
    180             download_func,
    181             url_or_urls,

~/proj/datasets/src/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types)
    223     # Singleton
    224     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 225         return function(data_struct)
    226
    227     disable_tqdm = bool(logger.getEffectiveLevel() > INFO)

~/proj/datasets/src/datasets/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    348     if is_remote_url(url_or_filename):
    349         # URL, so get it from the cache (downloading if necessary)
--> 350         output_path = get_from_cache(
    351             url_or_filename,
    352             cache_dir=cache_dir,

~/proj/datasets/src/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries)
    631         elif response is not None and response.status_code == 404:
    632             raise FileNotFoundError("Couldn't find file at {}".format(url))
--> 633         raise ConnectionError("Couldn't reach {}".format(url))
    634
    635     # Try a second time

ConnectionError: Couldn't reach https://ai2-s2-scifact.s3-us-west-2.amazonaws.com/release/2020-05-01/data.tar.gz
```

Hi ! This may be because you need to point `load_dataset` to the path of the dataset script that has the updated url:
```python
load_dataset("./datasets/scifact", "claims")
```

If you don't use a path to the updated script, then the old one is used by deffault

Nice, I did
```
load_dataset("./datasets/scifact", "claims")
```
and it worked. 

One more question about the way the code is being preprocessed. The way I've formatted the data, each entry is a claim, which may be associated with multiple evidence documents (similar to FEVER):
```
# My way
{'id': 70,
 'claim': 'Activation of PPM1D suppresses p53 function.',
 'evidence': {'5956380': [{'sentences': [5, 6], 'label': 'SUPPORT'}],
              '4414547': [{'sentences': [5], 'label': 'SUPPORT'}]},
 'cited_doc_ids': [5956380, 4414547]}
```

In the Hugginface data, each entry is a single claim / evidence document pair. So, the above entry is converted into two separate entries, like so:
```
# huggingface
[{'cited_doc_ids': [5956380, 4414547],
  'claim': 'Activation of PPM1D suppresses p53 function.',
  'evidence_doc_id': '5956380',
  'evidence_label': 'SUPPORT',
  'evidence_sentences': [5, 6],
  'id': 70},
 {'cited_doc_ids': [5956380, 4414547],
  'claim': 'Activation of PPM1D suppresses p53 function.',
  'evidence_doc_id': '4414547',
  'evidence_label': 'SUPPORT',
  'evidence_sentences': [5],
  'id': 70}]
```

Was this done by design? If not, would you mind if I modify the Huggingface code so that it more closely matches the format that people will get if they download the data from the SciFact repo?

Yes if you think the format is not convenient for training or evaluation we can change it.
Also I think we're doing something similar for FEVER: one example = one (claim, sentence) pair.

Let's merge this PR first and then feel free to open a new PR to change the format :) 

Thanks for merging!

I don't have super-strong feelings one way or the other in terms of the data, I think it's probably fine. I may revisit later.