DuplicatedKeysError on personal dataset
State: closed | Author: lucaguarro (NONE)
Labels: name: bug

## Describe the bug
Ever since today, I have been getting a DuplicatedKeysError while trying to load my dataset from my own script.
Error returned when running this line: `dataset = load_dataset('/content/drive/MyDrive/Thesis/Datasets/book_preprocessing/goodreads_maharjan_trimmed_and_nered/goodreadsnered.py')`
Note that my script was working fine with earlier versions of the Datasets library. Cannot say with 100% certainty if I have been doing something wrong with my dataset script this whole time or if this is simply a bug with the new version of datasets.

## Steps to reproduce the bug
I cannot provide code to reproduce the error as I am working with my own dataset. I can however provide my script if requested.

## Expected results
For my data to be loaded.

## Actual results
**DuplicatedKeysError** exception is raised
```
Downloading and preparing dataset good_reads_practice_dataset/main_domain (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/good_reads_practice_dataset/main_domain/1.1.0/64ff7c3fee2693afdddea75002eb6887d4fedc3d812ae3622128c8504ab21655...

---------------------------------------------------------------------------

DuplicatedKeysError                       Traceback (most recent call last)

<ipython-input-6-c342ea0dae9d> in <module>()
----> 1 dataset = load_dataset('/content/drive/MyDrive/Thesis/Datasets/book_preprocessing/goodreads_maharjan_trimmed_and_nered/goodreadsnered.py')

5 frames

/usr/local/lib/python3.7/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, **config_kwargs)
    749         try_from_hf_gcs=try_from_hf_gcs,
    750         base_path=base_path,
--> 751         use_auth_token=use_auth_token,
    752     )
    753 

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    573                     if not downloaded_from_gcs:
    574                         self._download_and_prepare(
--> 575                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    576                         )
    577                     # Sync info

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    650             try:
    651                 # Prepare split will record examples associated to the split
--> 652                 self._prepare_split(split_generator, **prepare_split_kwargs)
    653             except OSError as e:
    654                 raise OSError(

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _prepare_split(self, split_generator)
    990                     writer.write(example, key)
    991             finally:
--> 992                 num_examples, num_bytes = writer.finalize()
    993 
    994         split_generator.split_info.num_examples = num_examples

/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in finalize(self, close_stream)
    407         # In case current_examples < writer_batch_size, but user uses finalize()
    408         if self._check_duplicates:
--> 409             self.check_duplicate_keys()
    410             # Re-intializing to empty list for next batch
    411             self.hkey_record = []

/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in check_duplicate_keys(self)
    347         for hash, key in self.hkey_record:
    348             if hash in tmp_record:
--> 349                 raise DuplicatedKeysError(key)
    350             else:
    351                 tmp_record.add(hash)

DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 0
Keys should be unique and deterministic in nature
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.7.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.9
- PyArrow version: 3.0.0


Comments: Hi ! In your dataset script you must be yielding examples like
```python
for line in file:
    ...
    yield key, {...}
```

Since `datasets` 1.7.0 we enforce the keys to be unique.
However it looks like your examples generator creates duplicate keys: at least two examples have key 0.

You can fix that by making sure that your keys are unique.

For example if you use a counter to define the key of each example, make sure that your counter is not reset to 0 in during examples generation (between two open files for examples).

Let me know if you have other questions :)

Yup, I indeed was generating duplicate keys. Fixed it and now it's working.