SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Dataset Viewer issue for fgrezes/WIESP2022-NER

Comment:
https://huggingface.co/datasets/fgrezes/WIESP2022-NER

The error:

```
Message:       Couldn't find a dataset script at /src/services/worker/fgrezes/WIESP2022-NER/WIESP2022-NER.py or any data file in the same directory. Couldn't find 'fgrezes/WIESP2022-NER' on the Hugging Face Hub either: FileNotFoundError: Unable to resolve any data file that matches ['**test*', '**eval*'] in dataset repository fgrezes/WIESP2022-NER with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']
```

I understand the issue is not related to the dataset viewer in itself, but with the autodetection of the data files without a loading script in the datasets library. cc @lhoestq @albertvillanova @mariosasko
Issue: Dataset Viewer issue for fgrezes/WIESP2022-NER

Comment:
Apparently it finds `scoring-scripts/compute_seqeval.py` which matches `**eval*`, a regex that detects a test split. We should probably improve the regex because it's not supposed to catch this kind of files. It must also only check for files with supported extensions: txt, csv, png etc.
Issue: `to_pandas` doesn't take into account format.

Comment:
Thanks for opening a discussion :)

Note that you can use `.remove_columns(...)` to keep only the ones you're interested in before calling `.to_pandas()`
Issue: `to_pandas` doesn't take into account format.

Comment:
Yes I can do that thank you!

Do you think that conceptually my example should work? If not, I'm happy to close this issue. 

If yes, I can start working on it.
Issue: `to_pandas` doesn't take into account format.

Comment:
Hi! Instead of `with_format(columns=['a', 'b']).to_pandas()`, use `with_format("pandas", columns=["a", "b"])` for easy conversion of the parts of the dataset to pandas via indexing/slicing.

The full code:
```python
from datasets import Dataset

ds = Dataset.from_dict({'a': [1,2,3], 'b': [5,6,7], 'c': [8,9,10]})
pandas_df = ds.with_format("pandas", columns=['a', 'b'])[:]
```
Issue: Transcript string 'null' converted to [None] by load_dataset()

Comment:
Hi @mbarnig, thanks for reporting.

Please note that is an expected behavior by `pandas` (we use the `pandas` library to parse CSV files): https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html
```
By default the following values are interpreted as NaN: 
‘’, ‘#N/A’, ‘#N/A N/A’, ‘#NA’, ‘-1.#IND’, ‘-1.#QNAN’, ‘-NaN’, ‘-nan’, ‘1.#IND’, ‘1.#QNAN’, ‘<NA>’, ‘N/A’, ‘NA’, ‘NULL’, ‘NaN’, ‘n/a’, ‘nan’, ‘null’.
```
(see "null" in the last position in the above list).

In order to prevent `pandas` from performing that automatic conversion from the string "null" to a NaN value, you should pass the `pandas` parameter `keep_default_na=False`:
```python
In [2]: dataset = load_dataset('csv', data_files={'train': 'null-test.csv'}, keep_default_na=False)
In [3]: dataset["train"][0]["transcript"]
Out[3]: 'null'
```
Issue: Transcript string 'null' converted to [None] by load_dataset()

Comment:
@albertvillanova I also ran into this issue, it had me scratching my head for a while! In my case it was tripped by a literal "NA" comment collected from a user-facing form (e.g., this question does not apply to me). Thankfully this answer was here, but I feel it is such a common trap that it deserves to be noted in the official docs, maybe [here](https://huggingface.co/docs/datasets/loading#csv)? 

I'm happy to submit a PR if you agree!
Issue: BigBench: NonMatchingSplitsSizesError when passing a dataset configuration parameter

Comment:
Yup it can also work, and maybe it's simpler this way. Opening a PR to fix bigbench instead of https://github.com/huggingface/datasets/pull/4463
Issue: BigBench: NonMatchingSplitsSizesError when passing a dataset configuration parameter

Comment:
Hi @lhoestq,

Thank you for taking a look at this issue, and proposing a solution. 
Unfortunately, after trying the fix in #4465 I still see the same issue.

I think there is some subtlety where the config name gets overwritten somewhere when `BUILDER_CONFIGS`[(link)](https://github.com/huggingface/datasets/blob/master/datasets/bigbench/bigbench.py#L126) is defined. 

If I print out the `self.config.name` in the current version (with the fix in #4465), I see just the task name, but if I comment out `BUILDER_CONFIGS`, the `num_shots` and `max_examples` gets appended as was meant by #4465.

I haven't managed to track down where this happens, but I thought you might know? 

(Another comment on your fix: the `name` variable is used to fetch the task from the bigbench API, so modifying it causes an error if it's actually called. This can easily be fixed by having `config_name` variable in addition to the `task_name`)
Issue: Workflow for Tabular data

Comment:
I use below to load a dataset:
```
dataset = datasets.load_dataset("scikit-learn/auto-mpg")
df = pd.DataFrame(dataset["train"])
```
TBH as said, tabular folk split their own dataset, they sometimes have two splits, sometimes three. Maybe somehow avoiding it for tabular datasets might be good for later. (it's just UX improvement)
Issue: Workflow for Tabular data

Comment:
is very slow batch access of a dataset (tabular, csv) with many columns to be expected?
Issue: Workflow for Tabular data

Comment:
~20k! I was surprised batch loading with as few as 32 samples was really slow. I was speculating the columnar format was the cause -- or do you see good performance with this approx size of tabular data?
Issue: Workflow for Tabular data

Comment:
20k can be a lot for a columnar format but maybe we can optimize a few things.

It would be cool to profile the code to see if there's an unoptimized part of the code that slows everything down.

(it's also possible to kill the job when it accesses the batch, it often gives you the traceback at the location where the code was running)
Issue: Workflow for Tabular data

Comment:
@wconnell I'm not sure what you mean by my secret, I load them into a numpy array 😁 

An example dataset is [here](https://portal.gdc.cancer.gov/repository?facetTab=files&filters=%7B%22content%22%3A%5B%7B%22content%22%3A%7B%22field%22%3A%22cases.project.project_id%22%2C%22value%22%3A%5B%22TCGA-CESC%22%5D%7D%2C%22op%22%3A%22in%22%7D%2C%7B%22content%22%3A%7B%22field%22%3A%22files.data_category%22%2C%22value%22%3A%5B%22DNA%20Methylation%22%5D%7D%2C%22op%22%3A%22in%22%7D%5D%2C%22op%22%3A%22and%22%7D&searchTableTab=files) which is a dataset of DNA methylation reads. This dataset is about 950 rows and 450k columns.
Issue: Dataset Viewer issue for Yaxin/SemEval2015

Comment:
I understand that the issue is that a remote file (URL) is being loaded as a local file. Right @albertvillanova @lhoestq?

```
Message:       [Errno 2] No such file or directory: 'https://raw.githubusercontent.com/YaxinCui/ABSADataset/main/SemEval2015Task12Corrected/train/restaurants_train.xml'
```
Issue: Dataset Viewer issue for Yaxin/SemEval2015

Comment:
`xml.dom.minidom.parse` is not supported in streaming mode. I opened a PR here to fix it:
https://huggingface.co/datasets/Yaxin/SemEval2015/discussions/1

Please review the PR @WithYouTo and let me know if it works !
Issue: Trying to load FEVER dataset results in NonMatchingChecksumError

Comment:
Thanks for reporting @santhnm2. We are fixing it.

Data owners updated their URLs recently. We have to align with them, otherwise you do not download anything (that is why ignore_verifications does not work).
Issue: Trying to load FEVER dataset results in NonMatchingChecksumError

Comment:
Hello! Is there any update on this? I am having the same issue 6 months later.
Issue: New Preprocessing Feature  - Deduplication [Request]

Comment:
Hi! The [datasets_sql](https://github.com/mariosasko/datasets_sql) package lets you easily find distinct rows in a dataset (an example with `SELECT DISTINCT` is in the readme). Deduplication is (still) not part of the official API because it's hard to implement for datasets bigger than RAM while only using the native PyArrow ops.

(Btw, this is a duplicate of https://github.com/huggingface/datasets/issues/2514)
Issue: New Preprocessing Feature  - Deduplication [Request]

Comment:
Here is an example using the [datasets_sql](https://github.com/mariosasko/datasets_sql) mentioned 

```python 
from datasets_sql import query

dataset = load_dataset("imdb", split="train")

# If you dont have an id column just add one by enumerating
dataset=dataset.map(lambda x,i: {"id":i}, with_indices=True)

id_column='id'
unique_column='text'

# always selects min id
unique_dataset = query(f"SELECT dataset.* FROM dataset JOIN (SELECT MIN({id_column}) as unique_id FROM dataset group by {unique_column}) ON unique_id=dataset.{id_column}")
```
Not ideal for large datasets but good enough for basic cases.
Sure would be nice to have in the library 🤗
Issue: Dataset Viewer issue for openclimatefix/nimrod-uk-1km

Comment:
If I understand correctly, this is due to the key `split` missing in the line https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km/blob/main/nimrod-uk-1km.py#L41 of the script.
Maybe @albertvillanova could confirm.
Issue: Dataset Viewer issue for openclimatefix/nimrod-uk-1km

Comment:
Indeed there are several issues in this dataset loading script.

The one pointed out by @severo: for the default configuration "crops": https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km/blob/main/nimrod-uk-1km.py#L244
- The download manager downloads `_URL`
- But `_URL` is not defined:  https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km/blob/main/nimrod-uk-1km.py#L41
  ```python
  _URL = {'train': []}
  ```
- Afterwards, for each split, a different key in `_ULR` is used, but it only contains one key: "train"
  - "valid" key: https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km/blob/main/nimrod-uk-1km.py#L260
  - "test key: https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km/blob/main/nimrod-uk-1km.py#L269
  
These keys do not exist inside `_URL`, thus the error message reported in the viewer: 
```
Exception:     KeyError
Message:       'valid'
```
Issue: Dataset Viewer issue for openclimatefix/nimrod-uk-1km

Comment:
Would anyone want to submit a Hub PR (or open a Discussion for the authors to be aware) to this dataset? https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km
Issue: Dataset Viewer issue for openclimatefix/nimrod-uk-1km

Comment:
Hi, I'm the main author for that dataset, so I'll work on updating it! I was working on debugging some stuff awhile ago, which is what broke it.
Issue: Dataset Viewer issue for openclimatefix/nimrod-uk-1km

Comment:
I've opened a Discussion page, so that we can ask/answer and propose fixes until the script works properly: https://huggingface.co/datasets/openclimatefix/nimrod-uk-1km/discussions/1

CC: @julien-c @jacobbieker
Issue: TIMIT won't load after manual download: Errors about files that don't exist

Comment:
To have some context, please see:
- #4145

Please, also note that we have recently made some fixes to the script, which are in our GitHub master branch but not yet released:
- #4422
- #4425 
- #4436
Issue: TIMIT won't load after manual download: Errors about files that don't exist

Comment:
Thanks Albert! I'll try pulling `datasets` from the git repo instead of PyPI, and/or just wait for the next release.
Issue: TIMIT won't load after manual download: Errors about files that don't exist

Comment:
I'm closing this issue then. Please, feel free to reopen it again if the problem persists.
Issue: Load a local cached dataset that has been modified

Comment:
Hi! `datasets` caches every modification/loading, so you can either rerun the pipeline up to the `map` call or use `Dataset.from_file(modified_dataset)` to load the dataset directly from the cache file.
Issue: Add ability to load newer, cleaner version of Multi-News 

Comment:
Hi! Our versioning is based on Git revisions (the `revision` param in `load_dataset`), so you can just replace the old URL with the new one and open a PR :). I can also give you some pointers if needed.
Issue: Add ability to load newer, cleaner version of Multi-News 

Comment:
@mariosasko Awesome thanks! I will do that. Looks like this new version of the data is not available as a zip but as three files (train/dev/test). How is this usually handled in HF Datasets, should `_URL` be a dict with keys `train`, `val`, `test` perhaps?
Issue: Add ability to load newer, cleaner version of Multi-News 

Comment:
Yes! Let me help you with more detailed instructions.

In the first step, we need to update the URLs. One of the possible dictionary structures is as follows:
```python
_URLs = {
    "train": {"src": "https://drive.google.com/uc?export=download&id=1wHAWDOwOoQWSj7HYpyJ3Aeud8WhhaJ7P", "tgt": "https://drive.google.com/uc?export=download&id=1QVgswwhVTkd3VLCzajK6eVkcrSWEK6kq"}
    "val": ...
    "test": ...
}
```

(You can use this page to generate direct download links: https://sites.google.com/site/gdocs2direct/)

Then we move to the `split_generators` method:
```python
def _split_generators(self, dl_manager):
    """Returns SplitGenerators."""
    files = dl_manager.download(_URLs)
    return [
        datasets.SplitGenerator(
            name=datasets.Split.TRAIN,
            gen_kwargs={"src_file": files["train"]["src"], "tgt_file": files["train"]["tgt"]},
        ),
        ... # same for val and test
    ]
```
Finally, we adjust the signature of `_generate_examples`:
```python
def _generate_examples(self, src_file, tgt_file):
     """Yields examples."""
     with open(src_file, encoding="utf-8") as src_f, open(
         tgt_file, encoding="utf-8"
     ) as tgt_f:
         ... # the rest is the same
```

And that's it!

PS: Let me know if you need help updating the dummy data and regenerating the metadata file.
Issue: Add ability to load newer, cleaner version of Multi-News 

Comment:
Awesome! Thanks for the detailed help, that was straightforward with your instruction. However, I think I am being blocked by this issue: https://github.com/huggingface/datasets/issues/4428
Issue: Add loading variable number of columns for different splits

Comment:
Hi! Indeed the column is missing, but you shouldn't get an error? Have you made some modifications (locally) to the loading script? I've opened a PR to add the missing columns to the script.
Issue: Cannot load timit_asr data set

Comment:
@bhaddow we have also made a fix so that you don't have to convert to uppercase the file extensions of the LDC data.

Would you mind checking if it works OK now for you and reporting if there are any issues? Thanks.
Issue: Cannot load timit_asr data set

Comment:
Hi @albertvillanova  -It loads fine on a copy of the data from deepai - although I have to remove the copies of the .WAV files (with extension .WAV,wav). On a copy of the data that was obtained from the LDC, the glob still fails to find the files. The LDC copy looks like it was copied from CD, in 2004, so the structure may be different to a current download.
Issue: Cannot load timit_asr data set

Comment:
Ah, if I change the train/ and test/ directories to TRAIN/ and TEST/  then it works!
Issue: Cannot load timit_asr data set

Comment:
Thanks for your investigation and report, @bhaddow. I'm adding another fix for the TRAIN/train and TEST/test directory names.
Issue: Metric evaluation problems in multi-node, shared file system

Comment:
If you call `metric.compute` in a distributed setup like yours, then `metric.compute` is called in each process. `metric.compute` first calls `metric.add_batch`, and it looks like your error appears at that stage.

To make sure that all the processes have started writing their predictions/references at the same time, each process waits for process 0 to lock `slurm-{world_size}-0.arrow.lock`. Process 0 locks this file when `metric.add_batch` is called, so here when `metric.compute` is called.

Therefore your error can happen when process 0 takes too much time to call `metric.compute` compared to process 3 (>100 seconds by default). I haven't tried running your code but could it be the case ?

I guess it could also happen if you run multiple times the same distributed job at the same time with the same `experiment_id` because they would collide.
Issue: Metric evaluation problems in multi-node, shared file system

Comment:
We've finally been able to isolate the problem, it wasn't a timing problem, but rather a file locking one. 
The locks produced by calling `flock` where not visible between nodes (so the master node couldn't check other node's locks nor the other way around). 

We are now having issues with the pre-processing in our runner script, but are not related with the rendezvous process during the evaluation phase. We will let you know about it once we address it. 

Our solution to the rendezvous is as follows:
- We solved the problem by calling `lockf` instead of `flock`.
- We had to change slightly the `_check_all_processes_locks` method so that the main process (i.e. process 0) didn't check it's own lock (because `lockf` permits recursive locks and thus checking it only replaced the current lock with a new one). 

We use a shared file system between nodes using GPFS in our cluster setup. Maybe the difference between the behavior we see with respect to your usage in multi-node executions comes from that fact. Which file system scheme do you use for the multi-node executions? 

`lockf` seems to work in more settings than `flock`, so maybe we could write a PR so you could test it in your environment.
Issue: Metric evaluation problems in multi-node, shared file system

Comment:
Cool, I'm glad you managed to make evaluation work :)

I'm not completely aware of the differences between lockf and flock, but I've read somewhere that flock is preferable over lockf in multithreading and multiprocessing situations. Here we definitely are in such a situation so unless it is super important I don't think we will switch to lockf
Issue: Metric evaluation problems in multi-node, shared file system

Comment:
> * We had to change slightly the `_check_all_processes_locks` method so that the main process (i.e. process 0) didn't check it's own lock (because `lockf` permits recursive locks and thus checking it only replaced the current lock with a new one).

Hi @panserbjorn , Can you share your `_check_all_processes_locks` function? thanks!
Issue: Metric evaluation problems in multi-node, shared file system

Comment:
```
def _check_all_processes_locks(self):
        expected_lock_file_names = [
            os.path.join(self.data_dir, f"{self.experiment_id}-{self.num_process}-{process_id}.arrow.lock")
            for process_id in range(self.num_process)
        ]
        #for expected_lock_file_name in expected_lock_file_names: # OUR CHANGE process 0 shouldn't check its own lock
        for expected_lock_file_name in expected_lock_file_names[1:]:
            nofilelock = FileFreeLock(expected_lock_file_name)
            try:
                nofilelock.acquire(timeout=self.timeout)
            except Timeout:
                raise ValueError(
                    f"Expected to find locked file {expected_lock_file_name} from process {self.process_id} but it doesn't exist."
                )
            else:
                nofilelock.release()
```

### Changed files:
- metric.py file in the datasets library  
- filelock.py file in the datasets/utils library. 


Changes we made:

1. We changed the flock for lockf 
   flock and lockf both perform a lock over a file (like the lock for writing). 
   The difference is that flock only works in local file systems, but if you have a shared file system (like what we have in the clusters) the flock fails to “see” the lock of another node. The only disadvantage we had was that a single process couldn’t detect it’s own lock so we did the second change.
2. We prevented the process 0 (which is the one that coordinates the rendezvous) from checking its own lock on its arrow because it didn't work with lockf (as stated in the previous change). 
3. We made a second rendezvous so that all the process had the results of the metrics (other than the loss) and not only the process 0.
   What happened was that only process 0 computed the metric and that didn’t present any problem if you are using the loss. However, if you are using another metric, the only process which had the information to choose the best checkpoint at evaluation time was the process 0. But since the evaluation was performed over all processes, every process except the process 0 chose a bad check point (bad meaning it wasn’t the best one) because they didn’t have the information of the metric of the best checkpoint. 
   The consequence was that the evaluation was different from what would result if using only the best checkpoint, because each process chose a different checkpoint to run the evaluation and thus the numbers were often worse than the numbers that would be obtained if all processes choose the best checkpoint (correct one) to perform the evaluation of their samples. 
   We performed a second rendezvous so that all processes had the same best_metric and best_model as process 0 after the evaluation cycle.
Issue: Update `unittest` assertions over tuples from `assertEqual` to `assertTupleEqual`

Comment:
Hi! If the only goal is to improve readability, it's better to use `assertTupleEqual` than `assertSequenceEqual` for Python tuples. Also, note that this function is called internally by `assertEqual`, but I guess we can accept a PR to be more verbose.
Issue: Update `unittest` assertions over tuples from `assertEqual` to `assertTupleEqual`

Comment:
Hi @mariosasko, right! I'll update the issue title/desc with `assertTupleEqual` even though as you said it seems to be internally using `assertEqual` so I'm not sure whether it's worth it or not...

https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertTupleEqual
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
Hi ! As mentioned on the [forum](https://discuss.huggingface.co/t/how-to-wrap-a-generator-with-hf-dataset/18464), the simplest for now would be to define a [dataset script](https://huggingface.co/docs/datasets/dataset_script) which can contain your generator. But we can also explore adding something like `ds = Dataset.from_iterable(seqio_dataset)`
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
I like the idea of having `Dataset.from_iterable(iterable)` in the API. The only problem is that we also want to make this part cachable, which is tricky if `iterable` is a generator. 

Some resources on this issue:
* https://github.com/uqfoundation/dill/issues/311
* https://stackoverflow.com/questions/7180212/why-cant-generators-be-pickled
* https://github.com/tonyroberts/generator_tools - python package for pickling generators; pickles bytecode, so it creates version-specific dumps
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
For the caching maybe we can have `Dataset.from_generator` as TF and pickle+hash the generator function (not the generator object itself) ?

And then keep `Dataset.from_iterable` fo pickable objects like lists
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
@lhoestq, @mariosasko do you too have any examples where the dataset is a generator and needs to be wrapped into hf dataset ?
Issue: how to convert a dict generator into a huggingface dataset. 

Comment:
@lhoestq, following to my previous question ... what possibly could be done in this [link1](https://github.com/huggingface/datasets/issues/4417#issuecomment-1146627404) [link2](https://github.com/huggingface/datasets/issues/4417#issuecomment-1146627593) case? do you have any ideas?