hindi discourse analysis dataset commit
State: closed | Author: duttahritwik (CONTRIBUTOR)


Comments: That's a great dataset to have! We need a couple more things to be good to go:
- you should `make style` and `flake8 datasets` before pushing to make the code quality check happy :) 
- the dataset will need some dummy data which you should be able to auto-generate and test locally: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#automatically-add-code-metadata
- there's some good information in your current README, but we need the format to follow the template [here](https://github.com/huggingface/datasets/blob/master/templates/README.md) and to have YAML tags at the top, as described in the guide: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

LEt us know if you need any help!

Hi @yjernite 
I was successfully able to generate the dataset_info.json file using the command 
python datasets-cli test datasets/<your-dataset-folder> --save_infos --all_configs

But unfortunately, could not generate the dummy data

While running the command 
python datasets-cli dummy_data datasets/<your-dataset-folder> --auto_generate
I got an error as 

ValueError: Couldn't parse columns ['0', '1', '2', '3', '4', ......, '9982']. Maybe specify which json field must be used to read the data with --json_field <my_field>.

The thing is the dataset I am trying to upload is of the format 
{
   '0': {'Story_no': 15, 'Sentence': ' गाँठ से साढ़े तीन रुपये लग गये, जो अब पेट में जाकर खनकते भी नहीं! जो तेरी करनी मालिक! ” “इसमें मालिक की क्या करनी है? ”', 'Discourse Mode': 'Dialogue'},
  '1': {'Story_no': story_no, 'Sentence': sentence, 'Discourse Mode': discourse_mode},
   .......,
  '9982': {'Story_no': story_no, 'Sentence': sentence, 'Discourse Mode': discourse_mode}
}

Can you please suggest any errors I am making in the  _generate_examples method?

Thanks!

The dummy data generator doesn't support this kind of json format yet.
Can you create the dummy data manually please ? You can get the instructions by running the 
```
datasets-cli dummy_data ./datasets/dataset_name
```
command.

Hi, I created the dummy data manually but the tests are still failing it seems.
Can you suggest the format of JSON which is supported by dummy data generator?
I will have to modify my _generate_examples method accordingly.
Please advice on the same.
Thanks much.


Can you run `make style` to format the code for the CI please ?

Also about the dummy data, here is how to generate them:

We need a dummy_data.zip file in ./datasets/hindiDiscourse/dummy/1.0.0 (or replace hindiDiscourse by hindi_discourse since we have to rename the folder anyway)
To create the zip file, first go in this directory and create a folder named dummy_data.
Then inside the dummy_data folder create a file `discourse_dataset.json` and fill it with something like 5 examples.
Finally zip the dummy_data folder to end up with the dummy_data.zip file

Once it's done you can check if the dummy data test passes with 
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_hindi_discourse
```

If it passes you can then remove the dummy_data folder to keep only the dummy_data.zip file

Hi @duttahritwik did you manage to make the dummy data ?
Feel free to ping me if you have questions or if we can help

The error `tests/test_file_utils.py::TempSeedTest::test_tensorflow` just appeared because of tensorflow's update.
Once it's fixed on master we'll be free to merge this one

Ci is green on master :) 

merging since the CI is fixed on master