added saudinewsnet
State: closed | Author: abdulelahsm (CONTRIBUTOR)

I'm having issues in creating the dummy data. I'm still investigating how to fix it. I'll close the PR if I couldn't find a solution

Comments: Hi @abdulelahsm - This is an interesting dataset! But there are multiple issues with the PR. Some of them are listed below: 
-  default builder config is not defined. There should be atleast one builder config 
- URL is incorrectly constructed so the data files are not being downloaded 
- dataset_info.json file was not created

Please have a look at some existing merged datasets to get a reference on building the data loader. If you are still stuck, reach out. 


@skyprince999 I totally agree. Thx for the feedback!

Hi @abdulelahsm ! Thanks for adding this one :) 
you don't actually have to add builder configurations if you don't need them. It's fine as it is now

And as @skyprince999 noticed, the current URLs don't work. to download files.
You can use this one for example for the first batch instead:
https://github.com/parallelfold/SaudiNewsNet/raw/master/dataset/2015-07-21.zip

Feel free to ping me if you have questions or if you're ready for a review :) 

@lhoestq Hey, I tried using the first batch instead, the data was downloaded but I got this error, not sure why it can't find the path?

for content, I ran ``` "./datasets/saudinewsnet/test.py"```

which is a local test I'm running for the dataset, it contains the following code

```
from datasets import load_dataset

data = load_dataset("./datasets/saudinewsnet", split= "train")

print(data)

print(data[1])
```

this is the error I got 

```
2020-12-18 21:45:39.403908: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-12-18 21:45:39.403953: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Using custom data configuration default
Downloading and preparing dataset saudi_news_net/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/mesfas/.cache/huggingface/datasets/saudi_news_net/default/0.0.0/62ece5ef0a991415352d4b1efac681d75b5b3404064fd4f6a1d659499dab18f4...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.42M/3.42M [00:03<00:00, 1.03MB/s]
Traceback (most recent call last):
  File "/home/mesfas/opensource/datasets/src/datasets/builder.py", line 604, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/mesfas/opensource/datasets/src/datasets/builder.py", line 902, in _prepare_split
    for key, record in utils.tqdm(
  File "/home/mesfas/environments/ar_res_reviews/lib/python3.8/site-packages/tqdm/std.py", line 1133, in __iter__
    for obj in iterable:
  File "/home/mesfas/.cache/huggingface/modules/datasets_modules/datasets/saudinewsnet/62ece5ef0a991415352d4b1efac681d75b5b3404064fd4f6a1d659499dab18f4/saudinewsnet.py", line 108, in _generate_examples
    with open(filepath, encoding="utf-8").read() as f:
IsADirectoryError: [Errno 21] Is a directory: '/home/mesfas/.cache/huggingface/datasets/downloads/extracted/314fd983aa07d3dada9429911a805270c3285f48759d3584a1343c2d86260765'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./datasets/saudinewsnet/test.py", line 3, in <module>
    data = load_dataset("./datasets/saudinewsnet", split= "train")
  File "/home/mesfas/opensource/datasets/src/datasets/load.py", line 607, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/mesfas/opensource/datasets/src/datasets/builder.py", line 526, in download_and_prepare
    self._download_and_prepare(
  File "/home/mesfas/opensource/datasets/src/datasets/builder.py", line 606, in _download_and_prepare
    raise OSError(
OSError: Cannot find data file. 
Original error:
[Errno 21] Is a directory: '/home/mesfas/.cache/huggingface/datasets/downloads/extracted/314fd983aa07d3dada9429911a805270c3285f48759d3584a1343c2d86260765'
```


this is the split code 

```
    def _split_generators(self, dl_manager):
        """Returns SplitGenerators."""
        my_urls = _URL
        datadir = dl_manager.download_and_extract(my_urls)
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN,
                # These kwargs will be passed to _generate_examples
                gen_kwargs={
                    "filepath": datadir,
                    "split": "train"
                },
            ),
        ]
```
and this is how I'm generating the examples

```
    def _generate_examples(self, filepath, split):
        
        #logging.info("generating examples from = %s", filepath)
        with open(filepath, encoding="utf-8") as f:
            articles = json.load(f)
            for article in articles:
                title = article.get("title", "").strip()
                source = article.get("source", "").strip()
                date = article.get("date_extracted", "").strip()
                link = article.get("url", "").strip()
                author = article.get("author", "").strip()
                content = article.get("content", "").strip()

                yield id_, {
                    "title": title,
                    "source": source,
                    "date": date,
                    "link": link,
                    "author": author,
                    "content": content
                }
```

What's `_URL` ?

It looks like you are downloading an archive.
Therefore you may need to get to the file path using `filepath = os.path.join(datadir, "actual_file_name_inside_the_downloaded_archive")`

@lhoestq you were 100% right. Thank you. All fixed

@lhoestq ping!

@lhoestq  added the remaining 17 batches and modified the readme.md to reflect that + resolved the camel case comment

merging since the CI is fixed on master