Repository: huggingface/datasets
Pull request: Add dev-only config to Natural Questions dataset
As suggested by @lhoestq and @thomwolf, a new config has been added to Natural Questions dataset, so that only dev split can be downloaded. 

Fix #413.

Problem statement:
Is there a way to download only NQ dev?
Maybe I missed that in the docs, but is there a way to only download the dev set of natural questions (~1 GB)? 
As we want to benchmark QA models on different datasets, I would like to avoid downloading the 41GB of training data. 

I tried
```
dataset = nlp.load_dataset('natural_questions', split="validation", beam_runner="DirectRunner")
```
But this still triggered a big download of presumably the whole dataset. Is there any way of doing this or are splits / slicing options only available after downloading?

Thanks!

Hints from issue discussion:
Unfortunately it's not possible to download only the dev set of NQ.

I think we could add a way to download only the test set by adding a custom configuration to the processing script though.
Ok, got it. I think this could be a valuable feature - especially for large datasets like NQ, but potentially also others. 
For us, it will in this case make the difference of using the library or keeping the old downloads of the raw dev datasets.  
However, I don't know if that fits into your plans with the library and can also understand if you don't want to support this.
I don't think we could force this behavior generally since the dataset script authors are free to organize the file download as they want (sometimes the mapping between split and files can be very much nontrivial) but we can add an additional configuration for Natural Question indeed as @lhoestq indicate.

Commit message: Update dataset card

Patch to review:
diff --git a/datasets/natural_questions/README.md b/datasets/natural_questions/README.md
index ea4ca276741..849d75b1f47 100644
--- a/datasets/natural_questions/README.md
+++ b/datasets/natural_questions/README.md
@@ -1,11 +1,27 @@
 ---
+annotations_creators:
+- no-annotation
+language_creators:
+- crowdsourced
 languages:
 - en
-paperswithcode_id: natural-questions
+licenses:
+- cc-by-sa-3.0
+multilinguality:
+- monolingual
 pretty_name: Natural Questions
+size_categories:
+- 100K<n<1M
+source_datasets:
+- original
+task_categories:
+- question-answering
+task_ids:
+- open-domain-qa
+paperswithcode_id: natural-questions
 ---
 
-# Dataset Card for "natural_questions"
+# Dataset Card for Natural Questions
 
 ## Table of Contents
 - [Dataset Description](#dataset-description)
@@ -104,9 +120,10 @@ The data fields are the same among all splits.
 
 ### Data Splits
 
-| name  |train |validation|
-|-------|-----:|---------:|
-|default|307373|      7830|
+| name    |  train |validation|
+|---------|-------:|---------:|
+| default | 307373 |      7830|
+| dev     |      0 |      7830|
 
 ## Dataset Creation
 
@@ -160,7 +177,7 @@ The data fields are the same among all splits.
 
 ### Licensing Information
 
-[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+[Creative Commons Attribution-ShareAlike 3.0 Unported](https://creativecommons.org/licenses/by-sa/3.0/).
 
 ### Citation Information
 
diff --git a/datasets/natural_questions/natural_questions.py b/datasets/natural_questions/natural_questions.py
index 381ffeab106..693e3a71f90 100644
--- a/datasets/natural_questions/natural_questions.py
+++ b/datasets/natural_questions/natural_questions.py
@@ -51,12 +51,17 @@
     "validation": ["%s/dev/nq-dev-%02d.jsonl.gz" % (_BASE_DOWNLOAD_URL, i) for i in range(5)],
 }
 
+_VERSION = datasets.Version("0.0.3")
+
 
 class NaturalQuestions(datasets.BeamBasedBuilder):
     """Natural Questions: A Benchmark for Question Answering Research."""
 
-    VERSION = datasets.Version("0.0.2")
-    SUPPORTED_VERSIONS = [datasets.Version("0.0.1")]
+    BUILDER_CONFIGS = [
+        datasets.BuilderConfig(name="default", version=_VERSION),
+        datasets.BuilderConfig(name="dev", version=_VERSION, description="Only dev split"),
+    ]
+    DEFAULT_CONFIG_NAME = "default"
 
     def _info(self):
         return datasets.DatasetInfo(
@@ -108,20 +113,19 @@ def _info(self):
 
     def _split_generators(self, dl_manager, pipeline):
         """Returns SplitGenerators."""
-
-        files = dl_manager.download(_DOWNLOAD_URLS)
+        urls = _DOWNLOAD_URLS
+        if self.config.name == "dev":
+            urls = {"validation": urls["validation"]}
+        files = dl_manager.download(urls)
         if not pipeline.is_local():
             files = dl_manager.ship_files_with_pipeline(files, pipeline)
-
         return [
             datasets.SplitGenerator(
-                name=datasets.Split.TRAIN,
-                gen_kwargs={"filepaths": files["train"]},
-            ),
-            datasets.SplitGenerator(
-                name=datasets.Split.VALIDATION,
-                gen_kwargs={"filepaths": files["validation"]},
-            ),
+                name=split,
+                gen_kwargs={"filepaths": files[split]},
+            )
+            for split in [datasets.Split.TRAIN, datasets.Split.VALIDATION]
+            if split in files
         ]
 
     def _build_pcollection(self, pipeline, filepaths):
@@ -133,7 +137,7 @@ def _parse_example(line):
             html_bytes = ex_json["document_html"].encode("utf-8")
 
             def _parse_short_answer(short_ans):
-                """ "Extract text of short answer."""
+                """Extract text of short answer."""
                 ans_bytes = html_bytes[short_ans["start_byte"] : short_ans["end_byte"]]
                 # Remove non-breaking spaces.
                 ans_bytes = ans_bytes.replace(b"\xc2\xa0", b" ")


Code review comments:
diff_hunk: @@ -104,9 +120,10 @@ The data fields are the same among all splits.
 
 ### Data Splits
 
-| name  |train |validation|
-|-------|-----:|---------:|
-|default|307373|      7830|
+| name    |  train |validation|
+|---------|-------:|---------:|
+| default | 307373 |      7830|
+| dev     |      0 |      7830|
path: datasets/natural_questions/README.md
text: @user1:
maybe clearer this way ?
```suggestion
| dev     |      N/A |      7830|
```