Repository: deepjavalibrary/djl
Pull request #1545 (2022-03-26 09:39:09)

[basicdataset] Add WikiText-2 dataset
## Description ##

Add WikiText-2 dataset to basicdatasets.

- We referred to the code of cookStackexchange dataset. Function `WikiText2.getData()` will return the path of original text data.


dandansamax: Close #45 
dandansamax: BTW, could you help me upload the metadata to the server and write remote testcases? 
frankfliu: @dandansamax 

Thanks for your contribution. I have uploaded modified version of `metadata.json` file to our repo.
Please update your PR and push it again (I'm not able to push my changes to your repo, enabling `allow committer to update PR option` can make collaboration easier).

frankfliu: @dandansamax 

Please format the java code, there is white space in the test code:
```
./gradlew formatJava
```

And then commit the format changes.
dandansamax: Thank you so much for your patient help! I will enable edits from maintainers next time.
codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1545?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1545](https://codecov.io/gh/deepjavalibrary/djl/pull/1545?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (e11b754) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `1.32%`.
> The diff coverage is `51.05%`.

```diff
@@             Coverage Diff              @@
##             master    #1545      +/-   ##
============================================
- Coverage     72.08%   70.76%   -1.33%     
- Complexity     5126     5368     +242     
============================================
  Files           473      501      +28     
  Lines         21970    23427    +1457     
  Branches       2351     2545     +194     
============================================
+ Hits          15838    16579     +741     
- Misses         4925     5573     +648     
- Partials       1207     1275      +68     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1545?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1545/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1545/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <ø> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1545/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1545/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../modality/cv/translator/YoloTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1545/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1RyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | `8.33% <0.00%> (-1.67%)` | :arrow_down: |
| [...i/djl/modality/cv/translator/YoloV5Translator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1545/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1Y1VHJhbnNsYXRvci5qYXZh) | `5.69% <0.00%> (ø)` | |
| [...odality/cv/translator/YoloV5TranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1545/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1Y1VHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `8.33% <0.00%> (-1.67%)` | :arrow_down: |
| [...pi/src/main/java/ai/djl/ndarray/BytesSupplier.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1545/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL25kYXJyYXkvQnl0ZXNTdXBwbGllci5qYXZh) | `54.54% <0.00%> (-12.13%)` | :arrow_down: |
| [...pi/src/main/java/ai/djl/repository/Repository.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1545/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL3JlcG9zaXRvcnkvUmVwb3NpdG9yeS5qYXZh) | `83.33% <ø> (ø)` | |
| [...l/training/loss/SigmoidBinaryCrossEntropyLoss.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1545/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL3RyYWluaW5nL2xvc3MvU2lnbW9pZEJpbmFyeUNyb3NzRW50cm9weUxvc3MuamF2YQ==) | `64.00% <0.00%> (ø)` | |
| ... and [210 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1545/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1545?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1545?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Last update [2566416...e11b754](https://codecov.io/gh/deepjavalibrary/djl/pull/1545?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/basicdataset/src/main/java/ai/djl/basicdataset/nlp/WikiText2.java b/basicdataset/src/main/java/ai/djl/basicdataset/nlp/WikiText2.java
new file mode 100644
index 000000000..6a42db626
--- /dev/null
+++ b/basicdataset/src/main/java/ai/djl/basicdataset/nlp/WikiText2.java
@@ -0,0 +1,196 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.basicdataset.nlp;
+
+import ai.djl.Application;
+import ai.djl.basicdataset.BasicDatasets;
+import ai.djl.basicdataset.RawDataset;
+import ai.djl.ndarray.NDManager;
+import ai.djl.repository.Artifact;
+import ai.djl.repository.MRL;
+import ai.djl.repository.Repository;
+import ai.djl.training.dataset.Batch;
+import ai.djl.training.dataset.Dataset;
+import ai.djl.translate.TranslateException;
+import ai.djl.util.Progress;
+import java.io.IOException;
+import java.nio.file.Path;
+
+/**
+ * The WikiText language modeling dataset is a collection of over 100 million tokens extracted from
+ * the set of verified Good and Featured articles on Wikipedia.
+ */
+public class WikiText2 implements RawDataset<Path> {
+
+    private static final String VERSION = "1.0";
+    private static final String ARTIFACT_ID = "wikitext-2";
+
+    private Dataset.Usage usage;
+    private Path root;
+
+    private MRL mrl;
+    private boolean prepared;
+
+    WikiText2(Builder builder) {
+        this.usage = builder.usage;
+        mrl = builder.getMrl();
+    }
+
+    /**
+     * Creates a builder to build a {@link WikiText2}.
+     *
+     * @return a new {@link WikiText2.Builder} object
+     */
+    public static Builder builder() {
+        return new Builder();
+    }
+
+    /**
+     * Prepares the dataset for use with tracked progress.
+     *
+     * @param progress the progress tracker
+     * @throws IOException for various exceptions depending on the dataset
+     */
+    @Override
+    public void prepare(Progress progress) throws IOException {
+        if (prepared) {
+            return;
+        }
+
+        Artifact artifact = mrl.getDefaultArtifact();
+        mrl.prepare(artifact, progress);
+        Artifact.Item item;
+        item = artifact.getFiles().get("wikitext-2");
+        String path;
+        switch (usage) {
+            case TRAIN:
+                path = "wikitext-2/wiki.train.tokens";
+                break;
+            case TEST:
+                path = "wikitext-2/wiki.test.tokens";
+                break;
+            case VALIDATION:
+                path = "wikitext-2/wiki.valid.tokens";
+                break;
+            default:
+                throw new UnsupportedOperationException("Unsupported usage type.");
+        }
+
+        root = mrl.getRepository().getFile(item, path).toAbsolutePath();
+        prepared = true;
+    }
+
+    /**
+     * Fetches an iterator that can iterate through the {@link Dataset}. This method is not
+     * implemented for the WikiText2 dataset because the WikiText2 dataset is not suitable for
+     * iteration. If the method is called, it will directly return {@code null}.
+     *
+     * @param manager the dataset to iterate through
+     * @return an {@link Iterable} of {@link Batch} that contains batches of data from the dataset
+     */
+    @Override
+    public Iterable<Batch> getData(NDManager manager) throws IOException, TranslateException {
+        return null;
+    }
+
+    /**
+     * Get data from the WikiText2 dataset. This method will directly return the whole dataset.
+     *
+     * @return a {@link Path} object locating the WikiText2 dataset file
+     */
+    @Override
+    public Path getData() throws IOException {
+        prepare(null);
+        return root;
+    }
+
+    /** A builder to construct a {@link WikiText2} . */
+    public static final class Builder {
+
+        Repository repository;
+        String groupId;
+        String artifactId;
+        Dataset.Usage usage;
+
+        /** Constructs a new builder. */
+        Builder() {
+            repository = BasicDatasets.REPOSITORY;
+            groupId = BasicDatasets.GROUP_ID;
+            artifactId = ARTIFACT_ID;
+            usage = Dataset.Usage.TRAIN;
+        }
+
+        /**
+         * Sets the optional repository for the dataset.
+         *
+         * @param repository the new repository
+         * @return this builder
+         */
+        public Builder optRepository(Repository repository) {
+            this.repository = repository;
+            return this;
+        }
+
+        /**
+         * Sets optional groupId.
+         *
+         * @param groupId the groupId
+         * @return this builder
+         */
+        public Builder optGroupId(String groupId) {
+            this.groupId = groupId;
+            return this;
+        }
+
+        /**
+         * Sets the optional artifactId.
+         *
+         * @param artifactId the artifactId
+         * @return this builder
+         */
+        public Builder optArtifactId(String artifactId) {
+            if (artifactId.contains(":")) {
+                String[] tokens = artifactId.split(":");
+                groupId = tokens[0];
+                this.artifactId = tokens[1];
+            } else {
+                this.artifactId = artifactId;
+            }
+            return this;
+        }
+
+        /**
+         * Sets the optional usage for the dataset.
+         *
+         * @param usage the usage
+         * @return this builder
+         */
+        public Builder optUsage(Dataset.Usage usage) {
+            this.usage = usage;
+            return this;
+        }
+
+        /**
+         * Builds a new {@link WikiText2} object.
+         *
+         * @return the new {@link WikiText2} object
+         */
+        public WikiText2 build() {
+            return new WikiText2(this);
+        }
+
+        MRL getMrl() {
+            return repository.dataset(Application.NLP.ANY, groupId, artifactId, VERSION);
+        }
+    }
+}


Test patch:
diff --git a/basicdataset/src/test/java/ai/djl/basicdataset/WikiText2Test.java b/basicdataset/src/test/java/ai/djl/basicdataset/WikiText2Test.java
new file mode 100644
index 000000000..d066fc301
--- /dev/null
+++ b/basicdataset/src/test/java/ai/djl/basicdataset/WikiText2Test.java
@@ -0,0 +1,64 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.basicdataset;
+
+import ai.djl.basicdataset.nlp.WikiText2;
+import ai.djl.repository.Repository;
+import ai.djl.training.dataset.Dataset;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+public class WikiText2Test {
+    @Test
+    public void testWikiText2TrainRemote() throws IOException {
+        WikiText2 trainingSet = WikiText2.builder().optUsage(Dataset.Usage.TRAIN).build();
+        Path path = trainingSet.getData();
+        Assert.assertTrue(Files.isRegularFile(path));
+    }
+
+    @Test
+    public void testWikiText2TrainLocal() throws IOException {
+        Repository repository = Repository.newInstance("test", "src/test/resources/mlrepo");
+
+        WikiText2 trainingSet =
+                WikiText2.builder().optRepository(repository).optUsage(Dataset.Usage.TRAIN).build();
+        Path path = trainingSet.getData();
+        Assert.assertTrue(Files.isRegularFile(path));
+    }
+
+    @Test
+    public void testWikiText2TestLocal() throws IOException {
+        Repository repository = Repository.newInstance("test", "src/test/resources/mlrepo");
+
+        WikiText2 trainingSet =
+                WikiText2.builder().optRepository(repository).optUsage(Dataset.Usage.TEST).build();
+        Path path = trainingSet.getData();
+        Assert.assertTrue(Files.isRegularFile(path));
+    }
+
+    @Test
+    public void testWikiText2ValidationLocal() throws IOException {
+        Repository repository = Repository.newInstance("test", "src/test/resources/mlrepo");
+
+        WikiText2 trainingSet =
+                WikiText2.builder()
+                        .optRepository(repository)
+                        .optUsage(Dataset.Usage.VALIDATION)
+                        .build();
+        Path path = trainingSet.getData();
+        Assert.assertTrue(Files.isRegularFile(path));
+    }
+}
diff --git a/basicdataset/src/test/resources/mlrepo/dataset/nlp/ai/djl/basicdataset/wikitext-2/metadata.json b/basicdataset/src/test/resources/mlrepo/dataset/nlp/ai/djl/basicdataset/wikitext-2/metadata.json
new file mode 100644
index 000000000..f9c64dc80
--- /dev/null
+++ b/basicdataset/src/test/resources/mlrepo/dataset/nlp/ai/djl/basicdataset/wikitext-2/metadata.json
@@ -0,0 +1,31 @@
+{
+  "metadataVersion": "0.2",
+  "resourceType": "dataset",
+  "application": "nlp",
+  "groupId": "ai.djl.basicdataset",
+  "artifactId": "wikitext-2",
+  "name": "wikitext-2",
+  "description": "The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia.",
+  "website": "https://blog.salesforceairesearch.com/the-wikitext-long-term-dependency-language-modeling-dataset/",
+  "licenses": {
+    "license": {
+      "name": "Creative Commons Attribution-ShareAlike License",
+      "url": "https://en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0_Unported_License"
+    }
+  },
+  "artifacts": [
+    {
+      "version": "1.0",
+      "snapshot": false,
+      "name": "wikitext-2",
+      "files": {
+        "wikitext-2": {
+          "uri": "https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip",
+          "sha1Hash": "3c914d17d80b1459be871a5039ac23e752a53cbe",
+          "name": "",
+          "size": 4475746
+        }
+      }
+    }
+  ]
+}