Repository: deepjavalibrary/djl
Pull request #2169 (2022-11-20 00:21:41)

[integration] Move model zoo download test to canary
## Description ##

Brief description of what this PR is about

- If this change is a backward incompatible change, why must this change be made?
- Interesting edge cases to note here


codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/2169?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
Base: **72.08**% // Head: **71.42**% // Decreases project coverage by **`-0.66%`** :warning:
> Coverage data is based on head [(`e0d1081`)](https://codecov.io/gh/deepjavalibrary/djl/pull/2169?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) compared to base [(`bb5073f`)](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> Patch coverage: 71.90% of modified lines in pull request are covered.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff              @@
##             master    #2169      +/-   ##
============================================
- Coverage     72.08%   71.42%   -0.67%     
- Complexity     5126     6332    +1206     
============================================
  Files           473      628     +155     
  Lines         21970    27978    +6008     
  Branches       2351     3009     +658     
============================================
+ Hits          15838    19983    +4145     
- Misses         4925     6527    +1602     
- Partials       1207     1468     +261     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/2169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2169/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2169/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L011bHRpQm94UHJpb3IuamF2YQ==) | `76.00% <ø> (ø)` | |
| [...rc/main/java/ai/djl/modality/cv/output/Joints.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2169/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9Kb2ludHMuamF2YQ==) | `71.42% <ø> (ø)` | |
| [.../main/java/ai/djl/modality/cv/output/Landmark.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2169/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9MYW5kbWFyay5qYXZh) | `100.00% <ø> (ø)` | |
| [...main/java/ai/djl/modality/cv/output/Rectangle.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2169/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9SZWN0YW5nbGUuamF2YQ==) | `72.41% <0.00%> (ø)` | |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2169/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <0.00%> (-5.24%)` | :arrow_down: |
| [.../modality/cv/translator/ImageFeatureExtractor.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2169/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW1hZ2VGZWF0dXJlRXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../ai/djl/modality/cv/translator/YoloTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2169/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1RyYW5zbGF0b3IuamF2YQ==) | `27.77% <0.00%> (+18.95%)` | :arrow_up: |
| [...modality/cv/translator/wrapper/FileTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2169/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3Ivd3JhcHBlci9GaWxlVHJhbnNsYXRvci5qYXZh) | `44.44% <ø> (ø)` | |
| [...y/cv/translator/wrapper/InputStreamTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2169/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3Ivd3JhcHBlci9JbnB1dFN0cmVhbVRyYW5zbGF0b3IuamF2YQ==) | `44.44% <ø> (ø)` | |
| ... and [562 more](https://codecov.io/gh/deepjavalibrary/djl/pull/2169/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/2169?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/model-zoo/src/main/java/ai/djl/basicmodelzoo/cv/object_detection/yolo/YOLOV3.java b/model-zoo/src/main/java/ai/djl/basicmodelzoo/cv/object_detection/yolo/YOLOV3.java
index 1a329cf54..df094aaf0 100644
--- a/model-zoo/src/main/java/ai/djl/basicmodelzoo/cv/object_detection/yolo/YOLOV3.java
+++ b/model-zoo/src/main/java/ai/djl/basicmodelzoo/cv/object_detection/yolo/YOLOV3.java
@@ -38,15 +38,18 @@ import java.util.Arrays;
 import java.util.List;
 
 /**
- * {@code YOLOV3} contains a generic implementation of yolov3 adapted from
- * https://github.com/bubbliiiing/yolo3-pytorch (Original author bubbliiiing). Yolov3 is a fast and
- * accurate model for ObjectDetection tasks.
+ * {@code YOLOV3} contains a generic implementation of <a
+ * href="https://github.com/bubbliiiing/yolo3-pytorch">yolov3</a> (Original author bubbliiiing).
+ *
+ * <p>Yolov3 is a fast and accurate model for ObjectDetection tasks.
  *
  * @see <a href="https://arxiv.org/abs/1804.02767">for more information and knowledge about
  *     yolov3</a>
  */
 public final class YOLOV3 extends AbstractBlock {
+
     private static final byte VERSION = 1;
+
     private SequentialBlock darkNet53; // backBone of YOLOv3
 
     private Block lastLayer0;
@@ -334,7 +337,7 @@ public final class YOLOV3 extends AbstractBlock {
             } else if (!name.contains("Output")) {
                 current = block.getOutputShapes(current);
             } else { // name.contains("Output")
-                Shape[] output = block.getOutputShapes(current);
+                block.getOutputShapes(current);
             }
         }
     }
@@ -467,7 +470,7 @@ public final class YOLOV3 extends AbstractBlock {
          * Sets the number of classes for yolov3.
          *
          * @param numClasses the number of classes
-         * @return @return this {@code Builder}
+         * @return this {@code Builder}
          */
         public Builder setNumClasses(int numClasses) {
             this.numClasses = numClasses;


Test patch:
diff --git a/basicdataset/src/test/resources/mlrepo/dataset/tabular/ai/djl/basicdataset/movielens-100k/metadata.json b/basicdataset/src/test/resources/mlrepo/dataset/tabular/ai/djl/basicdataset/movielens-100k/metadata.json
index 2728d25cf..c2a29e32e 100644
--- a/basicdataset/src/test/resources/mlrepo/dataset/tabular/ai/djl/basicdataset/movielens-100k/metadata.json
+++ b/basicdataset/src/test/resources/mlrepo/dataset/tabular/ai/djl/basicdataset/movielens-100k/metadata.json
@@ -19,11 +19,11 @@
       "snapshot": false,
       "files": {
         "ml-100k": {
-          "uri": "https://files.grouplens.org/datasets/movielens/ml-100k.zip",
-          "sha1Hash": "cd4dcac4241c8a4ad7badc7ca635da8a69dddb83",
-          "size": "4924029"
+          "uri": "1.0/ml-100k.zip",
+          "sha1Hash": "c14dcdbe7e9ae1961f44f58b696ca295c4d3807e",
+          "size": "632294"
         }
       }
     }
   ]
-}
\ No newline at end of file
+}
diff --git a/extensions/aws-ai/src/test/java/ai/djl/aws/sagemaker/SageMakerTest.java b/extensions/aws-ai/src/test/java/ai/djl/aws/sagemaker/SageMakerTest.java
index 30ada2f7a..6d9d56366 100644
--- a/extensions/aws-ai/src/test/java/ai/djl/aws/sagemaker/SageMakerTest.java
+++ b/extensions/aws-ai/src/test/java/ai/djl/aws/sagemaker/SageMakerTest.java
@@ -59,7 +59,7 @@ public class SageMakerTest {
                             .optBucketName("djl-sm-test")
                             .optModelName("resnet18-jit")
                             .optContainerImage(
-                                    "125045733377.dkr.ecr.us-east-1.amazonaws.com/djl-serving")
+                                    "125045733377.dkr.ecr.us-east-1.amazonaws.com/djl-serving:cpu-nightly")
                             .optExecutionRole(
                                     "arn:aws:iam::125045733377:role/service-role/DJLSageMaker-ExecutionRole-20210213T1027050")
                             .build();
diff --git a/integration/src/main/java/ai/djl/integration/tests/model_zoo/ModelZooTest.java b/integration/src/main/java/ai/djl/integration/tests/model_zoo/ModelZooTest.java
deleted file mode 100644
index a8a07d1ef..000000000
--- a/integration/src/main/java/ai/djl/integration/tests/model_zoo/ModelZooTest.java
+++ /dev/null
@@ -1,71 +0,0 @@
-/*
- * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
- *
- * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
- * with the License. A copy of the License is located at
- *
- * http://aws.amazon.com/apache2.0/
- *
- * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
- * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
- * and limitations under the License.
- */
-package ai.djl.integration.tests.model_zoo;
-
-import ai.djl.Model;
-import ai.djl.ModelException;
-import ai.djl.ndarray.NDList;
-import ai.djl.repository.Artifact;
-import ai.djl.repository.zoo.Criteria;
-import ai.djl.repository.zoo.ModelLoader;
-import ai.djl.repository.zoo.ModelZoo;
-import ai.djl.testing.TestRequirements;
-import ai.djl.util.Utils;
-
-import org.testng.annotations.AfterClass;
-import org.testng.annotations.BeforeClass;
-import org.testng.annotations.Test;
-
-import java.io.IOException;
-import java.nio.file.Paths;
-import java.util.List;
-
-public class ModelZooTest {
-
-    @BeforeClass
-    public void setUp() {
-        // force downloading without cache in .djl.ai folder.
-        System.setProperty("DJL_CACHE_DIR", "build/cache");
-        String userHome = System.getProperty("user.home");
-        System.setProperty("ENGINE_CACHE_DIR", userHome + "/.djl.ai");
-    }
-
-    @AfterClass
-    public void tearDown() {
-        System.clearProperty("DJL_CACHE_DIR");
-        System.clearProperty("ENGINE_CACHE_DIR");
-    }
-
-    @Test
-    public void testDownloadModels() throws IOException, ModelException {
-        TestRequirements.nightly();
-        TestRequirements.notOffline();
-        TestRequirements.weekly();
-
-        for (ModelZoo zoo : ModelZoo.listModelZoo()) {
-            for (ModelLoader modelLoader : zoo.getModelLoaders()) {
-                List<Artifact> artifacts = modelLoader.listModels();
-                for (Artifact artifact : artifacts) {
-                    Criteria<NDList, NDList> criteria =
-                            Criteria.builder()
-                                    .setTypes(NDList.class, NDList.class)
-                                    .optFilters(artifact.getProperties())
-                                    .build();
-                    Model model = modelLoader.loadModel(criteria);
-                    model.close();
-                }
-                Utils.deleteQuietly(Paths.get("build/cache"));
-            }
-        }
-    }
-}
diff --git a/integration/src/main/java/ai/djl/integration/tests/nn/BlockCoreTest.java b/integration/src/main/java/ai/djl/integration/tests/nn/BlockCoreTest.java
index 9e2809681..d166c71b0 100644
--- a/integration/src/main/java/ai/djl/integration/tests/nn/BlockCoreTest.java
+++ b/integration/src/main/java/ai/djl/integration/tests/nn/BlockCoreTest.java
@@ -60,9 +60,9 @@ import org.testng.annotations.Test;
 
 import java.io.DataInputStream;
 import java.io.DataOutputStream;
-import java.io.File;
 import java.io.IOException;
 import java.nio.file.Files;
+import java.nio.file.Path;
 import java.util.Arrays;
 import java.util.stream.IntStream;
 
@@ -1050,11 +1050,14 @@ public class BlockCoreTest {
     private void testEncode(NDManager manager, Block block)
             throws IOException, MalformedModelException {
         PairList<String, Parameter> original = block.getParameters();
-        File temp = Files.createTempFile("block", ".param").toFile();
-        DataOutputStream os = new DataOutputStream(Files.newOutputStream(temp.toPath()));
-        block.saveParameters(os);
-        block.loadParameters(manager, new DataInputStream(Files.newInputStream(temp.toPath())));
-        Files.delete(temp.toPath());
+        Path temp = Files.createTempFile("block", ".param");
+        try (DataOutputStream os = new DataOutputStream(Files.newOutputStream(temp))) {
+            block.saveParameters(os);
+        }
+        try (DataInputStream is = new DataInputStream(Files.newInputStream(temp))) {
+            block.loadParameters(manager, is);
+        }
+        Files.delete(temp);
         PairList<String, Parameter> loaded = block.getParameters();
         int bound = original.size();
         for (int idx = 0; idx < bound; idx++) {