Repository: deepjavalibrary/djl
Pull request #2322 (2023-01-15 03:45:06)

[tokenizer] Allows Huggingface model zoo list models in offline mode
## Description ##

Brief description of what this PR is about

- If this change is a backward incompatible change, why must this change be made?
- Interesting edge cases to note here


codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
Base: **72.08**% // Head: **74.31**% // Increases project coverage by **`+2.22%`** :tada:
> Coverage data is based on head [(`bd8bca5`)](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) compared to base [(`bb5073f`)](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> Patch coverage: 75.10% of modified lines in pull request are covered.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff              @@
##             master    #2322      +/-   ##
============================================
+ Coverage     72.08%   74.31%   +2.22%     
- Complexity     5126     6707    +1581     
============================================
  Files           473      659     +186     
  Lines         21970    29165    +7195     
  Branches       2351     3017     +666     
============================================
+ Hits          15838    21673    +5835     
- Misses         4925     6021    +1096     
- Partials       1207     1471     +264     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L011bHRpQm94UHJpb3IuamF2YQ==) | `76.00% <ø> (ø)` | |
| [...rc/main/java/ai/djl/modality/cv/output/Joints.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9Kb2ludHMuamF2YQ==) | `71.42% <ø> (ø)` | |
| [.../main/java/ai/djl/modality/cv/output/Landmark.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9MYW5kbWFyay5qYXZh) | `100.00% <ø> (ø)` | |
| [...main/java/ai/djl/modality/cv/output/Rectangle.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9SZWN0YW5nbGUuamF2YQ==) | `72.41% <0.00%> (ø)` | |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <0.00%> (-5.24%)` | :arrow_down: |
| [.../modality/cv/translator/ImageFeatureExtractor.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW1hZ2VGZWF0dXJlRXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../ai/djl/modality/cv/translator/YoloTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1RyYW5zbGF0b3IuamF2YQ==) | `27.77% <0.00%> (+18.95%)` | :arrow_up: |
| [...ain/java/ai/djl/modality/cv/util/NDImageUtils.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3V0aWwvTkRJbWFnZVV0aWxzLmphdmE=) | `67.10% <0.00%> (+7.89%)` | :arrow_up: |
| [api/src/main/java/ai/djl/modality/nlp/Decoder.java](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L25scC9EZWNvZGVyLmphdmE=) | `63.63% <ø> (ø)` | |
| ... and [603 more](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/2322?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/extensions/tokenizers/build.gradle b/extensions/tokenizers/build.gradle
index 3a2f4313d..429bdd137 100644
--- a/extensions/tokenizers/build.gradle
+++ b/extensions/tokenizers/build.gradle
@@ -1,3 +1,5 @@
+import java.util.zip.GZIPInputStream
+
 group "ai.djl.huggingface"
 
 dependencies {
@@ -44,6 +46,27 @@ processResources {
         // write properties
         def propFile = file("${project.buildDir}/classes/java/main/native/lib/tokenizers.properties")
         propFile.text = "version=${tokenizers_version}-${version}\n"
+
+        url = "https://mlrepo.djl.ai/model/nlp"
+        def tasks = [
+                "fill_mask",
+                "question_answer",
+                "text_classification",
+                "text_embedding",
+                "token_classification",
+        ]
+        def prefix = "${project.buildDir}/classes/java/main/nlp"
+        for (String task : tasks) {
+            def file = new File("${prefix}/${task}/ai.djl.huggingface.pytorch.json")
+            if (file.exists()) {
+                project.logger.lifecycle("model zoo metadata alrady exists: ${task}")
+            } else {
+                project.logger.lifecycle("Downloading model zoo metadata: ${task}")
+                file.getParentFile().mkdirs()
+                def downloadPath = new URL("${url}/${task}/ai/djl/huggingface/pytorch/models.json.gz")
+                downloadPath.withInputStream { i -> file.withOutputStream { it << new GZIPInputStream(i) } }
+            }
+        }
     }
 }
 
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/zoo/HfModelZoo.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/zoo/HfModelZoo.java
index 88cb137b2..78346217f 100644
--- a/extensions/tokenizers/src/main/java/ai/djl/huggingface/zoo/HfModelZoo.java
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/zoo/HfModelZoo.java
@@ -19,6 +19,7 @@ import ai.djl.repository.Repository;
 import ai.djl.repository.Version;
 import ai.djl.repository.VersionRange;
 import ai.djl.repository.zoo.ModelZoo;
+import ai.djl.util.ClassLoaderUtils;
 import ai.djl.util.JsonUtils;
 import ai.djl.util.Utils;
 
@@ -28,10 +29,10 @@ import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.io.IOException;
+import java.io.InputStream;
 import java.io.Reader;
 import java.io.Writer;
 import java.lang.reflect.Type;
-import java.net.URL;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.time.Duration;
@@ -117,13 +118,30 @@ public class HfModelZoo extends ModelZoo {
 
             String url = REPO + path + "models.json.gz";
             Path tmp = Files.createTempFile(dir, "models", ".tmp");
-            try (GZIPInputStream gis = new GZIPInputStream(new URL(url).openStream())) {
+            try (GZIPInputStream gis = new GZIPInputStream(Utils.openUrl(url))) {
                 String json = Utils.toString(gis);
                 try (Writer writer = Files.newBufferedWriter(tmp)) {
                     writer.write(json);
                 }
                 Utils.moveQuietly(tmp, file);
                 return JsonUtils.GSON.fromJson(json, type);
+            } catch (IOException e) {
+                logger.warn("Failed to download Huggingface model zoo index: {}", app);
+                if (Files.exists(file)) {
+                    try (Reader reader = Files.newBufferedReader(file)) {
+                        return JsonUtils.GSON.fromJson(reader, type);
+                    }
+                }
+
+                String resource = app.getPath() + "/" + GROUP_ID + ".json";
+                try (InputStream is = ClassLoaderUtils.getResourceAsStream(resource)) {
+                    String json = Utils.toString(is);
+                    try (Writer writer = Files.newBufferedWriter(tmp)) {
+                        writer.write(json);
+                    }
+                    Utils.moveQuietly(tmp, file);
+                    return JsonUtils.GSON.fromJson(json, type);
+                }
             } finally {
                 Utils.deleteQuietly(tmp);
             }
diff --git a/extensions/tokenizers/src/main/python/huggingface_converter.py b/extensions/tokenizers/src/main/python/huggingface_converter.py
index 5b4749d28..f3b85c241 100644
--- a/extensions/tokenizers/src/main/python/huggingface_converter.py
+++ b/extensions/tokenizers/src/main/python/huggingface_converter.py
@@ -112,7 +112,7 @@ class HuggingfaceConverter:
             logging.info(f"Saving torchscript model: {model_name}.pt ...")
             model_file = os.path.join(temp_dir, f"{model_name}.pt")
             script_module.save(model_file)
-        except RuntimeError as e:
+        except (RuntimeError, ValueError) as e:
             logging.warning(f"Failed to trace model: {model_id}.")
             logging.warning(e, exc_info=True)
             return None


Test patch:
diff --git a/extensions/tokenizers/src/test/java/ai/djl/huggingface/zoo/ModelZooTest.java b/extensions/tokenizers/src/test/java/ai/djl/huggingface/zoo/ModelZooTest.java
index 89e19014d..e585a219f 100644
--- a/extensions/tokenizers/src/test/java/ai/djl/huggingface/zoo/ModelZooTest.java
+++ b/extensions/tokenizers/src/test/java/ai/djl/huggingface/zoo/ModelZooTest.java
@@ -31,7 +31,9 @@ import java.io.IOException;
 import java.io.Writer;
 import java.nio.file.Files;
 import java.nio.file.Path;
+import java.nio.file.Paths;
 import java.util.Map;
+import java.util.Set;
 import java.util.concurrent.ConcurrentHashMap;
 
 public class ModelZooTest {
@@ -62,6 +64,7 @@ public class ModelZooTest {
     public void testFutureVersion() throws IOException {
         System.setProperty("DJL_CACHE_DIR", "build/cache");
         try {
+            Utils.deleteQuietly(Paths.get("build/cache"));
             Map<String, Map<String, Object>> map = new ConcurrentHashMap<>();
             Map<String, Object> model = new ConcurrentHashMap<>();
             model.put("result", "failed");
@@ -96,4 +99,25 @@ public class ModelZooTest {
             System.clearProperty("DJL_CACHE_DIR");
         }
     }
+
+    @Test
+    public void testOffLine() throws IOException {
+        System.setProperty("DJL_CACHE_DIR", "build/cache");
+        System.setProperty("offline", "true");
+        try {
+            Utils.deleteQuietly(Paths.get("build/cache"));
+            // static variables cannot not be initialized properly if directly use new HfModelZoo()
+            ModelZoo.getModelZoo("ai.djl.huggingface.pytorch");
+
+            ModelZoo zoo = new HfModelZoo();
+            Assert.assertTrue(zoo.getModelLoaders().size() > 0);
+
+            Set<String> engines = zoo.getSupportedEngines();
+            Assert.assertEquals(engines.size(), 1);
+            Assert.assertEquals(engines.iterator().next(), "PyTorch");
+        } finally {
+            System.clearProperty("DJL_CACHE_DIR");
+            System.clearProperty("offline");
+        }
+    }
 }