Repository: deepjavalibrary/djl
Pull request #1899 (2022-08-12 17:29:19)

speech recognition translator
speech recognition app files

## Description ##

Brief description of what this PR is about

- Contains speech recognition translator and files
- currently has model loading error, waiting for pytorch 1.12 release


codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1899?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1899](https://codecov.io/gh/deepjavalibrary/djl/pull/1899?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (27cd068) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `2.17%`.
> The diff coverage is `65.36%`.

```diff
@@             Coverage Diff              @@
##             master    #1899      +/-   ##
============================================
- Coverage     72.08%   69.91%   -2.18%     
- Complexity     5126     5586     +460     
============================================
  Files           473      531      +58     
  Lines         21970    24659    +2689     
  Branches       2351     2674     +323     
============================================
+ Hits          15838    17240    +1402     
- Misses         4925     6109    +1184     
- Partials       1207     1310     +103     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1899?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [.../audio/translator/SpeechRecognitionTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1899/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2F1ZGlvL3RyYW5zbGF0b3IvU3BlZWNoUmVjb2duaXRpb25UcmFuc2xhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1899/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1899/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L011bHRpQm94UHJpb3IuamF2YQ==) | `76.00% <ø> (ø)` | |
| [...rc/main/java/ai/djl/modality/cv/output/Joints.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1899/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9Kb2ludHMuamF2YQ==) | `71.42% <ø> (ø)` | |
| [.../main/java/ai/djl/modality/cv/output/Landmark.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1899/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9MYW5kbWFyay5qYXZh) | `100.00% <ø> (ø)` | |
| [...main/java/ai/djl/modality/cv/output/Rectangle.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1899/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9SZWN0YW5nbGUuamF2YQ==) | `72.41% <0.00%> (ø)` | |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1899/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <0.00%> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1899/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1899/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../cv/translator/SemanticSegmentationTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1899/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU2VtYW50aWNTZWdtZW50YXRpb25UcmFuc2xhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| ... and [429 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1899/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)


Patch:
diff --git a/api/src/main/java/ai/djl/modality/audio/translator/SpeechRecognitionTranslator.java b/api/src/main/java/ai/djl/modality/audio/translator/SpeechRecognitionTranslator.java
new file mode 100644
index 000000000..6d53de694
--- /dev/null
+++ b/api/src/main/java/ai/djl/modality/audio/translator/SpeechRecognitionTranslator.java
@@ -0,0 +1,43 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+
+package ai.djl.modality.audio.translator;
+
+import ai.djl.modality.audio.Audio;
+import ai.djl.ndarray.NDArray;
+import ai.djl.ndarray.NDList;
+import ai.djl.ndarray.types.Shape;
+import ai.djl.translate.NoBatchifyTranslator;
+import ai.djl.translate.Translator;
+import ai.djl.translate.TranslatorContext;
+
+/**
+ * A {@link Translator} that post-process the {@link Audio} into {@link String} to get a text
+ * translation of the audio.
+ */
+public class SpeechRecognitionTranslator implements NoBatchifyTranslator<Audio, String> {
+
+    /** {@inheritDoc} */
+    @Override
+    public NDList processInput(TranslatorContext ctx, Audio input) throws Exception {
+        float[] data = input.getData();
+        NDArray array = ctx.getNDManager().create(data, new Shape(1, data.length));
+        return new NDList(array);
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public String processOutput(TranslatorContext ctx, NDList list) throws Exception {
+        return list.get(0).toStringArray()[0];
+    }
+}
diff --git a/api/src/main/java/ai/djl/modality/audio/translator/SpeechRecognitionTranslatorFactory.java b/api/src/main/java/ai/djl/modality/audio/translator/SpeechRecognitionTranslatorFactory.java
new file mode 100644
index 000000000..f09a415e6
--- /dev/null
+++ b/api/src/main/java/ai/djl/modality/audio/translator/SpeechRecognitionTranslatorFactory.java
@@ -0,0 +1,52 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.modality.audio.translator;
+
+import ai.djl.Model;
+import ai.djl.modality.audio.Audio;
+import ai.djl.translate.TranslateException;
+import ai.djl.translate.Translator;
+import ai.djl.translate.TranslatorFactory;
+import ai.djl.util.Pair;
+
+import java.lang.reflect.Type;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** A {@link TranslatorFactory} that creates a {@link SpeechRecognitionTranslator} instance. */
+public class SpeechRecognitionTranslatorFactory implements TranslatorFactory {
+
+    private static final Set<Pair<Type, Type>> SUPPORTED_TYPES = new HashSet<>();
+
+    static {
+        SUPPORTED_TYPES.add(new Pair<>(Audio.class, String.class));
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public Set<Pair<Type, Type>> getSupportedTypes() {
+        return SUPPORTED_TYPES;
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public Translator<?, ?> newInstance(
+            Class<?> input, Class<?> output, Model model, Map<String, ?> arguments)
+            throws TranslateException {
+        if (input == Audio.class && output == String.class) {
+            return new SpeechRecognitionTranslator();
+        }
+        throw new IllegalArgumentException("Unsupported input/output types.");
+    }
+}
diff --git a/api/src/main/java/ai/djl/modality/audio/translator/package-info.java b/api/src/main/java/ai/djl/modality/audio/translator/package-info.java
new file mode 100644
index 000000000..661af2334
--- /dev/null
+++ b/api/src/main/java/ai/djl/modality/audio/translator/package-info.java
@@ -0,0 +1,15 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+
+/** Contains utility classes for speech recognition processing. */
+package ai.djl.modality.audio.translator;
diff --git a/examples/docs/speech_recognition.md b/examples/docs/speech_recognition.md
new file mode 100644
index 000000000..0018455dd
--- /dev/null
+++ b/examples/docs/speech_recognition.md
@@ -0,0 +1,28 @@
+# Speech Recognition Example
+
+In this example, you learn how to use Speech Recognition using PyTorch.
+You can provide the model with a wav input file. 
+You can find the source code in [SpeechRecognition.java](https://github.com/deepjavalibrary/djl/blob/master/examples/src/main/java/ai/djl/examples/inference/SpeechRecognition.java).
+
+Example:
+
+Input audio: https://resources.djl.ai/audios/speech.wav
+
+Result:
+
+```text
+THE NEAREST SAID THE DISTRICT DOCTOR IS A GOOD ITALIAN ABBE WHO LIVES NEXT DOOR TO YOU SHALL I CALL ON HIM AS I PASS 
+```
+
+## Setup guide
+
+Follow [setup](../../docs/development/setup.md) to configure your development environment.
+
+### Run Inference
+
+```
+cd examples
+./gradlew run -Dmain=ai.djl.examples.inference.SpeechRecognition
+
+[INFO ] - Result: THE NEAREST SAID THE DISTRICT DOCTOR IS A GOOD ITALIAN ABBE WHO LIVES NEXT DOOR TO YOU SHALL I CALL ON HIM AS I PASS 
+```
diff --git a/examples/src/main/java/ai/djl/examples/inference/SpeechRecognition.java b/examples/src/main/java/ai/djl/examples/inference/SpeechRecognition.java
new file mode 100644
index 000000000..1561bab1b
--- /dev/null
+++ b/examples/src/main/java/ai/djl/examples/inference/SpeechRecognition.java
@@ -0,0 +1,73 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+
+package ai.djl.examples.inference;
+
+import ai.djl.ModelException;
+import ai.djl.inference.Predictor;
+import ai.djl.modality.audio.Audio;
+import ai.djl.modality.audio.AudioFactory;
+import ai.djl.modality.audio.translator.SpeechRecognitionTranslatorFactory;
+import ai.djl.repository.zoo.Criteria;
+import ai.djl.repository.zoo.ZooModel;
+import ai.djl.translate.TranslateException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+
+import javax.sound.sampled.UnsupportedAudioFileException;
+
+/**
+ * An example of inference using a speech recognition model.
+ *
+ * <p>See this <a
+ * href="https://github.com/deepjavalibrary/djl/blob/master/examples/docs/speech_recognition.md">doc</a>
+ * for information about this example.
+ */
+public final class SpeechRecognition {
+
+    public static final Logger logger = LoggerFactory.getLogger((SpeechRecognition.class));
+
+    private SpeechRecognition() {}
+
+    public static void main(String[] args)
+            throws UnsupportedAudioFileException, IOException, TranslateException, ModelException {
+        logger.info("Result: {}", predict());
+    }
+
+    public static String predict()
+            throws UnsupportedAudioFileException, IOException, ModelException, TranslateException {
+        // Load model.
+        // Wav2Vec2 model is a speech model that accepts a float array corresponding to the raw
+        // waveform of the speech signal.
+        String url = "https://resources.djl.ai/test-models/pytorch/wav2vec2.zip";
+        Criteria<Audio, String> criteria =
+                Criteria.builder()
+                        .setTypes(Audio.class, String.class)
+                        .optModelUrls(url)
+                        .optTranslatorFactory(new SpeechRecognitionTranslatorFactory())
+                        .optModelName("wav2vec2.ptl")
+                        .optEngine("PyTorch")
+                        .build();
+
+        // Read in audio file
+        String wave = "https://resources.djl.ai/audios/speech.wav";
+        Audio audio = AudioFactory.getInstance().fromUrl(wave);
+        try (ZooModel<Audio, String> model = criteria.loadModel();
+                Predictor<Audio, String> predictor = model.newPredictor()) {
+            return predictor.predict(audio);
+        }
+    }
+}


Test patch:
diff --git a/examples/src/test/java/ai/djl/examples/inference/SpeechRecognitionTest.java b/examples/src/test/java/ai/djl/examples/inference/SpeechRecognitionTest.java
new file mode 100644
index 000000000..a26d4dd7e
--- /dev/null
+++ b/examples/src/test/java/ai/djl/examples/inference/SpeechRecognitionTest.java
@@ -0,0 +1,40 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.examples.inference;
+
+import ai.djl.ModelException;
+import ai.djl.testing.TestRequirements;
+import ai.djl.translate.TranslateException;
+
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import java.io.IOException;
+
+import javax.sound.sampled.UnsupportedAudioFileException;
+
+public class SpeechRecognitionTest {
+
+    @Test
+    public void testSpeechRecognition()
+            throws ModelException, TranslateException, IOException, UnsupportedAudioFileException {
+        TestRequirements.nightly();
+        TestRequirements.engine("PyTorch");
+
+        String result = SpeechRecognition.predict();
+        Assert.assertEquals(
+                result,
+                "THE NEAREST SAID THE DISTRICT DOCTOR IS A GOOD ITALIAN ABBE WHO LIVES NEXT DOOR TO"
+                        + " YOU SHALL I CALL ON HIM AS I PASS ");
+    }
+}