Repository: deepjavalibrary/djl
Pull request #1974 (2022-08-29 16:08:26)

Add Audio API
## Description ##

Brief description of what this PR is about

Add Audio API

- If this change is a backward incompatible change, why must this change be made?
- Interesting edge cases to note here


codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1974?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1974](https://codecov.io/gh/deepjavalibrary/djl/pull/1974?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (3b75985) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `2.28%`.
> The diff coverage is `65.92%`.

```diff
@@             Coverage Diff              @@
##             master    #1974      +/-   ##
============================================
- Coverage     72.08%   69.80%   -2.29%     
- Complexity     5126     5833     +707     
============================================
  Files           473      581     +108     
  Lines         21970    25987    +4017     
  Branches       2351     2796     +445     
============================================
+ Hits          15838    18140    +2302     
- Misses         4925     6479    +1554     
- Partials       1207     1368     +161     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1974?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1974/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1974/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L011bHRpQm94UHJpb3IuamF2YQ==) | `76.00% <ø> (ø)` | |
| [...rc/main/java/ai/djl/modality/cv/output/Joints.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1974/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9Kb2ludHMuamF2YQ==) | `71.42% <ø> (ø)` | |
| [.../main/java/ai/djl/modality/cv/output/Landmark.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1974/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9MYW5kbWFyay5qYXZh) | `100.00% <ø> (ø)` | |
| [...main/java/ai/djl/modality/cv/output/Rectangle.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1974/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9SZWN0YW5nbGUuamF2YQ==) | `72.41% <0.00%> (ø)` | |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1974/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <0.00%> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1974/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1974/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../cv/translator/SemanticSegmentationTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1974/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU2VtYW50aWNTZWdtZW50YXRpb25UcmFuc2xhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../cv/translator/StyleTransferTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1974/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU3R5bGVUcmFuc2ZlclRyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | `40.00% <ø> (ø)` | |
| ... and [483 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1974/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)

frankfliu: Can you add some unit test?
xyang16: > 

Done
xyang16: > Can you add some unit test?

Done

Patch:
diff --git a/api/src/main/java/ai/djl/modality/audio/Audio.java b/api/src/main/java/ai/djl/modality/audio/Audio.java
new file mode 100644
index 000000000..4ad949845
--- /dev/null
+++ b/api/src/main/java/ai/djl/modality/audio/Audio.java
@@ -0,0 +1,73 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.modality.audio;
+
+/**
+ * {@code Audio} is a container of an audio in DJL. The raw data of the audio is wrapped in a float
+ * array.
+ */
+public class Audio {
+
+    private float[] data;
+    private float sampleRate;
+    private int channels;
+
+    /**
+     * Constructs a new {@code Audio} instance.
+     *
+     * @param data the wrapped float array data
+     */
+    public Audio(float[] data) {
+        this.data = data;
+    }
+
+    /**
+     * Constructs a new {@code Audio} instance.
+     *
+     * @param data the wrapped float array data
+     * @param sampleRate the sample rate
+     * @param channels number of channels
+     */
+    public Audio(float[] data, float sampleRate, int channels) {
+        this.data = data;
+        this.sampleRate = sampleRate;
+        this.channels = channels;
+    }
+
+    /**
+     * Returns the float array data.
+     *
+     * @return The float array data.
+     */
+    public float[] getData() {
+        return data;
+    }
+
+    /**
+     * Returns the sample rate.
+     *
+     * @return sample rate.
+     */
+    public float getSampleRate() {
+        return sampleRate;
+    }
+
+    /**
+     * Get the number of channels of an audio file.
+     *
+     * @return The number of channels of an audio file.
+     */
+    public int getChannels() {
+        return channels;
+    }
+}
diff --git a/api/src/main/java/ai/djl/modality/audio/AudioFactory.java b/api/src/main/java/ai/djl/modality/audio/AudioFactory.java
new file mode 100644
index 000000000..310134c57
--- /dev/null
+++ b/api/src/main/java/ai/djl/modality/audio/AudioFactory.java
@@ -0,0 +1,124 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.modality.audio;
+
+import ai.djl.ndarray.NDArray;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URI;
+import java.net.URL;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+
+/**
+ * {@code AudioFactory} contains audio creation mechanism on top of different platforms like PC and
+ * Android. System will choose appropriate Factory based on the supported audio type.
+ */
+public abstract class AudioFactory {
+
+    private static final Logger logger = LoggerFactory.getLogger(AudioFactory.class);
+
+    private static final String[] FACTORIES = {
+        "ai.djl.audio.FFmpegAudioFactory", "ai.djl.modality.audio.SampledAudioFactory"
+    };
+
+    private static AudioFactory factory = newInstance();
+
+    private static AudioFactory newInstance() {
+        for (String f : FACTORIES) {
+            try {
+                Class<? extends AudioFactory> clazz =
+                        Class.forName(f).asSubclass(AudioFactory.class);
+                return clazz.getDeclaredConstructor().newInstance();
+            } catch (ReflectiveOperationException e) {
+                logger.trace("", e);
+            }
+        }
+        throw new IllegalStateException("Failed to create AudioFactory!");
+    }
+
+    /**
+     * Gets new instance of Audio factory from the provided factory implementation.
+     *
+     * @return {@link AudioFactory}
+     */
+    public static AudioFactory getInstance() {
+        return factory;
+    }
+
+    /**
+     * Returns {@link Audio} from file.
+     *
+     * @param path the path to the audio
+     * @return {@link Audio}
+     * @throws IOException Audio not found or not readable
+     */
+    public abstract Audio fromFile(Path path) throws IOException;
+
+    /**
+     * Returns {@link Audio} from URL.
+     *
+     * @param url the URL to load from
+     * @return {@link Audio}
+     * @throws IOException URL is not valid.
+     */
+    public Audio fromUrl(URL url) throws IOException {
+        try (InputStream is = url.openStream()) {
+            return fromInputStream(is);
+        }
+    }
+
+    /**
+     * Returns {@link Audio} from URL.
+     *
+     * @param url the String represent URL to load from
+     * @return {@link Audio}
+     * @throws IOException URL is not valid.
+     */
+    public Audio fromUrl(String url) throws IOException {
+        URI uri = URI.create(url);
+        if (uri.isAbsolute()) {
+            return fromUrl(uri.toURL());
+        }
+        return fromFile(Paths.get(url));
+    }
+
+    /**
+     * Returns {@link Audio} from {@link InputStream}.
+     *
+     * @param is {@link InputStream}
+     * @return {@link Audio}
+     * @throws IOException image cannot be read from input stream.
+     */
+    public abstract Audio fromInputStream(InputStream is) throws IOException;
+
+    /**
+     * Returns {@link Audio} from raw data.
+     *
+     * @param data the raw data in float array form.
+     * @return {@link Audio}
+     */
+    public abstract Audio fromData(float[] data);
+
+    /**
+     * Returns {@link Audio} from {@link NDArray}.
+     *
+     * @param array the NDArray with CHW format
+     * @return {@link Audio}
+     */
+    public abstract Audio fromNDArray(NDArray array);
+}
diff --git a/api/src/main/java/ai/djl/modality/audio/SampledAudioFactory.java b/api/src/main/java/ai/djl/modality/audio/SampledAudioFactory.java
new file mode 100644
index 000000000..cdbeaccf2
--- /dev/null
+++ b/api/src/main/java/ai/djl/modality/audio/SampledAudioFactory.java
@@ -0,0 +1,117 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.modality.audio;
+
+import ai.djl.modality.cv.ImageFactory;
+import ai.djl.ndarray.NDArray;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.ShortBuffer;
+import java.nio.file.Path;
+
+import javax.sound.sampled.AudioFormat;
+import javax.sound.sampled.AudioInputStream;
+import javax.sound.sampled.AudioSystem;
+import javax.sound.sampled.UnsupportedAudioFileException;
+
+/**
+ * {@code SampledAudioFactory} is an implementation of {@link ImageFactory} using the Java Sampled
+ * Package.
+ *
+ * @see <a
+ *     href="http://google.com">https://docs.oracle.com/javase/tutorial/sound/sampled-overview.html</a>
+ */
+public class SampledAudioFactory extends AudioFactory {
+
+    /** {@inheritDoc} */
+    @Override
+    public Audio fromFile(Path path) throws IOException {
+        try (AudioInputStream ais = AudioSystem.getAudioInputStream(path.toFile())) {
+            AudioFormat format = ais.getFormat();
+            byte[] bytes = read(ais);
+            float[] floats =
+                    bytesToFloats(
+                            bytes,
+                            format.isBigEndian() ? ByteOrder.BIG_ENDIAN : ByteOrder.LITTLE_ENDIAN);
+            return new Audio(floats, format.getSampleRate(), format.getChannels());
+        } catch (UnsupportedAudioFileException e) {
+            throw new IOException("Unsupported Audio file", e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public Audio fromInputStream(InputStream is) throws IOException {
+        try (AudioInputStream ais = AudioSystem.getAudioInputStream(new BufferedInputStream(is))) {
+            AudioFormat format = ais.getFormat();
+            byte[] bytes = read(ais);
+            float[] floats =
+                    bytesToFloats(
+                            bytes,
+                            format.isBigEndian() ? ByteOrder.BIG_ENDIAN : ByteOrder.LITTLE_ENDIAN);
+            return new Audio(floats, format.getSampleRate(), format.getChannels());
+        } catch (UnsupportedAudioFileException e) {
+            throw new IOException("Unsupported Audio file", e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public Audio fromData(float[] data) {
+        return new Audio(data);
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public Audio fromNDArray(NDArray array) {
+        throw new UnsupportedOperationException("Not supported!");
+    }
+
+    private byte[] read(AudioInputStream ais) throws IOException {
+        AudioFormat format = ais.getFormat();
+        int frameSize = format.getFrameSize();
+
+        // Some audio formats may have unspecified frame size
+        if (frameSize == AudioSystem.NOT_SPECIFIED) {
+            frameSize = 1;
+        }
+
+        int size = (int) ais.getFrameLength() * frameSize;
+        byte[] ret = new byte[size];
+        byte[] buf = new byte[1024];
+        int offset = 0;
+        int read;
+        while ((read = ais.read(buf)) != -1) {
+            System.arraycopy(buf, 0, ret, offset, read);
+            offset += read;
+        }
+        return ret;
+    }
+
+    private float[] bytesToFloats(byte[] bytes, ByteOrder order) {
+        ShortBuffer buffer = ByteBuffer.wrap(bytes).order(order).asShortBuffer();
+        short[] shorts = new short[buffer.capacity()];
+        buffer.get(shorts);
+
+        // Feed in float values between -1.0f and 1.0f.
+        float[] floats = new float[shorts.length];
+        for (int i = 0; i < shorts.length; i++) {
+            floats[i] = ((float) shorts[i]) / (float) Short.MAX_VALUE;
+        }
+        return floats;
+    }
+}
diff --git a/api/src/main/java/ai/djl/modality/audio/package-info.java b/api/src/main/java/ai/djl/modality/audio/package-info.java
new file mode 100644
index 000000000..d9d2c2646
--- /dev/null
+++ b/api/src/main/java/ai/djl/modality/audio/package-info.java
@@ -0,0 +1,15 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+
+/** Contains utility classes for audio processing. */
+package ai.djl.modality.audio;
diff --git a/extensions/audio/src/main/java/ai/djl/audio/FFmpegAudioFactory.java b/extensions/audio/src/main/java/ai/djl/audio/FFmpegAudioFactory.java
new file mode 100644
index 000000000..ad6ffa071
--- /dev/null
+++ b/extensions/audio/src/main/java/ai/djl/audio/FFmpegAudioFactory.java
@@ -0,0 +1,94 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.audio;
+
+import ai.djl.modality.audio.Audio;
+import ai.djl.modality.audio.AudioFactory;
+import ai.djl.ndarray.NDArray;
+
+import org.bytedeco.javacv.FFmpegFrameGrabber;
+import org.bytedeco.javacv.Frame;
+import org.bytedeco.javacv.FrameGrabber;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ShortBuffer;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * {@code FFmpegAudioFactory} is a high performance implementation of {@link AudioFactory} using
+ * FFmpeg.
+ */
+public class FFmpegAudioFactory extends AudioFactory {
+
+    /** {@inheritDoc} */
+    @Override
+    public Audio fromFile(Path path) throws IOException {
+        try (FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(path.toFile())) {
+            grabber.start();
+            float[] floats = grab(grabber);
+            return new Audio(floats, grabber.getSampleRate(), grabber.getAudioChannels());
+        } catch (FrameGrabber.Exception e) {
+            throw new IOException("Unsupported Audio file", e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public Audio fromInputStream(InputStream is) throws IOException {
+        try (FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(is)) {
+            grabber.start();
+            float[] floats = grab(grabber);
+            return new Audio(floats, grabber.getSampleRate(), grabber.getAudioChannels());
+        } catch (FrameGrabber.Exception e) {
+            throw new IOException("Unsupported Audio file", e);
+        }
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public Audio fromData(float[] data) {
+        return new Audio(data);
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public Audio fromNDArray(NDArray array) {
+        throw new UnsupportedOperationException("Not supported!");
+    }
+
+    /**
+     * Grab frames from the audio using {@link FFmpegFrameGrabber}.
+     *
+     * @param grabber the {@link FFmpegFrameGrabber}.
+     * @return the float array read from the audio.
+     * @throws FFmpegFrameGrabber.Exception if error occurs
+     */
+    private float[] grab(FFmpegFrameGrabber grabber) throws FFmpegFrameGrabber.Exception {
+        List<Float> list = new ArrayList<>();
+        Frame frame;
+        while ((frame = grabber.grab()) != null) {
+            ShortBuffer buffer = (ShortBuffer) frame.samples[0];
+            for (int i = 0; i < buffer.limit(); i++) {
+                list.add(buffer.get() / (float) Short.MAX_VALUE);
+            }
+        }
+        float[] ret = new float[list.size()];
+        for (int i = 0; i < list.size(); i++) {
+            ret[i] = list.get(i);
+        }
+        return ret;
+    }
+}
diff --git a/extensions/audio/src/main/java/ai/djl/audio/dataset/AudioData.java b/extensions/audio/src/main/java/ai/djl/audio/dataset/AudioData.java
index dcd0d2cd5..60f580c54 100644
--- a/extensions/audio/src/main/java/ai/djl/audio/dataset/AudioData.java
+++ b/extensions/audio/src/main/java/ai/djl/audio/dataset/AudioData.java
@@ -18,7 +18,6 @@ import ai.djl.audio.processor.AudioProcessor;
 import ai.djl.audio.processor.LinearSpecgram;
 import ai.djl.ndarray.NDArray;
 import ai.djl.ndarray.NDManager;
-import ai.djl.repository.LocalRepository;
 
 import org.bytedeco.javacv.FFmpegFrameGrabber;
 import org.bytedeco.javacv.Frame;
@@ -44,7 +43,7 @@ import java.util.List;
  */
 public class AudioData {
 
-    private static final Logger logger = LoggerFactory.getLogger(LocalRepository.class);
+    private static final Logger logger = LoggerFactory.getLogger(AudioData.class);
 
     private int sampleRate;
     private int audioChannels;
@@ -63,13 +62,11 @@ public class AudioData {
     }
 
     /**
-     * Returns a good default {@link AudioData.Configuration} to use for the constructor with
-     * defaults.
+     * Returns a good default {@link Configuration} to use for the constructor with defaults.
      *
-     * @return a good default {@link AudioData.Configuration} to use for the constructor with
-     *     defaults.
+     * @return a good default {@link Configuration} to use for the constructor with defaults.
      */
-    public static AudioData.Configuration getDefaultConfiguration() {
+    public static Configuration getDefaultConfiguration() {
         float targetDb = -20f;
         float strideMs = 10f;
         float windowsMs = 20f;
@@ -78,9 +75,7 @@ public class AudioData {
                 Arrays.asList(
                         new AudioNormalizer(targetDb),
                         new LinearSpecgram(strideMs, windowsMs, sampleRate));
-        return new AudioData.Configuration()
-                .setProcessorList(defaultProcessors)
-                .setSampleRate(sampleRate);
+        return new Configuration().setProcessorList(defaultProcessors).setSampleRate(sampleRate);
     }
 
     /**
@@ -177,10 +172,7 @@ public class AudioData {
         return audioPaths.size();
     }
 
-    /**
-     * The configuration for creating a {@link AudioData} value in a {@link
-     * ai.djl.audio.dataset.AudioData}.
-     */
+    /** The configuration for creating a {@code AudioData} value in a {@code AudioData}. */
     public static final class Configuration {
 
         private int sampleRate;
@@ -211,13 +203,12 @@ public class AudioData {
         }
 
         /**
-         * Updates this {@link AudioData.Configuration} with the non-null values from another
-         * configuration.
+         * Updates this {@link Configuration} with the non-null values from another configuration.
          *
          * @param other the other configuration to use to update this
          * @return this configuration after updating
          */
-        public AudioData.Configuration update(AudioData.Configuration other) {
+        public Configuration update(AudioData.Configuration other) {
             processorList = other.processorList;
             return this;
         }
diff --git a/extensions/audio/src/main/java/ai/djl/audio/processor/AudioNormalizer.java b/extensions/audio/src/main/java/ai/djl/audio/processor/AudioNormalizer.java
index 55c4de4ed..5d76a5115 100644
--- a/extensions/audio/src/main/java/ai/djl/audio/processor/AudioNormalizer.java
+++ b/extensions/audio/src/main/java/ai/djl/audio/processor/AudioNormalizer.java
@@ -12,7 +12,7 @@
  */
 package ai.djl.audio.processor;
 
-import ai.djl.audio.AudioUtils;
+import ai.djl.audio.util.AudioUtils;
 import ai.djl.ndarray.NDArray;
 import ai.djl.ndarray.NDManager;
 
diff --git a/extensions/audio/src/main/java/ai/djl/audio/AudioUtils.java b/extensions/audio/src/main/java/ai/djl/audio/util/AudioUtils.java
similarity index 97%
rename from extensions/audio/src/main/java/ai/djl/audio/AudioUtils.java
rename to extensions/audio/src/main/java/ai/djl/audio/util/AudioUtils.java
index e2a2bcb7d..2e0603023 100644
--- a/extensions/audio/src/main/java/ai/djl/audio/AudioUtils.java
+++ b/extensions/audio/src/main/java/ai/djl/audio/util/AudioUtils.java
@@ -10,7 +10,7 @@
  * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
  * and limitations under the License.
  */
-package ai.djl.audio;
+package ai.djl.audio.util;
 
 import ai.djl.ndarray.NDArray;
 
diff --git a/extensions/audio/src/main/java/ai/djl/audio/util/package-info.java b/extensions/audio/src/main/java/ai/djl/audio/util/package-info.java
new file mode 100644
index 000000000..ec145a739
--- /dev/null
+++ b/extensions/audio/src/main/java/ai/djl/audio/util/package-info.java
@@ -0,0 +1,15 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+
+/** Contains audio utils for djl. */
+package ai.djl.audio.util;


Test patch:
diff --git a/api/src/test/java/ai/djl/modality/audio/AudioFactoryTest.java b/api/src/test/java/ai/djl/modality/audio/AudioFactoryTest.java
new file mode 100644
index 000000000..40ec26d27
--- /dev/null
+++ b/api/src/test/java/ai/djl/modality/audio/AudioFactoryTest.java
@@ -0,0 +1,81 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.modality.audio;
+
+import ai.djl.ndarray.NDArray;
+import ai.djl.ndarray.NDManager;
+import ai.djl.ndarray.types.Shape;
+import ai.djl.training.util.DownloadUtils;
+
+import org.testng.Assert;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+
+public class AudioFactoryTest {
+
+    private static final String URL = "https://resources.djl.ai/audios/test_01.wav";
+
+    @BeforeClass
+    public void setUp() throws IOException {
+        DownloadUtils.download(URL, "build/test/test_01.wav");
+    }
+
+    @Test
+    public void testFromFile() throws IOException {
+        Audio audio = AudioFactory.getInstance().fromFile(Paths.get("build/test/test_01.wav"));
+        Assert.assertEquals(audio.getSampleRate(), 16000f);
+        Assert.assertEquals(audio.getChannels(), 1);
+    }
+
+    @Test
+    public void testFromUrl() throws IOException {
+        Audio audio = AudioFactory.getInstance().fromUrl("build/test/test_01.wav");
+        Assert.assertEquals(audio.getSampleRate(), 16000f);
+        Assert.assertEquals(audio.getChannels(), 1);
+
+        audio = AudioFactory.getInstance().fromUrl(URL);
+        Assert.assertEquals(audio.getSampleRate(), 16000f);
+        Assert.assertEquals(audio.getChannels(), 1);
+    }
+
+    @Test
+    public void testFromInputStream() throws IOException {
+        try (InputStream is = Files.newInputStream(Paths.get("build/test/test_01.wav"))) {
+            Audio audio = AudioFactory.getInstance().fromInputStream(is);
+            Assert.assertEquals(audio.getSampleRate(), 16000f);
+            Assert.assertEquals(audio.getChannels(), 1);
+        }
+    }
+
+    @Test
+    public void testFromData() {
+        float[] data = {0.001f, 0.002f, 0.003f};
+        Audio audio = AudioFactory.getInstance().fromData(data);
+        Assert.assertEquals(audio.getData(), data);
+        Assert.assertEquals(audio.getSampleRate(), 0);
+        Assert.assertEquals(audio.getChannels(), 0);
+    }
+
+    @Test(expectedExceptions = UnsupportedOperationException.class)
+    public void testFromNDArray() {
+        try (NDManager manager = NDManager.newBaseManager()) {
+            NDArray array = manager.zeros(new Shape(1));
+            AudioFactory.getInstance().fromNDArray(array);
+        }
+    }
+}
diff --git a/api/src/test/java/ai/djl/modality/audio/package-info.java b/api/src/test/java/ai/djl/modality/audio/package-info.java
new file mode 100644
index 000000000..29a1b0cbc
--- /dev/null
+++ b/api/src/test/java/ai/djl/modality/audio/package-info.java
@@ -0,0 +1,15 @@
+/*
+ * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+
+/** Contains tests for the audio api. */
+package ai.djl.modality.audio;
diff --git a/extensions/audio/src/test/java/ai/djl/audio/FFmpegAudioFactoryTest.java b/extensions/audio/src/test/java/ai/djl/audio/FFmpegAudioFactoryTest.java
new file mode 100644
index 000000000..7cfe2ca63
--- /dev/null
+++ b/extensions/audio/src/test/java/ai/djl/audio/FFmpegAudioFactoryTest.java
@@ -0,0 +1,66 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.audio;
+
+import ai.djl.modality.audio.Audio;
+import ai.djl.modality.audio.AudioFactory;
+import ai.djl.ndarray.NDArray;
+import ai.djl.ndarray.NDManager;
+import ai.djl.ndarray.types.Shape;
+import ai.djl.training.util.DownloadUtils;
+
+import org.testng.Assert;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import java.io.IOException;
+import java.nio.file.Paths;
+
+public class FFmpegAudioFactoryTest {
+
+    private static final String URL = "https://resources.djl.ai/audios/test_01.wav";
+
+    @BeforeClass
+    public void setUp() throws IOException {
+        DownloadUtils.download(URL, "build/test/test_01.wav");
+    }
+
+    @Test
+    public void testFactory() throws IOException {
+        Audio audio = AudioFactory.getInstance().fromFile(Paths.get("build/test/test_01.wav"));
+        Assert.assertEquals(audio.getSampleRate(), 16000f);
+        Assert.assertEquals(audio.getChannels(), 1);
+
+        audio = AudioFactory.getInstance().fromUrl("build/test/test_01.wav");
+        Assert.assertEquals(audio.getSampleRate(), 16000f);
+        Assert.assertEquals(audio.getChannels(), 1);
+
+        audio = AudioFactory.getInstance().fromUrl(URL);
+        Assert.assertEquals(audio.getSampleRate(), 16000f);
+        Assert.assertEquals(audio.getChannels(), 1);
+
+        float[] data = {0.001f, 0.002f, 0.003f};
+        audio = AudioFactory.getInstance().fromData(data);
+        Assert.assertEquals(audio.getData(), data);
+        Assert.assertEquals(audio.getSampleRate(), 0);
+        Assert.assertEquals(audio.getChannels(), 0);
+    }
+
+    @Test(expectedExceptions = UnsupportedOperationException.class)
+    public void testFromNDArray() {
+        try (NDManager manager = NDManager.newBaseManager()) {
+            NDArray array = manager.zeros(new Shape(1));
+            AudioFactory.getInstance().fromNDArray(array);
+        }
+    }
+}
diff --git a/extensions/audio/src/test/java/ai/djl/audio/package-info.java b/extensions/audio/src/test/java/ai/djl/audio/package-info.java
new file mode 100644
index 000000000..24d79f6c5
--- /dev/null
+++ b/extensions/audio/src/test/java/ai/djl/audio/package-info.java
@@ -0,0 +1,13 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.audio;
diff --git a/extensions/audio/src/test/java/ai/djl/audio/processor/AudioProcessorTest.java b/extensions/audio/src/test/java/ai/djl/audio/processor/AudioProcessorTest.java
index 0c5894dd5..df9fa801f 100644
--- a/extensions/audio/src/test/java/ai/djl/audio/processor/AudioProcessorTest.java
+++ b/extensions/audio/src/test/java/ai/djl/audio/processor/AudioProcessorTest.java
@@ -14,8 +14,8 @@
 package ai.djl.audio.processor;
 
 import ai.djl.Device;
-import ai.djl.audio.AudioUtils;
 import ai.djl.audio.dataset.AudioData;
+import ai.djl.audio.util.AudioUtils;
 import ai.djl.ndarray.NDArray;
 import ai.djl.ndarray.NDManager;
 import ai.djl.ndarray.types.Shape;