Repository: deepjavalibrary/djl
Pull request #2639 (2023-06-07 18:05:26)

[fasttext] Adds maxOS M1 supports
Fixes: #2638

codecov-commenter: ## [Codecov](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
Patch coverage: **`71.38`**% and project coverage change: **`+1.50`** :tada:
> Comparison is base [(`bb5073f`)](https://app.codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) 72.08% compared to head [(`09c8ad8`)](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) 73.59%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please [install the Github App Integration](https://github.com/apps/codecov) for your organization. [Read more](https://about.codecov.io/blog/codecov-is-updating-its-github-integration/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff              @@
##             master    #2639      +/-   ##
============================================
+ Coverage     72.08%   73.59%   +1.50%     
- Complexity     5126     6959    +1833     
============================================
  Files           473      685     +212     
  Lines         21970    30392    +8422     
  Branches       2351     3147     +796     
============================================
+ Hits          15838    22366    +6528     
- Misses         4925     6500    +1575     
- Partials       1207     1526     +319     
```


| [Impacted Files](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L011bHRpQm94UHJpb3IuamF2YQ==) | `76.00% <ø> (ø)` | |
| [.../main/java/ai/djl/modality/cv/output/Landmark.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9MYW5kbWFyay5qYXZh) | `100.00% <ø> (ø)` | |
| [...djl/modality/cv/transform/RandomFlipLeftRight.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zZm9ybS9SYW5kb21GbGlwTGVmdFJpZ2h0LmphdmE=) | `25.00% <0.00%> (-25.00%)` | :arrow_down: |
| [...djl/modality/cv/transform/RandomFlipTopBottom.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zZm9ybS9SYW5kb21GbGlwVG9wQm90dG9tLmphdmE=) | `25.00% <0.00%> (-25.00%)` | :arrow_down: |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <0.00%> (-5.24%)` | :arrow_down: |
| [.../modality/cv/translator/ImageFeatureExtractor.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW1hZ2VGZWF0dXJlRXh0cmFjdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../ai/djl/modality/cv/translator/YoloTranslator.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1RyYW5zbGF0b3IuamF2YQ==) | `27.77% <0.00%> (+18.95%)` | :arrow_up: |
| [...ain/java/ai/djl/modality/cv/util/NDImageUtils.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3V0aWwvTkRJbWFnZVV0aWxzLmphdmE=) | `67.10% <0.00%> (+7.89%)` | :arrow_up: |
| [api/src/main/java/ai/djl/modality/nlp/Decoder.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L25scC9EZWNvZGVyLmphdmE=) | `63.63% <ø> (ø)` | |
| ... and [228 more](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

... and [354 files with indirect coverage changes](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2639?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/.github/workflows/native_s3_fasttext.yml b/.github/workflows/native_s3_fasttext.yml
index b1baef0a2..948ebabf4 100644
--- a/.github/workflows/native_s3_fasttext.yml
+++ b/.github/workflows/native_s3_fasttext.yml
@@ -72,3 +72,35 @@ jobs:
         run: |
           FASTTEXT_VERSION="$(cat gradle.properties | awk -F '=' '/fasttext_version/ {print $2}')"
           aws s3 sync extensions/fasttext/jnilib s3://djl-ai/publish/fasttext-${FASTTEXT_VERSION}/jnilib/
+
+  build-fasttext-jni-arm64-osx:
+    if: ${{ github.repository == 'deepjavalibrary/djl' && always() }}
+    runs-on: [ self-hosted, ARM64, macOS ]
+    steps:
+      - uses: actions/checkout@v3
+      - name: Set up JDK 11
+        uses: actions/setup-java@v3
+        with:
+          java-version: 11
+          distribution: corretto
+          architecture: aarch64
+      - uses: actions/cache@v3
+        with:
+          path: ~/.gradle/caches
+          key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}
+          restore-keys: |
+            ${{ runner.os }}-gradle-
+      - name: Release JNI prep
+        run: |
+          ./gradlew :extensions:fasttext:compileJNI
+          ./gradlew -Pjni :extensions:fasttext:test
+      - name: Configure AWS Credentials
+        uses: aws-actions/configure-aws-credentials@v1-node16
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: us-east-2
+      - name: Copy files to S3 with the AWS CLI
+        run: |
+          FASTTEXT_VERSION="$(cat gradle.properties | awk -F '=' '/fasttext_version/ {print $2}')"
+          aws s3 sync extensions/fasttext/jnilib s3://djl-ai/publish/fasttext-${FASTTEXT_VERSION}/jnilib/
diff --git a/extensions/fasttext/build.gradle b/extensions/fasttext/build.gradle
index 9bdd65add..7e5d844de 100644
--- a/extensions/fasttext/build.gradle
+++ b/extensions/fasttext/build.gradle
@@ -16,7 +16,8 @@ processResources {
         def url = "https://publish.djl.ai/fasttext-${fasttext_version}/jnilib/${djl_version}"
         def files = [
                 "linux-x86_64": "libjni_fasttext.so",
-                "osx-x86_64"  : "libjni_fasttext.dylib"
+                "osx-x86_64"  : "libjni_fasttext.dylib",
+                "osx-aarch64" : "libjni_fasttext.dylib"
         ]
         def jnilibDir = "${project.projectDir}/jnilib/${djl_version}"
         files.each { entry ->
diff --git a/extensions/fasttext/build.sh b/extensions/fasttext/build.sh
index 7366bad42..0b04d8171 100755
--- a/extensions/fasttext/build.sh
+++ b/extensions/fasttext/build.sh
@@ -4,22 +4,26 @@ set -e
 WORK_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 NUM_PROC=1
 if [[ -n $(command -v nproc) ]]; then
-    NUM_PROC=$(nproc)
+  NUM_PROC=$(nproc)
 elif [[ -n $(command -v sysctl) ]]; then
-    NUM_PROC=$(sysctl -n hw.ncpu)
+  NUM_PROC=$(sysctl -n hw.ncpu)
 fi
 PLATFORM=$(uname | tr '[:upper:]' '[:lower:]')
 
 VERSION=v0.9.2
 
 pushd "$WORK_DIR"
-if [ ! -d "fastText" ];
-then
+if [ ! -d "fastText" ]; then
   git clone https://github.com/facebookresearch/fastText.git -b $VERSION
+
+  if [[ $PLATFORM == 'darwin' ]]; then
+    if [[ $(uname -m) == 'arm64' ]]; then
+      sed -i '' -E "s/-march=native/-mcpu=apple-m1/g" fastText/CMakeLists.txt
+    fi
+  fi
 fi
 
-if [ ! -d "build" ];
-then
+if [ ! -d "build" ]; then
   mkdir build
 fi
 cd build
@@ -33,8 +37,13 @@ popd
 
 # for nightly ci
 if [[ $PLATFORM == 'darwin' ]]; then
-  mkdir -p build/jnilib/osx-x86_64
-  cp -f build/libjni_fasttext.dylib build/jnilib/osx-x86_64/
+  if [[ $(uname -m) == 'arm64' ]]; then
+    mkdir -p build/jnilib/osx-aarch64
+    cp -f build/libjni_fasttext.dylib build/jnilib/osx-aarch64/
+  else
+    mkdir -p build/jnilib/osx-x86_64
+    cp -f build/libjni_fasttext.dylib build/jnilib/osx-x86_64/
+  fi
 elif [[ $PLATFORM == 'linux' ]]; then
   mkdir -p build/jnilib/linux-x86_64
   cp -f build/libjni_fasttext.so build/jnilib/linux-x86_64/


Test patch:
diff --git a/extensions/fasttext/src/test/java/ai/djl/fasttext/CookingStackExchangeTest.java b/extensions/fasttext/src/test/java/ai/djl/fasttext/CookingStackExchangeTest.java
index 6517e8378..5b421ff43 100644
--- a/extensions/fasttext/src/test/java/ai/djl/fasttext/CookingStackExchangeTest.java
+++ b/extensions/fasttext/src/test/java/ai/djl/fasttext/CookingStackExchangeTest.java
@@ -57,7 +57,6 @@ public class CookingStackExchangeTest {
     @BeforeClass
     public void setUp() {
         TestRequirements.notWindows(); // fastText is not supported on windows
-        TestRequirements.notArm();
     }
 
     @Test