Repository: pytorch/executorch
Pull request #9483: Enable CoreML by default on macOS wheel builds
Author: login: jathu
Labels: name: CLA Signed
description: This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
name: module: build/install
description: Issues related to the cmake and buck2 builds, and to installing ExecuTorch
name: ciflow/trunk
name: topic: not user facing
Created: 2025-03-21T02:49:03Z | Merged: 2025-03-26T19:04:03Z

### Summary
Context: https://github.com/pytorch/executorch/pull/9481

* Include the `executorchcoreml` pybinding in the builds
* Remove separate installation option
* Turn on CoreML by default for macOS builds
* Add a dependency on coremltools for macOS

### Test plan

CI

```
$ rm -rf cmake-out pip-out dist && ./install_executorch.sh
$ ./examples/models/llama/install_requirements.sh
$ .ci/scripts/test_llama.sh -model stories110M -build_tool cmake -dtype fp32 -mode coreml
$ .ci/scripts/test_llama.sh -model stories110M -build_tool cmake -dtype fp32 -mode xnnpack+custom+quantize_kv
```

cc @larryliu0820 @lucylq


Diff:
diff --git a/backends/apple/coreml/compiler/coreml_preprocess.py b/backends/apple/coreml/compiler/coreml_preprocess.py
index c7828888ee5..e9afd819d94 100644
--- a/backends/apple/coreml/compiler/coreml_preprocess.py
+++ b/backends/apple/coreml/compiler/coreml_preprocess.py
@@ -16,8 +16,8 @@
 
 import coremltools as ct
 import coremltools.optimize as cto
-import executorchcoreml
 
+from executorch.backends.apple.coreml import executorchcoreml
 from executorch.exir.backend.backend_details import (
     BackendDetails,
     ExportedProgram,
diff --git a/backends/apple/coreml/runtime/inmemoryfs/setup.py b/backends/apple/coreml/runtime/inmemoryfs/setup.py
deleted file mode 100644
index c93022ed341..00000000000
--- a/backends/apple/coreml/runtime/inmemoryfs/setup.py
+++ /dev/null
@@ -1,52 +0,0 @@
-import os
-
-import pathlib
-import sys
-
-REPO_ROOT = pathlib.Path(__file__).resolve().parent.parent.parent.parent.parent.parent
-PYBIND11_DIR_PATH = REPO_ROOT / "third-party" / "pybind11"
-sys.path.append(str(PYBIND11_DIR_PATH.absolute()))
-
-from pybind11.setup_helpers import build_ext, Pybind11Extension
-from setuptools import setup
-
-__version__ = "0.0.1"
-
-cxx_std = int(os.environ.get("CMAKE_CXX_STANDARD", "17"))
-
-ext_modules = [
-    Pybind11Extension(
-        "executorchcoreml",
-        [
-            "../util/json_util.cpp",
-            "inmemory_filesystem.cpp",
-            "inmemory_filesystem_py.cpp",
-            "inmemory_filesystem_utils.cpp",
-            "memory_buffer.cpp",
-            "memory_stream.cpp",
-            "reversed_memory_stream.cpp",
-        ],
-        define_macros=[("VERSION_INFO", __version__)],
-        cxx_std=cxx_std,
-        extra_compile_args=["-mmacosx-version-min=10.15", "-g"],
-        include_dirs=[
-            "../../third-party/nlohmann_json/single_include",
-            ".",
-            "../util",
-        ],
-    ),
-]
-
-setup(
-    name="executorchcoreml",
-    version=__version__,
-    description="CoreML extension for executorch",
-    long_description="",
-    author="Apple Inc.",
-    ext_modules=ext_modules,
-    extras_require={"test": "pytest"},
-    cmdclass={"build_ext": build_ext},
-    include_package_data=True,
-    zip_safe=False,
-    python_requires=">=3.9",
-)
diff --git a/backends/apple/coreml/scripts/install_inmemoryfs.sh b/backends/apple/coreml/scripts/install_inmemoryfs.sh
deleted file mode 100644
index 1fb9dd1c4d5..00000000000
--- a/backends/apple/coreml/scripts/install_inmemoryfs.sh
+++ /dev/null
@@ -1,25 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright © 2023 Apple Inc. All rights reserved.
-#
-# This source code is licensed under the BSD-style license found in the
-# LICENSE file in the root directory of this source tree.
-
-SCRIPT_DIR_PATH="$(
-    cd -- "$(dirname "$0")" >/dev/null 2>&1
-    pwd -P
-)"
-
-EXECUTORCH_ROOT_PATH=$(realpath "$SCRIPT_DIR_PATH/../../../../")
-COREML_DIR_PATH="$EXECUTORCH_ROOT_PATH/backends/apple/coreml"
-
-red=`tput setaf 1`
-green=`tput setaf 2`
-
-echo "${green}ExecuTorch: Installing inmemoryfs extension."
-pip install "$COREML_DIR_PATH/runtime/inmemoryfs"
-STATUS=$?
-if [ $STATUS -ne 0 ]; then
-    echo "${red}ExecuTorch: Failed to install inmemoryfs extension."
-    exit 1
-fi
\ No newline at end of file
diff --git a/backends/apple/coreml/scripts/install_requirements.sh b/backends/apple/coreml/scripts/install_requirements.sh
index 140ba09c702..5b3f4e3f31a 100755
--- a/backends/apple/coreml/scripts/install_requirements.sh
+++ b/backends/apple/coreml/scripts/install_requirements.sh
@@ -10,6 +10,10 @@ SCRIPT_DIR_PATH="$(
     pwd -P
 )"
 
+# TODO(jathu): remove the need to fetch coremltools to build deps for coreml_executor_runner.
+# Keep this version in sync with: pyproject.toml
+COREMLTOOLS_VERSION="8.1"
+
 red=`tput setaf 1`
 green=`tput setaf 2`
 
@@ -24,7 +28,7 @@ rm -rf "$COREML_DIR_PATH/third-party"
 mkdir "$COREML_DIR_PATH/third-party"
 
 echo "${green}ExecuTorch: Cloning coremltools."
-git clone --depth 1 --branch 8.1 "https://github.com/apple/coremltools.git" $COREMLTOOLS_DIR_PATH
+git clone --depth 1 --branch "${COREMLTOOLS_VERSION}" "https://github.com/apple/coremltools.git" $COREMLTOOLS_DIR_PATH
 cd $COREMLTOOLS_DIR_PATH
 
 STATUS=$?
@@ -43,16 +47,7 @@ fi
 
 mkdir "$COREMLTOOLS_DIR_PATH/build"
 cmake -S "$COREMLTOOLS_DIR_PATH" -B "$COREMLTOOLS_DIR_PATH/build"
-cmake --build "$COREMLTOOLS_DIR_PATH/build" --parallel
-
-echo "${green}ExecuTorch: Installing coremltools."
-pip install "$COREMLTOOLS_DIR_PATH"
-
-STATUS=$?
-if [ $STATUS -ne 0 ]; then
-    echo "${red}ExecuTorch: Failed to install coremltools."
-    exit 1
-fi
+cmake --build "$COREMLTOOLS_DIR_PATH/build" --parallel --target mlmodel
 
 echo "${green}ExecuTorch: Cloning nlohmann."
 git clone https://github.com/nlohmann/json.git "$COREML_DIR_PATH/third-party/nlohmann_json"
@@ -62,8 +57,6 @@ if [ $STATUS -ne 0 ]; then
     exit 1
 fi
 
-sh "$COREML_DIR_PATH/scripts/install_inmemoryfs.sh"
-
 echo "${green}ExecuTorch: Copying protobuf files."
 mkdir -p "$COREML_DIR_PATH/runtime/sdk/format/"
 cp -rf "$PROTOBUF_FILES_DIR_PATH" "$COREML_DIR_PATH/runtime/sdk/format/"
diff --git a/examples/apple/coreml/scripts/extract_coreml_models.py b/examples/apple/coreml/scripts/extract_coreml_models.py
index d2812d37ab0..b3778a22625 100644
--- a/examples/apple/coreml/scripts/extract_coreml_models.py
+++ b/examples/apple/coreml/scripts/extract_coreml_models.py
@@ -10,12 +10,9 @@
 
 from typing import List, Optional
 
-import executorchcoreml
-
+from executorch.backends.apple.coreml import executorchcoreml
 from executorch.backends.apple.coreml.compiler import CoreMLBackend
-
 from executorch.exir._serialize._program import deserialize_pte_binary
-
 from executorch.exir.schema import (
     BackendDelegate,
     BackendDelegateDataReference,
diff --git a/pyproject.toml b/pyproject.toml
index 90640c8dbdd..b2bd5c06944 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -67,6 +67,8 @@ dependencies=[
   "sympy",
   "tabulate",
   "typing-extensions",
+  # Keep this version in sync with: ./backends/apple/coreml/scripts/install_requirements.sh
+  "coremltools==8.1; platform_system == 'Darwin'",
 ]
 
 [project.urls]
diff --git a/setup.py b/setup.py
index a1c94aee4e9..871fdf329c2 100644
--- a/setup.py
+++ b/setup.py
@@ -84,6 +84,10 @@ def _cmake_args_defines() -> Dict[str, str]:
     return result
 
 
+def _is_macos() -> bool:
+    return sys.platform == "darwin"
+
+
 class ShouldBuild:
     """Indicates whether to build various components."""
 
@@ -125,7 +129,7 @@ def pybindings(cls) -> bool:
 
     @classmethod
     def coreml(cls) -> bool:
-        return cls._is_cmake_arg_enabled("EXECUTORCH_BUILD_COREML", default=False)
+        return cls._is_cmake_arg_enabled("EXECUTORCH_BUILD_COREML", default=_is_macos())
 
     @classmethod
     def mps(cls) -> bool:
@@ -720,8 +724,14 @@ def run(self):
                 "-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON",  # add quantized ops to pybindings.
                 "-DEXECUTORCH_BUILD_KERNELS_QUANTIZED_AOT=ON",
             ]
+
             if ShouldBuild.training():
                 build_args += ["--target", "_training_lib"]
+
+            if ShouldBuild.coreml():
+                cmake_args += ["-DEXECUTORCH_BUILD_COREML=ON"]
+                build_args += ["--target", "executorchcoreml"]
+
             build_args += ["--target", "portable_lib"]
             # To link backends into the portable_lib target, callers should
             # add entries like `-DEXECUTORCH_BUILD_XNNPACK=ON` to the CMAKE_ARGS
@@ -855,6 +865,14 @@ def get_ext_modules() -> List[Extension]:
                     "executorch.extension.training.pybindings._training_lib",
                 )
             )
+        if ShouldBuild.coreml():
+            ext_modules.append(
+                BuiltExtension(
+                    src="executorchcoreml.*",
+                    src_dir="backends/apple/coreml",
+                    modpath="executorch.backends.apple.coreml.executorchcoreml",
+                )
+            )
     if ShouldBuild.llama_custom_ops():
         ext_modules.append(
             BuiltFile(


Reviews:
user: login: mergennachin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T15:44:32Z
user: login: mergennachin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T15:51:00Z
user: login: cymbalrush
body: Thank you!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T17:30:00Z
user: login: cymbalrush
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T17:32:40Z
user: login: mergennachin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T18:56:35Z
user: login: mergennachin
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T18:56:50Z
user: login: jathu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T19:03:30Z

Review comments:
diff_hunk: @@ -67,6 +67,8 @@ dependencies=[
   "sympy",
   "tabulate",
   "typing-extensions",
+  # Keep this version in sync with: ./backends/apple/coreml/scripts/install_requirements.sh
+  "coremltools==8.1; platform_system == 'Darwin'",
path: pyproject.toml
user: login: mergennachin
body: Niiice!
created_at: 2025-03-26T15:44:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,16 +47,7 @@ fi
 
 mkdir "$COREMLTOOLS_DIR_PATH/build"
 cmake -S "$COREMLTOOLS_DIR_PATH" -B "$COREMLTOOLS_DIR_PATH/build"
-cmake --build "$COREMLTOOLS_DIR_PATH/build" --parallel
-
-echo "${green}ExecuTorch: Installing coremltools."
-pip install "$COREMLTOOLS_DIR_PATH"
-
-STATUS=$?
-if [ $STATUS -ne 0 ]; then
-    echo "${red}ExecuTorch: Failed to install coremltools."
-    exit 1
-fi
+cmake --build "$COREMLTOOLS_DIR_PATH/build" --parallel --target mlmodel
path: backends/apple/coreml/scripts/install_requirements.sh
user: login: mergennachin
body: @YifanShenSZ @cymbalrush -- any reason why we still need to build mlmodel target from source, instead of using directly from pip install coremltools?
created_at: 2025-03-26T15:51:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,16 +47,7 @@ fi
 
 mkdir "$COREMLTOOLS_DIR_PATH/build"
 cmake -S "$COREMLTOOLS_DIR_PATH" -B "$COREMLTOOLS_DIR_PATH/build"
-cmake --build "$COREMLTOOLS_DIR_PATH/build" --parallel
-
-echo "${green}ExecuTorch: Installing coremltools."
-pip install "$COREMLTOOLS_DIR_PATH"
-
-STATUS=$?
-if [ $STATUS -ne 0 ]; then
-    echo "${red}ExecuTorch: Failed to install coremltools."
-    exit 1
-fi
+cmake --build "$COREMLTOOLS_DIR_PATH/build" --parallel --target mlmodel
path: backends/apple/coreml/scripts/install_requirements.sh
user: login: cymbalrush
body: It shouldn't be required any more but let me confirm
created_at: 2025-03-26T17:32:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,16 +47,7 @@ fi
 
 mkdir "$COREMLTOOLS_DIR_PATH/build"
 cmake -S "$COREMLTOOLS_DIR_PATH" -B "$COREMLTOOLS_DIR_PATH/build"
-cmake --build "$COREMLTOOLS_DIR_PATH/build" --parallel
-
-echo "${green}ExecuTorch: Installing coremltools."
-pip install "$COREMLTOOLS_DIR_PATH"
-
-STATUS=$?
-if [ $STATUS -ne 0 ]; then
-    echo "${red}ExecuTorch: Failed to install coremltools."
-    exit 1
-fi
+cmake --build "$COREMLTOOLS_DIR_PATH/build" --parallel --target mlmodel
path: backends/apple/coreml/scripts/install_requirements.sh
user: login: mergennachin
body: @jathu 

Okay, let's not block it, but create an GH issue to track this...
created_at: 2025-03-26T18:56:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -43,16 +47,7 @@ fi
 
 mkdir "$COREMLTOOLS_DIR_PATH/build"
 cmake -S "$COREMLTOOLS_DIR_PATH" -B "$COREMLTOOLS_DIR_PATH/build"
-cmake --build "$COREMLTOOLS_DIR_PATH/build" --parallel
-
-echo "${green}ExecuTorch: Installing coremltools."
-pip install "$COREMLTOOLS_DIR_PATH"
-
-STATUS=$?
-if [ $STATUS -ne 0 ]; then
-    echo "${red}ExecuTorch: Failed to install coremltools."
-    exit 1
-fi
+cmake --build "$COREMLTOOLS_DIR_PATH/build" --parallel --target mlmodel
path: backends/apple/coreml/scripts/install_requirements.sh
user: login: jathu
body: Done https://github.com/pytorch/executorch/issues/9651
created_at: 2025-03-26T19:03:30Z
author_association: CONTRIBUTOR

PR comments:
user: login: pytorch-bot[bot]
created_at: 2025-03-21T02:49:07Z
body: <!-- drci-comment-start -->

## :link: Helpful Links
### :test_tube: See artifacts and rendered test results at [hud.pytorch.org/pr/pytorch/executorch/9483](https://hud.pytorch.org/pr/pytorch/executorch/9483)
* :page_facing_up: Preview [Python docs built from this PR](https://docs-preview.pytorch.org/pytorch/executorch/9483/index.html)

Note: Links to docs will display an error until the docs builds have been completed.


## :x: 2 New Failures, 1 Pending
As of commit 6488b4455c14454be8d11d4765495df15feed740 with merge base 265b9b784eef08acf36e5140f9e51005dc210505 (<sub><sub><img alt="image" width=70 src="https://img.shields.io/date/1742999270?label=&color=FFFFFF&style=flat-square"></sub></sub>):
<details open><summary><b>NEW FAILURES</b> - The following jobs have failed:</summary><p>

* [trunk / test-arm-backend-delegation / linux-job](https://hud.pytorch.org/pr/pytorch/executorch/9483#39454562013) ([gh](https://github.com/pytorch/executorch/actions/runs/14087279001/job/39454562013))
    `RuntimeError: Command docker exec -t a163b9ffe417b5c07bb25e13c9071ae6c161ead5abbee6a108f5d7953d6e0f12 /exec failed with exit code 1`
* [trunk / test-arm-reference-delegation / linux-job](https://hud.pytorch.org/pr/pytorch/executorch/9483#39454565185) ([gh](https://github.com/pytorch/executorch/actions/runs/14087279001/job/39454565185))
    `backends/arm/test/ops/test_bmm.py::TestBMM::test_bmm_single_input_tosa_MI_2`
</p></details>


This comment was automatically generated by Dr. CI and updates every 15 minutes.
<!-- drci-comment-end -->
author_association: NONE
user: login: mergennachin
created_at: 2025-03-25T15:46:59Z
body: coreml has its own install_requirements.sh script that downloads a coremltools repo. What happens to that?
author_association: CONTRIBUTOR
user: login: jathu
created_at: 2025-03-25T16:01:21Z
body: > coreml has its own install_requirements.sh script that downloads a coremltools repo. What happens to that?

In the next diff, I will remove that and install it through pip for only macOS
author_association: CONTRIBUTOR
user: login: jathu
created_at: 2025-03-26T15:07:51Z
body: I will update the docs on usage of the install script in another PR.
author_association: CONTRIBUTOR
user: login: mergennachin
created_at: 2025-03-26T15:42:19Z
body: This is awesome! 

cc @YifanShenSZ  @cymbalrush 
author_association: CONTRIBUTOR