Repository: OpenAssetIO/OpenAssetIO (language: python)

Issue:
Add .pyi stub files to Python package
## What

Following on from #747, add `.pyi` stub files to our Python package covering the `_openassetio` Python extension module.

## Why

IDEs can use `.pyi` stub files located in the package to greatly enhance code-completion and documentation for Python C extension modules.

## Acceptance Criteria

- Installing via `pip install` includes an `_openassetio` directory containing `.pyi` stub files (satisfying [PEP 561](https://peps.python.org/pep-0561/)), which PyCharm (and other IDEs) can detect and use to enhance code completion.
- `.pyi` stubs are included in the binary wheels on PyPI.
-  `.pyi` stubs are installed when doing a from-source install i.e. `pip install .`
- ~~`.pyi` stubs are installed (to the source tree, just like the extension module itself) when doing a `--editable` install.~~
- Installing via `cmake --install`  bundles the `.pyi` stubs in the install tree.

# Notes

See #747 for investigation with proposed implementation.


Discussion / hints:
You'll also need a `py.typed` file as documented in https://peps.python.org/pep-0561/.
> You'll also need a `py.typed` file as documented in https://peps.python.org/pep-0561/.

Thanks for this. :thinking: At first glance, the PEP appeared to me to focus on out-of-tree stub files.  But on a second look it does seem to cover the case of bundling stubs alongside. PyCharm doesn't seem to care about the existence of a `py.typed` file, but it is the standard so we should include it.
`py.typed` will tell type checkers that the project has types. PyCharm might (and probably) bases itself on this and other logic to cover a lot of cases and improve the IDE experience. For example, see https://mypy.readthedocs.io/en/stable/installed_packages.html#creating-pep-561-compatible-packages.

Fix patch:
diff --git a/.github/workflows/build-wheels.yml b/.github/workflows/build-wheels.yml
index 98850999d..9fed1e2c6 100644
--- a/.github/workflows/build-wheels.yml
+++ b/.github/workflows/build-wheels.yml
@@ -72,6 +72,8 @@ jobs:
           PIP_VERBOSE: 1
           # Required as we make use of c++17 features
           MACOSX_DEPLOYMENT_TARGET: 10.15
+          # Ensure .pyi stub tests won't be skipped
+          OPENASSETIO_TEST_ENABLE_PYTHON_STUBGEN: 1
 
       - uses: actions/upload-artifact@v3
         with:
diff --git a/CMakeLists.txt b/CMakeLists.txt
index fff7fcb04..db553e0c4 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -127,6 +127,8 @@ if (OPENASSETIO_ENABLE_PYTHON)
         " discoverability and prevent overwrite by package managers such as pip")
     option(OPENASSETIO_ENABLE_PYTHON_INSTALL_DIST_INFO
         "${OPENASSETIO_ENABLE_PYTHON_INSTALL_DIST_INFO_desc}" ON)
+
+    option(OPENASSETIO_ENABLE_PYTHON_STUBGEN "Enable Python .pyi stub file generation" ON)
 endif ()
 
 # Enable C bindings, built as a separate library that depends on the
@@ -323,6 +325,7 @@ message(STATUS "Enable Python module build         = ${OPENASSETIO_ENABLE_PYTHON
 if (OPENASSETIO_ENABLE_PYTHON)
     message(STATUS "Python install .dist-info metadata = ${OPENASSETIO_ENABLE_PYTHON_INSTALL_DIST_INFO}")
     message(STATUS "Python relative install dir        = ${OPENASSETIO_PYTHON_SITEDIR}")
+    message(STATUS "Generate .pyi stubs                = ${OPENASSETIO_ENABLE_PYTHON_STUBGEN}")
 endif ()
 message(STATUS "Create test targets                = ${OPENASSETIO_ENABLE_TESTS}")
 message(STATUS "Create Python venv during tests    = ${OPENASSETIO_ENABLE_PYTHON_TEST_VENV}")
diff --git a/CMakePresets.json b/CMakePresets.json
index 99ea59ea2..669a20437 100644
--- a/CMakePresets.json
+++ b/CMakePresets.json
@@ -31,7 +31,10 @@
         "enable-sanitizers",
         "enable-targets-all",
         "enable-tests"
-      ]
+      ],
+      "cacheVariables": {
+        "OPENASSETIO_ENABLE_PYTHON_STUBGEN": "OFF"
+      }
     },
     {
       "name": "test",
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 9ddffea44..e8942dab0 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -27,6 +27,11 @@ v1.0.0-beta.x.y
   logger.
   [#1014](https://github.com/OpenAssetIO/OpenAssetIO/issues/1014)
 
+### Improvements
+
+- Added `.pyi` stub files to the Python package to aid IDE code
+  completion for Python bindings of C++ types.
+  [#1252](https://github.com/OpenAssetIO/OpenAssetIO/issues/1252)
 
 v1.0.0-beta.2.1
 ---------------
diff --git a/cmake/Testing.cmake b/cmake/Testing.cmake
index b853e65fe..4e0e26e27 100644
--- a/cmake/Testing.cmake
+++ b/cmake/Testing.cmake
@@ -108,8 +108,11 @@ if (OPENASSETIO_ENABLE_PYTHON)
     #-------------------------------------------------------------------
     # Common environment variables for pytest tests.
 
-    set(_pytest_env
-        OPENASSETIO_TEST_CPP_PLUGINS_SUBDIR=${OPENASSETIO_TEST_CPP_PLUGINS_SUBDIR})
+    set(
+        _pytest_env
+        OPENASSETIO_TEST_CPP_PLUGINS_SUBDIR=${OPENASSETIO_TEST_CPP_PLUGINS_SUBDIR}
+        OPENASSETIO_TEST_ENABLE_PYTHON_STUBGEN=$<BOOL:${OPENASSETIO_ENABLE_PYTHON_STUBGEN}>
+    )
 
     #-------------------------------------------------------------------
     # Gather ASan-specific environment variables to prepend to the
diff --git a/resources/build/Dockerfile b/resources/build/Dockerfile
index a8b8a8a6b..c88f7e75a 100644
--- a/resources/build/Dockerfile
+++ b/resources/build/Dockerfile
@@ -74,8 +74,9 @@ COPY --from=openassetio-dependencies /usr/local/cmake/pcre2-config-version.cmake
 COPY --from=openassetio-dependencies /usr/local/lib64/cmake/trompeloeil /usr/local/lib64/cmake/trompeloeil
 COPY --from=openassetio-dependencies /usr/local/include/trompeloeil.hpp /usr/local/include/trompeloeil.hpp
 
-# Update CMake. See cmake_minimum_required in top-level CMakeLists.txt.
-RUN pip install cmake==3.28.3
+# * Update CMake. See cmake_minimum_required in top-level CMakeLists.txt.
+# * Add pybind11-stubgen for generating .pyi stub files
+RUN pip install cmake==3.28.3 pybind11-stubgen==2.5.1
 
 LABEL org.opencontainers.image.name="openassetio-build"
 LABEL org.opencontainers.image.title="OpenAssetIO VFX CY2022 Build Docker Image"
diff --git a/resources/build/bootstrap-ubuntu-20.04.sh b/resources/build/bootstrap-ubuntu-20.04.sh
index 0ea22aa14..258e5d154 100644
--- a/resources/build/bootstrap-ubuntu-20.04.sh
+++ b/resources/build/bootstrap-ubuntu-20.04.sh
@@ -8,7 +8,7 @@ sudo apt-get update
 sudo apt-get install -y build-essential pkgconf clang-format-12 clang-tidy-12 python3-pip ccache
 
 # Install additional build tools.
-sudo pip3 install -r "$WORKSPACE/resources/build/requirements.txt"
+pip3 install -r "$WORKSPACE/resources/build/requirements.txt"
 # Use explicit predictable conan root path, where packages are cached.
 export CONAN_USER_HOME="$HOME/conan"
 # Create default conan profile so we can configure it before install.
diff --git a/resources/build/requirements.txt b/resources/build/requirements.txt
index fbea7e958..32ad34d4d 100644
--- a/resources/build/requirements.txt
+++ b/resources/build/requirements.txt
@@ -2,3 +2,4 @@ conan==1.59.0
 cmake==3.28.3
 ninja==1.10.2.3
 pip>=21.3
+pybind11-stubgen==2.5.1
diff --git a/src/openassetio-python/cmodule/CMakeLists.txt b/src/openassetio-python/cmodule/CMakeLists.txt
index fac04b101..83e5fd9f0 100644
--- a/src/openassetio-python/cmodule/CMakeLists.txt
+++ b/src/openassetio-python/cmodule/CMakeLists.txt
@@ -124,3 +124,102 @@ if (OPENASSETIO_ENABLE_TESTS)
     target_compile_definitions(openassetio-python-module PRIVATE OPENASSETIO_ENABLE_TESTS)
     target_link_libraries(openassetio-python-module PRIVATE openassetio-python-module-test)
 endif ()
+
+
+#-----------------------------------------------------------------------
+# Generate .pyi stubs
+#
+# pybind11-stubgen will create openassetio/_openassetio subdirectories
+# in the build output directory. We use this knowledge to assemble a
+# Python pseudo-package in the build output directory, such that stubgen
+# can import the module to do its thing then dump its output alongside.
+
+if (OPENASSETIO_ENABLE_PYTHON_STUBGEN)
+    # Check pybind11-stubgen is available.
+    execute_process(
+        COMMAND "${Python_EXECUTABLE}" -m pybind11_stubgen --help
+        OUTPUT_VARIABLE _pybind11_stubgen_output
+        ERROR_VARIABLE _pybind11_stubgen_output
+        RESULT_VARIABLE _pybind11_stubgen_exit_code
+    )
+
+    # Fatal error if pybind11-stubgen isn't available.
+    if (NOT _pybind11_stubgen_exit_code EQUAL "0")
+        message(
+            FATAL_ERROR
+            "OPENASSETIO_ENABLE_PYTHON_STUBGEN=${OPENASSETIO_ENABLE_PYTHON_STUBGEN} but"
+            " pybind11-stubgen not found: status=${_pybind11_stubgen_exit_code}:"
+            " ${_pybind11_stubgen_output}"
+        )
+    endif()
+
+    # Create temporary pseudo-package directory.
+    add_custom_command(
+        TARGET openassetio-python-module POST_BUILD
+        COMMAND
+        "${CMAKE_COMMAND}" -E make_directory "${CMAKE_CURRENT_BINARY_DIR}/openassetio"
+    )
+
+    if (WIN32)
+        # On Windows, copy openassetio.dll dependency into the
+        # pseudo-package directory, so that stubgen can import
+        # _openassetio.
+        # Note that this must be done _after_ the build, otherwise
+        # Windows (Visual Studio CMake generator) complains that the
+        # build directory already exists.
+        add_custom_command(
+            TARGET openassetio-python-module POST_BUILD
+            COMMAND
+            "${CMAKE_COMMAND}" -E copy_if_different
+            "$<TARGET_FILE:openassetio-core>"
+            "${CMAKE_CURRENT_BINARY_DIR}/openassetio/"
+        )
+    endif()
+
+    # Execute commands to generate .pyi stubs.
+    add_custom_command(
+        TARGET openassetio-python-module POST_BUILD
+        COMMAND
+        "${CMAKE_COMMAND}" -E echo "Generating .pyi stubs with pybind11-stubgen..."
+        # Copy the Python extension module under the pseudo-package
+        # directory.
+        COMMAND
+        "${CMAKE_COMMAND}" -E copy_if_different
+        "$<TARGET_FILE:openassetio-python-module>"
+        "${CMAKE_CURRENT_BINARY_DIR}/openassetio/"
+        # Ensure we have a py.typed file at the root of our package, to
+        # signal to IDEs that stubs exist.
+        COMMAND
+        "${CMAKE_COMMAND}" -E touch "${CMAKE_CURRENT_BINARY_DIR}/openassetio/py.typed"
+        # Execute pybind11-stubgen, modifying PYTHONPATH so it can
+        # locate the openassetio._openassetio module.
+        COMMAND
+        "${CMAKE_COMMAND}" -E env
+        --modify "PYTHONPATH=path_list_prepend:${CMAKE_CURRENT_BINARY_DIR}"
+        --
+        "${Python_EXECUTABLE}" -m pybind11_stubgen
+        # Fail the build and abort if any errors generating stubs.
+        --exit-code
+        # For whatever reason, stubgen fails to resolve PathType.
+        --enum-class-locations PathType:openassetio._openassetio.utils
+        -o "${CMAKE_CURRENT_BINARY_DIR}" openassetio._openassetio
+        VERBATIM
+    )
+
+    # Add stub files to the Python extension module installation
+    # component, so that
+    # `cmake --install --component openassetio-python-module` will
+    # include the stubs.
+    install(
+        # pybind11-stubgen generates stubs for openassetio._openassetio
+        # under an `openassetio/_openassetio` directory structure.
+        DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/openassetio/_openassetio"
+        DESTINATION "${_install_subdir}"
+        COMPONENT openassetio-python-module
+    )
+    install(
+        FILES "${CMAKE_CURRENT_BINARY_DIR}/openassetio/py.typed"
+        DESTINATION "${_install_subdir}"
+        COMPONENT openassetio-python-module
+    )
+endif ()
diff --git a/src/openassetio-python/pyproject.toml b/src/openassetio-python/pyproject.toml
index 99df02886..c7375d8f8 100644
--- a/src/openassetio-python/pyproject.toml
+++ b/src/openassetio-python/pyproject.toml
@@ -7,7 +7,9 @@ requires = [
     # CMake 3.29 PyPI package requires importlib_metadata, which is not
     # available in a fresh Python 3.7 environment.
     "cmake==3.28.3",
-    "ninja>=1.10.2.4"
+    "ninja>=1.10.2.4",
+    # For generating .pyi stub files.
+    "pybind11-stubgen==2.5.1"
 ]
 build-backend = "setuptools.build_meta"
 
@@ -141,9 +143,9 @@ target-version = ["py39"]
 [tool.cibuildwheel]
 test-requires = ["pytest==7.4.4", "pytest-subtests==0.11.0"]
 test-command = "pytest {package}/tests/package"
+environment-pass = ["PIP_VERBOSE", "OPENASSETIO_TEST_ENABLE_PYTHON_STUBGEN"]
 
 [tool.cibuildwheel.linux]
 # Linux runs in a docker container, with the project at top level
 before-build = "resources/build/bootstrap-cibuildwheel-manylinux-2014.sh"
 environment = { CMAKE_TOOLCHAIN_FILE=".conan/conan_paths.cmake" }
-environment-pass = ["PIP_VERBOSE"]
diff --git a/src/openassetio-python/setup.py b/src/openassetio-python/setup.py
index b147ede59..fb0d7592a 100644
--- a/src/openassetio-python/setup.py
+++ b/src/openassetio-python/setup.py
@@ -96,6 +96,7 @@ def __cmake(self, args: List[str]):
 setup(
     packages=find_packages(where="package"),
     package_dir={"": "package"},
+    package_data={"": ["py.typed", "_openassetio/*.pyi"]},
     ext_modules=[Extension("openassetio._openassetio", sources=[])],
     cmdclass={"build_ext": build_ext},
     # See pyproject.toml for other metadata fields.


Test patch:
diff --git a/src/openassetio-python/tests/cmodule/resources/_testutils/gilTest.cpp b/src/openassetio-python/tests/cmodule/resources/_testutils/gilTest.cpp
index 8c9a36adb..f345981b5 100644
--- a/src/openassetio-python/tests/cmodule/resources/_testutils/gilTest.cpp
+++ b/src/openassetio-python/tests/cmodule/resources/_testutils/gilTest.cpp
@@ -212,6 +212,11 @@ void registerRunInThread(py::module_& mod) {
       [](const std::function<void()>& func) { std::async(std::launch::async, func).get(); },
       py::arg("func"), py::call_guard<py::gil_scoped_release>());
 
+  py::class_<Flag, PyFlag>{mod, "Flag"}
+      .def(py::init())
+      .def("get", &Flag::get)
+      .def("set", &Flag::set);
+
   mod.def(
       "flagInThread",
       [](Flag& flag) {
@@ -221,11 +226,6 @@ void registerRunInThread(py::module_& mod) {
       },
       py::arg("func"), py::call_guard<py::gil_scoped_release>());
 
-  py::class_<Flag, PyFlag>{mod, "Flag"}
-      .def(py::init())
-      .def("get", &Flag::get)
-      .def("set", &Flag::set);
-
   auto gil = mod.def_submodule("gil");
 
   gil.def("wrapInThreadedManagerInterface", &ThreadedManagerInterface::make);
diff --git a/src/openassetio-python/tests/package/_openassetio/test_pyi.py b/src/openassetio-python/tests/package/_openassetio/test_pyi.py
new file mode 100644
index 000000000..be47ff4bc
--- /dev/null
+++ b/src/openassetio-python/tests/package/_openassetio/test_pyi.py
@@ -0,0 +1,76 @@
+#
+#   Copyright 2024 The Foundry Visionmongers Ltd
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+#
+"""
+Smoke test for .pyi stub files
+"""
+# pylint: disable=missing-function-docstring,invalid-name
+# pylint: disable=redefined-outer-name
+import ast
+import os
+import pathlib
+
+import pytest
+
+import openassetio
+
+
+expected_pyi_files = (
+    "__init__.pyi",
+    "access.pyi",
+    "constants.pyi",
+    "errors.pyi",
+    "hostApi.pyi",
+    "log.pyi",
+    "managerApi.pyi",
+    "pluginSystem.pyi",
+    "trait.pyi",
+    "utils.pyi",
+)
+
+
+def test_expected_pyi_files_generated(pyi_dir: pathlib.Path):
+    actual_pyi_files = set(p.name for p in pyi_dir.glob("*") if p.name != "_testutils")
+    assert actual_pyi_files == set(expected_pyi_files)
+
+
+@pytest.mark.parametrize("pyi_filename", expected_pyi_files)
+def test_pyi_files_have_valid_python_ast(pyi_dir: pathlib.Path, pyi_filename):
+    # Cannot import the module, since in pybind11-stubgen v2.5.1 the
+    # ordering of statements causes errors. E.g. exception classes
+    # inheriting from OpenAssetIOException come before the definition of
+    # OpenAssetIOException in the .pyi file. So just check the AST is
+    # valid.
+    with open(pyi_dir / pyi_filename, encoding="utf-8") as pyi_file:
+        ast.parse(pyi_file.read())
+
+
+def test_py_typed_exists():
+    assert pathlib.Path(openassetio.__file__).with_name("py.typed").is_file()
+
+
+@pytest.fixture(scope="module", autouse=True)
+def skip_if_stubs_disabled(pyi_dir):
+    """
+    Disable stub file tests if stubgen is not explicitly enabled and
+    stubs directory does not exist.
+    """
+    if os.environ.get("OPENASSETIO_TEST_ENABLE_PYTHON_STUBGEN") != "1" and not pyi_dir.is_dir():
+        pytest.skip("Skipping .pyi stub tests as stubs directory not found")
+
+
+@pytest.fixture(scope="session")
+def pyi_dir():
+    return pathlib.Path(openassetio.__file__).with_name("_openassetio")