Repository: pantsbuild/pants

Issue:
Show a useful error message if you try to build lambdas on MacOS
Right now it fails in all sorts of subtle ways relating to failures to resolve requirements and picking weird interpreters (e.g., due to the issue explained in https://github.com/pantsbuild/pex/issues/957).

It hypothetically possible to build lambdas on MacOS if all transitive requirements have prebuilt linux wheels, but that is too narrow a case to support, and it's subtle and tricky to get right even if we wanted to. Better to just say no in a helpful and explicit way.


Discussion:
@benjyw Started looking at this. Were you thinking a logger.error message or raising an exception - I'm not sure what the convention is elsewhere. 
@riisi  I think you want to raise an exception, in order to not proceed with the build.
Edit: including the helpful message in the exception..
Got it, thanks. Will draft something soon.

On Mon, 22 Nov 2021 at 10:33, Andreas Stenius ***@***.***>
wrote:

> @riisi <https://github.com/riisi> I think you want to raise an exception,
> in order to not proceed with the build.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pantsbuild/pants/issues/11941#issuecomment-975381618>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHGSB3ER2YC772B5VMBS62LUNIL7FANCNFSM43D3HQ5Q>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>

I think you could use `FallibleProcessResult` instead of `ProcessResult` so that the error is not eager, and then inspect the error code and error message to determine dynamically if you should mention macOS as a culprit? 

Or as a first step, you could always include that macOS warning if the user is on macOS, but use "hedge words"—like "may" or "it's possible"—to make clear that that is only one possible reason the build failed.
I don't think parsing error messages is great, there are so many different errors that can occur. On error we can check the platform and mention MacOS as a likely culprit. 
Managed to replicate this locally (i.e. building a lambda with package
without a wheel on mac). I was initially assuming that it would be possible
to try/catch in the lambda rule, however, I now understand that depends on
`FallibleProcessResult`.

It looks like the `build_pex` rule would need to be modified - in which
case should this warning be logged there, and be applicable for packaging
pex as well as lambdas?

Unless there's another way to catch rule engine errors that I'm missing?


On Mon, 22 Nov 2021 at 17:48, Benjy Weinberger ***@***.***>
wrote:

> I don't think parsing error messages is great, there are so many different
> errors that can occur. On error we can check the platform and mention MacOS
> as a likely culprit.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pantsbuild/pants/issues/11941#issuecomment-975771917>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHGSB3BRCZWAAMONGMHDGV3UNJ65DANCNFSM43D3HQ5Q>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>

Yeah, `build_pex` would need to be modified to use `FallibleProcessResult`, which is almost never what we want. It's convenient that the result is a built PEX and callers don't need to handle the failure case.

Instead, I think I still recommend this:

> Or as a first step, you could always include that macOS warning if the user is on macOS, but use "hedge words"—like "may" or "it's possible"—to make clear that that is only one possible reason the build failed.

Basically use `logger.warning()` with a variant of the warning info box at https://www.pantsbuild.org/docs/awslambda-python#step-3-run-package. 

(We assume some users do not read the docs. Our UX and error messages should be good enough that you can figure things out on your own. At least, that's the aspiration!)

Resolving patch:
diff --git a/src/python/pants/backend/awslambda/python/rules.py b/src/python/pants/backend/awslambda/python/rules.py
--- a/src/python/pants/backend/awslambda/python/rules.py
+++ b/src/python/pants/backend/awslambda/python/rules.py
@@ -27,6 +27,7 @@
     PackageFieldSet,
 )
 from pants.core.target_types import FileSourceField
+from pants.engine.platform import Platform
 from pants.engine.process import ProcessResult
 from pants.engine.rules import Get, MultiGet, collect_rules, rule
 from pants.engine.target import (
@@ -52,8 +53,21 @@ class PythonAwsLambdaFieldSet(PackageFieldSet):
 
 @rule(desc="Create Python AWS Lambda", level=LogLevel.DEBUG)
 async def package_python_awslambda(
-    field_set: PythonAwsLambdaFieldSet, lambdex: Lambdex, union_membership: UnionMembership
+    field_set: PythonAwsLambdaFieldSet,
+    lambdex: Lambdex,
+    platform: Platform,
+    union_membership: UnionMembership,
 ) -> BuiltPackage:
+    if platform.is_macos:
+        logger.warning(
+            "AWS Lambdas built on macOS may fail to build. If your lambda uses any third-party"
+            " dependencies without binary wheels (bdist) for Linux available, it will fail to"
+            " build. If this happens, you will either need to update your dependencies to only use"
+            " dependencies with pre-built wheels, or find a Linux environment to run ./pants"
+            " package. (See https://realpython.com/python-wheels/ for more about wheels.)\n\n(If"
+            " the build does not raise an exception, it's safe to use macOS.)"
+        )
+
     output_filename = field_set.output_path.value_or_default(
         # Lambdas typically use the .zip suffix, so we use that instead of .pex.
         file_ending="zip",
@@ -63,12 +77,12 @@ async def package_python_awslambda(
     # (Running the "hello world" lambda in the example code will report the platform, and can be
     # used to verify correctness of these platform strings.)
     py_major, py_minor = field_set.runtime.to_interpreter_version()
-    platform = f"linux_x86_64-cp-{py_major}{py_minor}-cp{py_major}{py_minor}"
+    platform_str = f"linux_x86_64-cp-{py_major}{py_minor}-cp{py_major}{py_minor}"
     # set pymalloc ABI flag - this was removed in python 3.8 https://bugs.python.org/issue36707
     if py_major <= 3 and py_minor < 8:
-        platform += "m"
+        platform_str += "m"
     if (py_major, py_minor) == (2, 7):
-        platform += "u"
+        platform_str += "u"
 
     additional_pex_args = (
         # Ensure we can resolve manylinux wheels in addition to any AMI-specific wheels.
@@ -81,7 +95,7 @@ async def package_python_awslambda(
         addresses=[field_set.address],
         internal_only=False,
         output_filename=output_filename,
-        platforms=PexPlatforms([platform]),
+        platforms=PexPlatforms([platform_str]),
         additional_args=additional_pex_args,
         additional_lockfile_args=additional_pex_args,
     )
diff --git a/src/python/pants/backend/google_cloud_function/python/rules.py b/src/python/pants/backend/google_cloud_function/python/rules.py
--- a/src/python/pants/backend/google_cloud_function/python/rules.py
+++ b/src/python/pants/backend/google_cloud_function/python/rules.py
@@ -28,6 +28,7 @@
     PackageFieldSet,
 )
 from pants.core.target_types import FileSourceField
+from pants.engine.platform import Platform
 from pants.engine.process import ProcessResult
 from pants.engine.rules import Get, MultiGet, collect_rules, rule
 from pants.engine.target import (
@@ -56,8 +57,19 @@ class PythonGoogleCloudFunctionFieldSet(PackageFieldSet):
 async def package_python_google_cloud_function(
     field_set: PythonGoogleCloudFunctionFieldSet,
     lambdex: Lambdex,
+    platform: Platform,
     union_membership: UnionMembership,
 ) -> BuiltPackage:
+    if platform.is_macos:
+        logger.warning(
+            "Google Cloud Functions built on macOS may fail to build. If your function uses any"
+            " third-party dependencies without binary wheels (bdist) for Linux available, it will"
+            " fail to build. If this happens, you will either need to update your dependencies to"
+            " only use dependencies with pre-built wheels, or find a Linux environment to run"
+            " ./pants package. (See https://realpython.com/python-wheels/ for more about"
+            " wheels.)\n\n(If the build does not raise an exception, it's safe to use macOS.)"
+        )
+
     output_filename = field_set.output_path.value_or_default(
         # Cloud Functions typically use the .zip suffix, so we use that instead of .pex.
         file_ending="zip",
@@ -67,10 +79,10 @@ async def package_python_google_cloud_function(
     # (Running the "hello world" cloud function in the example code will report the platform, and can be
     # used to verify correctness of these platform strings.)
     py_major, py_minor = field_set.runtime.to_interpreter_version()
-    platform = f"linux_x86_64-cp-{py_major}{py_minor}-cp{py_major}{py_minor}"
+    platform_str = f"linux_x86_64-cp-{py_major}{py_minor}-cp{py_major}{py_minor}"
     # set pymalloc ABI flag - this was removed in python 3.8 https://bugs.python.org/issue36707
     if py_major <= 3 and py_minor < 8:
-        platform += "m"
+        platform_str += "m"
 
     additional_pex_args = (
         # Ensure we can resolve manylinux wheels in addition to any AMI-specific wheels.
@@ -83,7 +95,7 @@ async def package_python_google_cloud_function(
         addresses=[field_set.address],
         internal_only=False,
         output_filename=output_filename,
-        platforms=PexPlatforms([platform]),
+        platforms=PexPlatforms([platform_str]),
         additional_args=additional_pex_args,
         additional_lockfile_args=additional_pex_args,
     )


Test patch:
diff --git a/src/python/pants/backend/awslambda/python/rules_test.py b/src/python/pants/backend/awslambda/python/rules_test.py
--- a/src/python/pants/backend/awslambda/python/rules_test.py
+++ b/src/python/pants/backend/awslambda/python/rules_test.py
@@ -3,6 +3,7 @@
 
 from __future__ import annotations
 
+import sys
 from io import BytesIO
 from textwrap import dedent
 from zipfile import ZipFile
@@ -72,7 +73,9 @@ def create_python_awslambda(
     "major_minor_interpreter",
     all_major_minor_python_versions(Lambdex.default_interpreter_constraints),
 )
-def test_create_hello_world_lambda(rule_runner: RuleRunner, major_minor_interpreter: str) -> None:
+def test_create_hello_world_lambda(
+    rule_runner: RuleRunner, major_minor_interpreter: str, caplog
+) -> None:
     rule_runner.write_files(
         {
             "src/python/foo/bar/hello_world.py": dedent(
@@ -105,6 +108,8 @@ def handler(event, context):
     names = set(zipfile.namelist())
     assert "lambdex_handler.py" in names
     assert "foo/bar/hello_world.py" in names
+    if sys.platform == "darwin":
+        assert "AWS Lambdas built on macOS may fail to build." in caplog.text
 
 
 def test_warn_files_targets(rule_runner: RuleRunner, caplog) -> None:
diff --git a/src/python/pants/backend/google_cloud_function/python/rules_test.py b/src/python/pants/backend/google_cloud_function/python/rules_test.py
--- a/src/python/pants/backend/google_cloud_function/python/rules_test.py
+++ b/src/python/pants/backend/google_cloud_function/python/rules_test.py
@@ -3,6 +3,7 @@
 
 from __future__ import annotations
 
+import sys
 from io import BytesIO
 from textwrap import dedent
 from zipfile import ZipFile
@@ -81,7 +82,9 @@ def create_python_google_cloud_function(
     "major_minor_interpreter",
     all_major_minor_python_versions(Lambdex.default_interpreter_constraints),
 )
-def test_create_hello_world_lambda(rule_runner: RuleRunner, major_minor_interpreter: str) -> None:
+def test_create_hello_world_lambda(
+    rule_runner: RuleRunner, major_minor_interpreter: str, caplog
+) -> None:
     rule_runner.write_files(
         {
             "src/python/foo/bar/hello_world.py": dedent(
@@ -115,6 +118,8 @@ def handler(event, context):
     names = set(zipfile.namelist())
     assert "main.py" in names
     assert "foo/bar/hello_world.py" in names
+    if sys.platform == "darwin":
+        assert "Google Cloud Functions built on macOS may fail to build." in caplog.text
 
 
 def test_warn_files_targets(rule_runner: RuleRunner, caplog) -> None: