Repository: pantsbuild/pants

Issue:
google_cloud_function backend: __path__ attribute not found on 'main' while trying to find 'main.handler'
**Describe the bug**

Pants version: 2.12.0

package build with `pants.backend.google_cloud_function.python` does not work.


```
python_google_cloud_function(
    name="cloud_function",
    runtime="python39",
    handler="function.py:main",
    type="http",
)
```
Error when deploy into GCP cloud function:
```
ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Function failed on loading user code. This is likely due to a bug in the user code. Error message: Error: please examine your function logs to see the error cause: https://cloud.google.com/functions/docs/monitoring/logging#viewing_logs. Additional troubleshooting documentation can be found at https://cloud.google.com/functions/docs/troubleshooting#logging. Please visit https://cloud.google.com/functions/docs/troubleshooting for in-depth troubleshooting documentation.
```
Checking run log:
```
Traceback (most recent call last): File "/opt/python3.9/lib/python3.9/importlib/util.py", line 96, in find_spec parent_path = parent.__path__ AttributeError: module 'main' has no attribute '__path__'

The above exception was the direct cause of the following exception:
Traceback (most recent call last): 
File "/opt/python3.9/lib/python3.9/pkgutil.py", line 495, in find_loader spec = importlib.util.find_spec(fullname) 
File "/opt/python3.9/lib/python3.9/importlib/util.py", line 98, in find_spec raise ModuleNotFoundError( ModuleNotFoundError: __path__ attribute not found on 'main' while trying to find 'main.handler'

The above exception was the direct cause of the following exception:

[...] Line 481, in get_loader return find_loader(fullname) File "/opt/python3.9/lib/python3.9/pkgutil.py", line 501, in find_loader raise ImportError(msg.format(fullname, type(ex), ex)) from ex 
ImportError: Error while finding loader for 'main.handler' (<class 'ModuleNotFoundError'>: __path__ attribute not found on 'main' while trying to find 'main.handler')
```

A minimal reproducible repo: https://github.com/stephentt42/rp-issue-pantsbuild-01 You can see deploy script in `deploy.sh`


I really appreciate any help.

**OS**
* Linux



Discussion:
Thanks for providing the example repository - that is a huge help.

I repro:
```
$ GCP_PROJECT=pants-issues-16242 GCS_BUCKET=rp-issue-pantsbuild-01 ./deploy.sh 
17:58:32.53 [INFO] Wrote dist/cloud_function.zip
    Runtime: python39
    Handler: main.handler
Copying file://dist/cloud_function.zip [Content-Type=application/zip]...
/ [1 files][472.5 KiB/472.5 KiB]                                                
Operation completed over 1 objects/472.5 KiB.                                    
API [cloudfunctions.googleapis.com] not enabled on project [1032983878945]. Would you like to enable and retry (this will take a few minutes)? (y/N)?  y

Enabling service [cloudfunctions.googleapis.com] on project [1032983878945]...
Operation "operations/acf.p2-1032983878945-37f800c3-13db-4cd3-b047-867978183111" finished successfully.
API [cloudbuild.googleapis.com] not enabled on project [pants-issues-16242]. Would you like to enable and retry (this will take a few minutes)? (y/N)?  y

Enabling service [cloudbuild.googleapis.com] on project [pants-issues-16242]...
Operation "operations/acf.p2-1032983878945-0a042946-234e-4a5d-8a79-e694d04b9f80" finished successfully.
Deploying function (may take a while - up to 2 minutes)...⠧                                                                                                                                                                                                                     
For Cloud Build Logs, visit: https://console.cloud.google.com/cloud-build/builds;region=us-central1/97bcdcb1-6990-4c90-a2cc-a555a92d71f6?project=1032983878945
Deploying function (may take a while - up to 2 minutes)...failed.                                                                                                                                                                                                               
ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Function failed on loading user code. This is likely due to a bug in the user code. Error message: Error: please examine your function logs to see the error cause: https://cloud.google.com/functions/docs/monitoring/logging#viewing_logs. Additional troubleshooting documentation can be found at https://cloud.google.com/functions/docs/troubleshooting#logging. Please visit https://cloud.google.com/functions/docs/troubleshooting for in-depth troubleshooting documentation.
```

And the logs show:
```
2022-07-21 18:00:43.887 MDT
topstops6o0bhe82pw0h Traceback (most recent call last): File "/layers/google.python.pip/pip/bin/functions-framework", line 8, in <module> sys.exit(_cli()) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/_cli.py", line 37, in _cli app = create_app(target, source, signature_type) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/__init__.py", line 263, in create_app _app = flask.Flask(target, template_folder=template_folder) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 397, in __init__ super().__init__( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/scaffold.py", line 113, in __init__ root_path = get_root_path(self.import_name) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/helpers.py", line 721, in get_root_path loader = pkgutil.get_loader(import_name) File "/opt/python3.9/lib/python3.9/pkgutil.py", line 481, in get_loader return find_loader(fullname) File "/opt/python3.9/lib/python3.9/pkgutil.py", line 501, in find_loader raise ImportError(msg.format(fullname, type(ex), ex)) from ex ImportError: Error while finding loader for 'main.handler' (<class 'ModuleNotFoundError'>: __path__ attribute not found on 'main' while trying to find 'main.handler')
Traceback (most recent call last): File "/layers/google.python.pip/pip/bin/functions-framework", line 8, in <module> sys.exit(_cli()) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/click/core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/_cli.py", line 37, in _cli app = create_app(target, source, signature_type) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/__init__.py", line 263, in create_app _app = flask.Flask(target, template_folder=template_folder) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 397, in __init__ super().__init__( File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/scaffold.py", line 113, in __init__ root_path = get_root_path(self.import_name) File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/helpers.py", line 721, in get_root_path loader = pkgutil.get_loader(import_name) File "/opt/python3.9/lib/python3.9/pkgutil.py", line 481, in get_loader return find_loader(fullname) File "/opt/python3.9/lib/python3.9/pkgutil.py", line 501, in find_loader raise ImportError(msg.format(fullname, type(ex), ex)) from ex ImportError: Error while finding loader for 'main.handler' (<class 'ModuleNotFoundError'>: __path__ attribute not found on 'main' while trying to find 'main.handler')
```

But with this diff:
```diff
diff --git a/deploy.sh b/deploy.sh
index 66e6cd4..6c7c747 100755
--- a/deploy.sh
+++ b/deploy.sh
@@ -6,6 +6,6 @@ set -o pipefail
 gsutil cp dist/cloud_function.zip gs://$GCS_BUCKET/
 gcloud functions deploy topstops \
     --source gs://$GCS_BUCKET/cloud_function.zip \
-    --entry-point main.handler \
+    --entry-point handler \
     --runtime python39 --trigger-http \
     --allow-unauthenticated --project $GCP_PROJECT
```

Success:
```
$ GCP_PROJECT=pants-issues-16242 GCS_BUCKET=rp-issue-pantsbuild-01 ./deploy.sh 
18:14:17.74 [INFO] Wrote dist/cloud_function.zip
    Runtime: python39
    Handler: main.handler
Copying file://dist/cloud_function.zip [Content-Type=application/zip]...
/ [1 files][472.5 KiB/472.5 KiB]                                                
Operation completed over 1 objects/472.5 KiB.                                    
Deploying function (may take a while - up to 2 minutes)...⠛                                                                                                                                                                                                                     
For Cloud Build Logs, visit: https://console.cloud.google.com/cloud-build/builds;region=us-central1/9e05c598-4298-43c1-b110-9dfc83c51ee2?project=1032983878945
Deploying function (may take a while - up to 2 minutes)...done.                                                                                                                                                                                                                 
availableMemoryMb: 256
buildId: 9e05c598-4298-43c1-b110-9dfc83c51ee2
buildName: projects/1032983878945/locations/us-central1/builds/9e05c598-4298-43c1-b110-9dfc83c51ee2
dockerRegistry: CONTAINER_REGISTRY
entryPoint: handler
httpsTrigger:
  securityLevel: SECURE_OPTIONAL
  url: https://us-central1-pants-issues-16242.cloudfunctions.net/topstops
ingressSettings: ALLOW_ALL
labels:
  deployment-tool: cli-gcloud
name: projects/pants-issues-16242/locations/us-central1/functions/topstops
runtime: python39
serviceAccountEmail: pants-issues-16242@appspot.gserviceaccount.com
sourceArchiveUrl: gs://rp-issue-pantsbuild-01/cloud_function.zip
status: ACTIVE
timeout: 60s
updateTime: '2022-07-22T00:16:01.829Z'
versionId: '4'
```

So the issue is in your `--entry-point` argument. It appears GCF for Python mandates the entrypoint will live in main.py and so the `main` module is assumed and should be left out. Just give your function name and all is well.

@stephentt42 I'm going to close this as answered, but please speak up if you have further questions or difficulties getting this working.
Aha, @stephentt42 is it this bit of documentation that led you astray?: https://www.pantsbuild.org/docs/google-cloud-function-python#step-4-upload-to-google-cloud
Thank you for your help. It works as expected.

Maybe in [this docs](https://www.pantsbuild.org/docs/google-cloud-function-python) the line:

> You must specify the handler as `main.handler`. 

create some confusion.

But I am truly grateful for your help. Thanks a lot. :D 

Hm, so what should the docs be? I'm going to re-open this issue because it sounds like we still have a "docs bug"
Per my investigation above it seems like `s/main.handler/handler/`. That said, my investigation was narrow and directly tied to @stephentt42's sample repo and as a GCF total noob. What we need is someone who actually knows the toolsets to definitively suss this. Barring that, it needs one of us noobs to invest more time playing with GCF and making sure this is actually generally correct advice.
A continual pitfall in Pants is not knowing the tools we support well and / or dogfooding that support we add.
OK. This is all pretty baroque, but the function need not be in a top-level `main.py`, it just defaults that way. It can be in any file iff:
1. You configure a `GOOGLE_FUNCTION_SOURCE` build environment variable (`gcloud functions deploy {--build-env-vars-file,--set-build-env-vars}`).
2. That file's parent directory is acceptable as the `sys.path` entry for locating other requirements in the deployment zip.

Either way, using the default top-level zip file of `main.py` or some other file in the zip pointed to by the `GOOGLE_FUNCTION_SOURCE` build env var, the handler is expected to be just the function name within that file.

The practical upshot is you really need to be using a top-level `main.py` which is what Pants Lambdex GCF integration sets up; so this is a simple doc change in the end.

Resolving patch:
diff --git a/src/python/pants/backend/google_cloud_function/python/rules.py b/src/python/pants/backend/google_cloud_function/python/rules.py
--- a/src/python/pants/backend/google_cloud_function/python/rules.py
+++ b/src/python/pants/backend/google_cloud_function/python/rules.py
@@ -151,7 +151,7 @@ async def package_python_google_cloud_function(
         ProcessResult,
         VenvPexProcess(
             lambdex_pex,
-            argv=("build", "-M", "main.py", "-e", handler.val, output_filename),
+            argv=("build", "-M", "main.py", "-H", "handler", "-e", handler.val, output_filename),
             input_digest=pex_result.digest,
             output_files=(output_filename,),
             description=f"Setting up handler in {output_filename}",
@@ -162,9 +162,15 @@ async def package_python_google_cloud_function(
     if field_set.runtime.value:
         extra_log_data.append(("Runtime", field_set.runtime.value))
     extra_log_data.extend(("Complete platform", path) for path in complete_platforms)
-    # The GCP-facing handler function is always main.handler, which is the
-    # wrapper injected by lambdex that manages invocation of the actual handler.
-    extra_log_data.append(("Handler", "main.handler"))
+    # The GCP-facing handler function is always `main.handler` (We pass `-M main.py -H handler` to
+    # Lambdex to ensure this), which is the wrapper injected by Lambdex that manages invocation of
+    # the actual user-supplied handler function. This arrangement works well since GCF assumes the
+    # handler function is housed in `main.py` in the root of the zip (you can re-direct this by
+    # setting a `GOOGLE_FUNCTION_SOURCE` Google Cloud build environment variable; e.g.:
+    # `gcloud functions deploy {--build-env-vars-file,--set-build-env-vars}`, but it's non-trivial
+    # to do this right or with intended effect) and the handler name you configure GCF with is just
+    # the unqualified function name, which we log here.
+    extra_log_data.append(("Handler", "handler"))
 
     first_column_width = 4 + max(len(header) for header, _ in extra_log_data)
     artifact = BuiltPackageArtifact(


Test patch:
diff --git a/src/python/pants/backend/google_cloud_function/python/rules_test.py b/src/python/pants/backend/google_cloud_function/python/rules_test.py
--- a/src/python/pants/backend/google_cloud_function/python/rules_test.py
+++ b/src/python/pants/backend/google_cloud_function/python/rules_test.py
@@ -103,7 +103,7 @@ def complete_platform(rule_runner: RuleRunner) -> bytes:
         {
             "pex_exe/BUILD": dedent(
                 """\
-                python_requirement(name="req", requirements=["pex==2.1.66"])
+                python_requirement(name="req", requirements=["pex==2.1.99"])
                 pex_binary(dependencies=[":req"], script="pex")
                 """
             ),
@@ -162,7 +162,7 @@ def handler(event, context):
         expected_extra_log_lines=(
             "              Runtime: python37",
             "    Complete platform: src/python/foo/bar/platform.json",
-            "              Handler: main.handler",
+            "              Handler: handler",
         ),
         extra_args=[f"--lambdex-interpreter-constraints=['=={major_minor_interpreter}.*']"],
     )
@@ -225,7 +225,7 @@ def handler(event, context):
         Address("src/py/project", target_name="lambda"),
         expected_extra_log_lines=(
             "    Runtime: python37",
-            "    Handler: main.handler",
+            "    Handler: handler",
         ),
     )
     assert caplog.records