Repository: google/clusterfuzz

Issue:
error deploying version 2.5.7
This is with python 3.7 on v2.5.7. 

Trace:
```Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/gunicorn/workers/gthread.py", line 92, in init_process
    super().init_process()
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/gunicorn/util.py", line 359, in import_app
    mod = importlib.import_module(module)
  File "/opt/python3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/srv/main.py", line 42, in <module>
    import server
  File "/srv/server.py", line 19, in <module>
    from google.cloud import ndb
  File "third_party/google/cloud/ndb/__init__.py", line 28, in <module>
    from google.cloud.ndb.client import Client
  File "third_party/google/cloud/ndb/client.py", line 18, in <module>
    import grpc
  File "third_party/grpc/__init__.py", line 23, in <module>
    from grpc._cython import cygrpc as _cygrpc
ImportError: cannot import name 'cygrpc' from 'grpc._cython' (third_party/grpc/_cython/__init__.py)```


Discussion:
Hi there! 

We haven't seen this before, but usually I've seen this in the past if there's a mismatch in the Python version used for for deployment vs the actual production environment. 

Are you deploying with Python 3.7 as well as using the same version in production? 
> Hi there!
> 
> We haven't seen this before, but usually I've seen this in the past if there's a mismatch in the Python version used for for deployment vs the actual production environment.
> 
> Are you deploying with Python 3.7 as well as using the same version in production

Yes, this is deployed with Python 3.7, I also tried with 3.8/3.9 but got the same errors. What's odd is that I can deploy a previous version without any issues using the same environment.
Do unit tests work locally? Does `cygrpc.cpython-37m-x86_64-linux-gnu.so` exist in `src/third_party/grpc/_cython` ? 


> Do unit tests work locally? 

The unit tests mostly works except for `test_profiler (tests.core.metrics.profiler_test.ProfilerTest)`
```
ImportError: cannot import name '_profiler' from 'googlecloudprofiler' (./clusterfuzz/src/third_party/googlecloudprofiler/__init__.py)
```
> Does `cygrpc.cpython-37m-x86_64-linux-gnu.so` exist in `src/third_party/grpc/_cython` ?

It exists within `deployment/linux-3.zip`, but it also contains the darwin version of it `cygrpc.cpython-37m-darwin.so`.

That's really strange -- we definitely are not able to reproduce this. 

The main change we made (that I can think of) which may affect this is https://github.com/google/clusterfuzz/commit/fb88290102f60c6ddcb3c5b38ea21d360f207b5f.

Would you be able to check if this is what's causing the issues for you?
After some testing, the issue seems to be related to naming the appengine requirements file `gae_requirements.txt` instead of `requirements.txt`, any idea why that'd matter?
I believe `gcloud app deploy` understands and uses `requirements.txt`, while we did not want this and installed/vendored our own third_party dependencies. 

So this definitely works if you just rename the `gae_requirements.txt` file to `requirements.txt` ?

We can experiment to see if we can just use `requirements.txt` for all App Engine deployments.

I took another look, and I believe the root cause is indeed due to the lack of `cygrpc.cpython-37m-x86_64-linux-gnu.so` in `src/appengine/third_party` while deploying from Mac OS. Seems like previously the issue was avoided as appengine was installing dependencies from `requirements.txt` and not using what's in third_party possibly.

So for deployment to work from a non-linux it seems like we'd have to download the linux version of grpcio into `third_party` during deployment
I see! Yes -- deployment was intended to only be supported from Linux. Sadly none of us use macOS for development so that path is not well tested. 
We can take a look into if can just use `requirements.txt` directly without trying to install the dependencies ourselves. 
> I believe `gcloud app deploy` understands and uses `requirements.txt`, while we did not want this and installed/vendored our own third_party dependencies.
> 
> So this definitely works if you just rename the `gae_requirements.txt` file to `requirements.txt` ?
> 
> We can experiment to see if we can just use `requirements.txt` for all App Engine deployments.

I believe requirements.txt would work, but there are some additional dependencies that needed to be pinned like werkzeug and pyparsing 

Resolving patch:
diff --git a/src/local/butler/common.py b/src/local/butler/common.py
--- a/src/local/butler/common.py
+++ b/src/local/butler/common.py
@@ -315,8 +315,7 @@ def install_dependencies(platform_name=None, is_reproduce_tool_setup=False):
   """Install dependencies for bots."""
   _pipfile_to_requirements('src', 'src/requirements.txt')
   # Hack: Use "dev-packages" to specify App Engine only packages.
-  _pipfile_to_requirements(
-      'src', 'src/appengine/gae_requirements.txt', dev=True)
+  _pipfile_to_requirements('src', 'src/appengine/requirements.txt', dev=True)
 
   _install_pip('src/requirements.txt', 'src/third_party')
   if platform_name:
@@ -325,8 +324,7 @@ def install_dependencies(platform_name=None, is_reproduce_tool_setup=False):
         'src/third_party',
         platform_name=platform_name)
 
-  _install_pip('src/appengine/gae_requirements.txt',
-               'src/appengine/third_party')
+  _install_pip('src/appengine/requirements.txt', 'src/appengine/third_party')
 
   # Only the previous dependencies are needed for reproduce tool installation.
   if is_reproduce_tool_setup:


Test patch: