Repository: ray-project/ray

Issue:
[autoscaler] GCP error missing required parameter body
<!--
General questions should be asked on the mailing list ray-dev@googlegroups.com.
Questions about how to use Ray should be asked on
[StackOverflow](https://stackoverflow.com/questions/tagged/ray).

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux
- **Ray installed from (source or binary)**:
- **Ray version**: 0.6.2
- **Python version**: 3.6
- **Exact command to reproduce**: ray up gcp_trainer.yaml

<!--
You can obtain the Ray version with

python -c "import ray; print(ray.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
Code worked well until today (no update). I got an error at the beginning after the getIamPolicy function in /autoscaler/gcp/config.py. I have all the rights / permissions in my GCP. 
The yaml file is similar to the example-full.yaml

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->
file_cache is unavailable when using oauth2client >= 4.0.0
Traceback (most recent call last):
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py", line 36, in autodetect
    from google.appengine.api import memcache
ModuleNotFoundError: No module named 'google.appengine'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in <module>
    from oauth2client.contrib.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in <module>
    from oauth2client.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py", line 41, in autodetect
    from . import file_cache
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in <module>
    'file_cache is unavailable when using oauth2client >= 4.0.0')
ImportError: file_cache is unavailable when using oauth2client >= 4.0.0
URL being requested: GET https://www.googleapis.com/discovery/v1/apis/cloudresourcemanager/v1/rest
/opt/tools/anaconda3/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
file_cache is unavailable when using oauth2client >= 4.0.0
Traceback (most recent call last):
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py", line 36, in autodetect
    from google.appengine.api import memcache
ModuleNotFoundError: No module named 'google.appengine'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in <module>
    from oauth2client.contrib.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in <module>
    from oauth2client.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py", line 41, in autodetect
    from . import file_cache
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in <module>
    'file_cache is unavailable when using oauth2client >= 4.0.0')
ImportError: file_cache is unavailable when using oauth2client >= 4.0.0
URL being requested: GET https://www.googleapis.com/discovery/v1/apis/iam/v1/rest
/opt/tools/anaconda3/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
file_cache is unavailable when using oauth2client >= 4.0.0
Traceback (most recent call last):
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py", line 36, in autodetect
    from google.appengine.api import memcache
ModuleNotFoundError: No module named 'google.appengine'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 33, in <module>
    from oauth2client.contrib.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.contrib.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 37, in <module>
    from oauth2client.locked_file import LockedFile
ModuleNotFoundError: No module named 'oauth2client.locked_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/__init__.py", line 41, in autodetect
    from . import file_cache
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery_cache/file_cache.py", line 41, in <module>
    'file_cache is unavailable when using oauth2client >= 4.0.0')
ImportError: file_cache is unavailable when using oauth2client >= 4.0.0
URL being requested: GET https://www.googleapis.com/discovery/v1/apis/compute/v1/rest
/opt/tools/anaconda3/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
URL being requested: GET https://cloudresourcemanager.googleapis.com/v1/projects/d-dls-dlsi?alt=json
URL being requested: GET https://iam.googleapis.com/v1/projects/d-dls-dlsi/serviceAccounts/ray-autoscaler-sa-v1@d-dls-dlsi.iam.gserviceaccount.com?alt=json
Traceback (most recent call last):
  File "/opt/tools/anaconda3/bin/ray", line 10, in <module>
    sys.exit(main())
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/ray/scripts/scripts.py", line 744, in main
    return cli()
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/ray/scripts/scripts.py", line 463, in create_or_update
    no_restart, restart_only, yes, cluster_name)
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/ray/autoscaler/commands.py", line 43, in create_or_update_cluster
    config = _bootstrap_config(config)
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/ray/autoscaler/commands.py", line 65, in _bootstrap_config
    resolved_config = bootstrap_config(config)
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/ray/autoscaler/gcp/config.py", line 109, in bootstrap_gcp
    config = _configure_iam_role(config)
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/ray/autoscaler/gcp/config.py", line 169, in _configure_iam_role
    _add_iam_policy_binding(service_account, DEFAULT_SERVICE_ACCOUNT_ROLES)
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/ray/autoscaler/gcp/config.py", line 381, in _add_iam_policy_binding
    policy = crm.projects().getIamPolicy(resource=project_id).execute()
  File "/opt/tools/anaconda3/lib/python3.6/site-packages/googleapiclient/discovery.py", line 730, in method
    raise TypeError('Missing required parameter "%s"' % name)
TypeError: Missing required parameter "body"



Discussion:
Looks like https://github.com/googleapis/google-api-python-client/issues/416 is a fix. Odd that this didn't fail before. 

Can you try it out to see if it works? You can follow these instructions for setting up an environment (it'll take 3 minutes) - https://ray.readthedocs.io/en/latest/tune-contrib.html#setting-up-a-development-environment
This issue appeared for me too on several different machines independently starting at the same time a couple of days ago. I believe something in the Google API changed recently and the Google API python client library wasn't changed accordingly.

The fix for autoscaler is to add `body={}` argument to the `getIamPolicy` here: https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/gcp/config.py#L386.

Resolving patch:
diff --git a/python/ray/autoscaler/gcp/config.py b/python/ray/autoscaler/gcp/config.py
--- a/python/ray/autoscaler/gcp/config.py
+++ b/python/ray/autoscaler/gcp/config.py
@@ -383,7 +383,8 @@ def _add_iam_policy_binding(service_account, roles):
     email = service_account["email"]
     member_id = "serviceAccount:" + email
 
-    policy = crm.projects().getIamPolicy(resource=project_id).execute()
+    policy = crm.projects().getIamPolicy(
+        resource=project_id, body={}).execute()
 
     already_configured = True
 


Test patch: