Repository: googleapis/google-cloud-python

Issue:
Bigtable: 'test_create_instance_w_two_clusters' flakes with '504 Deadline Exceeded'
/cc @sduskis, @vikas-jamdar 

From:

- https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/7977
- https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/7978

```python
___________ TestInstanceAdminAPI.test_create_instance_w_two_clusters ___________

self = <tests.system.TestInstanceAdminAPI testMethod=test_create_instance_w_two_clusters>

    def test_create_instance_w_two_clusters(self):
        from google.cloud.bigtable import enums
        from google.cloud.bigtable.table import ClusterState
        _PRODUCTION = enums.Instance.Type.PRODUCTION
        ALT_INSTANCE_ID = 'dif' + unique_resource_id('-')
        instance = Config.CLIENT.instance(ALT_INSTANCE_ID,
                                          instance_type=_PRODUCTION,
                                          labels=LABELS)
    
        ALT_CLUSTER_ID_1 = ALT_INSTANCE_ID + '-c1'
        ALT_CLUSTER_ID_2 = ALT_INSTANCE_ID + '-c2'
        LOCATION_ID_2 = 'us-central1-f'
        STORAGE_TYPE = enums.StorageType.HDD
        cluster_1 = instance.cluster(
            ALT_CLUSTER_ID_1, location_id=LOCATION_ID, serve_nodes=SERVE_NODES,
            default_storage_type=STORAGE_TYPE)
        cluster_2 = instance.cluster(
            ALT_CLUSTER_ID_2, location_id=LOCATION_ID_2,
            serve_nodes=SERVE_NODES, default_storage_type=STORAGE_TYPE)
        operation = instance.create(clusters=[cluster_1, cluster_2])
        # We want to make sure the operation completes.
        operation.result(timeout=10)
    
        # Make sure this instance gets deleted after the test case.
        self.instances_to_delete.append(instance)
    
        # Create a new instance instance and make sure it is the same.
        instance_alt = Config.CLIENT.instance(ALT_INSTANCE_ID)
        instance_alt.reload()
    
        self.assertEqual(instance, instance_alt)
        self.assertEqual(instance.display_name, instance_alt.display_name)
        self.assertEqual(instance.type_, instance_alt.type_)
    
        clusters, failed_locations = instance_alt.list_clusters()
        self.assertEqual(failed_locations, [])
    
        clusters.sort(key=lambda x: x.name)
        alt_cluster_1, alt_cluster_2 = clusters
    
        self.assertEqual(cluster_1.location_id, alt_cluster_1.location_id)
        self.assertEqual(alt_cluster_1.state, enums.Cluster.State.READY)
        self.assertEqual(cluster_1.serve_nodes, alt_cluster_1.serve_nodes)
        self.assertEqual(cluster_1.default_storage_type,
                         alt_cluster_1.default_storage_type)
        self.assertEqual(cluster_2.location_id, alt_cluster_2.location_id)
        self.assertEqual(alt_cluster_2.state, enums.Cluster.State.READY)
        self.assertEqual(cluster_2.serve_nodes, alt_cluster_2.serve_nodes)
        self.assertEqual(cluster_2.default_storage_type,
                         alt_cluster_2.default_storage_type)
    
        # Test list clusters in project via 'client.list_clusters'
        clusters, failed_locations = Config.CLIENT.list_clusters()
        self.assertFalse(failed_locations)
        found = set([cluster.name for cluster in clusters])
        self.assertTrue({alt_cluster_1.name,
                         alt_cluster_2.name,
                         Config.CLUSTER.name}.issubset(found))
    
        temp_table_id = 'test-get-cluster-states'
        temp_table = instance.table(temp_table_id)
>       temp_table.create()

tests/system.py:280: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
google/cloud/bigtable/table.py:218: in create
    table=table, initial_splits=splits)
google/cloud/bigtable_admin_v2/gapic/bigtable_table_admin_client.py:327: in create_table
    request, retry=retry, timeout=timeout, metadata=metadata)
../api_core/google/api_core/gapic_v1/method.py:139: in __call__
    return wrapped_func(*args, **kwargs)
../api_core/google/api_core/retry.py:260: in retry_wrapped_func
    on_error=on_error,
../api_core/google/api_core/retry.py:177: in retry_target
    return target()
../api_core/google/api_core/timeout.py:206: in func_with_timeout
    return func(*args, **kwargs)
../api_core/google/api_core/grpc_helpers.py:61: in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

value = DeadlineExceeded('Deadline Exceeded',)
from_value = <_Rendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEED...all.cc","file_line":1099,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

    def raise_from(value, from_value):
>       raise value
E       DeadlineExceeded: 504 Deadline Exceeded

../.nox/sys-2-7/lib/python2.7/site-packages/six.py:737: DeadlineExceeded
```


Discussion:
We probably need to increase the time out from the 130 seconds it is now, to more like 15 minutes.  Create table can take longer than 2 minutes, depending on conditions.
@sduskis The deadline in [bigtable_table_admin_client_config.py](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/64caa5f0a319313ccc5dbe515ae91c4029548447/bigtable/google/cloud/bigtable_admin_v2/gapic/bigtable_table_admin_client_config.py#L39) is set via autosynth.  We would need to get the upstream configuration fixed, rather than changing here manually.
Understood.  I am in progress in making those upstream changes now.
Another two failures today:

- https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/8187
- https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/8188
FYI, my internal changes touched more than just `CreateTable`.  I wanted to fix other RPCs as well.  Hopefully, we'll get a fix out this week.
@tseaver, I got the changes out.  See this [googleapis commit](https://github.com/googleapis/googleapis/commit/59036b030f4a02b1eef2425d76064992b50c4d9e#diff-05c73320211201124119d8c88eb6d5d2) for details.

A synth regen should hopefully fix this flake.

Resolving patch:
diff --git a/bigtable/google/cloud/bigtable_admin_v2/gapic/bigtable_instance_admin_client_config.py b/bigtable/google/cloud/bigtable_admin_v2/gapic/bigtable_instance_admin_client_config.py
--- a/bigtable/google/cloud/bigtable_admin_v2/gapic/bigtable_instance_admin_client_config.py
+++ b/bigtable/google/cloud/bigtable_admin_v2/gapic/bigtable_instance_admin_client_config.py
@@ -23,12 +23,12 @@
                     "retry_params_name": "default"
                 },
                 "GetInstance": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "idempotent",
                     "retry_params_name": "default"
                 },
                 "ListInstances": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "idempotent",
                     "retry_params_name": "default"
                 },
@@ -43,8 +43,8 @@
                     "retry_params_name": "default"
                 },
                 "DeleteInstance": {
-                    "timeout_millis": 60000,
-                    "retry_codes_name": "idempotent",
+                    "timeout_millis": 150000,
+                    "retry_codes_name": "non_idempotent",
                     "retry_params_name": "default"
                 },
                 "CreateCluster": {
@@ -53,12 +53,12 @@
                     "retry_params_name": "default"
                 },
                 "GetCluster": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "idempotent",
                     "retry_params_name": "default"
                 },
                 "ListClusters": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "idempotent",
                     "retry_params_name": "default"
                 },
@@ -68,47 +68,47 @@
                     "retry_params_name": "default"
                 },
                 "DeleteCluster": {
-                    "timeout_millis": 60000,
-                    "retry_codes_name": "idempotent",
+                    "timeout_millis": 120000,
+                    "retry_codes_name": "non_idempotent",
                     "retry_params_name": "default"
                 },
                 "CreateAppProfile": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "non_idempotent",
                     "retry_params_name": "default"
                 },
                 "GetAppProfile": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "idempotent",
                     "retry_params_name": "default"
                 },
                 "ListAppProfiles": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "idempotent",
                     "retry_params_name": "default"
                 },
                 "UpdateAppProfile": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "idempotent",
                     "retry_params_name": "default"
                 },
                 "DeleteAppProfile": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "non_idempotent",
                     "retry_params_name": "default"
                 },
                 "GetIamPolicy": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "idempotent",
                     "retry_params_name": "default"
                 },
                 "SetIamPolicy": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "non_idempotent",
                     "retry_params_name": "default"
                 },
                 "TestIamPermissions": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "idempotent",
                     "retry_params_name": "default"
                 }
diff --git a/bigtable/google/cloud/bigtable_admin_v2/gapic/bigtable_table_admin_client_config.py b/bigtable/google/cloud/bigtable_admin_v2/gapic/bigtable_table_admin_client_config.py
--- a/bigtable/google/cloud/bigtable_admin_v2/gapic/bigtable_table_admin_client_config.py
+++ b/bigtable/google/cloud/bigtable_admin_v2/gapic/bigtable_table_admin_client_config.py
@@ -14,31 +14,13 @@
                     "rpc_timeout_multiplier": 1.0,
                     "max_rpc_timeout_millis": 20000,
                     "total_timeout_millis": 600000
-                },
-                "create_table": {
-                    "initial_retry_delay_millis": 100,
-                    "retry_delay_multiplier": 1.3,
-                    "max_retry_delay_millis": 60000,
-                    "initial_rpc_timeout_millis": 130000,
-                    "rpc_timeout_multiplier": 1.0,
-                    "max_rpc_timeout_millis": 130000,
-                    "total_timeout_millis": 3600000
-                },
-                "drop_row_range": {
-                    "initial_retry_delay_millis": 100,
-                    "retry_delay_multiplier": 1.3,
-                    "max_retry_delay_millis": 60000,
-                    "initial_rpc_timeout_millis": 900000,
-                    "rpc_timeout_multiplier": 1.0,
-                    "max_rpc_timeout_millis": 900000,
-                    "total_timeout_millis": 3600000
                 }
             },
             "methods": {
                 "CreateTable": {
-                    "timeout_millis": 130000,
+                    "timeout_millis": 900000,
                     "retry_codes_name": "non_idempotent",
-                    "retry_params_name": "create_table"
+                    "retry_params_name": "default"
                 },
                 "CreateTableFromSnapshot": {
                     "timeout_millis": 60000,
@@ -46,29 +28,29 @@
                     "retry_params_name": "default"
                 },
                 "ListTables": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "idempotent",
                     "retry_params_name": "default"
                 },
                 "GetTable": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 120000,
                     "retry_codes_name": "idempotent",
                     "retry_params_name": "default"
                 },
                 "DeleteTable": {
-                    "timeout_millis": 60000,
-                    "retry_codes_name": "idempotent",
+                    "timeout_millis": 120000,
+                    "retry_codes_name": "non_idempotent",
                     "retry_params_name": "default"
                 },
                 "ModifyColumnFamilies": {
-                    "timeout_millis": 60000,
+                    "timeout_millis": 900000,
                     "retry_codes_name": "non_idempotent",
                     "retry_params_name": "default"
                 },
                 "DropRowRange": {
                     "timeout_millis": 900000,
                     "retry_codes_name": "non_idempotent",
-                    "retry_params_name": "drop_row_range"
+                    "retry_params_name": "default"
                 },
                 "GenerateConsistencyToken": {
                     "timeout_millis": 60000,
diff --git a/bigtable/google/cloud/bigtable_admin_v2/gapic/enums.py b/bigtable/google/cloud/bigtable_admin_v2/gapic/enums.py
--- a/bigtable/google/cloud/bigtable_admin_v2/gapic/enums.py
+++ b/bigtable/google/cloud/bigtable_admin_v2/gapic/enums.py
@@ -32,6 +32,71 @@ class StorageType(enum.IntEnum):
     HDD = 2
 
 
+class Instance(object):
+    class State(enum.IntEnum):
+        """
+        Possible states of an instance.
+
+        Attributes:
+          STATE_NOT_KNOWN (int): The state of the instance could not be determined.
+          READY (int): The instance has been successfully created and can serve requests
+          to its tables.
+          CREATING (int): The instance is currently being created, and may be destroyed
+          if the creation process encounters an error.
+        """
+        STATE_NOT_KNOWN = 0
+        READY = 1
+        CREATING = 2
+
+    class Type(enum.IntEnum):
+        """
+        The type of the instance.
+
+        Attributes:
+          TYPE_UNSPECIFIED (int): The type of the instance is unspecified. If set when creating an
+          instance, a ``PRODUCTION`` instance will be created. If set when updating
+          an instance, the type will be left unchanged.
+          PRODUCTION (int): An instance meant for production use. ``serve_nodes`` must be set
+          on the cluster.
+          DEVELOPMENT (int): The instance is meant for development and testing purposes only; it has
+          no performance or uptime guarantees and is not covered by SLA.
+          After a development instance is created, it can be upgraded by
+          updating the instance to type ``PRODUCTION``. An instance created
+          as a production instance cannot be changed to a development instance.
+          When creating a development instance, ``serve_nodes`` on the cluster must
+          not be set.
+        """
+        TYPE_UNSPECIFIED = 0
+        PRODUCTION = 1
+        DEVELOPMENT = 2
+
+
+class Cluster(object):
+    class State(enum.IntEnum):
+        """
+        Possible states of a cluster.
+
+        Attributes:
+          STATE_NOT_KNOWN (int): The state of the cluster could not be determined.
+          READY (int): The cluster has been successfully created and is ready to serve requests.
+          CREATING (int): The cluster is currently being created, and may be destroyed
+          if the creation process encounters an error.
+          A cluster may not be able to serve requests while being created.
+          RESIZING (int): The cluster is currently being resized, and may revert to its previous
+          node count if the process encounters an error.
+          A cluster is still capable of serving requests while being resized,
+          but may exhibit performance as if its number of allocated nodes is
+          between the starting and requested states.
+          DISABLED (int): The cluster has no backing nodes. The data (tables) still
+          exist, but no operations can be performed on the cluster.
+        """
+        STATE_NOT_KNOWN = 0
+        READY = 1
+        CREATING = 2
+        RESIZING = 3
+        DISABLED = 4
+
+
 class Table(object):
     class TimestampGranularity(enum.IntEnum):
         """
@@ -104,68 +169,3 @@ class State(enum.IntEnum):
         STATE_NOT_KNOWN = 0
         READY = 1
         CREATING = 2
-
-
-class Instance(object):
-    class State(enum.IntEnum):
-        """
-        Possible states of an instance.
-
-        Attributes:
-          STATE_NOT_KNOWN (int): The state of the instance could not be determined.
-          READY (int): The instance has been successfully created and can serve requests
-          to its tables.
-          CREATING (int): The instance is currently being created, and may be destroyed
-          if the creation process encounters an error.
-        """
-        STATE_NOT_KNOWN = 0
-        READY = 1
-        CREATING = 2
-
-    class Type(enum.IntEnum):
-        """
-        The type of the instance.
-
-        Attributes:
-          TYPE_UNSPECIFIED (int): The type of the instance is unspecified. If set when creating an
-          instance, a ``PRODUCTION`` instance will be created. If set when updating
-          an instance, the type will be left unchanged.
-          PRODUCTION (int): An instance meant for production use. ``serve_nodes`` must be set
-          on the cluster.
-          DEVELOPMENT (int): The instance is meant for development and testing purposes only; it has
-          no performance or uptime guarantees and is not covered by SLA.
-          After a development instance is created, it can be upgraded by
-          updating the instance to type ``PRODUCTION``. An instance created
-          as a production instance cannot be changed to a development instance.
-          When creating a development instance, ``serve_nodes`` on the cluster must
-          not be set.
-        """
-        TYPE_UNSPECIFIED = 0
-        PRODUCTION = 1
-        DEVELOPMENT = 2
-
-
-class Cluster(object):
-    class State(enum.IntEnum):
-        """
-        Possible states of a cluster.
-
-        Attributes:
-          STATE_NOT_KNOWN (int): The state of the cluster could not be determined.
-          READY (int): The cluster has been successfully created and is ready to serve requests.
-          CREATING (int): The cluster is currently being created, and may be destroyed
-          if the creation process encounters an error.
-          A cluster may not be able to serve requests while being created.
-          RESIZING (int): The cluster is currently being resized, and may revert to its previous
-          node count if the process encounters an error.
-          A cluster is still capable of serving requests while being resized,
-          but may exhibit performance as if its number of allocated nodes is
-          between the starting and requested states.
-          DISABLED (int): The cluster has no backing nodes. The data (tables) still
-          exist, but no operations can be performed on the cluster.
-        """
-        STATE_NOT_KNOWN = 0
-        READY = 1
-        CREATING = 2
-        RESIZING = 3
-        DISABLED = 4


Test patch: