Repository: mars-project/mars

Issue:
[BUG] missing a default timeout parameter and timeout error mismatch when deploying on yarn 
**Describe the bug**
missing a default timeout parameter and timeout error mismatch when deploying on yarn 

**To Reproduce**
To help us reproducing this bug, please provide information below:
1. Your Python version: 3.7.9
2. The version of Mars you use: pymars[distributed]==0.6.0
3. Versions of crucial packages, such as numpy, scipy and protobuf: numpy==1.19.4, scipy==1.5.4, protobuf==3.14.0, pyarrow==2.0.0
4. Full stack of the error.
```bash
Traceback (most recent call last):
  File "test_yarn.py", line 10, in <module>
    cluster = new_cluster(environment='python:///opt/anaconda3/envs/pymodel/bin/python', scheduler_num=1, web_num=1,worker_num=4, worker_cpu=8, worker_mem='28g',min_worker_num=2)
  File "/opt/anaconda3/envs/pymodel/lib/python3.7/site-packages/mars/deploy/yarn/client.py", line 148, in new_cluster
    timeout=timeout - (time.time() - check_start_time))
TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'
```
By adding a timeout parameter, the error disappear. But sometimes timeout error occurs as follows:
```bash
Traceback (most recent call last):
  File "test_yarn.py", line 10, in <module>
    cluster = new_cluster(environment='python:///opt/anaconda3/envs/pymodel/bin/python', scheduler_num=1, web_num=1,worker_num=4, worker_cpu=8, worker_mem='28g',min_worker_num=2, timeout=1)
  File "/opt/anaconda3/envs/pymodel/lib/python3.7/site-packages/mars/deploy/yarn/client.py", line 148, in new_cluster
    timeout=timeout - (time.time() - check_start_time))
  File "/opt/anaconda3/envs/pymodel/lib/python3.7/site-packages/mars/deploy/utils.py", line 31, in wait_services_ready
    raise TimeoutError('Wait kubernetes cluster start timeout')
TimeoutError: Wait kubernetes cluster start timeout
``` 
5. Minimized code to reproduce the error.
```python
import os
from mars.deploy.yarn import new_cluster
from mars.session import Session
import mars.tensor as mt
import mars.dataframe as md

os.environ['JAVA_HOME'] = '/usr/java/jdk1.8.0_191-amd64'
os.environ['HADOOP_HOME'] = "/usr/bin/hadoop"
cluster = new_cluster(environment='python:///opt/anaconda3/envs/pymodel/bin/python', scheduler_num=1, web_num=1,worker_num=4, worker_cpu=8, timeout=1)
print(Session.default_or_local().endpoint)
a = mt.random.rand(10, 10)
print(a.dot(a.T).execute())
```

**Expected behavior**
1. give timeout a default value, such as 60.
2. when timeout error occurs, print the correct info, E.g. "Wait yarn cluster start timeout"  




Discussion:
When timeout is not specified, it might be better to wait infinitely till KeyboardInterrupt is raised.
@wjsi  waitting infinitely till KeyboardInterrupt is better. By the way, TimeoutError raised by the func `wait_services_ready`, how do you like adding a parameter naming `platform` to raise the corresponding cluster timout such as k8s or yarn.
> @wjsi waitting infinitely till KeyboardInterrupt is better. By the way, TimeoutError raised by the func `wait_services_ready`, how do you like adding a parameter naming `platform` to raise the corresponding cluster timeout such as k8s or yarn.

It is cleaner to raise a general error inside ``wait_services_ready``. Different clients for different platforms can catch this error and raise their own using ``raise ... from None``.
Could you please submit a PR to address this issue? @smartguo 
> Could you please submit a PR to address this issue? @smartguo

My pleasure, I will try my best.

Resolving patch:
diff --git a/mars/deploy/kubernetes/client.py b/mars/deploy/kubernetes/client.py
--- a/mars/deploy/kubernetes/client.py
+++ b/mars/deploy/kubernetes/client.py
@@ -272,6 +272,7 @@ def _wait_services_ready(self):
             ]
         wait_services_ready(selectors, limits,
                             lambda sel: self._get_ready_pod_count(sel),
+                            platform='kubernetes',
                             timeout=self._timeout)
         logger.info('All service pods ready.')
 
diff --git a/mars/deploy/utils.py b/mars/deploy/utils.py
--- a/mars/deploy/utils.py
+++ b/mars/deploy/utils.py
@@ -15,7 +15,7 @@
 import time
 
 
-def wait_services_ready(selectors, min_counts, count_fun, timeout=None):
+def wait_services_ready(selectors, min_counts, count_fun, platform, timeout=None):
     readies = [0] * len(selectors)
     start_time = time.time()
     while True:
@@ -28,5 +28,5 @@ def wait_services_ready(selectors, min_counts, count_fun, timeout=None):
         if all_satisfy:
             break
         if timeout and timeout + start_time < time.time():
-            raise TimeoutError('Wait kubernetes cluster start timeout')
+            raise TimeoutError('Wait %s cluster start timeout' % platform)
         time.sleep(1)
diff --git a/mars/deploy/yarn/client.py b/mars/deploy/yarn/client.py
--- a/mars/deploy/yarn/client.py
+++ b/mars/deploy/yarn/client.py
@@ -132,7 +132,7 @@ def _override_envs(src, updates):
                 break
             except skein.ApplicationNotRunningError:  # pragma: no cover
                 time.sleep(0.5)
-                if time.time() - check_start_time > timeout:
+                if timeout and time.time() - check_start_time > timeout:
                     raise
 
         logger.debug('Application client for %s at %s retrieved', app_id, app_client.address)
@@ -145,7 +145,8 @@ def _override_envs(src, updates):
 
         wait_services_ready(services, limits,
                             lambda svc: _get_ready_container_count(app_client, svc),
-                            timeout=timeout - (time.time() - check_start_time))
+                            platform='yarn',
+                            timeout=None if not timeout else timeout - (time.time() - check_start_time))
         web_endpoint_kv = app_client.kv.get_prefix(YarnWebApplication.service_name)
         web_endpoint = random.choice([to_str(v).split('@', 1)[0] for v in web_endpoint_kv.values()])
         return YarnClusterClient(skein_client, app_client.id, 'http://' + web_endpoint,


Test patch: