Repository: ray-project/ray

Issue:
Initial sets of tasks are slow on large cluster.
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 16.04
- **Ray installed from (source or binary)**: source
- **Ray version**: Testing this PR https://github.com/ray-project/ray/pull/1257, but I'm pretty sure it happens with the current master at e583d5a421c6287505b885246bed731a957cd0bb.
- **Python version**: Python 3.5.2, Anaconda 4.2.0 (64-bit)

I first start Ray on the head node with

```
ray start --head --redis-port=6379 --redis-max-clients=65504
```

Then I start Ray on the workers with

```
parallel-ssh -h workers.txt -P -I -p 200 < start_worker.sh
```

using a `start_worker.sh` containing

```
export PATH=/home/ubuntu/anaconda3/bin/:$PATH
ray start --redis-address=<head-ip-address>:6379
```

Then I start a driver (on the head node) with

```python
import ray
import time

ray.init(redis_address=<head-node-ip>)

@ray.remote
def f():
    time.sleep(0.01)
    return ray.services.get_node_ip_address()
```

Then I run

```
%time l = set(ray.get([f.remote() for _ in range(1000)]))
```

in the driver a number of times. The first couple times take on the order of 50 seconds. Subsequent runs take around 170 milliseconds.

In `/tmp/raylogs`, a number of processes contain lines like

```
[WARN] (/home/ubuntu/ray/src/common/state/db_client_table.cc:54) calling redis_get_cached_db_client in a loop in with 1 manager IDs took 1038 milliseconds.
```


Discussion:

Resolving patch:
diff --git a/python/ray/worker.py b/python/ray/worker.py
--- a/python/ray/worker.py
+++ b/python/ray/worker.py
@@ -1476,12 +1476,12 @@ def print_error_messages(worker):
 
     worker.error_message_pubsub_client = worker.redis_client.pubsub()
     # Exports that are published after the call to
-    # error_message_pubsub_client.psubscribe and before the call to
+    # error_message_pubsub_client.subscribe and before the call to
     # error_message_pubsub_client.listen will still be processed in the loop.
-    worker.error_message_pubsub_client.psubscribe("__keyspace@0__:ErrorKeys")
+    worker.error_message_pubsub_client.subscribe("__keyspace@0__:ErrorKeys")
     num_errors_received = 0
 
-    # Get the exports that occurred before the call to psubscribe.
+    # Get the exports that occurred before the call to subscribe.
     with worker.lock:
         error_keys = worker.redis_client.lrange("ErrorKeys", 0, -1)
         for error_key in error_keys:
@@ -1589,13 +1589,13 @@ def fetch_and_execute_function_to_run(key, worker=global_worker):
 def import_thread(worker, mode):
     worker.import_pubsub_client = worker.redis_client.pubsub()
     # Exports that are published after the call to
-    # import_pubsub_client.psubscribe and before the call to
+    # import_pubsub_client.subscribe and before the call to
     # import_pubsub_client.listen will still be processed in the loop.
-    worker.import_pubsub_client.psubscribe("__keyspace@0__:Exports")
+    worker.import_pubsub_client.subscribe("__keyspace@0__:Exports")
     # Keep track of the number of imports that we've imported.
     num_imported = 0
 
-    # Get the exports that occurred before the call to psubscribe.
+    # Get the exports that occurred before the call to subscribe.
     with worker.lock:
         export_keys = worker.redis_client.lrange("Exports", 0, -1)
         for key in export_keys:
@@ -1627,7 +1627,7 @@ def import_thread(worker, mode):
     try:
         for msg in worker.import_pubsub_client.listen():
             with worker.lock:
-                if msg["type"] == "psubscribe":
+                if msg["type"] == "subscribe":
                     continue
                 assert msg["data"] == b"rpush"
                 num_imports = worker.redis_client.llen("Exports")


Test patch: