Repository: ray-project/ray

Issue:
[Serve] Backend Actor can't scale up to 2 replicas if it uses an endpoint handle internally
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?
A backend using an endpoint handle internally appears to hang when `client.update_backend_config` is called to change the number of replicas from 1 to 2.
*Ray version and other system information (Python version, TensorFlow version, OS):*
Ray 1.0.0, Python 3.6, Mac OS (Also Py3.7 and Linux)

### Reproduction (REQUIRED)
There are three 
Please provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):
```python
import ray
from ray import serve
from ray.serve import BackendConfig

ray.init()
client = serve.start() # serve.start(detached=True) fixes it

def hello_world(_):
    return b"Hello World"

class ForwardActor:
    def __init__(self):
        # client = serve.connect() fixes it
        self.handle = client.get_handle("hello_world") # replacing this with self.handle = 1 fixes it

    def __call__(self, _):
        return 1 

client.create_backend("hello_world", hello_world)
client.create_endpoint("hello_world", backend="hello_world")

client.create_backend("ForwardActor", ForwardActor)
client.create_endpoint("ForwardActor", backend="ForwardActor")

forwarders_config = BackendConfig(num_replicas=2) # replacing this with num_replicas = 1 fixes it

#Hangs on this line
client.update_backend_config("ForwardActor", forwarders_config)
```
Actually, the same problem happens if you set the num_replicas to 2 right from the start in `create_backend`.

This is the line where it hangs: https://github.com/ray-project/ray/blob/master/python/ray/serve/controller.py#L435

If we cannot run your script, we cannot fix your issue.

- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [x] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).



Discussion:
This looks pretty problematic. My initial guess would be that there's a deadlock between the code that starts the handle in the backend worker (which is calling into the controller) and the code that waits for the new replicas to start up.
I will try to isolated it into a Ray Core issue. Seems like something to do with actor ref counting
Minimal version from Serve:
```python
from ray import serve

client = serve.start()

class ForwardActor:
    def __init__(self):
        self.client = client

client.create_backend("ForwardActor", ForwardActor, config={"num_replicas": 2})
```

Resolving patch:
diff --git a/python/ray/serve/api.py b/python/ray/serve/api.py
--- a/python/ray/serve/api.py
+++ b/python/ray/serve/api.py
@@ -7,7 +7,7 @@
 from ray.serve.controller import ServeController
 from ray.serve.handle import RayServeHandle
 from ray.serve.utils import (block_until_http_ready, format_actor_name,
-                             get_random_letters)
+                             get_random_letters, logger)
 from ray.serve.exceptions import RayServeException
 from ray.serve.config import BackendConfig, ReplicaConfig, BackendMetadata
 from ray.actor import ActorHandle
@@ -53,8 +53,16 @@ def shutdown_serve_client():
 
     def __del__(self):
         if not self._detached:
+            logger.info("Shutting down Ray Serve because client went out of "
+                        "scope. To prevent this, either keep a reference to "
+                        "the client object or use serve.start(detached=True).")
             self.shutdown()
 
+    def __reduce__(self):
+        raise RayServeException(
+            ("Ray Serve client cannot be serialized. Please use "
+             "serve.connect() to get a client from within a backend."))
+
     def shutdown(self) -> None:
         """Completely shut down the connected Serve instance.
 


Test patch: