Repository: ray-project/ray

Issue:
Allow `ray up` to take URL as argument.
To get started with the autoscaler, we tell people to run a line like `ray up ray/python/ray/autoscaler/aws/example-full.yaml`. In practice, the steps are

1. Figure out that this means you have to clone the Ray repository.
2. Clone the Ray repository.
3. Figure out the new correct relative file path based on where you cloned it.
4. Run `ray up ...`.

Instead, we could let people just copy and paste the following command:

```
ray up https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/aws/example-full.yaml
```

In this blog post https://towardsdatascience.com/10x-faster-parallel-python-without-python-multiprocessing-e5017c93cce1, I posted benchmarks at the end that they could run with the autoscaler, but they still have to `wget` the relevant file. It'd be much easier if they could just use the URL.

Could also just write stuff like the following, but one line would be better.

```
wget https://raw.githubusercontent.com/ray-project/ray/master/python/ray/autoscaler/aws/example-full.yaml
ray up example-full.yaml
```


Discussion:
Actually, even better, maybe we can have default cluster profiles available. 

For example, 

```
ray up examples.tune
ray up examples.sgd
ray up examples.small
```
The url thing is a standard practice in kubernetes world. for example https://aws.amazon.com/premiumsupport/knowledge-center/eks-cluster-kubernetes-dashboard/
@richardliaw the thing I like about the URL approach is that it is extensible, so if I write a blog post, I can host my own cluster yaml somewhere and then give people one command to run to reproduce the experiments.
This might be within my capability (found via `good first issue` tag).  Can someone reference me to the `ray up` section of the codebase?
Sure! Here's the code that correspond to `ray up` 

https://github.com/ray-project/ray/blob/b23b6addfcfd4be7a27e58aefbecb3fd7dd6510c/python/ray/scripts/scripts.py#L523-L566

https://github.com/ray-project/ray/blob/b23b6addfcfd4be7a27e58aefbecb3fd7dd6510c/python/ray/scripts/scripts.py#L999
Thanks! Yeah, I think I can do that.  I'll take a stab over the weekend and get you a PR.

Resolving patch:
diff --git a/python/ray/scripts/scripts.py b/python/ray/scripts/scripts.py
--- a/python/ray/scripts/scripts.py
+++ b/python/ray/scripts/scripts.py
@@ -7,6 +7,8 @@
 import subprocess
 import sys
 import time
+import urllib
+import urllib.parse
 
 import ray.services as services
 from ray.autoscaler.commands import (
@@ -566,6 +568,16 @@ def create_or_update(cluster_config_file, min_workers, max_workers, no_restart,
     if restart_only or no_restart:
         assert restart_only != no_restart, "Cannot set both 'restart_only' " \
             "and 'no_restart' at the same time!"
+    if urllib.parse.urlparse(cluster_config_file).scheme in ("http", "https"):
+        try:
+            response = urllib.request.urlopen(cluster_config_file, timeout=5)
+            content = response.read()
+            file_name = cluster_config_file.split("/")[-1]
+            with open(file_name, "wb") as f:
+                f.write(content)
+            cluster_config_file = file_name
+        except urllib.error.HTTPError as e:
+            logger.info("Error downloading file: ", e)
     create_or_update_cluster(cluster_config_file, min_workers, max_workers,
                              no_restart, restart_only, yes, cluster_name)
 


Test patch:
diff --git a/python/ray/tests/test_autoscaler_yaml.py b/python/ray/tests/test_autoscaler_yaml.py
--- a/python/ray/tests/test_autoscaler_yaml.py
+++ b/python/ray/tests/test_autoscaler_yaml.py
@@ -2,6 +2,8 @@
 import os
 import unittest
 import yaml
+import urllib
+import tempfile
 
 from ray.autoscaler.autoscaler import fillout_defaults, validate_config
 from ray.test_utils import recursive_fnmatch
@@ -25,6 +27,21 @@ def testValidateDefaultConfig(self):
             except Exception:
                 self.fail("Config did not pass validation test!")
 
+    def testValidateNetworkConfig(self):
+        web_yaml = "https://raw.githubusercontent.com/ray-project/ray/" \
+            "master/python/ray/autoscaler/aws/example-full.yaml"
+        response = urllib.request.urlopen(web_yaml, timeout=5)
+        content = response.read()
+        with tempfile.TemporaryFile() as f:
+            f.write(content)
+            f.seek(0)
+            config = yaml.safe_load(f)
+        config = fillout_defaults(config)
+        try:
+            validate_config(config)
+        except Exception:
+            self.fail("Config did not pass validation test!")
+
     def _test_invalid_config(self, config_path):
         with open(os.path.join(RAY_PATH, config_path)) as f:
             config = yaml.safe_load(f)