Repository: OpenMined/PySyft

Issue:
Implement Auto-Scaling on Google Cloud
### Description

In this project, you will implement functionality necessary to automatically spin-up Google cloud machines, load a PyGrid instance, run a training job, and tear down the instance upon completion (depositing the results into another long-running instance). The primary feature will be the ability to run a “hyperparameter sweep”, as exemplified below.

```python
parameters = {“alpha” : [0, 0.01, 0.02], “batch_size” : [32,64]}

gcloud = gr.GoogleCloud(credentials)

cluster = gcloud.LazyCluster(n_machines=10, 
                            type="n-series", 
                            priority="low",
                            eviction_policy="Delete",
                            max_price_usd=0.55,
                            reboot_max_price_usd=0.65)

cluster.sweep(model=model,
             parameters=parameters,
             optim=optim.SGD,
             results_node=grid['my node']
             )
```

### Context

I will be mentoring this project as part of the [Google Summer of Code](https://summerofcode.withgoogle.com/)

Also, this year we have some plans for PyGrid, along with our other projects, including a focus on production: Deployment on Cloud Providers (GCP, Amazon, Azure). You can check out the PyGrid Roadmap [here](https://github.com/OpenMined/Roadmap/tree/master/pygrid_team).

### Required Skills:

- Knowledge of [PyTorch](https://pytorch.org/) and Deep Learning
- Familiarity (or willingness to become familiar) with [PySyft](https://github.com/OpenMined/PySyft)
- Familiarity (or willingness to become familiar) with [Google Cloud’s Infrastructure](https://cloud.google.com/docs)


**Difficulty**: Beginner

While this is a sizeable project, the core functionality is relatively straightforward (provision machines, install and start PyGrid servers, and run training scripts). Furthermore, there are plenty of tutorials on how to run PyGrid servers and how to run Google Cloud machines.

If you are new to PyGrid and want to learn more about it, try out the tutorials. A comprehensive list of tutorials can be found [here](https://github.com/OpenMined/PySyft/tree/master/examples/tutorials/grid). These tutorials cover how to create a PyGrid node and what operations you can perform.


### Useful Links

- PyTorch - https://pytorch.org/blog
- PySyft - https://github.com/OpenMined/PySyft
- PyGrid - https://github.com/OpenMined/PyGrid
- GCP Docs - https://cloud.google.com/docs
- GCloud APIs for Python - https://cloud.google.com/python/docs/reference
- Google APIs (Python Client) - https://github.com/googleapis/google-api-python-client 
- Google Summer of Code - https://summerofcode.withgoogle.com
  - [Timeline](https://developers.google.com/open-source/gsoc/timeline)
  - [Student Guide](https://google.github.io/gsocguides/student/)
  - [OpenMined Org.](https://summerofcode.withgoogle.com/organizations/5561599089180672)

_List of all GSoC project ideas [here](https://docs.google.com/document/d/1OA8277RB9ZN8u7Wgn6NqZK7tzixnZbSlkfulyqoo2G4/edit?usp=sharing)._


Discussion:
Note: we want to allow for automatic node teardown after calling .sweep() as an option - where it deposits the results into a "main cluster node". In other words, if I spin up a cluster of 10 workers, this is ACTUALLY a cluster of 10 workers and 1 master, where the master can store resources produced by the 10 workers.
I'd like to work on this project


Cool, @carlodavid012! I saw that you already know PySyft. I also think that basic PyGrid concepts can be useful. Feel free to chat with me if you need anything!
I am working on it.
I have no experience related to Cloud but I am trying to learn PySyft .
I really want to contribute to this project.
Please can you help me .
Great! There are a lot of interested people.  I think you could also talk to each other about the issue. :rocket: :nerd_face: 
Is there any particular channel for this project or should I message in gsoc slack group
Yess @neeravjain24! Don't worry if you're not familiar with GCloud. There's a lot of tutorials, docs and codelabs to learn. I also recommend taking a look at the PyGrid tutorial. The links are in the issue description.

GCloud links:
https://cloud.google.com/getting-started
https://codelabs.developers.google.com/cloud/
> Is there any particular channel for this project or should I message in gsoc slack group

You can use the #gsoc channel :slightly_smiling_face: 
I would like to work on this project and I am willing to learn PySyft. Also, I am learning about GCP already. 
I'd like to contribute to this project
Great, guys. I'm thinking of creating a slack channel for this issue. It is even better to discuss it there too. What do you think? :slightly_smiling_face: 
Hey guys, 
I am pursuing B.Tech from University of Delhi and am fully interested in contributing to this project. I have also started working on this.
This issue has been marked stale because it has been open 30 days with no activity. Leave a comment or remove the `stale` label to unmark it. Otherwise, this will be closed in 7 days.

Resolving patch:
diff --git a/syft/grid/utils/autoscale/gcloud.py b/syft/grid/utils/autoscale/gcloud.py
--- a/syft/grid/utils/autoscale/gcloud.py
+++ b/syft/grid/utils/autoscale/gcloud.py
@@ -5,6 +5,11 @@
 import terrascript.data
 import terrascript.provider
 import terrascript.resource
+
+# syft dependencies
+from syft.grid.clients.data_centric_fl_client import DataCentricFLClient
+
+# terraform utils
 from utils.script import terraform_script
 from utils.notebook import terraform_notebook
 
@@ -25,7 +30,9 @@ def __init__(self, credentials, project_id, region):
         self.provider = "google"
         self.config = terrascript.Terrascript()
         self.config += terrascript.provider.google(
-            credentials=self.credentials, project=self.project_id, region=self.region
+            credentials=self.credentials,
+            project=self.project_id,
+            region=self.region,
         )
         with open("main.tf.json", "w") as main_config:
             json.dump(self.config, main_config, indent=2, sort_keys=False)
@@ -107,9 +114,9 @@ def create_gridnetwork(self, name, machine_type, zone, apply=True):
         self.expose_port(name="pygrid", apply=False)
 
         image = terrascript.data.google_compute_image(
-            name + "pytorch",
-            family="pytorch-latest-gpu-debian-10",
-            project="deeplearning-platform-release",
+            name + "container-optimized-os",
+            family="cos-81-lts",
+            project="cos-cloud",
         )
         self.config += image
 
@@ -125,11 +132,14 @@ def create_gridnetwork(self, name, machine_type, zone, apply=True):
             },
             metadata_startup_script="""
                 #!/bin/bash
-                apt-get update
-                apt-get -y upgrade
-                sudo -i bash -c 'pip install git+https://github.com/OpenMined/PyGridNetwork.git'
-                sudo -i bash -c 'echo Starting PyGridNetwork & \
-                python -m gridnetwork --port=80 --start_local_db'""",
+                sleep 30;
+                docker pull openmined/grid-network:production;
+                docker run \
+                -e PORT=80 \
+                -e DATABASE_URL=sqlite:///databasenode.db \
+                --name gridnetwork\
+                -p 80:80 \
+                -d openmined/grid-network:production;""",
         )
         self.config += node
 
@@ -158,15 +168,17 @@ def create_gridnode(self, name, machine_type, zone, gridnetwork_name=None, apply
             outputs = json.load(out)["outputs"]
 
         gridnetwork_ip = outputs[gridnetwork_name + "-ip"]["value"]
-        pygrid_network_address = "--gateway_url=http://" + gridnetwork_ip if gridnetwork_ip else ""
+        pygrid_network_address = "=http://" + gridnetwork_ip if gridnetwork_ip else ""
+        host = "curl ifconfig.co" if gridnetwork_ip else "hostname -I"
 
         image = terrascript.data.google_compute_image(
-            name + "pytorch",
-            family="pytorch-latest-gpu-debian-10",
-            project="deeplearning-platform-release",
+            name + "container-optimized-os",
+            family="cos-81-lts",
+            project="cos-cloud",
         )
         self.config += image
 
+        # HOST environment variable is set to internal IP address
         node = terrascript.resource.google_compute_instance(
             name,
             name=name,
@@ -176,13 +188,17 @@ def create_gridnode(self, name, machine_type, zone, gridnetwork_name=None, apply
             network_interface={"network": "default", "access_config": {}},
             metadata_startup_script=f"""
                 #!/bin/bash
-                apt-get update
-                apt-get -y upgrade
-                sudo -i bash -c 'pip install notebook==5.7.8'
-                sudo -i bash -c 'pip install git+https://github.com/OpenMined/PyGridNode.git'
-                sudo -i bash -c 'echo Starting Node {name} \
-                joined with PyGridNetwork at {gridnetwork_ip} & \
-                python -m gridnode --id={name} --port=80 {pygrid_network_address}'""",
+                sleep 30;
+                docker pull openmined/grid-node:production;
+                docker run \
+                -e NODE_ID={name} \
+                -e HOST = "$({host})" \
+                -e PORT=80 \
+                -e NETWORK={pygrid_network_address} \
+                -e DATABASE_URL=sqlite:///databasenode.db \
+                --name gridnode \
+                -p 80:80 \
+                -d openmined/grid-node:production;""",
         )
         self.config += node
         with open("main.tf.json", "w") as main_config:
@@ -211,19 +227,22 @@ def create_cluster(
             zone: zone of your GCP project
             reserve_ip_name: name of the reserved ip created using reserve_ip
             target_size: number of wokers to be created(N workers + 1 master)
-            eviction_policy: "delete" to teardown the cluster after calling .sweep() else None
+            eviction_policy: "delete" to teardown the cluster after calling
+            cluster.sweep() else None
             apply: to call terraform apply at the end
         """
-        self.expose_port("pygrid", apply=False)
+        self.expose_port("pygrid", ports=[80, 3000], apply=False)
 
         with open("terraform.tfstate", "r") as out:
             outputs = json.load(out)["outputs"]
+
         gridnetwork_ip = outputs[reserve_ip_name + "-ip"]["value"]
+        pygrid_network_address = "http://" + gridnetwork_ip
 
         image = terrascript.data.google_compute_image(
-            name + "pytorch",
-            family="pytorch-latest-gpu-debian-10",
-            project="deeplearning-platform-release",
+            name + "container-optimized-os",
+            family="cos-81-lts",
+            project="cos-cloud",
         )
         self.config += image
 
@@ -234,16 +253,29 @@ def create_cluster(
             zone=zone,
             boot_disk={"initialize_params": {"image": "${" + image.self_link + "}"}},
             network_interface={"network": "default", "access_config": {"nat_ip": gridnetwork_ip}},
-            metadata_startup_script="""
+            metadata_startup_script=f"""
                 #!/bin/bash
-                apt-get update
-                apt-get -y upgrade
-                sudo -i bash -c 'pip install git+https://github.com/OpenMined/PyGridNetwork.git'
-                sudo -i bash -c 'echo Starting PyGridNetwork & \
-                python -m gridnetwork --port=80 --start_local_db'""",
+                sleep 30;
+                docker pull openmined/grid-network:production && \
+                docker run \
+                -e PORT=80 \
+                -e DATABASE_URL=sqlite:///databasenode.db \
+                --name gridnetwork \
+                -p 80:80 \
+                -d openmined/grid-network:production;
+                docker pull openmined/grid-node:production;
+                docker run \
+                -e NODE_ID={name+"-network-node"} \
+                -e HOST="$(hostname -I)" \
+                -e PORT=3000 \
+                -e NETWORK={pygrid_network_address} \
+                -e DATABASE_URL=sqlite:///databasenode.db \
+                --name gridnode \
+                -p 3000:3000 \
+                -d openmined/grid-node:production;""",
         )
 
-        pygrid_network_address = "http://" + gridnetwork_ip
+        # HOST environment variable is set to external IP address
         instance_template = terrascript.resource.google_compute_instance_template(
             name + "-template",
             name=name + "-template",
@@ -252,14 +284,17 @@ def create_cluster(
             network_interface={"network": "default", "access_config": {}},
             metadata_startup_script=f"""
                 #!/bin/bash
-                apt-get update
-                apt-get -y upgrade
-                sudo -i bash -c 'pip install notebook==5.7.8'
-                sudo -i bash -c 'pip install git+https://github.com/OpenMined/PyGridNode.git'
-                sudo -i bash -c 'echo Starting Node {name} \
-                joined with PyGridNetwork at {pygrid_network_address} & \
-                python -m gridnode --id={name} --port=80 \
-                --gateway_url={pygrid_network_address}'""",
+                sleep 30;
+                docker pull openmined/grid-node:production;
+                docker run \
+                -e NODE_ID={name} \
+                -e HOST="$(curl ifconfig.co)" \
+                -e PORT=80 \
+                -e NETWORK={pygrid_network_address} \
+                -e DATABASE_URL=sqlite:///databasenode.db \
+                --name gridnode \
+                -p 80:80 \
+                -d openmined/grid-node:production;""",
             lifecycle={"create_before_destroy": True},
         )
         self.config += instance_template
@@ -281,7 +316,12 @@ def create_cluster(
             else:
                 terraform_script.apply()
 
-        return Cluster(name, self.provider, gridnetwork_ip, eviction_policy=eviction_policy)
+        return Cluster(
+            name,
+            self.provider,
+            gridnetwork_ip,
+            eviction_policy=eviction_policy,
+        )
 
     def compute_instance(self, name, machine_type, zone, image_family, apply=True):
         """
@@ -329,22 +369,55 @@ def __init__(self, name, provider, gridnetwork_ip, eviction_policy=None):
             name: name of the cluster
             provider: terrafrom provider for the cluster
             gridnetwork_ip: ip of grid network instance
-            eviction_policy: "delete" to teardown the cluster after calling .sweep() else None
+            eviction_policy: "delete" to teardown the cluster after calling
+            cluster.sweep() else None
         """
         self.name = name
         self.provider = provider
         self.gridnetwork_ip = gridnetwork_ip
+        self.gridnetwork_node_ip = gridnetwork_ip + ":3000"
         self.master = name + "-master"
         self.cluster = name + "-cluster"
         self.template = name + "-template"
         self.eviction_policy = eviction_policy
+        self.network_node = None
         self.config = None
 
-    def sweep(self, apply=True):
+    def sweep(
+        self,
+        model,
+        hook,
+        model_id=None,
+        mpc=False,
+        allow_download=False,
+        allow_remote_inference=False,
+        apply=True,
+    ):
         """
         args:
+            model : A jit model or Syft Plan.
+            hook : A PySyft hook
+            model_id (str): An integer/string representing the model id.
+            If it isn't provided and the model is a Plan we use model.id,
+            if the model is a jit model we raise an exception.
+            allow_download (bool) : Allow to copy the model to run it locally.
+            allow_remote_inference (bool) : Allow to run remote inferences.
             apply: to call terraform apply at the end
         """
+        print("Connecting to network-node")
+        self.network_node = DataCentricFLClient(hook, self.gridnetwork_node_ip)
+        print("Sending model to node")
+        self.network_node.serve_model(
+            model=model,
+            model_id=model_id,
+            mpc=mpc,
+            allow_download=allow_download,
+            allow_remote_inference=allow_remote_inference,
+        )
+        print("Model sent, disconnecting node now")
+        self.network_node.close()
+        print("Node disconnected")
+
         with open("main.tf.json", "r") as main_config:
             self.config = json.load(main_config)
 


Test patch: