Repository: mantl/mantl

Issue:
ci build should use terraform modules
It appears the CI build is using legacy terraform (at least for AWS and GCE). It should use the newer terraform modules to be consistent with our recommended configuration and documentation. 

Legacy:

./terraform/aws/aws.tf
./terraform/gce/gce.tf

The ./terraform/aws.sample.tf and ./terraform/gce.sample.tf show how to use the terraform modules.

Docker Swarm on top of Mantl
## Why

I propose that support for Docker Swarm should be added to Mantl.
By having Swarm on Mantl, all the (available) resources on a cluster are exposed via the Docker API directly. Developers are already used to this API / model of working.

Swarm is ideal to run one-time-jobs in production easily. Think about database migrations (which could be run detached), or running some interactive script to generate a report (in which the attached input/output is really usefull).

Finally, Swarm could allow for easy development of microservices on top of Mantl. A developer could write the code for microservices on his local MackBook Air, and use Docker Compose to start a set of microservices via Swarm on top of Mantl, which would use way more memory and CPU than his MacBook Air could provide.
## How

Swarm registers itself as a Mesos framework, uses it to allocate resources and uses the Docker Executor to start containers. So far, so good! However, it also requires access to the Docker API on each worker node to provide services like pulling/pushing of images and showing the logs. At the moment, only the `/var/run/sock` unix domain socket is enabled in Mantl. To expose the Docker API in a secure way, I propose to add a proxy container (just like is done with Consul), to avoid some issues in Swarm [1].
Furthermore, Mesos Framework authentication should be implemented in Swarm [2]. After some investigation, It appears that adding this will take about 100 lines of Go code.
To make sure that a crash of a single Swarm daemon does not take down all containers started via Swarm, Swarm’s high availability option should be used [3]. I suggest that Swarm should run on each Mesos master node, to provide the same level of availability as Mesos itself.

[1] https://github.com/docker/swarm/issues/1089
[2] https://github.com/docker/swarm/issues/1944
[3] https://docs.docker.com/swarm/multi-manager-setup/



Discussion:

Our contract with developers is a docker-compose file.  This would help a lot in crossing between development (compose) and production (marathon).

I'd love to see swarm running on mantl. Our preference is to be an addon installed locally instead of a framework on top of mesos.

@stevendborrelli what do you mean exactly? Swarm on mesos _is_ registering itself as a mesos framework. If it would be added to Mantl as a addon, I imagine it would be in a HA setup, with swarm daemons running on the same nodes as the mesos masters.

After some research, I can't really find any information on integrating Swarm with Mesos's scheduling. Swarm will run jobs separately from Mesos/Marathon (which is fine, just something to be noted). 

Unless you can point me to docs for using Mesos + Swarm @moretea? I haven't found anything mentioning Swarm as a framework. 

Here's a crazy thought: Swarm is basically installed by running a Docker container on each control and worker node. Maybe we can start these jobs via Marathon? 

I have done some work on this, but haven't gotten swarm to elect a leader. It seems that it's having trouble connecting to Consul, though when I try the API interactions from the console using curl, they work just fine. 

Branch here: https://github.com/CiscoCloud/mantl/tree/feature/docker-swarm
Error log here: http://pastebin.com/raw/yAisqgfR

+1 for THE SWARM

+1

@distributorofpain @lguminski If either of you has more experience with Swarm and would like to help debug my branch, that'd be greatly apprciated!

Would love to help... i am a bit of a newb though, so i cant say i would know where to begin helping...

Being a linux sysops guy, i would think i could help though :)

Top of my head looking at the log file output( this is where my lack of understanding of mantl core setup could get annoying):
1. Does mantl run consul on 8500?  Does it require a certificate to talk to?  I only ask because i see environmental variables like this:  CONSUL_SSL=true   CONSUL_SSL_VERIFY=false
2. I see thats running on 127.0.0.1:8500  If thats in docker, wouldnt you stipulate an outside port?
3. Is that container setup to run on the bridge network?
4. I am assuming you went with the docker container solution mentioned above.....

@distributorofpain You were dead on with 2 :smile: It's always obvious afterwards, huh? I'm making more progress now. 

Also, more details on Swarm on Mesos: https://github.com/docker/swarm/blob/master/cluster/mesos/README.md. In particular:

> The Docker Swarm on Mesos integration is experimental and not production ready, please use with caution.

For now, I'll get swarm running on its own, and Mesos integration can be a goal for the future. 

Second set of eyes thing :)  

I just started playing with docker last week.  I had to setup a registry and client to test it.  I really like docker, i can see why people are running with it.  Heck even mantl brought in kubernetes now :)

 I saw the demo on youtube of docker swarm running on messoshpere, very impressive.  Everyone is really charging forward with this stuff.

@moretea @distributorofpain @lguminski I put in a PR and would love some feedback! 


Resolving patch:
diff --git a/roles/lvm/files/mantl-storage-setup.py b/roles/lvm/files/mantl-storage-setup.py
--- a/roles/lvm/files/mantl-storage-setup.py
+++ b/roles/lvm/files/mantl-storage-setup.py
@@ -139,6 +139,15 @@ def parse_size(size):
     fail("size not specified")
 
 
+def wait_for_device(dev):
+    print "--> Wait for device {}".format(dev)
+    for sec in range(0, 60):
+        if os.path.exists(dev):
+            print "--> Device {} appears in {} seconds".format(dev, sec)
+            break
+        os.sleep(1)
+
+
 def process_volume(sec, params):
     lv = params.get(sec, "volume")
     vg = params.get(sec, "group")
@@ -176,6 +185,7 @@ def process_fs(sec, params):
 
     fstype = params.get(sec, "fstype")
 
+    wait_for_device(dev)
     try:
         exist_fs = check_output(["blkid", "-c", "/dev/null", "-o", "value", "-s", "TYPE", dev]).strip()
     except subprocess.CalledProcessError:


Test patch:
diff --git a/testing/terraform/aws.tf b/testing/terraform/aws.tf
--- a/testing/terraform/aws.tf
+++ b/testing/terraform/aws.tf
@@ -1,19 +1,163 @@
 variable "build_number" {}
+variable "amis" {
+  default = {
+    us-east-1      = "ami-6d1c2007"
+    us-west-2      = "ami-d2c924b2"
+    us-west-1      = "ami-af4333cf"
+    eu-central-1   = "ami-9bf712f4"
+    eu-west-1      = "ami-7abd0209"
+    ap-southeast-1 = "ami-f068a193"
+    ap-southeast-2 = "ami-fedafc9d"
+    ap-northeast-1 = "ami-eec1c380"
+    sa-east-1      = "ami-26b93b4a"
+  }
+}
+variable "availability_zones"  {
+  default = "a,b,c"
+}
+variable "control_count" { default = 3 }
+variable "datacenter" {default = "aws-us-west-2"}
+variable "edge_count" { default = 2 }
+variable "region" {default = "us-west-2"}
+variable "short_name" {default = "mantl-ci"}
+variable "long_name" {default = "ciscocloud-mantl-ci"}
+variable "ssh_username" {default = "centos"}
+variable "worker_count" { default = 4 }
+variable "kubeworker_count" { default = 2 }
+variable "dns_subdomain" { default = ".dev" }
+variable "dns_domain" { default = "my-domain.com" }
+variable "dns_zone_id" { default = "XXXXXXXXXXXX" }
+variable "control_type" { default = "m3.medium" }
+variable "edge_type" { default = "m3.medium" }
+variable "worker_type" { default = "m3.large" }
+variable "kubeworker_type" { default = "m3.large" }
 
 provider "aws" {
-  region = "us-west-1"
+  region = "${var.region}"
 }
 
-module "aws-mantl-testing" {
-  source = "./terraform/aws"
-  availability_zone = "us-west-1b"
-  ssh_username = "centos"
-  source_ami = "ami-af4333cf"
-  short_name = "mantl-ci-${var.build_number}"
-  long_name = "ciscocloud-mantl-ci-${var.build_number}"
+# _local is for development only s3 or something else should be used
+# https://github.com/hashicorp/terraform/blob/master/state/remote/remote.go#L47
+# https://www.terraform.io/docs/state/remote.html
+#resource "terraform_remote_state" "vpc" {
+#  backend = "_local"
+#  config {
+#    path = "./vpc/terraform.tfstate"
+#  }
+# }
+
+# s3 example
+#resource  "terraform_remote_state" "vpc" {
+# backend = "s3"
+#  config {
+#    bucket = "mybucketname"
+#   key = "name_of_state_file"
+#  }
+#}
+
+module "vpc" {
+  source ="./terraform/aws/vpc"
+  availability_zones = "${var.availability_zones}"
+  short_name = "${var.short_name}-${var.build_number}"
+  long_name = "${var.long_name}"
+  region = "${var.region}"
+}
+
+module "ssh-key" {
+  source ="./terraform/aws/ssh"
+  short_name = "${var.short_name}-${var.build_number}"
+}
+
+module "security-groups" {
+  source = "./terraform/aws/security_groups"
+  short_name = "${var.short_name}-${var.build_number}"
+  vpc_id = "${module.vpc.vpc_id}"
+}
+
+module "iam-profiles" {
+  source = "./terraform/aws/iam"
+  short_name = "${var.short_name}-${var.build_number}"
+}
+
+module "control-nodes" {
+  source = "./terraform/aws/instance"
+  count = "${var.control_count}"
+  datacenter = "${var.datacenter}"
+  role = "control"
+  ec2_type = "${var.control_type}"
+  iam_profile = "${module.iam-profiles.control_iam_instance_profile}"
+  ssh_username = "${var.ssh_username}"
+  source_ami = "${lookup(var.amis, var.region)}"
+  short_name = "${var.short_name}-${var.build_number}"
+  ssh_key_pair = "${module.ssh-key.ssh_key_name}"
+  availability_zones = "${module.vpc.availability_zones}"
+  security_group_ids = "${module.vpc.default_security_group},${module.security-groups.ui_security_group},${module.security-groups.control_security_group}"
+  vpc_subnet_ids = "${module.vpc.subnet_ids}"
+  # uncomment below it you want to use remote state for vpc variables
+  #availability_zones = "${terraform_remote_state.vpc.output.availability_zones}"
+  #security_group_ids = "${terraform_remote_state.vpc.output.default_security_group},${module.security-groups.ui_security_group},${module.security-groups.control_security_group}"
+  #vpc_subnet_ids = "${terraform_remote_state.vpc.output.subnet_ids}"
+}
+
+module "edge-nodes" {
+  source = "./terraform/aws/instance"
+  count = "${var.edge_count}"
+  datacenter = "${var.datacenter}"
+  role = "edge"
+  ec2_type = "${var.edge_type}"
+  ssh_username = "${var.ssh_username}"
+  source_ami = "${lookup(var.amis, var.region)}"
+  short_name = "${var.short_name}-${var.build_number}"
+  ssh_key_pair = "${module.ssh-key.ssh_key_name}"
+  availability_zones = "${module.vpc.availability_zones}"
+  security_group_ids = "${module.vpc.default_security_group},${module.security-groups.edge_security_group}"
+  vpc_subnet_ids = "${module.vpc.subnet_ids}"
+  # uncomment below it you want to use remote state for vpc variables
+  #availability_zones = "${terraform_remote_state.vpc.output.availability_zones}"
+  #security_group_ids = "${terraform_remote_state.vpc.output.default_security_group},${module.security-groups.edge_security_group}"
+  #vpc_subnet_ids = "${terraform_remote_state.vpc.output.subnet_ids}"
+}
+
+module "worker-nodes" {
+  source = "./terraform/aws/instance"
+  count = "${var.worker_count}"
+  count_format = "%03d"
+  datacenter = "${var.datacenter}"
+  data_ebs_volume_size = "100"
+  role = "worker"
+  ec2_type = "${var.worker_type}"
+  iam_profile = "${module.iam-profiles.worker_iam_instance_profile}"
+  ssh_username = "${var.ssh_username}"
+  source_ami = "${lookup(var.amis, var.region)}"
+  short_name = "${var.short_name}-${var.build_number}"
+  ssh_key_pair = "${module.ssh-key.ssh_key_name}"
+  availability_zones = "${module.vpc.availability_zones}"
+  security_group_ids = "${module.vpc.default_security_group},${module.security-groups.worker_security_group}"
+  vpc_subnet_ids = "${module.vpc.subnet_ids}"
+  # uncomment below it you want to use remote state for vpc variables
+  #availability_zones = "${terraform_remote_state.vpc.output.availability_zones}"
+  #security_group_ids = "${terraform_remote_state.vpc.output.default_security_group},${module.security-groups.worker_security_group}"
+  #vpc_subnet_ids = "${terraform_remote_state.vpc.output.subnet_ids}"
+}
 
-  control_count = 3
-  worker_count = 2
-  edge_count = 1
-  kubeworker_count = 2
+module "kubeworker-nodes" {
+  source = "./terraform/aws/instance"
+  count = "${var.kubeworker_count}"
+  count_format = "%03d"
+  datacenter = "${var.datacenter}"
+  data_ebs_volume_size = "100"
+  role = "kubeworker"
+  ec2_type = "${var.kubeworker_type}"
+  iam_profile = "${module.iam-profiles.worker_iam_instance_profile}"
+  ssh_username = "${var.ssh_username}"
+  source_ami = "${lookup(var.amis, var.region)}"
+  short_name = "${var.short_name}-${var.build_number}"
+  ssh_key_pair = "${module.ssh-key.ssh_key_name}"
+  availability_zones = "${module.vpc.availability_zones}"
+  security_group_ids = "${module.vpc.default_security_group},${module.security-groups.worker_security_group}"
+  vpc_subnet_ids = "${module.vpc.subnet_ids}"
+  # uncomment below it you want to use remote state for vpc variables
+  #availability_zones = "${terraform_remote_state.vpc.output.availability_zones}"
+  #security_group_ids = "${terraform_remote_state.vpc.output.default_security_group},${module.security-groups.worker_security_group}"
+  #vpc_subnet_ids = "${terraform_remote_state.vpc.output.subnet_ids}"
 }
diff --git a/testing/terraform/do.tf b/testing/terraform/do.tf
--- a/testing/terraform/do.tf
+++ b/testing/terraform/do.tf
@@ -1,22 +1,50 @@
 variable "build_number" {}
+variable "region" { default = "sfo1" } # Must have metadata support
 
-provider "digitalocean" {
-}
+provider "digitalocean" { }
 
-module "do-mantl-keypair" {
+module "do-keypair" {
   name = "ci-${var.build_number}"
   source = "./terraform/digitalocean/keypair"
   public_key_filename = "~/.ssh/id_rsa.pub"
 }
 
-module "do-mantl-hosts" {
+module "control-nodes" {
+  source = "./terraform/digitalocean/instance"
+  name = "ci-${var.build_number}"
+  region = "${var.region}"
+  keypair_id = "${module.do-keypair.keypair_id}"
+
+  role = "control"
+  count_format = "%02d"
+  count = "3"
+}
+
+module "worker-nodes" {
+  source = "./terraform/digitalocean/instance"
+  name = "ci-${var.build_number}"
+  region = "${var.region}"
+  keypair_id = "${module.do-keypair.keypair_id}"
+
+  role = "worker"
+}
+
+module "kubeworker-nodes" {
+  source = "./terraform/digitalocean/instance"
   name = "ci-${var.build_number}"
-  source = "./terraform/digitalocean/hosts"
-  ssh_key = "${module.do-mantl-keypair.keypair_id}"
-  region_name = "sfo1" # this must be a region with metadata support
-
-  control_count = 3
-  worker_count = 2
-  edge_count = 1
-  kubeworker_count = 2
+  region = "${var.region}"
+  keypair_id = "${module.do-keypair.keypair_id}"
+
+  role = "kubeworker"
+}
+
+module "edge-nodes" {
+  source = "./terraform/digitalocean/instance"
+  name = "ci-${var.build_number}"
+  region = "${var.region}"
+  keypair_id = "${module.do-keypair.keypair_id}"
+
+  role = "edge"
+  count = "1"
+  size = "2gb"
 }
diff --git a/testing/terraform/gce.tf b/testing/terraform/gce.tf
--- a/testing/terraform/gce.tf
+++ b/testing/terraform/gce.tf
@@ -1,17 +1,16 @@
 variable "build_number" {}
-
 variable "control_count" { default = 3 }
-variable "datacenter" {default = "mantl-dc"}
+variable "control_type" { default = "n1-standard-1" }
+variable "datacenter" {default = "gce"}
 variable "edge_count" { default = 1}
-variable "image" {default = "centos-7-v20160119"}
-variable "long_name" {default = "ciscocloud-mantl-ci-0-0"}
-variable "short_name" {default = "mantl-ci-0-0"}
+variable "edge_type" { default = "n1-standard-1" }
+variable "image" {default = "centos-7-v20160606"}
+variable "kubeworker_count" {default = 2}
+variable "long_name" {default = "ciscocloud-mantl-ci"}
+variable "short_name" {default = "mantl-ci"}
 variable "ssh_key" {default = "~/.ssh/id_rsa.pub"}
 variable "ssh_user" {default = "centos"}
-variable "kubeworker_count" {default = 0}
 variable "worker_count" {default = 2}
-variable "control_type" { default = "n1-standard-1" }
-variable "edge_type" { default = "n1-standard-1" }
 variable "worker_type" { default = "n1-standard-2" }
 variable "zones" {
   default = "us-central1-a,us-central1-b"
@@ -25,8 +24,8 @@ provider "google" {
 module "gce-network" {
   source = "./terraform/gce/network"
   network_ipv4 = "10.0.0.0/16"
-  long_name = "ciscocloud-mantl-ci-${var.build_number}"
-  short_name = "mantl-ci-${var.build_number}"
+  long_name = "${var.long_name}-${var.build_number}"
+  short_name = "${var.short_name}-${var.build_number}"
 }
 
 # remote state example
@@ -50,10 +49,11 @@ module "control-nodes" {
   network_name = "${module.gce-network.network_name}"
   #network_name = "${terraform_remote_state.gce-network.output.network_name}"
   role = "control"
-  short_name = "mantl-ci-${var.build_number}"
+  short_name = "${var.short_name}-${var.build_number}"
   ssh_user = "${var.ssh_user}"
   ssh_key = "${var.ssh_key}"
   zones = "${var.zones}"
+  volume_type = "pd-ssd"
 }
 
 module "edge-nodes" {
@@ -65,7 +65,7 @@ module "edge-nodes" {
   network_name = "${module.gce-network.network_name}"
   #network_name = "${terraform_remote_state.gce-network.output.network_name}"
   role = "edge"
-  short_name = "mantl-ci-${var.build_number}"
+  short_name = "${var.short_name}-${var.build_number}"
   ssh_user = "${var.ssh_user}"
   ssh_key = "${var.ssh_key}"
   zones = "${var.zones}"
@@ -80,7 +80,22 @@ module "worker-nodes" {
   network_name = "${module.gce-network.network_name}"
   #network_name = "${terraform_remote_state.gce-network.output.network_name}"
   role = "worker"
-  short_name = "mantl-ci-${var.build_number}"
+  short_name = "${var.short_name}-${var.build_number}"
+  ssh_user = "${var.ssh_user}"
+  ssh_key = "${var.ssh_key}"
+  zones = "${var.zones}"
+}
+
+module "kubeworker-nodes" {
+  source = "./terraform/gce/instance"
+  count = "${var.kubeworker_count}"
+  datacenter = "${var.datacenter}"
+  image = "${var.image}"
+  machine_type = "n1-highmem-2"
+  network_name = "${module.gce-network.network_name}"
+  #network_name = "${terraform_remote_state.gce-network.output.network_name}"
+  role = "kubeworker"
+  short_name = "${var.short_name}-${var.build_number}"
   ssh_user = "${var.ssh_user}"
   ssh_key = "${var.ssh_key}"
   zones = "${var.zones}"
@@ -89,5 +104,5 @@ module "worker-nodes" {
 module "network-lb" {
   source = "./terraform/gce/lb"
   instances = "${module.edge-nodes.instances}"
-  short_name = "mantl-ci-${var.build_number}"
+  short_name = "${var.short_name}-${var.build_number}"
 }