GCPにBOSHの環境を構築する


本当の目的は[Deploying Cloud Foundry on Google Compute Engine](https://github.com/cloudfoundry-incubator/bosh-google-cpi-release/tree/master/docs/cloudfoundry)に従ってGCP上にCloud Foundryの環境を構築することなのですが、その前に、前提条件となるBOSHの動作環境が必要なので、[Deploy BOSH on Google Cloud Platform](https://github.com/cloudfoundry-incubator/bosh-google-cpi-release/blob/master/docs/bosh/README.md)に従って構築します。

## 関連ソフトウェア

この作業の中ではBOSH以外にも以下のソフトウェアが使用されています。

* [Terraform](https://www.terraform.io/)
	* オーケストレーションツール

## BOSH環境構築

以降の作業はブラウザからGoogle Cloud Shellを起動して実行します。

使用するGCPのアカウントやリージョンなどの情報を環境変数に設定します。

基本ドキュメントのままです。

```
export project_id=$(gcloud config list 2>/dev/null | grep project | sed -e 's/project = //g')
export region=us-east1
export zone=us-east1-d
export service_account_email=terraform@${project_id}.iam.gserviceaccount.com
gcloud config set compute/zone ${zone}
gcloud config set compute/region ${region}
gcloud iam service-accounts create terraform
gcloud iam service-accounts keys create ~/terraform.key.json \
    --iam-account ${service_account_email}
gcloud projects add-iam-policy-binding ${project_id} \
  --member serviceAccount:${service_account_email} \
  --role roles/owner
export GOOGLE_CREDENTIALS=$(cat ~/terraform.key.json)
```

必要なソースをgitリポジトリからクローンします。

```
git clone https://github.com/cloudfoundry-incubator/bosh-google-cpi-release.git
cd bosh-google-cpi-release/docs/bosh
```

`main.tf`を覗いてみます。Terraformの設定ファイルです。
サブネット、ルート、ファイヤーウォール、GCEインスタンスなどの設定が記述してあります。


```
$ cat main.tf
variable "projectid" {
    type = "string"
}

variable "region" {
    type = "string"
    default = "us-east1"
}

variable "zone" {
    type = "string"
    default = "us-east1-d"
}

variable "prefix" {
    type = "string"
    default = ""
}

variable "service_account_email" {
    type = "string"
    default = ""
}

provider "google" {
    project = "${var.projectid}"
    region = "${var.region}"
}

resource "google_compute_network" "bosh" {
  name       = "${var.prefix}bosh"
}

resource "google_compute_route" "nat-primary" {
  name        = "${var.prefix}nat-primary"
  dest_range  = "0.0.0.0/0"
  network       = "${google_compute_network.bosh.name}"
  next_hop_instance = "${google_compute_instance.nat-instance-private-with-nat-primary.name}"
  next_hop_instance_zone = "${var.zone}"
  priority    = 800
  tags = ["no-ip"]
}

// Subnet for the BOSH director
resource "google_compute_subnetwork" "bosh-subnet-1" {
  name          = "${var.prefix}bosh-${var.region}"
  ip_cidr_range = "10.0.0.0/24"
  network       = "${google_compute_network.bosh.self_link}"
}

// Allow SSH to BOSH bastion
resource "google_compute_firewall" "bosh-bastion" {
  name    = "${var.prefix}bosh-bastion"
  network = "${google_compute_network.bosh.name}"

  allow {
    protocol = "icmp"
  }

  allow {
    protocol = "tcp"
    ports    = ["22"]
  }

  target_tags = ["bosh-bastion"]
}

// Allow all traffic within subnet
resource "google_compute_firewall" "intra-subnet-open" {
  name    = "${var.prefix}intra-subnet-open"
  network = "${google_compute_network.bosh.name}"

  allow {
    protocol = "icmp"
  }

  allow {
    protocol = "tcp"
    ports    = ["1-65535"]
  }

  allow {
    protocol = "udp"
    ports    = ["1-65535"]
  }

  source_tags = ["internal"]
}

// BOSH bastion host
resource "google_compute_instance" "bosh-bastion" {
  name         = "${var.prefix}bosh-bastion"
  machine_type = "n1-standard-1"
  zone         = "${var.zone}"

  tags = ["bosh-bastion", "internal"]

  disk {
    image = "ubuntu-1404-trusty-v20161020"
  }

  network_interface {
    subnetwork = "${google_compute_subnetwork.bosh-subnet-1.name}"
    access_config {
      // Ephemeral IP
    }
  }

  metadata_startup_script = <<EOT
#!/bin/bash
cat > /etc/motd <<EOF




#    #    ##    #####   #    #     #    #    #   ####
#    #   #  #   #    #  ##   #     #    ##   #  #    #
#    #  #    #  #    #  # #  #     #    # #  #  #
# ## #  ######  #####   #  # #     #    #  # #  #  ###
##  ##  #    #  #   #   #   ##     #    #   ##  #    #
#    #  #    #  #    #  #    #     #    #    #   ####

Startup scripts have not finished installing, and the tools you need
are not ready yet. Please log out and log back in again in a few moments.
This warning will not appear when the system is ready.
EOF

apt-get update
apt-get install -y build-essential zlibc zlib1g-dev ruby ruby-dev openssl libxslt-dev libxml2-dev libssl-dev libreadline6 libreadline6-dev libyaml-dev libsqlite3-dev sqlite3 jq git unzip
gem install bosh_cli
curl -o /tmp/cf.tgz https://s3.amazonaws.com/go-cli/releases/v6.20.0/cf-cli_6.20.0_linux_x86-64.tgz
tar -zxvf /tmp/cf.tgz && mv cf /usr/bin/cf && chmod +x /usr/bin/cf
curl -o /usr/bin/bosh-init https://s3.amazonaws.com/bosh-init-artifacts/bosh-init-0.0.96-linux-amd64
chmod +x /usr/bin/bosh-init

cat > /etc/profile.d/bosh.sh <<'EOF'
#!/bin/bash
# Misc vars
export prefix=${var.prefix}
export ssh_key_path=$HOME/.ssh/bosh

# Vars from Terraform
export subnetwork=${google_compute_subnetwork.bosh-subnet-1.name}
export network=${google_compute_network.bosh.name}


# Vars from metadata service
export project_id=$$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/project/project-id)
export zone=$$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/zone)
export zone=$${zone##*/}
export region=$${zone%-*}

# Configure gcloud
gcloud config set compute/zone $${zone}
gcloud config set compute/region $${region}
EOF

# Clone repo
mkdir /share
git clone https://github.com/cloudfoundry-incubator/bosh-google-cpi-release.git /share
chmod -R 777 /share

# Install Terraform
wget https://releases.hashicorp.com/terraform/0.7.7/terraform_0.7.7_linux_amd64.zip
unzip terraform*.zip -d /usr/local/bin
rm /etc/motd
EOT

  service_account {
    email = "${var.service_account_email}"
    scopes = ["cloud-platform"]
  }
}

// NAT server (primary)
resource "google_compute_instance" "nat-instance-private-with-nat-primary" {
  name         = "${var.prefix}nat-instance-primary"
  machine_type = "n1-standard-1"
  zone         = "${var.zone}"

  tags = ["nat", "internal"]

  disk {
    image = "ubuntu-1404-trusty-v20161020"
  }

  network_interface {
    subnetwork = "${google_compute_subnetwork.bosh-subnet-1.name}"
    access_config {
      // Ephemeral IP
    }
  }

  can_ip_forward = true

  metadata_startup_script = <<EOT
#!/bin/bash
sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
EOT
}
```

作成されるリソースを確認します。

```
docker run -i -t \
  -e "GOOGLE_CREDENTIALS=${GOOGLE_CREDENTIALS}" \
  -v `pwd`:/$(basename `pwd`) \
  -w /$(basename `pwd`) \
  hashicorp/terraform:light plan \
    -var service_account_email=${service_account_email} \
    -var projectid=${project_id} \
    -var region=${region} \
    -var zone=${zone}
```

作成するリソースがわかりやすく表示されています。

```stdout
+ google_compute_firewall.bosh-bastion
    allow.#:                   "2"
    allow.1367131964.ports.#:  "0"
    allow.1367131964.protocol: "icmp"
    allow.803338340.ports.#:   "1"
    allow.803338340.ports.0:   "22"
    allow.803338340.protocol:  "tcp"
    name:                      "bosh-bastion"
    network:                   "bosh"
    project:                   "<computed>"
    self_link:                 "<computed>"
    target_tags.#:             "1"
    target_tags.1860295641:    "bosh-bastion"

+ google_compute_route.nat-primary
    dest_range:             "0.0.0.0/0"
    name:                   "nat-primary"
    network:                "bosh"
    next_hop_instance:      "nat-instance-primary"
    next_hop_instance_zone: "us-east1-d"
    next_hop_network:       "<computed>"
    priority:               "800"
    self_link:              "<computed>"
    tags.#:                 "1"
    tags.1472900357:        "no-ip"

+ google_compute_subnetwork.bosh-subnet-1
    gateway_address: "<computed>"
    ip_cidr_range:   "10.0.0.0/24"
    name:            "bosh-us-east1"
    network:         "<computed>"
    self_link:       "<computed>"
Plan: 7 to add, 0 to change, 0 to destroy.
```

リソースを作成します。

```
docker run -i -t \
  -e "GOOGLE_CREDENTIALS=${GOOGLE_CREDENTIALS}" \
  -v `pwd`:/$(basename `pwd`) \
  -w /$(basename `pwd`) \
  hashicorp/terraform:light apply \
    -var service_account_email=${service_account_email} \
    -var projectid=${project_id} \
    -var region=${region} \
    -var zone=${zone}
```

Cloud shellで確認してみると、リソースが作成されています。

```
$ gcloud compute networks list 
NAME     MODE    IPV4_RANGE  GATEWAY_IPV4
bosh     custom
default  auto

$ gcloud compute networks subnets list 
NAME           REGION           NETWORK  RANGE
default        asia-northeast1  default  10.146.0.0/20
default        us-west1         default  10.138.0.0/20
bosh-us-east1  us-east1         bosh     10.0.0.0/24
default        us-east1         default  10.142.0.0/20
default        europe-west1     default  10.132.0.0/20
default        asia-east1       default  10.140.0.0/20
default        us-central1      default  10.128.0.0/20

$ gcloud compute instances list 
NAME                  ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
bosh-bastion          us-east1-d  n1-standard-1               10.0.0.2     XXX.XXX.XXX.XXX  RUNNING
nat-instance-primary  us-east1-d  n1-standard-1               10.0.0.3     XXX.XXX.XXX.XXX   RUNNING
```

bash-bastionにsshでログインします。

```
gcloud compute ssh bosh-bastion
```

VM上でGoogle Cloud SDKを最新版にアップデートします。

```
sudo gcloud components update
```

BOSHが使うサービスアカウントを作成します。

```
export service_account=bosh-user
export service_account_email=${service_account}@${project_id}.iam.gserviceaccount.com
gcloud iam service-accounts create ${service_account}
```

ここで以下のように権限エラーが出た場合は、現在ログインしているアカウントを確認し、iamの操作権限を持っているアカウントに変更します。

```
$ gcloud iam service-accounts create ${service_account}
ERROR: (gcloud.iam.service-accounts.create) You do not have permission to access project [cf-test] (or it may not exist
): Google Identity and Access Management (IAM) API has not been used in project cf-test before or it is disabled. Enabl
e it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=cf-test then retry.
 If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
```
```
$ gcloud config list
Your active configuration is: [default]
[compute]
region = us-east1
zone = us-east1-d
[core]
account = terraform@cf-test.iam.gserviceaccount.com
project = cf-test

$ gcloud config set core/account tommarute@example.com
Updated property [core/account].

$ gcloud auth login

$ gcloud config list
Your active configuration is: [default]

[compute]
region = us-east1
zone = us-east1-d
[core]
account = tommarute@example.com
project = cf-test
```

アカウントに権限を設定します。

```
gcloud projects add-iam-policy-binding ${project_id} \
  --member serviceAccount:${service_account_email} \
  --role roles/compute.instanceAdmin
gcloud projects add-iam-policy-binding ${project_id} \
  --member serviceAccount:${service_account_email} \
  --role roles/compute.storageAdmin
gcloud projects add-iam-policy-binding ${project_id} \
  --member serviceAccount:${service_account_email} \
  --role roles/storage.admin
gcloud projects add-iam-policy-binding ${project_id} \
  --member serviceAccount:${service_account_email} \
  --role  roles/compute.networkAdmin
gcloud projects add-iam-policy-binding ${project_id} \
  --member serviceAccount:${service_account_email} \
  --role roles/iam.serviceAccountActor
```

パスワードなしのsshキーを作成してアップロードします。

```
ssh-keygen -t rsa -f ~/.ssh/bosh -C bosh
gcloud compute project-info add-metadata --metadata-from-file \
         sshKeys=<( gcloud compute project-info describe --format=json | jq -r '.commonInstanceMetadata.items[] | select(.key == "sshKeys") | .value' & echo "bosh:$(cat ~/.ssh/bosh.pub)" )

```

インストールされたBOSHのバージョンを確認します。

```
bosh-init -v
```

マニフェストテンプレートを作成するディレクトリを作ります。

```
mkdir google-bosh-director
cd google-bosh-director
```

テンプレートを作成します。
10.0.0.6がbosh-directorになるようです。
DirectorはVMの管理を行います。
CPIというのはBOSHがIaaSを操作するインタフェースです。

```
$ cat << EOF > manifest.yml.erb
> ---
> name: bosh
> 
> releases:
>   - name: bosh
>     url: https://bosh.io/d/github.com/cloudfoundry/bosh?v=257.3
>     sha1: e4442afcc64123e11f2b33cc2be799a0b59207d0
>   - name: bosh-google-cpi
>     url: https://bosh.io/d/github.com/cloudfoundry-incubator/bosh-google-cpi-release?v=25.6.0
>     sha1: 3e01539a1228d62b8015feb388df2234978efaf6
> 
> resource_pools:
>   - name: vms
>     network: private
>     stemcell:
>       url: https://bosh.io/d/stemcells/bosh-google-kvm-ubuntu-trusty-go_agent?v=3263.7
>       sha1: a09ce8b4acfa9876f52ee7b4869b4b23f27d5ace
>     cloud_properties:
>       zone: <%= ENV['zone'] %>
>       machine_type: n1-standard-1
>       root_disk_size_gb: 40
>       root_disk_type: pd-standard
>       service_account: <%= ENV['service_account_email'] %>
> 
> disk_pools:
>   - name: disks
>     disk_size: 32_768
>     cloud_properties:
>       type: pd-standard
> 
> networks:
>   - name: vip
>     type: vip
>   - name: private
>     type: manual
>     subnets:
>     - range: 10.0.0.0/29
>       gateway: 10.0.0.1
>       static: [10.0.0.3-10.0.0.7]
>       cloud_properties:
>         network_name: <%= ENV['network'] %>
>         subnetwork_name: <%= ENV['subnetwork'] %>
>         ephemeral_external_ip: false
>         tags:
>           - internal
>           - no-ip
> 
> jobs:
>   - name: bosh
>     instances: 1
> 
>     templates:
>       - name: nats
>         release: bosh
>       - name: postgres
>         release: bosh
>       - name: powerdns
>         release: bosh
>       - name: blobstore
>         release: bosh
>       - name: director
>         release: bosh
>       - name: health_monitor
>         release: bosh
>       - name: google_cpi
>         release: bosh-google-cpi
> 
>     resource_pool: vms
>     persistent_disk_pool: disks
> 
>     networks:
>       - name: private
>         static_ips: [10.0.0.6]
>         default:
>           - dns
>           - gateway
> 
>     properties:
>       nats:
>         address: 127.0.0.1
>         user: nats
>         password: nats-password
> 
>       postgres: &db
>         listen_address: 127.0.0.1
>         host: 127.0.0.1
>         user: postgres
>         password: postgres-password
>         database: bosh
>         adapter: postgres
> 
>       dns:
>         address: 10.0.0.6
>         domain_name: microbosh
>         db: *db
>         recursor: 169.254.169.254
> 
>       blobstore:
>         address: 10.0.0.6
>         port: 25250
>         provider: dav
>         director:
>           user: director
>           password: director-password
>         agent:
>           user: agent
>           password: agent-password
> 
>       director:
>         address: 127.0.0.1
>         name: micro-google
>         db: *db
>         cpi_job: google_cpi
>         user_management:
>           provider: local
>           local:
>             users:
>               - name: admin
>                 password: admin
>               - name: hm
>                 password: hm-password
>       hm:
>         director_account:
>           user: hm
>           password: hm-password
>         resurrector_enabled: true
> 
>       google: &google_properties
>         project: <%= ENV['project_id'] %>
> 
>       agent:
>         mbus: nats://nats:nats-password@10.0.0.6:4222
>         ntp: *ntp
>         blobstore:
>            options:
>              endpoint: http://10.0.0.6:25250
>              user: agent
>              password: agent-password
> 
>       ntp: &ntp
>         - 169.254.169.254
> 
> cloud_provider:
>   template:
>     name: google_cpi
>     release: bosh-google-cpi
> 
>   template:
>   ssh_tunnel:
>     host: 10.0.0.6
>     port: 22
>     user: bosh
>     private_key: <%= ENV['ssh_key_path'] %>
> 
>   mbus: https://mbus:mbus-password@10.0.0.6:6868
> 
>   properties:
>     google: *google_properties
>     agent: {mbus: "https://mbus:mbus-password@0.0.0.0:6868"}
>     blobstore: {provider: local, path: /var/vcap/micro_bosh/data/cache}
>     ntp: *ntp
> EOF
```

テンプレートからYAMLファイルを作成します。

```
erb manifest.yml.erb > manifest.yml
```

BOSH Directorを作成するためのマニフェストを展開します。

```
bosh-init deploy manifest.yml
```

BOSH CLIにリモートの環境を指定します。

```
bosh target 10.0.0.6
# 認証はadmin/admin
```

[次回](http://qiita.com/tommarute/items/2720b1b37c7442c66faf)はCloud Foundryの環境を構築します。

## 参考

* https://github.com/cloudfoundry-incubator/bosh-google-cpi-release/blob/master/docs/cloudfoundry/README.md
* https://github.com/cloudfoundry-incubator/bosh-google-cpi-release/blob/master/docs/bosh/README.md
* http://www.slideshare.net/i_yudai/bosh-13404516
* https://blog.ik.am/entries/379