Repository: GoogleCloudPlatform/PerfKitBenchmarker

Issue:
cluster_boot seems to over-report boot time
The [code docs say that cluster_boot reports how long it takes for a VM to become responsive to SSH](https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/c1e4945ad34bd47c409ebb55d538d22f5463a643/perfkitbenchmarker/benchmarks/cluster_boot_benchmark.py#L48), but I experience much faster ssh response times for GCE than PKB is reporting.

PKB for gcloud:

```
CLUSTER_BOOT:
  num_cpus="1" num_vms="10" # n1-standard-1
  Boot Time                            78.790919 seconds                        (machine_instance="0")
  Boot Time                            82.916165 seconds                        (machine_instance="1")
  Boot Time                            82.787568 seconds                        (machine_instance="2")
  Boot Time                            82.920172 seconds                        (machine_instance="3")
  Boot Time                            78.791941 seconds                        (machine_instance="4")
  ...
```

Manually:

``` sh
$ START=`date +%s`; CREATE=`gcloud compute instances create zb-boot-1 --format json --zone us-central1-a --image ubuntu-14-04 --boot-disk-auto-delete --boot-disk-type pd-standard --boot-disk-size 10 --machine-type n1-standard-1 --no-restart-on-failure --preemptible`; echo "running: $((`date +%s`-$START))"; gcloud compute --quiet ssh zb-boot-1 --command "pwd"; echo "ssh response: $((`date +%s`-$START))"; gcloud compute instances delete zb-boot-1 --quiet
ssh response: 52
```

49 to 52 seconds is pretty consistent for that image, machine type and disk type, which is to say cluster_boot over-reports by about 30 seconds. I haven't followed PKB's code too closely, but is the "boot time" including the time to create the network and all other resources, not just the VM? I think just the VM is the most metric useful if people are evaluating how quickly a VM can be contribute to an autoscaling group, for example. (That's my use case, at least.)

For AWS EC2, standard EBS, m4.large, ubuntu (ami-0fbce65), cluster_boot reports ~84 seconds. Manually I get 67 seconds until ping, 80 seconds until SSH. Not off by nearly as much.

(Aside, cluster_boot is running stuff like `sudo apt-get update`. It doesn't seem to be timing those commands, but they waste time if you're trying to make these benchmarks run faster.)

(Very happy this benchmark kit exists, thanks for releasing it!)



Discussion:
Thanks for reporting this. For GCP: right now I believe the network creation is not included in the reported time, but creating a firewall rule to allow SSH access is, which is causing these numbers to appear higher. I'll check into it.


Resolving patch:
diff --git a/perfkitbenchmarker/benchmark_spec.py b/perfkitbenchmarker/benchmark_spec.py
--- a/perfkitbenchmarker/benchmark_spec.py
+++ b/perfkitbenchmarker/benchmark_spec.py
@@ -325,11 +325,11 @@ def PrepareVm(self, vm):
     Args:
         vm: The BaseVirtualMachine object representing the VM.
     """
+    for port in vm.remote_access_ports:
+      vm.AllowPort(port)
     vm.Create()
     logging.info('VM: %s', vm.ip_address)
     logging.info('Waiting for boot completion.')
-    for port in vm.remote_access_ports:
-      vm.AllowPort(port)
     vm.AddMetadata(benchmark=self.name, perfkit_uuid=self.uuid,
                    benchmark_uid=self.uid)
     vm.WaitForBootCompletion()


Test patch: