Repository: openshift/openshift-ansible

Issue:
GCE client certificates need to include the hostname
After changes in #3225, masters can't contact nodes (this is via exec from the test suite)

```
    [91mExpected error:
        <*errors.errorString | 0xc8204ee5a0>: {
            s: "host command failed: error running &{/data/src/github.com/openshift/origin/_output/local/bin/linux/amd64/kubectl [kubectl --server=https://internal-api.pr79.origin-ci-int-gce.dev.rhcloud.com:8443 --kubeconfig=/tmp/cluster-admin.kubeconfig exec --namespace=extended-test-scoped-router-3f3wx-f762m execpod -- /bin/sh -c \n\t\tset -e\n\t\tfor i in $(seq 1 180); do\n\t\t\tcode=$( curl -s -o /dev/null -w '%{http_code}\\n' --header 'Host: 172.16.2.54' \"http://172.16.2.54:1936/healthz\" )\n\t\t\techo $code\n\t\t\tif [[ $code -eq 200 ]]; then\n\t\t\t\texit 0\n\t\t\tfi\n\t\t\tif [[ $code -ne 503 ]]; then\n\t\t\t\texit 1\n\t\t\tfi\n\t\t\tsleep 1\n\t\tdone\n\t\t] []  <nil>  Error from server: error dialing backend: x509: certificate is valid for ci-pr79-ig-n-72h9.c.openshift-gce-devel-ci.internal, 10.128.0.3, 104.198.163.129, not ci-pr79-ig-n-72h9\n [] <nil> 0xc8216a4000 exit status 1 <nil> true [0xc8200325d0 0xc820032680 0xc820032728] [0xc8200325d0 0xc820032680 0xc820032728] [0xc820032678 0xc820032718] [0xaf79e0 0xaf79e0] 0xc820626ea0}:\nCommand stdout:\n\nstderr:\nError from server: error dialing backend: x509: certificate is valid for ci-pr79-ig-n-72h9.c.openshift-gce-devel-ci.internal, 10.128.0.3, 104.198.163.129, not ci-pr79-ig-n-72h9\n\nerror:\nexit status 1\n\n",
        }
        host command failed: error running &{/data/src/github.com/openshift/origin/_output/local/bin/linux/amd64/kubectl [kubectl --server=https://internal-api.pr79.origin-ci-int-gce.dev.rhcloud.com:8443 --kubeconfig=/tmp/cluster-admin.kubeconfig exec --namespace=extended-test-scoped-router-3f3wx-f762m execpod -- /bin/sh -c 
        		set -e
        		for i in $(seq 1 180); do
        			code=$( curl -s -o /dev/null -w '%{http_code}\n' --header 'Host: 172.16.2.54' "http://172.16.2.54:1936/healthz" )
        			echo $code
        			if [[ $code -eq 200 ]]; then
        				exit 0
        			fi
        			if [[ $code -ne 503 ]]; then
        				exit 1
        			fi
        			sleep 1
        		done
        		] []  <nil>  Error from server: error dialing backend: x509: certificate is valid for ci-pr79-ig-n-72h9.c.openshift-gce-devel-ci.internal, 10.128.0.3, 104.198.163.129, not ci-pr79-ig-n-72h9
         [] <nil> 0xc8216a4000 exit status 1 <nil> true [0xc8200325d0 0xc820032680 0xc820032728] [0xc8200325d0 0xc820032680 0xc820032728] [0xc820032678 0xc820032718] [0xaf79e0 0xaf79e0] 0xc820626ea0}:
        Command stdout:
        
        stderr:
        Error from server: error dialing backend: x509: certificate is valid for ci-pr79-ig-n-72h9.c.openshift-gce-devel-ci.internal, 10.128.0.3, 104.198.163.129, not ci-pr79-ig-n-72h9
        
        error:
        exit status 1
        
        
    not to have occurred[0m
```

That means cert generation for masters is wrong as well.


Discussion:
@detiber @sdodson 
From https://ci.openshift.redhat.com/jenkins/job/zz_origin_gce_image/79/consoleFull
After some digging, I realized that Kubelet sucks and can't set a hostname (reported via node API object) that is different than its node name (the only parameter is hostnameOverride which maps to nodeName in origin).  

This means that we have to change cert generation to use the short name on GCE as well.  Setting `openshift_hostname` to the value from the metadata did not work.  Not sure if this was a mistake in how I configured the value as `openshift_hostname: "{{ openshift.node.nodename }}"`.  Led to a failure in etcd generation

```
TASK [etcd_server_certificates : Sign and create the server crt] ***************
task path: /usr/share/ansible/openshift-ansible/roles/etcd_server_certificates/tasks/main.yml:49
Tuesday 31 January 2017  05:00:40 +0000 (0:00:00.032)       0:02:11.197 *******
Using module file /usr/share/ansible/openshift-ansible/roles/etcd_common/library/delegated_serial_command.py
<104.197.65.51> ESTABLISH SSH CONNECTION FOR USER: cloud-user
<104.197.65.51> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=900s -o GSSAPIAuthentication=no -o StrictHostKeyChecking=no -o 'IdentityFile="/home/cloud-user/.ssh/google_compute_engine"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=cloud-user -o ConnectTimeout=10 -o ControlPath=/var/tmp/%h-%r 104.197.65.51 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-xufzjfmjkofxqyjebsfthxtfntrsvnrg; SAN=IP:10.128.0.5 /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_fyPOkZ/ansible_module_delegated_serial_command.py", line 274, in <module>
    main()
  File "/tmp/ansible_fyPOkZ/ansible_module_delegated_serial_command.py", line 204, in main
    os.chdir(chdir)
OSError: [Errno 2] No such file or directory: '/etc/etcd/generated_certs/etcd-ci-prclaytontest-ig-m-sh2v'
```
Actually, that was on a re-run, not on a fresh cluster.  So changing openshift_hostname for regen = bad.  But this may get me going.
Spoke too soon:

```
 {"failed": true, "msg": "the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: {{ openshift.node.nodename }}: 'openshift' is undefined\n\nThe error appears to have been in '/usr/share/ansible/openshift-ansible/roles/openshift_facts/tasks/main.yml': line 18, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Gather Cluster facts and set is_containerized if needed\n  ^ here\n"}
```

after setting it via vars on the playbook include for byo config.  Hrm....
I don't know how to set the value so it is resolved after openshift facts is run, but provides an input to openshift facts.  I guess this has to be fixed in openshift_node_certificates, and it has to use the right value from the provider for each node.

Resolving patch:
diff --git a/roles/openshift_facts/library/openshift_facts.py b/roles/openshift_facts/library/openshift_facts.py
--- a/roles/openshift_facts/library/openshift_facts.py
+++ b/roles/openshift_facts/library/openshift_facts.py
@@ -195,8 +195,7 @@ def hostname_valid(hostname):
     if (not hostname or
             hostname.startswith('localhost') or
             hostname.endswith('localdomain') or
-            hostname.endswith('novalocal') or
-            len(hostname.split('.')) < 2):
+            hostname.endswith('novalocal')):
         return False
 
     return True
@@ -332,7 +331,8 @@ def normalize_gce_facts(metadata, facts):
     facts['network']['ip'] = facts['network']['interfaces'][0]['ips'][0]
     pub_ip = facts['network']['interfaces'][0]['public_ips'][0]
     facts['network']['public_ip'] = pub_ip
-    facts['network']['hostname'] = metadata['instance']['hostname']
+    # Split instance hostname from GCE metadata to use the short instance name
+    facts['network']['hostname'] = metadata['instance']['hostname'].split('.')[0]
 
     # TODO: attempt to resolve public_hostname
     facts['network']['public_hostname'] = facts['network']['public_ip']


Test patch: