Repository: openshift/openshift-ansible

Issue:
OpenShift 3.9 origin install on aws fails memory_availability check due to skopeo not finding etcd on registry.access.redhat.com
#### Description

Provide a brief description of your issue here. For example:

On a simple AWS CentOS install,  memory availability check fails as follows:

CHECK [memory_availability : ec2-54-210-205-226.compute-1.amazonaws.com] ***********************************************************************************************************************
fatal: [ec2-54-210-205-226.compute-1.amazonaws.com]: FAILED! => {"changed": true, "checks": {"disk_availability": {}, "docker_image_availability": {"changed": true, "failed": true, "failures": [["OpenShiftCheckException", "One or more required container images are not available:\n    registry.access.redhat.com/rhel7/etcd\nChecked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>\nDefault registries searched: docker.io\n"]], "msg": "One or more required container images are not available:\n    registry.access.redhat.com/rhel7/etcd\nChecked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>\nDefault registries searched: docker.io\n"}, "docker_storage": {"changed": true, "data_pct_used": 0.03787602219626169, "data_threshold": 90.0, "data_total": "21.36 GB", "data_total_bytes": 22935125360.64, "data_used": "19.92 MB", "data_used_bytes": 20887633.92, "metadata_pct_used": 0.00015596679096317636, "metadata_threshold": 90.0, "metadata_total": "54.53 MB", "metadata_total_bytes": 57178849.28, "metadata_used": "49.15 kB", "metadata_used_bytes": 50329.6, "msg": "Thinpool usage is within thresholds.", "vg_free": "30.00g", "vg_free_bytes": 32212254720.0}, "memory_availability": {}, "package_availability": {"skipped": true, "skipped_reason": "Not active for this host"}, "package_version": {"skipped": true, "skipped_reason": "Not active for this host"}}, "msg": "One or more checks failed", "playbook_context": "install"}


If I set 

osm_etcd_image=registry.redhat.io/rhel7/etcd

The memory test passes, but the etcd image gets downloaded from fedora which is where it's supposed to come from in an origin

##### Version
```
openshift-ansible-3.9.15-1-16-g87b143960

ansible 2.5.0
```

##### Steps To Reproduce
1. set up infrastructure using the following cloud formation template:
https://s3.amazonaws.com/ocp-cloud-formation-template/CloudFormationTemplateOpenShift.yaml
2. [step 2] run the following playbook to set up hosts:
```
---
- hosts: nodes
  gather_facts: no
  pre_tasks:
    - name: 'install python2'
      raw: sudo yum install -y python
      
  tasks:

    - name: upgrade packages
      yum: state=latest name={{ item }}
      with_items:
        # - kernel*
        - docker-1.13.1
        - NetworkManager
        - emacs
        - wget
        - git
        - net-tools
        - bind-utils
        - iptables-services
        - bridge-utils
        - bash-completion
        - kexec-tools
        - sos
        - psacct
        - atomic

    - name: enable network-manager
      shell: systemctl enable NetworkManager && systemctl start NetworkManager

    - name: docker storage conf file
      copy:
        content: "DEVS=/dev/xvdb\nVG=docker-vg\n"
        dest: /etc/sysconfig/docker-storage-setup

    - name: docker-storage-setup
      shell: docker-storage-setup

    - name: enable docker
      shell: systemctl enable docker && systemctl start docker

```
Use an inventory file something like this, substituting the proper hostnames and aws access/secret keys

```
[OSEv3:children]
masters
etcd
nodes

[OSEv3:vars]
ansible_ssh_user=centos
ansible_sudo=true
ansible_become=true

debug_level=4

openshift_deployment_type=origin

openshift_release=v3.9
openshift_image_tag=v3.9.0

# *************
# If you leave this commented out, you see the issue... uncommenting it makes the problem go away
# *************
# osm_etcd_image=registry.redhat.io/rhel7/etcd

# Only needed for Container Platform
#ocp_version=3.9

openshift_cloudprovider_kind=aws
openshift_clusterid=sdb_cluster_1

openshift_cloudprovider_aws_access_key= ...
openshift_cloudprovider_aws_secret_key= ...

containerized=true

os_sdn_network_plugin_name='redhat/openshift-ovs-multitenant'

openshift_install_examples=true

openshift_docker_options='--selinux-enabled --insecure-registry 172.30.0.0/16'
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

osm_cluster_network_cidr=10.128.0.0/14
openshift_portal_net=172.30.0.0/16


#openshift_disable_check=disk_availability,docker_storage,memory_availability

openshift_repos_enable_testing=True

[masters]
ec2-54-210-205-226.compute-1.amazonaws.com

[etcd]
ec2-54-210-205-226.compute-1.amazonaws.com

[nodes]
ec2-54-210-205-226.compute-1.amazonaws.com openshift_node_labels="{'region':'infra','zone':'east'}" openshift_schedulable=true
ec2-52-87-203-168.compute-1.amazonaws.com openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
ec2-54-144-236-48.compute-1.amazonaws.com openshift_node_labels="{'region': 'primary', 'zone': 'east'}"

```
then run prerequisites.yml and deploy_cluster.yml

##### Expected Results
Since I'm trying to install origin, the access check shouldn't look at redhat.com anyway...

```
Example command and output or error messages
```

##### Observed Results
Describe what is actually happening.

```
CHECK [memory_availability : ec2-54-210-205-226.compute-1.amazonaws.com] ***********************************************************************************************************************
fatal: [ec2-54-210-205-226.compute-1.amazonaws.com]: FAILED! => {"changed": true, "checks": {"disk_availability": {}, "docker_image_availability": {"changed": true, "failed": true, "failures": [["OpenShiftCheckException", "One or more required container images are not available:\n    registry.access.redhat.com/rhel7/etcd\nChecked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>\nDefault registries searched: docker.io\n"]], "msg": "One or more required container images are not available:\n    registry.access.redhat.com/rhel7/etcd\nChecked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>\nDefault registries searched: docker.io\n"}, "docker_storage": {"changed": true, "data_pct_used": 0.03787602219626169, "data_threshold": 90.0, "data_total": "21.36 GB", "data_total_bytes": 22935125360.64, "data_used": "19.92 MB", "data_used_bytes": 20887633.92, "metadata_pct_used": 0.00015596679096317636, "metadata_threshold": 90.0, "metadata_total": "54.53 MB", "metadata_total_bytes": 57178849.28, "metadata_used": "49.15 kB", "metadata_used_bytes": 50329.6, "msg": "Thinpool usage is within thresholds.", "vg_free": "30.00g", "vg_free_bytes": 32212254720.0}, "memory_availability": {}, "package_availability": {"skipped": true, "skipped_reason": "Not active for this host"}, "package_version": {"skipped": true, "skipped_reason": "Not active for this host"}}, "msg": "One or more checks failed", "playbook_context": "install"}

```

##### Additional Information

Provide any additional information which may help us diagnose the
issue.

* Your operating system and version, ie: RHEL 7.2, Fedora 23 (`$ cat /etc/redhat-release`)
* Your inventory file (especially any non-standard configuration parameters)
* Sample code, etc

```
EXTRA INFORMATION GOES HERE
```



Discussion:
One more comment:  Probably a shot in the dark but...I read through 

https://github.com/openshift/openshift-ansible/blob/eb105467a2549343cc83f525367891bd6e967b50/roles/openshift_health_checker/openshift_checks/docker_image_availability.py

And it looks kinda broken to me because it's explicitly looking for the redhat repo, not the fedora repo when it does the check:  line 190 is looking for osm_etcd_image rather than using the dictionary defined in 

https://github.com/openshift/openshift-ansible/blob/06d3c6894495dbd4584b4f1fc6a286cf2366842c/roles/etcd/defaults/main.yaml

Resolving patch:
diff --git a/roles/openshift_health_checker/openshift_checks/docker_image_availability.py b/roles/openshift_health_checker/openshift_checks/docker_image_availability.py
--- a/roles/openshift_health_checker/openshift_checks/docker_image_availability.py
+++ b/roles/openshift_health_checker/openshift_checks/docker_image_availability.py
@@ -187,7 +187,7 @@ def add_var_or_default_img(var_name, comp_name):
                 add_var_or_default_img("osm_image", image_info["name"])
             if 'oo_etcd_to_config' in host_groups:
                 # special case, note default is the same for origin/enterprise and has no image tag
-                etcd_img = self.get_var("osm_etcd_image", default="registry.access.redhat.com/rhel7/etcd")
+                etcd_img = self.get_var("etcd_image", default="registry.access.redhat.com/rhel7/etcd")
                 required.add(self.template_var(etcd_img))
 
         return required


Test patch:
diff --git a/roles/openshift_health_checker/test/docker_image_availability_test.py b/roles/openshift_health_checker/test/docker_image_availability_test.py
--- a/roles/openshift_health_checker/test/docker_image_availability_test.py
+++ b/roles/openshift_health_checker/test/docker_image_availability_test.py
@@ -301,7 +301,7 @@ def test_registry_console_image(task_vars, expected):
     ), (
         dict(
             group_names=['oo_etcd_to_config'],
-            osm_etcd_image='spam/etcd',
+            etcd_image='spam/etcd',
         ),
         set(['spam/etcd']),
     ),