Repository: openshift/openshift-ansible

Issue:
Unattended atomic-openshift-installer fails
I'd like to reiterate @ganhuang's concerns with https://github.com/openshift/openshift-ansible/pull/1778#issuecomment-223903971 that have not been addressed and I'm currently experiencing in production. Specifically, the installer.cfg.yml generated by "atomic-openshift-installer install":

``` yml
ansible_callback_facts_yaml: /home/ghuang/.config/openshift/.ansible/callback_facts.yaml
ansible_config: /usr/share/atomic-openshift-utils/ansible.cfg
ansible_inventory_path: /home/ghuang/.config/openshift/hosts
ansible_log_path: /tmp/ansible.log
ansible_ssh_user: root
deployment:
  hosts:
  - connect_to: 10.x.x.x
    containerized: true
    hostname: 10.x.x.x
    ip: 10.x.x.x
    public_hostname: 10.x.x.x
    public_ip: 10.x.x.x
    roles:
    - master
    - node
    - storage
  master_routingconfig_subdomain: ''
  proxy_exclude_hosts: ''
  proxy_http: ''
  proxy_https: ''
  roles: {}
variant: openshift-enterprise
variant_version: '3.2'
version: v2
```

Trigger the unattended mode with installer.cfg.yml above

```
# atomic-openshift-installer -u install
Any roles assigned to hosts must be defined.
```

The workaround to this is to manually change the empty roles dict from:

``` yml
roles: {}
```

to 

``` yml
roles:
  master:
    containerized: false
  storage:
    containerized: false
  node:
    containerized: false
```

However, while I've gotten past the original problem, I'm now encountering a different error. I manually added a master lb to the installer.cfg.yml:

``` yml
- connect_to: lb.sample.com
  hostname: lb.sample.com
  ip: 1.2.3.4
  public_hostname: lb.sample.com
  public_ip: 1.2.3.4
  roles:
  - master_lb
```

and the following role in the same file:

``` yml
deployment:
  ...
  roles:
    ...
    master_lb:
      preconfigured: true
```

Re-running `atomic-openshift-installer --unattended install` yields the following:

```
*** Installation Summary ***

Hosts:
- 10.10.10.7
  - OpenShift Master
  - OpenShift Node (Unscheduled)
  - Etcd Member
  - Storage
- 10.10.10.8
  - OpenShift Master
  - OpenShift Node (Unscheduled)
  - Etcd Member
- 10.10.10.9
  - OpenShift Master
  - OpenShift Node (Unscheduled)
  - Etcd Member
- 10.10.10.10
  - OpenShift Node (Dedicated)
- 10.10.10.11
  - OpenShift Node (Dedicated)
- lb.sample.com
  - Load Balancer (Preconfigured)

Total OpenShift Masters: 3
Total OpenShift Nodes: 5

NOTE: Multiple Masters specified, this will be an HA deployment with a separate
etcd cluster. You will be prompted to provide the FQDN of a load balancer and
a host for storage once finished entering hosts.


WARNING: A minimum of 3 dedicated Nodes are recommended for an HA
deployment.

Gathering information from hosts...
ERROR: /root/.config/openshift/hosts:6: child group is not defined: (master_lb)
There was a problem fetching the required information. Please see /tmp/ansible.log for details
```

As you can see, the installer recognizes the `master_lb` role based on the summary. However, it is not producing the correct group in the hosts file resulting in the error at the bottom.



Discussion:
It looks like this has been fixed in the enterprise-3.2 branch. i'll try that one and report back.

HEAD of `enterprise-3.2` (https://github.com/openshift/openshift-ansible/commit/fe61946d893deab8f0937f8222ea3fb9c11722fc) does not fix the issue. The problem is that cli_installer.py expects a `master_lb` role in installer.cfg so I can't skip that part. However, even if I follow its requirements, `openshift-ansible` does not translate that to a section in the `hosts` file. But since it still writes `master_lb` under the `OSEv3:children` section, Ansible errors out saying it can't find the `master_lb` group in the hosts file.

Adding `master_lb` to `ROLES_TO_GROUPS_MAP` as follows seems to do the trick:

```
ROLES_TO_GROUPS_MAP = {
    'master': 'masters',
    'node': 'nodes',
    'storage': 'nfs',
    'master_lb': ''
}
```

However, Ansible is still failing because lines 52 to 55 (https://github.com/openshift/openshift-ansible/blob/enterprise-3.2/utils/src/ooinstall/openshift_ansible.py#L52) was not moved to `write_inventory_vars()` along with the rest of the code that got moved there in https://github.com/openshift/openshift-ansible/commit/296d1283202fe92de3045eab8bbc60db192f3a46. The result is that the `deployment_type` variable is now under `[master:vars]` instead of `[OSEv3:vars]`.

The fix that work for me are as follows:

``` diff
diff --git a/utils/src/ooinstall/openshift_ansible.py b/utils/src/ooinstall/openshift_ansible.py
index 3529550..da81d03 100644
--- a/utils/src/ooinstall/openshift_ansible.py
+++ b/utils/src/ooinstall/openshift_ansible.py
@@ -14,7 +14,8 @@ CFG = None
 ROLES_TO_GROUPS_MAP = {
     'master': 'masters',
     'node': 'nodes',
-    'storage': 'nfs'
+    'storage': 'nfs',
+    'master_lb': ''
 }

 VARIABLES_MAP = {
@@ -49,11 +50,6 @@ def generate_inventory(hosts):

     write_inventory_vars(base_inventory, multiple_masters, proxy)

-    # Find the correct deployment type for ansible:
-    ver = find_variant(CFG.settings['variant'],
-        version=CFG.settings.get('variant_version', None))[1]
-    base_inventory.write('deployment_type={}\n'.format(ver.ansible_key))
-
     if 'OO_INSTALL_ADDITIONAL_REGISTRIES' in os.environ:
         base_inventory.write('openshift_docker_additional_registries={}\n'
           .format(os.environ['OO_INSTALL_ADDITIONAL_REGISTRIES']))
@@ -162,10 +158,15 @@ def write_inventory_vars(base_inventory, multiple_masters, proxy):

     write_proxy_settings(base_inventory)

+    # Find the correct deployment type for ansible:
+    ver = find_variant(CFG.settings['variant'],
+        version=CFG.settings.get('variant_version', None))[1]
+    base_inventory.write('deployment_type={}\n'.format(ver.ansible_key))
+
     for name, role_obj in CFG.deployment.roles.iteritems():
         if role_obj.variables:
             group_name = ROLES_TO_GROUPS_MAP.get(name, name)
-            base_inventory.write("\n[{}:vars]\n".format(group_name))
+            base_inventory.write("\n[{}:vars]".format(group_name))
             for variable, value in role_obj.variables.iteritems():
                 inventory_var = VARIABLES_MAP.get(variable, variable)
                 if value:
```

Is ooinstall/atomic-openshift-install still the supported way of installing openshift or is the team working on something new?

Several improvements( #2222 , #2223 , #2226 , #2228 , #2229 ) have been made this week that I think address most of these issues. I'll go through each thing you list here and make sure we're all good though.

Update: The master_lb issue remains. I'm working on a clean fix to that now. 

Thanks, @smunilla. Sat down with @abutcher yesterday (Well, actually we were at a stand up table. Way healthier!) at Ansible Fest and he helped me with a workaround. Happy to answer any questions you might have if you need more info on the issue.

@relaxdiego I just pushed up #2239 which, in addition to cleaning up how we write the inventory, should solve the issues you've been running into.


Resolving patch:
diff --git a/utils/src/ooinstall/cli_installer.py b/utils/src/ooinstall/cli_installer.py
--- a/utils/src/ooinstall/cli_installer.py
+++ b/utils/src/ooinstall/cli_installer.py
@@ -121,7 +121,7 @@ def collect_hosts(oo_cfg, existing_env=False, masters_set=False, print_summary=T
     click.echo(message)
 
     hosts = []
-    roles = set(['master', 'node', 'storage'])
+    roles = set(['master', 'node', 'storage', 'etcd'])
     more_hosts = True
     num_masters = 0
     while more_hosts:
@@ -133,6 +133,7 @@ def collect_hosts(oo_cfg, existing_env=False, masters_set=False, print_summary=T
         if not masters_set:
             if click.confirm('Will this host be an OpenShift Master?'):
                 host_props['roles'].append('master')
+                host_props['roles'].append('etcd')
                 num_masters += 1
 
                 if oo_cfg.settings['variant_version'] == '3.0':
@@ -827,21 +828,25 @@ def uninstall(ctx):
     oo_cfg = ctx.obj['oo_cfg']
     verbose = ctx.obj['verbose']
 
-    if len(oo_cfg.deployment.hosts) == 0:
+    if hasattr(oo_cfg, 'deployment'):
+        hosts = oo_cfg.deployment.hosts
+    elif hasattr(oo_cfg, 'hosts'):
+        hosts = oo_cfg.hosts
+    else:
         click.echo("No hosts defined in: %s" % oo_cfg.config_path)
         sys.exit(1)
 
     click.echo("OpenShift will be uninstalled from the following hosts:\n")
     if not ctx.obj['unattended']:
         # Prompt interactively to confirm:
-        for host in oo_cfg.deployment.hosts:
+        for host in hosts:
             click.echo("  * %s" % host.connect_to)
         proceed = click.confirm("\nDo you wish to proceed?")
         if not proceed:
             click.echo("Uninstall cancelled.")
             sys.exit(0)
 
-    openshift_ansible.run_uninstall_playbook(verbose)
+    openshift_ansible.run_uninstall_playbook(hosts, verbose)
 
 
 @click.command()
diff --git a/utils/src/ooinstall/oo_config.py b/utils/src/ooinstall/oo_config.py
--- a/utils/src/ooinstall/oo_config.py
+++ b/utils/src/ooinstall/oo_config.py
@@ -17,11 +17,9 @@
     'variant_version',
     ]
 
-DEPLOYMENT_PERSIST_SETTINGS = [
-    'master_routingconfig_subdomain',
-    'proxy_http',
-    'proxy_https',
-    'proxy_exclude_hosts',
+DEPLOYMENT_VARIABLES_BLACKLIST = [
+    'hosts',
+    'roles',
 ]
 
 DEFAULT_REQUIRED_FACTS = ['ip', 'public_ip', 'hostname', 'public_hostname']
@@ -191,10 +189,11 @@ def _read_config(self):
                     except KeyError:
                         continue
 
-                for setting in DEPLOYMENT_PERSIST_SETTINGS:
+                for setting in loaded_config['deployment']:
                     try:
-                        self.deployment.variables[setting] = \
-                            str(loaded_config['deployment'][setting])
+                        if setting not in DEPLOYMENT_VARIABLES_BLACKLIST:
+                            self.deployment.variables[setting] = \
+                                str(loaded_config['deployment'][setting])
                     except KeyError:
                         continue
 
@@ -306,21 +305,20 @@ def persist_settings(self):
             if setting in self.settings and self.settings[setting]:
                 p_settings[setting] = self.settings[setting]
 
-
         p_settings['deployment'] = {}
         p_settings['deployment']['hosts'] = []
         p_settings['deployment']['roles'] = {}
 
-        for setting in DEPLOYMENT_PERSIST_SETTINGS:
-            if setting in self.deployment.variables:
-                p_settings['deployment'][setting] = self.deployment.variables[setting]
-
         for host in self.deployment.hosts:
             p_settings['deployment']['hosts'].append(host.to_dict())
 
         for name, role in self.deployment.roles.iteritems():
             p_settings['deployment']['roles'][name] = role.variables
 
+        for setting in self.deployment.variables:
+            if setting not in DEPLOYMENT_VARIABLES_BLACKLIST:
+                p_settings['deployment'][setting] = self.deployment.variables[setting]
+
         try:
             p_settings['variant'] = self.settings['variant']
             p_settings['variant_version'] = self.settings['variant_version']
diff --git a/utils/src/ooinstall/openshift_ansible.py b/utils/src/ooinstall/openshift_ansible.py
--- a/utils/src/ooinstall/openshift_ansible.py
+++ b/utils/src/ooinstall/openshift_ansible.py
@@ -1,5 +1,3 @@
-# TODO: Temporarily disabled due to importing old code into openshift-ansible
-# repo. We will work on these over time.
 # pylint: disable=bad-continuation,missing-docstring,no-self-use,invalid-name,global-statement,global-variable-not-assigned
 
 import socket
@@ -14,7 +12,9 @@
 ROLES_TO_GROUPS_MAP = {
     'master': 'masters',
     'node': 'nodes',
-    'storage': 'nfs'
+    'etcd': 'etcd',
+    'storage': 'nfs',
+    'master_lb': 'lb'
 }
 
 VARIABLES_MAP = {
@@ -34,75 +34,52 @@ def set_config(cfg):
 
 def generate_inventory(hosts):
     global CFG
+
     masters = [host for host in hosts if host.is_master()]
-    nodes = [host for host in hosts if host.is_node()]
-    new_nodes = [host for host in hosts if host.is_node() and host.new_host]
-    proxy = determine_proxy_configuration(hosts)
-    storage = determine_storage_configuration(hosts)
     multiple_masters = len(masters) > 1
+
+    new_nodes = [host for host in hosts if host.is_node() and host.new_host]
     scaleup = len(new_nodes) > 0
 
+    lb = determine_lb_configuration(hosts)
+
     base_inventory_path = CFG.settings['ansible_inventory_path']
     base_inventory = open(base_inventory_path, 'w')
 
-    write_inventory_children(base_inventory, multiple_masters, proxy, scaleup)
-
-    write_inventory_vars(base_inventory, multiple_masters, proxy)
-
-
-
-    base_inventory.write('\n[masters]\n')
-    for master in masters:
-        write_host(master, base_inventory)
-
-    if len(masters) > 1:
-        base_inventory.write('\n[etcd]\n')
-        for master in masters:
-            write_host(master, base_inventory)
-
-    base_inventory.write('\n[nodes]\n')
+    write_inventory_children(base_inventory, scaleup)
 
-    for node in nodes:
-        # Let the fact defaults decide if we're not a master:
-        schedulable = None
+    write_inventory_vars(base_inventory, multiple_masters, lb)
 
-        # If the node is also a master, we must explicitly set schedulablity:
-        if node.is_master():
-            schedulable = node.is_schedulable_node(hosts)
-        write_host(node, base_inventory, schedulable)
-
-    if not getattr(proxy, 'preconfigured', True):
-        base_inventory.write('\n[lb]\n')
-        write_host(proxy, base_inventory)
 
+    #write_inventory_hosts
+    for role in CFG.deployment.roles:
+        # write group block
+        group = ROLES_TO_GROUPS_MAP.get(role, role)
+        base_inventory.write("\n[{}]\n".format(group))
+        # write each host
+        group_hosts = [host for host in hosts if role in host.roles]
+        for host in group_hosts:
+            schedulable = host.is_schedulable_node(hosts)
+            write_host(host, role, base_inventory, schedulable)
 
     if scaleup:
         base_inventory.write('\n[new_nodes]\n')
         for node in new_nodes:
-            write_host(node, base_inventory)
-
-    if storage:
-        base_inventory.write('\n[nfs]\n')
-        write_host(storage, base_inventory)
+            write_host(node, 'new_nodes', base_inventory)
 
     base_inventory.close()
     return base_inventory_path
 
-def determine_proxy_configuration(hosts):
-    proxy = next((host for host in hosts if host.is_master_lb()), None)
-    if proxy:
-        if proxy.hostname == None:
-            proxy.hostname = proxy.connect_to
-            proxy.public_hostname = proxy.connect_to
+def determine_lb_configuration(hosts):
+    lb = next((host for host in hosts if host.is_master_lb()), None)
+    if lb:
+        if lb.hostname == None:
+            lb.hostname = lb.connect_to
+            lb.public_hostname = lb.connect_to
 
-    return proxy
+    return lb
 
-def determine_storage_configuration(hosts):
-    storage = next((host for host in hosts if host.is_storage()), None)
-
-    return storage
-
-def write_inventory_children(base_inventory, multiple_masters, proxy, scaleup):
+def write_inventory_children(base_inventory, scaleup):
     global CFG
 
     base_inventory.write('\n[OSEv3:children]\n')
@@ -112,13 +89,10 @@ def write_inventory_children(base_inventory, multiple_masters, proxy, scaleup):
 
     if scaleup:
         base_inventory.write('new_nodes\n')
-    if multiple_masters:
-        base_inventory.write('etcd\n')
-    if not getattr(proxy, 'preconfigured', True):
-        base_inventory.write('lb\n')
+
 
 # pylint: disable=too-many-branches
-def write_inventory_vars(base_inventory, multiple_masters, proxy):
+def write_inventory_vars(base_inventory, multiple_masters, lb):
     global CFG
     base_inventory.write('\n[OSEv3:vars]\n')
 
@@ -135,11 +109,11 @@ def write_inventory_vars(base_inventory, multiple_masters, proxy):
     if CFG.settings['ansible_ssh_user'] != 'root':
         base_inventory.write('ansible_become=yes\n')
 
-    if multiple_masters and proxy is not None:
+    if multiple_masters and lb is not None:
         base_inventory.write('openshift_master_cluster_method=native\n')
-        base_inventory.write("openshift_master_cluster_hostname={}\n".format(proxy.hostname))
+        base_inventory.write("openshift_master_cluster_hostname={}\n".format(lb.hostname))
         base_inventory.write(
-            "openshift_master_cluster_public_hostname={}\n".format(proxy.public_hostname))
+            "openshift_master_cluster_public_hostname={}\n".format(lb.public_hostname))
 
     if CFG.settings.get('variant_version', None) == '3.1':
         #base_inventory.write('openshift_image_tag=v{}\n'.format(CFG.settings.get('variant_version')))
@@ -195,9 +169,12 @@ def write_proxy_settings(base_inventory):
 
 
 # pylint: disable=too-many-branches
-def write_host(host, inventory, schedulable=None):
+def write_host(host, role, inventory, schedulable=None):
     global CFG
 
+    if host.preconfigured:
+        return
+
     facts = ''
     if host.ip:
         facts += ' openshift_ip={}'.format(host.ip)
@@ -215,14 +192,13 @@ def write_host(host, inventory, schedulable=None):
     if host.node_labels:
         facts += ' openshift_node_labels="{}"'.format(host.node_labels)
 
+
     # Distinguish between three states, no schedulability specified (use default),
     # explicitly set to True, or explicitly set to False:
-    if schedulable is None:
+    if role != 'node' or schedulable is None:
         pass
-    elif schedulable:
-        facts += ' openshift_schedulable=True'
-    elif not schedulable:
-        facts += ' openshift_schedulable=False'
+    else:
+        facts += " openshift_schedulable={}".format(schedulable)
 
     installer_host = socket.gethostname()
     if installer_host in [host.connect_to, host.hostname, host.public_hostname]:
@@ -303,10 +279,10 @@ def run_ansible(playbook, inventory, env_vars, verbose=False):
     return subprocess.call(args, env=env_vars)
 
 
-def run_uninstall_playbook(verbose=False):
+def run_uninstall_playbook(hosts, verbose=False):
     playbook = os.path.join(CFG.settings['ansible_playbook_directory'],
         'playbooks/adhoc/uninstall.yml')
-    inventory_file = generate_inventory(CFG.hosts)
+    inventory_file = generate_inventory(hosts)
     facts_env = os.environ.copy()
     if 'ansible_log_path' in CFG.settings:
         facts_env['ANSIBLE_LOG_PATH'] = CFG.settings['ansible_log_path']


Test patch:
diff --git a/utils/test/cli_installer_tests.py b/utils/test/cli_installer_tests.py
--- a/utils/test/cli_installer_tests.py
+++ b/utils/test/cli_installer_tests.py
@@ -815,9 +815,9 @@ def test_full_run(self, load_facts_mock, run_playbook_mock):
         self.assert_inventory_host_var(inventory, 'nodes', '10.0.0.1',
                                  'openshift_schedulable=False')
         self.assert_inventory_host_var_unset(inventory, 'nodes', '10.0.0.2',
-                                 'openshift_schedulable')
+                                 'openshift_schedulable=True')
         self.assert_inventory_host_var_unset(inventory, 'nodes', '10.0.0.3',
-                                 'openshift_schedulable')
+                                 'openshift_schedulable=True')
 
     # interactive with config file and some installed some uninstalled hosts
     @patch('ooinstall.openshift_ansible.run_main_playbook')
@@ -939,7 +939,7 @@ def test_ha_dedicated_node(self, load_facts_mock, run_playbook_mock):
         self.assert_inventory_host_var(inventory, 'nodes', '10.0.0.3',
                                        'openshift_schedulable=False')
         self.assert_inventory_host_var_unset(inventory, 'nodes', '10.0.0.4',
-                                             'openshift_schedulable')
+                                             'openshift_schedulable=True')
 
         self.assertTrue(inventory.has_section('etcd'))
         self.assertEquals(3, len(inventory.items('etcd')))
@@ -1122,9 +1122,9 @@ def test_gen_inventory(self, load_facts_mock, run_playbook_mock):
         self.assert_inventory_host_var(inventory, 'nodes', '10.0.0.1',
                                  'openshift_schedulable=False')
         self.assert_inventory_host_var_unset(inventory, 'nodes', '10.0.0.2',
-                                 'openshift_schedulable')
+                                 'openshift_schedulable=True')
         self.assert_inventory_host_var_unset(inventory, 'nodes', '10.0.0.3',
-                                 'openshift_schedulable')
+                                 'openshift_schedulable=True')
 
 
 # TODO: test with config file, attended add node