opened
achuzhoy
username_0
BM deployment fails with:
 level=debug msg="Fetching \"Install Config\"..."
level=debug msg="Loading \"Install Config\"..."
level=debug msg="  Loading \"SSH Key\"..."
level=debug msg="  Loading \"Base Domain\"..."
level=debug msg="    Loading \"Platform\"..."
level=debug msg="  Loading \"Cluster Name\"..."
level=debug msg="    Loading \"Base Domain\"..."
level=debug msg="  Loading \"Pull Secret\"..."
level=debug msg="  Loading \"Platform\"..."
level=debug msg="Using \"Install Config\" loaded from state file"
level=debug msg="Reusing previously-fetched \"Install Config\""
level=error msg="Cannot fetch the bootstrap and control plane host addresses from state file for baremetal platform"
level=error msg="boostrap host address and at least one control plane host address must be provided"
level=fatal msg="waiting for Kubernetes API: context deadline exceeded"

Checking the bootkube.service on bootstrap VM:


[core@dhcp-19-134-201 ~]$  journalctl -b -f -u bootkube.service
-- Logs begin at Mon 2019-06-17 15:11:03 UTC. --
Jun 17 15:14:01 dhcp-19-134-201.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1922]: Writing asset: /assets/config-bootstrap/manifests/0000_10_config-operator_01_image.crd.yaml
Jun 17 15:14:02 dhcp-19-134-201.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1922]: Rendering MCO manifests...
Jun 17 15:14:05 dhcp-19-134-201.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1922]: I0617 15:14:05.944776       1 bootstrap.go:86] Version: 4.0.0-alpha.0-459-gfb707ca3-dirty (fb707ca3c8740c787435d63f8ae843449c69ce60)
Jun 17 15:14:05 dhcp-19-134-201.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1922]: I0617 15:14:05.947813       1 bootstrap.go:141] manifests/machineconfigcontroller/controllerconfig.yaml
Jun 17 15:14:05 dhcp-19-134-201.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1922]: I0617 15:14:05.949450       1 bootstrap.go:141] manifests/master.machineconfigpool.yaml
Jun 17 15:14:05 dhcp-19-134-201.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1922]: I0617 15:14:05.949634       1 bootstrap.go:141] manifests/worker.machineconfigpool.yaml
Jun 17 15:14:05 dhcp-19-134-201.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1922]: I0617 15:14:05.949793       1 bootstrap.go:141] manifests/bootstrap-pod-v2.yaml
Jun 17 15:14:05 dhcp-19-134-201.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1922]: I0617 15:14:05.950033       1 bootstrap.go:141] manifests/machineconfigserver/csr-bootstrap-role-binding.yaml
Jun 17 15:14:05 dhcp-19-134-201.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1922]: I0617 15:14:05.950241       1 bootstrap.go:141] manifests/machineconfigserver/kube-apiserver-serving-ca-configmap.yaml
Jun 17 15:14:06 dhcp-19-134-201.qe1.kni.lab.eng.bos.redhat.com bootkube.sh[1922]: Waiting for etcd member names to be available from SRV ...
BM deployment fails with "level=error msg="Cannot fetch the bootstrap and control plane host addresses from state file for baremetal platform" level=error msg="boostrap host address and at least one control plane host address must be provided""
issue
created
russellb
username_1
Here's your next hint.  The masters most likely never came up properly, so
the bootstrap VM never discovers them at this step.

The next place you need to look is the consoles of masters.

-- 
Russell Bryant
comment
created
stbenjam
username_2
https://github.com/openshift-metal3/kni-installer/issues/79
comment
created
mcornea
username_3
On the master's console we can see:

```shell
ignition[1322]: GET error: Get https://10.19.134.13:22623/config/master: dial tcp 10.19.134.13:22623: connect: connection refused
```

which is odd because we can reach this URL from the provision host:
```shell
[centos@r640-u01 ~]$ curl https://10.19.134.13:22623/config/master -I -k
HTTP/1.1 200 OK
Content-Length: 160998
Content-Type: application/json
Date: Mon, 17 Jun 2019 19:35:08 GMT
```

Digging more we can see that no incoming packets on this port are reaching the provisionhost VM and the bootstrap VM respectively:
```shell
[root@r640-u01 centos]# tcpdump -i baremetal tcp port 22623
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on baremetal, link-type EN10MB (Ethernet), capture size 262144 bytes
```

Which probably means that the API VIP was set on some other machine. Checking for VRRP traffic we can see incoming packets from a worker node which doesn't seem right:

```shell
[root@r640-u01 centos]# tcpdump -i baremetal host 224.0.0.18
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on baremetal, link-type EN10MB (Ethernet), capture size 262144 bytes
15:39:31.385012 IP openshift-worker-1.qe1.kni.lab.eng.bos.redhat.com > 224.0.0.18: VRRPv2, Advertisement, vrid 178, prio 40, authtype simple, intvl 1s, length 20
15:39:31.385525 IP openshift-worker-1.qe1.kni.lab.eng.bos.redhat.com > 224.0.0.18: VRRPv2, Advertisement, vrid 173, prio 90, authtype simple, intvl 1s, length 20
```
Most probably the worker node is a leftover from a previous deployment and we can see that the API VIP is set on the worker node's interface(which explains the Connection refused message from the master nodes):

```shell
[root@openshift-worker-1 core]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 98:03:9b:61:7c:58 brd ff:ff:ff:ff:ff:ff
    inet 172.22.0.80/24 brd 172.22.0.255 scope global dynamic noprefixroute eno1
       valid_lft 3292sec preferred_lft 3292sec
    inet6 fe80::7e54:fe62:367:7ca2/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 98:03:9b:61:7c:59 brd ff:ff:ff:ff:ff:ff
    inet 10.19.134.6/24 brd 10.19.134.255 scope global dynamic noprefixroute eno2
       valid_lft 54261sec preferred_lft 54261sec
    inet 10.19.134.13/24 scope global secondary eno2
       valid_lft forever preferred_lft forever
    inet 10.19.134.14/24 scope global secondary eno2
       valid_lft forever preferred_lft forever
    inet 10.19.134.15/24 scope global secondary eno2
       valid_lft forever preferred_lft forever
    inet6 2620:52:0:1386:bb6f:b509:e1c5:e1b2/64 scope global dynamic noprefixroute 
       valid_lft 2591557sec preferred_lft 604357sec
    inet6 fe80::7b56:7f5:8cda:fc91/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
```

I think as a quick workaround we can turn off the worker nodes manually before starting a fresh deployment but we probably need a more robust solution to avoid this situation, e.g: should the worker nodes even be allowed to take the VIPs?
comment
created
hardys
username_4
I don't think so, particularly for the API VIP - @celebdor and @username_6 how do we handle the worker/master difference for the MCO integrated keepalived integration?
comment
created
bkunal
username_5
Looks like I am even hitting this issue. 

Here is what logs(06_create_cluster-2019-08-22-121134.log) says: 
========================================

evel=debug msg="module.masters.ironic_deployment.openshift-master-deployment[2]: Creation complete after 3m2s [id=6e423853-94d3-466b-b1f2-90d729824d75]"
level=debug
level=debug msg="Apply complete! Resources: 12 added, 0 changed, 0 destroyed."
level=debug msg="OpenShift Installer v4.2.0-201908220219-dirty"
level=debug msg="Built from commit 1f939d01fc96469c2f2205d9e71d6324b85b8a69"
level=info msg="Waiting up to 30m0s for the Kubernetes API at https://api.ostest.test.metalkube.org:6443..."
level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
level=debug msg="Still waiting for the Kubernetes API: the server could not find the requested resource"
level=debug msg="Still waiting for the Kubernetes API: Get https://api.ostest.test.metalkube.org:6443/version?timeout=32s: dial tcp 192.168.111.5:6443: connect: connection refused"
level=info msg="API v1.14.0+b7c8409 up"
level=info msg="Waiting up to 30m0s for bootstrapping to complete..."
level=debug msg="Fetching \"Install Config\"..."
level=debug msg="Loading \"Install Config\"..."
level=debug msg="  Loading \"SSH Key\"..."
level=debug msg="  Loading \"Base Domain\"..."
level=debug msg="    Loading \"Platform\"..."
level=debug msg="  Loading \"Cluster Name\"..."
level=debug msg="    Loading \"Base Domain\"..."
level=debug msg="  Loading \"Pull Secret\"..."
level=debug msg="  Loading \"Platform\"..."
level=debug msg="Using \"Install Config\" loaded from state file"
level=debug msg="Reusing previously-fetched \"Install Config\""
level=error msg="Cannot fetch the bootstrap and control plane host addresses from state file for baremetal platform"
level=error msg="bootstrap host address and at least one control plane host address must be provided"
level=fatal msg="failed to wait for bootstrapping to complete: timed out waiting for the condition"
==============================================================

$ journalctl -b -f -u bootkube.service
Hint: You are currently not seeing messages from other users and the system.
      Users in the 'systemd-journal' group can see all messages. Pass -q to
      turn off this notice.
No journal files were opened due to insufficient permissions.
[metal@dell-r440-12 ~]$ sudo journalctl -b -f -u bootkube.service
-- Logs begin at Wed 2019-08-21 14:26:24 IST. --

=============================================================
$ virsh net-dhcp-leases baremetal
error: failed to get network 'baremetal'
error: Network not found: no network with matching name 'baremetal'

[metal@dell-r440-12 ~]$ ip addr  show | grep baremetal
58: baremetal: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.111.1/24 brd 192.168.111.255 scope global baremetal
59: baremetal-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master baremetal state DOWN group default qlen 1000
61: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master baremetal state UNKNOWN group default qlen 1000
76: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master baremetal state UNKNOWN group default qlen 1000
78: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master baremetal state UNKNOWN group default qlen 1000
80: vnet7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master baremetal state UNKNOWN group default qlen 1000
[metal@dell-r440-12 ~]$
comment
created
bkunal
username_5
I remember last time I did face the same issue and redoing everything again had resolved. But this time I am not so lucky and I am ending into this again and again.
comment
created
bkunal
username_5
Any help on this will be highly appreciated?
comment
created
bkunal
username_5
How to change this timeout value? I think 30 min is not enough and maybe trying with higher value should be a good idea.
comment
created
bcrochet
username_6
The only VIP that workers are configured for is the Ingress VIP. They should never be able to get either the API or DNS VIPs.
comment
closed
stbenjam
username_2
issue
created
stbenjam
username_2
There's been no activity on this since August, and the platform has undergone a lot of changes. If there's still an issue please reopen.
comment