Repository: openshift/openshift-ansible

Issue #3775: failed installation - kernel: vxlan: Cannot bind port 4789, err=-97
Created: 2017-03-25T15:44:32Z
Description
After attempting a multi-master installation, nodes will not start.
Version
Please put the following version information in the code block
indicated below.
ansible 2.2.0.0
config file = /etc/ansible/ansible.cfg
configured module search path = Default w/o overrides
openshift-ansible-3.6.9-1
master branch
installing via playbooks
Steps To Reproduce
ansible-playbook ~/openshift-ansible/playbooks/byo/openshift-cluster/config.yml
Expected Results
Successful installation.
Observed Results
Describe what is actually happening.
FAILED - RETRYING: TASK: openshift_node : Start and enable node (1 retries left).
fatal: [REPLACED_HOST_NAME]: FAILED! => {"attempts": 1, "changed": false, "failed": true, "msg": "Unable to start service origin-node: Job for origin-node.service failed because the control process exited with error code. See "systemctl status origin-node.service" and "journalctl -xe" for details.\n"}
For long output or logs, consider using a gist
Additional Information
Excerpt from journalctl -xe
Mar 25 11:29:52 MASTER-NODE-REPLACED-NAME origin-node[23418]: F0325 11:29:52.237522   23418 node.go:350] error: SDN node startup failed: Allocated ofport (-1) did not match request (1)
Mar 25 11:29:52 MASTER-NODE-REPLACED-NAME systemd[1]: origin-node.service: main process exited, code=exited, status=255/n/a
Mar 25 11:29:52 MASTER-NODE-REPLACED-NAME systemd[1]: Failed to start Origin Node.
Mar 25 11:29:52 MASTER-NODE-REPLACED-NAME systemd[1]: Unit origin-node.service entered failed state.
Mar 25 11:29:52 MASTER-NODE-REPLACED-NAME systemd[1]: origin-node.service failed.
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME systemd[1]: origin-node.service holdoff time over, scheduling restart.
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME systemd[1]: Cannot add dependency job for unit iptables.service, ignoring: Unit is masked.
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME systemd[1]: Starting Origin Node...
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME origin-node[23484]: I0325 11:29:57.762622   23484 node.go:61] Initializing SDN node of type "redhat/openshift-ovs-subnet" with configured hostname "MASTER-NODE-REPLACED-NAME" (IP ""), iptables sync period "30s"
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME origin-node[23484]: I0325 11:29:57.773608   23484 docker.go:418] Connecting to docker on unix:///var/run/docker.sock
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME origin-node[23484]: I0325 11:29:57.773638   23484 docker.go:438] Start docker client with request timeout=2m0s
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME origin-node[23484]: I0325 11:29:57.778087   23484 start_node.go:303] Starting node MASTER-NODE-REPLACED-NAME (v1.4.1)
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME origin-node[23484]: I0325 11:29:57.779433   23484 start_node.go:312] Connecting to API server https://MASTER-REPLACED-NAME:8443
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME origin-node[23484]: I0325 11:29:57.780783   23484 docker.go:418] Connecting to docker on unix:///var/run/docker.sock
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME origin-node[23484]: I0325 11:29:57.780855   23484 docker.go:438] Start docker client with request timeout=0s
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME dockerd-current[16015]: time="2017-03-25T11:29:57.781496605-04:00" level=info msg="{Action=_ping, LoginUID=4294967295, PID=23484}"
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME origin-node[23484]: I0325 11:29:57.782374   23484 node.go:141] Connecting to Docker at unix:///var/run/docker.sock
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME origin-node[23484]: I0325 11:29:57.880860   23484 manager.go:140] cAdvisor running in container: "/system.slice/origin-node.service"
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME origin-node[23484]: W0325 11:29:57.956063   23484 manager.go:148] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME ovs-vsctl[23532]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-br br0 -- add-br br0 -- set Bridge br0 fail-mode=secure protocols=OpenFlow13
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME ovs-vsctl[23533]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-port br0 vxlan0
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME ovs-vsctl[23534]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --may-exist add-port br0 vxlan0 -- set Interface vxlan0 ofport_request=1 type=vxlan "options:remote_ip=\"flow\"" "options:key=\"flow\""
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME kernel: vxlan: Cannot bind port 4789, err=-97
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME origin-node[23484]: F0325 11:29:57.996792   23484 node.go:350] error: SDN node startup failed: Allocated ofport (-1) did not match request (1)
Mar 25 11:29:57 MASTER-NODE-REPLACED-NAME systemd[1]: origin-node.service: main process exited, code=exited, status=255/n/a
Mar 25 11:29:58 MASTER-NODE-REPLACED-NAME systemd[1]: Failed to start Origin Node.
Mar 25 11:29:58 MASTER-NODE-REPLACED-NAME systemd[1]: Unit origin-node.service entered failed state.
Mar 25 11:29:58 MASTER-NODE-REPLACED-NAME systemd[1]: origin-node.service failed.

Red Hat Enterprise Linux Server release 7.3 (Maipo)
Inventory file:
# Create an OSEv3 group that contains the masters and nodes groups
[OSEv3:children]
masters
nodes
etcd
lb

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_ssh_user=root
deployment_type=origin
enable_excluders=false

# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
# openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/openshift/openshift-passwd'}]

# master cluster ha variables using pacemaker or RHEL HA
openshift_master_cluster_hostname=cloud-master.REPLACED.DOMAIN
openshift_master_cluster_public_hostname=cloud-master.REPLACED.DOMAIN
openshift_master_cluster_method=native

# apply updated node defaults
openshift_node_kubelet_args={'pods-per-core': ['10'], 'max-pods': ['250'], 'image-gc-high-threshold': ['90'], 'image-gc-low-threshold': ['80']}

# override the default controller lease ttl
#osm_controller_lease_ttl=30

# enable ntp on masters to ensure proper failover
openshift_clock_enabled=true

# osm_default_subdomain=cloud.REPLACED.DOMAIN

# host group for masters
[masters]
cloud-m1e.REPLACED.DOMAIN
cloud-m2e.REPLACED.DOMAIN
cloud-m1w.REPLACED.DOMAIN

# host group for etcd
[etcd]
cloud-e1e.REPLACED.DOMAIN
cloud-e1w.REPLACED.DOMAIN
cloud-e2w.REPLACED.DOMAIN

# Specify load balancer host
[lb]
cloud-master.REPLACED.DOMAIN

# host group for nodes, includes region info
[nodes]
cloud-m1e.REPLACED.DOMAIN openshift_schedulable=False openshift_node_labels="{'region': 'infra', 'zone': 'default-east'}"
cloud-m2e.REPLACED.DOMAIN openshift_schedulable=False openshift_node_labels="{'region': 'infra', 'zone': 'default-east'}"
cloud-m1w.REPLACED.DOMAIN openshift_schedulable=False openshift_node_labels="{'region': 'infra', 'zone': 'default-west'}"
cloud-h1e.REPLACED.DOMAIN openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
cloud-h2e.REPLACED.DOMAIN openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
cloud-h1w.REPLACED.DOMAIN openshift_node_labels="{'region': 'primary', 'zone': 'west'}"
cloud-h2w.REPLACED.DOMAIN openshift_node_labels="{'region': 'primary', 'zone': 'west'}"

Duplicate issue #4008: Unable to start service origin-node => vxlan mod problem in redhat 7.3 kernel
Created: 2017-04-26T09:12:30Z
Description
Hi.
I don't know if i have to post here or directly on openshift-origin github.
While running ansible,  I get an error and ansible stop : can't start origin-node
We have 3 masters, 2 nodes and 1 haproxy VMs running RedHat 7.3
Version

Your ansible version per ansible --version
ansible 2.2.0.0

oc version
oc v1.4.1
kubernetes v1.4.0+776c994
features: Basic-Auth GSSAPI Kerberos SPNEGO


openvswitch version

Name        : openvswitch
Arch        : x86_64
Version     : 2.4.0
Release     : 2.el7
Size        : 7.8 M
Repo        : installed
From repo   : centos-openshift-origin


Steps To Reproduce

Running advanced install on RedHat 7.3

Expected Results
Ansible script end correctly
Observed Results
Describe what is actually happening.
Unable to start service origin-node:

2017-04-19 11:49:43,965 p=32070 u=xxx |  fatal: [VM_name]: FAILED! => {
    "attempts": 1,
    "changed": false,
    "failed": true
}

MSG:

Unable to start service origin-node:

Additional Information
/etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)

ansible inventory
[OSEv3:children]
masters
nodes
etcd
lb

[OSEv3:vars]
ansible_ssh_user=my_user
ansible_become=true
ansible_become_user=root
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
deployment_type=origin
openshift_version=1.4.1-1.el7.x86_64
openshift_pkg_version=1.4.1-1.el7.x86_64
openshift_master_cluster_method=native
openshift_master_cluster_public_hostname=my_public_hostname
openshift_master_default_subdomain=my_subdomain
openshift_use_sdn=true
os_sdn_network_plugin_name='redhat/openshift-ovs-multitenant'
debug_level=10
openshift_docker_options="--log-driver json-file --log-opt max-size=1M --log-opt max-file=5"

openshift_docker_additional_registries=my_registry
openshift_docker_insecure_registries=my_registry
oreg_url=my_registry/openshift/origin-${component}:${version}
openshift_examples_modify_imagestreams=true

openshift_router_selector='region=primary'
openshift_registry_selector='region=primary'

[masters]
my_master01 openshift_ip=10.110.163.84
my_master02 openshift_ip=10.110.163.85
my_master03 openshift_ip=10.110.163.86

[etcd]
my_master01 openshift_ip=10.110.163.84
my_master02 openshift_ip=10.110.163.85
my_master03 openshift_ip=10.110.163.86

[lb]
my_lb06 openshift_ip=10.110.163.89

[nodes]
my_node0[4:5] openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
my_node04 openshift_ip=10.110.163.87 openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
my_node05 openshift_ip=10.110.163.88 openshift_node_labels="{'region': 'secondary', 'zone': 'east'}"

investigation
Looking at the log on the node01 :
origin-node: I0418 14:42:01.898212   27603 node.go:61] Initializing SDN node of type "redhat/openshift-ovs-subnet" with configured hostname "xxx" (IP ""), iptables sync period "30s"
origin-node: I0418 14:42:01.903478   27603 docker.go:418] Connecting to docker on unix:///var/run/docker.sock
origin-node: I0418 14:42:01.903506   27603 docker.go:438] Start docker client with request timeout=2m0s
origin-node: I0418 14:42:01.908249   27603 start_node.go:303] Starting node xxx (v1.4.1)
origin-node: I0418 14:42:01.909493   27603 start_node.go:312] Connecting to API server xxxx
origin-node: I0418 14:42:01.910924   27603 docker.go:418] Connecting to docker on unix:///var/run/docker.sock
origin-node: I0418 14:42:01.910947   27603 docker.go:438] Start docker client with request timeout=0s
dockerd-current: time="2017-04-18T14:42:01.911643465+02:00" level=info msg="{Action=_ping, LoginUID=4294967295, PID=27603}"
origin-node: I0418 14:42:01.912391   27603 node.go:141] Connecting to Docker at unix:///var/run/docker.sock
origin-node: I0418 14:42:01.944164   27603 manager.go:140] cAdvisor running in container: "/system.slice/origin-node.service"
ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-br br0 -- add-br br0 -- set Bridge br0 fail-mode=secure protocols=OpenFlow13
ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-port br0 vxlan0
ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --may-exist add-port br0 vxlan0 -- set Interface vxlan0 ofport_request=1 type=vxlan "options:remote_ip=\"flow\"" "options:key=\"flow\""
origin-node: F0418 14:42:01.991245   27603 node.go:350] error: SDN node startup failed: Allocated ofport (-1) did not match request (1)
systemd[1]: origin-node.service: main process exited, code=exited, status=255/n/a
systemd[1]: Failed to start Origin Node.
systemd[1]: Unit origin-node.service entered failed state.

Command ovs-vsctl get Interface vxlan0 ofport return -1 instead of 1 so openvswitch stop its work here
When we try manually, the same error occur.
So we try tu use gre mod instead of vxlan, and no error appear.  ovs-vsctl get Interface vxlan0 ofport return 1
Now, look at the vxlan modinfo:
on redhat 7.3
modinfo vxlan

filename:       /lib/modules/3.10.0-514.10.2.el7.x86_64/kernel/drivers/net/vxlan.ko
alias:          rtnl-link-vxlan
description:    Driver for VXLAN encapsulated traffic
author:         Stephen Hemminger <stephen@networkplumber.org>
version:        0.1
license:        GPL
rhelversion:    7.3
srcversion:     1DAD77C1FDAA6663DD8687D
depends:        udp_tunnel,ip6_udp_tunnel
intree:         Y
vermagic:       3.10.0-514.10.2.el7.x86_64 SMP mod_unload modversions
signer:         Red Hat Enterprise Linux kernel signing key
sig_key:        AB:35:DD:65:18:C9:6E:E1:64:47:78:64:6C:D6:1E:DB:91:AE:86:6B
sig_hashalgo:   sha256
parm:           udp_port:Destination UDP port (ushort)
parm:           log_ecn_error:Log packets received with corrupted ECN (bool)

on redhat 7.1
modinfo vxlan

filename:       /lib/modules/3.10.0-229.14.1.el7.x86_64/kernel/drivers/net/vxlan.ko
alias:          rtnl-link-vxlan
author:         Stephen Hemminger <stephen@networkplumber.org>
version:        0.1
license:        GPL
rhelversion:    7.1
srcversion:     E8E51CB27B5E20E94C52EE9
depends:        ip_tunnel
intree:         Y
vermagic:       3.10.0-229.14.1.el7.x86_64 SMP mod_unload modversions
signer:         Red Hat Enterprise Linux kernel signing key
sig_key:        44:02:00:8F:2B:3F:DA:1F:6C:1C:64:BA:2C:10:BF:0C:FE:EC:AB:B7
sig_hashalgo:   sha256
parm:           udp_port:Destination UDP port (ushort)
parm:           log_ecn_error:Log packets received with corrupted ECN (bool)

=> The depends parameter are differents.
Seeing this, we reboot our node on the 7.1 kernel and try to create manually a vxlan tunnel, it's now working.
Re-run ansible on master => now it's working
We use this workaround, but it is not a real solution to run on RH 7.1 kernel with RH 7.3 installed.

Closing comment (2017-04-28T17:07:08Z): Dupe of #3775