Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kubelet fails when Kubernetes master gets started using cloud-provider openstack und kubeadm init command. says: What you expected to happen: kubelet starts and manages the loadbalancer and its security group. How to reproduce it (as minimally and precisely as possible): Setup an instance on Openstack. Place the following cloud-config in And : As well as : Run: For further information: +-----------------+
--------------------------+ +-----------------+
--------------------------+ +-----------------+
--------------------------+ Environment: Kubernetes version (use ): 1.9.1 Cloud provider or hardware configuration: Openstack OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): x86_64 Install tools: kubeadm
/sig cloud-provider
Looks like auto-probing for for (s), please see [1] and [2]. Apologies, the deprecation should have been handled better. [1] [2]
If I understand you correctly, this cloud-config would be enough: If I try this, no Loadbalancer gets created at all and the service which requests type LoadBalancer says: The service I use there is the nginx helm chart ().
Can you please comment? /assign
/sig openstack
From that, '' is your service name and '-202f-4d44-a538-' is your subnet id, can you ensure '-202f-4d44-a538-' exist? Is it "-c60c-48cb-a146-"? I suspect that failed to autoprobe subnet. Can you give me more logs from kube controller manager
Actually, -202f-4d44-a538- is a network, not a subnet. The subnet has the ID -c1d5-4cc1-8b98-
And here the relevant logs from kube-controller-manager,first: And then it keeps repeating:
Oh, I see. Oops, should get 'sub network id', not 'network id'. I will fix it soon.
I tried the fix in 1.9.2, but it does not work. I tried multiple setting combinations of providing and/or/neither , but none works. The Service does not get a LoadBalancer IP assigned. Here are the logs and settings for the config version neither providing nor , since it seemed that's the way you recommended to me: cat /etc/kubernetes/pki/cloud-config kubectl get nodes journalctl -u kubelet.service kubectl describe svc ingress-nginx-ingress-controller Can you please re-open the issue? Do you need any other information?
Is it ok when you disable manage-security-group(manage-security-group=false)?
And please review .
/reopen
Yes, when I disable "manage-security-group" it assigns a LoadBalancer correctly. It also works, as soon as I run . I tried to check , but since I don't know go, I don't really understand what you are doing there and how it relates to my issue.
I check gophercloud.ErrResourceNotFound is not a interface, so the '' will return error and excute "" which is 'error occurred finding security group: ...'.
Okay, so you "just" print a better error message, right? But my question is, why does the error occur at all? In my understanding the kubelet should create a security-group on itself, if configured manage-security-groups=true and do not provide a security-group. However I also tried to define a valid security group id on my own, but in that case I again find these logs in kubelet (same as described first):
Okay, so you "just" print a better error message, right? But my question is, why does the error occur at all? In my understanding the kubelet should create a security-group on itself, if configured manage-security-groups=true and do not provide a security-group. No, not 'just print a better error message', openstack cloud provider can ensure security group for LB. Firstly, openstack cloud provider check whether security group does not exist, if not exist, it will create security group for LB. The "gophercloud.ErrResourceNotFound" means that security group not exist.
Now OpenStack cloud provider has not "node-security-group" variable. You should not specify it.
Please check this note:
Well, to me it does not look like the removal of any resources is the problem, rather that they do not get created at all. I can offer you a direct debugging session using Google Hangout or so, maybe that way it is easier to detect the real problem?
Have you see this log? It means the security group we need to create does not exist, we should create it, not raise error, right? And I get 'non-interface type ErrResourceNotFound on left' error, 'err.(*gophercloud.ErrResourceNotFound)' is incorrect.
Yes, exactly. I also tried to fix the error manually, so first I created the sg-group . After that kubelet complained about empty pool-targets. After fixing that, it complained about invalid endpoint which pointed to the nginx server, don't know how to fix that one yet. I just want to point out: All that errors only occur if I activate manageSecurityGroups, otherwise the kubelet is able to assign a LoadBalancer to the IngressController.
Hi, how is going? I've deployed k8s 1.9.5 on OpenStack with kubespray and I'm struggling with this issue. If I do not set the LBAAS is created but it's not assigned to any Security Group. So, the service is not reachable from the outside world. I can fix it by hand creating the Security Group and assign it to the VIP. If I do set kubelet fails to start. According to it's a valid parameter... (?) And finally, if I do as in his last comment, the LBAAS remains in a PENDING state forever... Well, I wonder if there is any workaround to deal with this... I presume you're working on the fix, aren't you? Any information will be appreciate. Thanks
I'm hitting the same issue as yours with openstack loadbalancer with 1.9.4. Anyone know if fixes this issue? If so, when will it be part of a release?
Any updates on this?
I just tried to use this feature in kubernetes 1.10.2 and it still seems to be broken. The security groups do not get created and the service creation is stuck at pending without an ip address. I used this configuration and i also tried all the alternatives listed here. Either nothing gets created or the controller-manager crashes if i add node-security-group.
It's an issue with as well. The behaviour is as described by
Bump
fixes this for us.
So, If I do not enable manage-security-groups, This should work? I am getting the following error: Error creating load balancer (will retry): failed to ensure load balancer for service default/lbs-test: error creating LB floatingip {FloatingNetworkID:YYYY FloatingIP: PortID:XXXX FixedIP: TenantID:}: Resource not found
Yes. Don't specify and at all in config. The LoadBalancer(s) will by default apply the security rules in the security-group .
You need to specify to automatically add / alter security groups. I get the following error after I've upgraded to 1.9.10:
I ran into the same issue. It seems to be related to how k8's openstack driver deals with adding floating ips to the LB. For now I set the lb to internal in the service metadata then manually associate a floating ip . This could also be a queens magnum issue not providing all required cloud config info. I am not sure yet.

Discussion:
} // getNodeSecurityGroupIDForLB lists node-security-groups for specific nodes <del> func getNodeSecurityGroupIDForLB(compute *gophercloud.ServiceClient, nodes []*v1.Node) ([]string, error) { nodeSecurityGroupIDs := sets.NewString() </del> <ins> func getNodeSecurityGroupIDForLB(compute *gophercloud.ServiceClient, network *gophercloud.ServiceClient, nodes []*v1.Node) ([]string, error) { secGroupNames := sets.NewString() </ins> for _, node := range nodes { nodeName := types.NodeName(node.Name) srv, err := getServerByName(compute, nodeName) if err != nil { <del> return nodeSecurityGroupIDs.List(), err </del> <ins> return []string{}, err </ins> } // use the first node-security-groups

// case 1: node1:SG1  node2:SG2  return SG1,SG2 // case 2: node1:SG1,SG2  node2:SG3,SG4  return SG1,SG3 // case 3: node1:SG1,SG2  node2:SG2,SG3  return SG1,SG2 <del> securityGroupName := srv.SecurityGroups[0]["name"] nodeSecurityGroupIDs.Insert(securityGroupName.(string)) </del> <ins> secGroupNames.Insert(srv.SecurityGroups[0]["name"].(string)) </ins> } <del> return nodeSecurityGroupIDs.List(), nil </del> <ins> secGroupIDs := make([]string, secGroupNames.Len()) for i, name := range secGroupNames.List() { secGroupID, err := groups.IDFromName(network, name) if err != nil { return []string{}, err } secGroupIDs[i] = secGroupID } return secGroupIDs, nil </ins> } // isSecurityGroupNotFound return true while 'err' is object of gophercloud.ErrResourceNotFound

// find node-security-group for service var err error if len(lbaas.opts.NodeSecurityGroupIDs) == 0 { <del> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, nodes) </del> <ins> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, lbaas.network, nodes) </ins> if err != nil { return fmt.Errorf("failed to find node-security-group for loadbalancer service %s/%s: %v", apiService.Namespace, apiService.Name, err) }

originalNodeSecurityGroupIDs := lbaas.opts.NodeSecurityGroupIDs var err error <del> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, nodes) </del> <ins> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, lbaas.network, nodes) </ins> if err != nil { return fmt.Errorf("failed to find node-security-group for loadbalancer service %s/%s: %v", apiService.Namespace, apiService.Name, err) }