Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Environment: On GCE with the win-bridge cni plugin On a node when creating a pod for the first time, running podsandbox would fail with the error message below. "failed to create the new HNSEndpoint: hnsCall failed in Win32: The object already exists. (0x1392)" Subsequent retry would succeed, and launching other pods afterwards would also, generally, be fine. I've found , but in that issue, the creation failed permanently. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): master (1.14, ) Cloud provider or hardware configuration: GCE OS (e.g: ): Windows 1809 Kernel (e.g. ): Install tools: Others: /cc
cc:
We've seen this before -- what's the root-cause?
So this happens only for the first time right? Could you please collect the following logs for us? On the node before creating the pod for the first time. start-birstranfer run collectlogs.ps1 then start the trace by running the following command C:kdebug repro the issue run "netsh trace stop" then do again collectlogs.ps1 and send us both before and after collectlogs.ps1 and C: cc
thank you for the detailed steps to capture the traces. I've uploaded the trace + log files at Files from both and are included. There was about a ten-minute delay between the node joining the cluster and the deployment of a pod to start the repro. The timestamp on the node just before creating the pod was Friday, March 1, 2019 6:56:06 PM. Please let us know if you have questions or need more information. Thanks for taking a look!
Thanks Logs were super useful in root causing the issue. You are using L2Bridge in host gateway mode, so u r allocating .2 ip as the Gateway Ip for this host. I am seeing a request to create an endpoint with an IP "10.64.1.2" "{"Name":"1885009e399aa542ad60bab063262196a804252010ab401967a6575cc7ccd1bd_l2bridge","VirtualNetwork":"64736A7F-E9F0-4465-B861-7879393DDD22","Policies":[{"ExceptionList":["10.64.0.0/14","10.0.0.0/16","10.40.0.0/22"],"Type":"OutBoundNAT"},{"DestinationPrefix":"10.0.0.0/16","NeedEncap":true,"Type":"ROUTE"},{"DestinationPrefix":"10.40.0.3/32","NeedEncap":true,"Type":"ROUTE"}],"IPAddress":"10.64.1.2","DNSSuffix":"","DNSServerList":"10.0.0.10","GatewayAddress":"10.64.1.2"}" But this IP is assigned to CBR0(Gateway) and an endpoint already exists on HNS with that IP. so when a container request is coming with the same IP we fail saying "the object already exists in our Address MAP" Now the question is why are we creating a container with an IP 10.64.1.2 when that IP is allocated for the Gateway NIC on this host? u can see that in Before logs on HNS - ID IPAddress MacAddress IsRemoteEnd point -- --------- ---------- ----------- 428D83D1-6138-459F-9B77-4F8B48A6FF6B 10.64.1.2 00-15-5D-6E-A4-DD First Entry is the endpoint that already exists but u are also sending a request to create a container with the same IP So basically ur IPAM is not excluding the gateway NIC and hence we are seeing that only for the first time u create the container. Hope this helps in root causing the issue on ur end. Again thanks for the logs. Super Useful..

Discussion:
Log-Output ("using mgmt IP ${mgmt_ip} and mgmt subnet ${mgmt_subnet} for " + "CNI config") <ins> # We reserve .1 and .2 for gateways. Start the CIDR range from ".3" so that # IPAM does not allocate those IPs to pods. $cidr_range_start = ` ${env:POD_CIDR}.substring(0, ${env:POD_CIDR}.lastIndexOf('.')) + '.3' </ins> # Explanation of the CNI config values: #   CLUSTER_CIDR: the cluster CIDR from which pod CIDRs are allocated. #   POD_CIDR: the pod CIDR assigned to this node. <ins> #   CIDR_RANGE_START: start of the pod CIDR range. </ins> #   MGMT_SUBNET: the subnet on which the Windows pods + kubelet will #     communicate with the rest of the cluster without NAT (i.e. the subnet #     that VM internal IPs are allocated from).

}, "ipam":  { "type": "host-local", <del> "subnet": "POD_CIDR" </del> <ins> "subnet": "POD_CIDR", "rangeStart": "CIDR_RANGE_START" </ins> }, "dns":  { "Nameservers":  [

} ] }'.replace('POD_CIDR', ${env:POD_CIDR}).` <ins> replace('CIDR_RANGE_START', ${cidr_range_start}).` </ins> replace('DNS_SERVER_IP', ${kube_env}['DNS_SERVER_IP']).` replace('DNS_DOMAIN', ${kube_env}['DNS_DOMAIN']).` replace('MGMT_IP', ${mgmt_ip}).`