Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: controller-manager is stuck in deleting an incorrectly configured LB type service. it cannot be recovered until restarted. What you expected to happen: controller-manager should recover. At least it should allow us to create the service without restarting it. How to reproduce it (as minimally and precisely as possible): put a random IP in above yaml and make sure the resource group doen't exist. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig azure
I also want to add that the above repro is stuck in 403. It's possible for it to be stuck in 404 as well when the SP has the right permission.
Can you share the log for 404?
Thanks for reporting the issue. /assign Do you mean other services couldn't be configured when this happens?
correct.
FYI, we had the same issue when the Idle Timeout annotation was set to "3" which isn't a supported case. Will this PR also cover this edge case ?
Unfortunately , this is still not fixed. The idle timeout is not 403 or 404. We need another fix for this (and possibly more for other annotations)

Discussion:
servicehelpers "k8s.io/cloud-provider/service/helpers" "k8s.io/klog/v2" azcache "k8s.io/legacy-cloud-providers/azure/cache" <ins> "k8s.io/legacy-cloud-providers/azure/retry" </ins> utilnet "k8s.io/utils/net" )

klog.V(5).Infof("Delete service (%s): START clusterName=%q", serviceName, clusterName) serviceIPToCleanup, err := az.findServiceIPAddress(ctx, clusterName, service, isInternal) <del> if err != nil { </del> <ins> if err != nil && !retry.HasStatusForbiddenOrIgnoredError(err) { </ins> return err }

return err } <del> if _, err := az.reconcileLoadBalancer(clusterName, service, nil, false /* wantLb */); err != nil { </del> <ins> if _, err := az.reconcileLoadBalancer(clusterName, service, nil, false /* wantLb */); err != nil && !retry.HasStatusForbiddenOrIgnoredError(err) { </ins> return err }

service           v1.Service isInternalSvc     bool expectCreateError bool <ins> wrongRGAtDelete   bool </ins> }{ { desc:    "external service should be created and deleted successfully",

service:           getResourceGroupTestService("service4", "random-rg", "1.2.3.4", 80), expectCreateError: true, }, <ins> { desc:              "annotated service with different resourceGroup shouldn't be created but should be deleted successfully", service:           getResourceGroupTestService("service5", "random-rg", "", 80), expectCreateError: true, wrongRGAtDelete:   true, }, </ins> } ctrl := gomock.NewController(t)

expectedLBs = make([]network.LoadBalancer, 0) setMockLBs(az, ctrl, &expectedLBs, "service", 1, i+1, c.isInternalSvc) // finally, delete it. <ins> if c.wrongRGAtDelete { az.LoadBalancerResourceGroup = "nil" } </ins> err = az.EnsureLoadBalancerDeleted(context.TODO(), testClusterName, &c.service) expectedLBs = make([]network.LoadBalancer, 0) mockLBsClient := mockloadbalancerclient.NewMockInterface(ctrl)

return strings.Contains(err.Error(), "Retriable: true") } <ins> // HasStatusForbiddenOrIgnoredError return true if the given error code is part of the error message // This should only be used when trying to delete resources func HasStatusForbiddenOrIgnoredError(err error) bool { if err == nil { return false } if strings.Contains(err.Error(), fmt.Sprintf("HTTPStatusCode: %d", http.StatusNotFound)) { return true } if strings.Contains(err.Error(), fmt.Sprintf("HTTPStatusCode: %d", http.StatusForbidden)) { return true } return false } </ins>

result = IsErrorRetriable(fmt.Errorf("Retriable: true")) assert.Equal(t, true, result) } <ins> func TestHasErrorCode(t *testing.T) { // false case result := HasStatusForbiddenOrIgnoredError(fmt.Errorf("HTTPStatusCode: 408")) assert.False(t, result) // true case 404 result = HasStatusForbiddenOrIgnoredError(fmt.Errorf("HTTPStatusCode: %d", http.StatusNotFound)) assert.True(t, result) // true case 403 result = HasStatusForbiddenOrIgnoredError(fmt.Errorf("HTTPStatusCode: %d", http.StatusForbidden)) assert.True(t, result) } </ins>