jiebi/CodeConvo huggingface dataset:
Code change:
} mockCadvisor.On("VersionInfo").Return(versionInfo, nil) expectedNode := &api.Node{ <del> ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}, </ins> Spec:       api.NodeSpec{}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{

Related discussion:
text: Looks like some kubelet unit tests perform a lookupIp which ends up depending on the health of the local resolver. Specifically has this failing: Which leads to the test in: And we recently swapped out the node names from testnode to 127.0.0.1, so probably the culprit?
/ref
Interesting. I did swap out the node names in the unit test as part of to be 127.0.0.1, party because I thought that it wouldn't have any problems resolving during unit tests.
/cc
Code change:
} mockCadvisor.On("VersionInfo").Return(versionInfo, nil) expectedNode := &api.Node{ <del> ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}, </ins> Spec:       api.NodeSpec{}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{

Related discussion:
text: cc
What is the environment where you are running the tests? They pass for me on my mac, a linux box, and shippable seems to be passing them as well.
I couldn 't reproduce the failure either.
I'm out right now; i'll try them when i get back. For the record, I run hack/test- Paul Morie wrote:
Ahem FTR This acronym is going to go quad-plat Paul Morie wrote:
I ran on both platforms and saw the tests pass.
I'm still seeing it. Won't have bandwidth to dig in personally until tomorrow.
seeing a similar failure after a git pull
I knew I wasn't crazy
environment: fedora 21, golang: 1.4, docker: 1.6
I think I see the problem. When I set the node name to in the test I was assuming it would parse as a valid ip address and not end up at the LookupIP call. But only the node name (and not the kubelet hostname) is being set (except in my new test where I ) so it's trying to do a lookup. Do you think you could test just changing the default hostname from "testnode" to "127.0.0.1" and see if that fixes it for you?
Trying now
I still get failures but no panic / threaddump.
gist coming
this looks like again?
Good that it isn't panicing any longer. I can dig into the test failures tomorrow (I think it should be relatively simple to resolve).
cool, thanks for looking
Passes in master now for me. Closing.
FYI, they will break again once goes in, and then we can merge my PR to see if that fixes it for you.
:+1:
is in -- are these tests breaking again?
I just rebased which I think will fix them.
They are breaking again.
is on it.
Code change:
. "github.com/onsi/gomega" ) <ins> func runLivenessTest(c *client.Client, podDescr *api.Pod) { defer GinkgoRecover() ns := "e2e-test-" + string(util.NewUUID()) By(fmt.Sprintf("Creating pod %s in namespace %s", podDescr.Name, ns)) _, err := c.Pods(ns).Create(podDescr) Expect(err).NotTo(HaveOccurred(), fmt.Sprintf("creating pod %s", podDescr.Name)) // At the end of the test, clean up by removing the pod. defer func() { By("deleting the pod") c.Pods(ns).Delete(podDescr.Name) }() // Wait until the pod is not pending. (Here we need to check for something other than // 'Pending' other than checking for 'Running', since when failures occur, we go to // 'Terminated' which can cause indefinite blocking.) By("waiting for the pod to be something other than pending") err = waitForPodNotPending(c, ns, podDescr.Name, 60*time.Second) Expect(err).NotTo(HaveOccurred(), fmt.Sprintf("starting pod %s in namespace %s", podDescr.Name, ns)) By(fmt.Sprintf("Started pod %s in namespace %s", podDescr.Name, ns)) // Check the pod's current state and verify that restartCount is present. By("checking the pod's current state and verifying that restartCount is present") pod, err := c.Pods(ns).Get(podDescr.Name) Expect(err).NotTo(HaveOccurred(), fmt.Sprintf("getting pod %s in namespace %s", podDescr.Name, ns)) initialRestartCount := pod.Status.Info["liveness"].RestartCount By(fmt.Sprintf("Initial restart count of pod %s is %d", podDescr.Name, initialRestartCount)) // Wait for at most 48 * 5 = 240s = 4 minutes until restartCount is incremented pass := false for i := 0; i < 48; i++ { // Wait until restartCount is incremented. time.Sleep(5 * time.Second) pod, err = c.Pods(ns).Get(podDescr.Name) Expect(err).NotTo(HaveOccurred(), fmt.Sprintf("getting pod %s", podDescr.Name)) restartCount := pod.Status.Info["liveness"].RestartCount By(fmt.Sprintf("Restart count of pod %s in namespace %s is now %d", podDescr.Name, ns, restartCount)) if restartCount > initialRestartCount { By(fmt.Sprintf("Restart count of pod %s in namespace %s increased from %d to %d during the test", podDescr.Name, ns, initialRestartCount, restartCount)) pass = true break } } if !pass { Fail(fmt.Sprintf("Did not see the restart count of pod %s in namespace %s increase from %d during the test", podDescr.Name, ns, initialRestartCount)) } } </ins> var _ = Describe("Pods", func() { <del> var ( c *client.Client ) </del> <ins> var c *client.Client </ins> BeforeEach(func() { var err error

Related discussion:
text: Park of ginkgo-ization
Per IRC, is working on this
Pushed an update yesterday to address comments.
what's the status of this? There's an issue open to de-flake this test (), but it's tough to do that if the code is in the middle of being moved to ginkgo.
This has been merged.
Code change:
}, expected: &v1.Secret{ ObjectMeta: metav1.ObjectMeta{ <del> Name: "foo-94759gc65b", </del> <ins> Name: "foo-548cm7fgdh", </ins> }, Data: map[string][]byte{ <del> v1.DockerConfigKey: secretData, </del> <ins> v1.DockerConfigJsonKey: secretData, </ins> }, <del> Type: v1.SecretTypeDockercfg, </del> <ins> Type: v1.SecretTypeDockerConfigJson, </ins> }, expectErr: false, },

Related discussion:
text: /kind bug We're having trouble retrieving images from a private registry on DockerCloud. For nearly a year the following has worked flawlessly. Last Friday I pulled down the 1.9.0 CLI (via brew) and since then have not been able to access any image in a private registry on DockerCloud. I've tried this on 10+ brand new clusters created with kops v1.8.0. The Kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0 (which is not supported by kops yet) and all exhibit the same problem. We create the secret as such: And verify it using: Which yields: And create a test service with with these contents: The pod fails immediatly with: I've reviewed the 1.9.0 release notes and nothing jumps out as the root cause. For reference I've tried putting quotes around the value when creating the secret to no avail. I've also tried prefixing the value with , so the full value would read: also to no avail. Environment: Kubernetes version: Cloud provider and installation info: AWS cluster built with Kops 1.8.0, using (to isolate this issue) nearly the default settings for everything.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have same problem, and I "solved" it creating a docker-registry secret in an old kubernetes version and pasting it on new kubernetes
I'm recently (I know it worked perfectly about last month) facing issues with as well. For me it ended in a similar manner saying: I was able to confirm, that this is a kubernetes CLI related issue and not end user. Based on and this I was able to create properly formatted auth information looking like this: . When encoding this one-liner using (MacOS workstation) and apply like this works fine and images can be pulled from private repository again.
Code change:
}, expected: &v1.Secret{ ObjectMeta: metav1.ObjectMeta{ <del> Name: "foo-94759gc65b", </del> <ins> Name: "foo-548cm7fgdh", </ins> }, Data: map[string][]byte{ <del> v1.DockerConfigKey: secretData, </del> <ins> v1.DockerConfigJsonKey: secretData, </ins> }, <del> Type: v1.SecretTypeDockercfg, </del> <ins> Type: v1.SecretTypeDockerConfigJson, </ins> }, expectErr: false, },

Related discussion:
text: However this breaks all automation and can't be considered as solution Confirmed working with kubectl 1.8.5 and server 1.8.4 (guess 1.7.9 as well)
looks like there's an extra "auths" element when adding the secret using 1.9 kubectl
Added in /assign
Many thanks for the quick fix on this. Any idea when this might get rolled out? Also, it might be a good idea if this was to the known issue list for v1.9.0. It would have saved us a bunch of time trying to track down the root cause. Just trying to make sure no one else needs to do the same.
linking to the "v1.9.0 known issues" list
ack, asking Mehdy since he is the patch manager for v1.9 branch. Mehdy, any idea when we can ship 1.9.1 with this fix?
workarounds for folks until 1.9.1 is released: if you have a .dockerconfigjson for your private registry already, you can manually specify the type and data key (e.g. ) if you don't have a .dockerconfigjson file already, you can fix up the secret produced by manually: the type from to the data key from to the modified secret with
1 will be released tomorrow
I can confirm solution works, thanks so much!
I am still having issues in 1.9.1. This worked in the 1.8.x branches: results in the following error:
you have to remove the gitlab-registry secret before you can apply an update... type is immutable, and the fix in 1.9.1 required changing the type of the secret
Code change:
} } <del> azClientConfig := &azClientConfig{ subscriptionID:                 config.SubscriptionID, resourceManagerEndpoint:        env.ResourceManagerEndpoint, servicePrincipalToken:          servicePrincipalToken, rateLimiterReader:              operationPollRateLimiter, rateLimiterWriter:              operationPollRateLimiterWrite, CloudProviderBackoffRetries:    config.CloudProviderBackoffRetries, CloudProviderBackoffDuration:   config.CloudProviderBackoffDuration, ShouldOmitCloudProviderBackoff: config.shouldOmitCloudProviderBackoff(), } </del> az := Cloud{ Config:                 *config, Environment:            *env,

Related discussion:
text: Currently there is no way to run without any identity (be it service principal or E/MSI). In reality code running on a worker node can use via - and only via - without the need to access ARM. Here is the proposed change a new mode authN = this mode, is basically telling the cloud provider to work any identity, this means only IMDS calls is allowed. This mode can be detected from if no SPN set and both MSI settings are set to . to use a config map within the cluster. This config map can be loaded on the cluster statically. or created - if not existent - in that then only is used in The net new result of the above is users bootstrapping clusters can configure the worker node to use without any identity on said worker nodes.
/sig azure These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by ‍
In-tree AzureDisk volume plugin is also depending on Azure credential to fetch lun id. we should figure out how to solve it without credential
fetch disk lun id is also a dependency in kubelet code now, after my investigation, it could be removed on Linux node, while it cannot be removed on Windows node, need to change the k/k upstream code, there is a discussion on going here: , let' wait for the discussion result.
For Linux: 1- use IMDS my understanding is new api versions will have the disk/lun pairs. 2- don't use arm for the return of is the second parameter passed to . For Windows: I don't understand the dep on ARM for this. The above still apply, we just need to find a way to encode the value as ut return from that allow us to find the disk by on a windows machine. might be a good source for this. Ideally speaking. The value should be the same. But i will leave it for you to decide.
could be a good idea since currently it's a LUN num returned by Attach() and then it could be changed to the windows disk num, I will make the code change on v1.15 later, thanks for discussion.
Update: PR would remove the VM API calls when getting the LUN number for AzureDisk.
Code change:
// InstanceShutdownByProviderID returns true if the instance is in safe state to detach volumes func (vs *VSphere) InstanceShutdownByProviderID(ctx context.Context, providerID string) (bool, error) { <del> return false, cloudprovider.NotImplemented </del> <ins> nodeName, err := vs.GetNodeNameFromProviderID(providerID) if err != nil { glog.Errorf("Error while getting nodename for providerID %s", providerID) return false, err } vsi, err := vs.getVSphereInstance(convertToK8sType(nodeName)) if err != nil { return false, err } // Ensure client is logged in and session is valid if err := vs.nodeManager.vcConnect(ctx, vsi); err != nil { return false, err } vm, err := vs.getVMFromNodeName(ctx, convertToK8sType(nodeName)) if err != nil { glog.Errorf("Failed to get VM object for node: %q. err: +%v", nodeName, err) return false, err } isActive, err := vm.IsActive(ctx) if err != nil { glog.Errorf("Failed to check whether node %q is active. err: %+v.", nodeName, err) return false, err } return !isActive, nil </ins> } // InstanceID returns the cloud provider ID of the node with the specified Name.

Related discussion:
text: Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/priority important-longterm
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/triage accepted /kind feature
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this can be closed now as we have non-graceful node shutdown.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
if s.ServiceClusterIPRange.IP == nil { glog.Fatal("No --service-cluster-ip-range specified") } <ins> var ones, bits = s.ServiceClusterIPRange.Mask.Size() if bits-ones > 20 { glog.Fatal("Specified --service-cluster-ip-range is too large") } </ins> } func newEtcd(etcdConfigFile string, etcdServerList []string, interfacesFunc meta.VersionInterfacesFunc, defaultVersion, storageVersion, pathPrefix string) (etcdStorage storage.Interface, err error) {

Related discussion:
text: If you spec a /8 for your service Cluster IPs, the bitmask becomes 2 MB, which is too big for etcd. We should enforce that the prefix be /12 or smaller or something like that.
Yes
Code change:
} <ins> func WaitForRootPaths(t *testing.T, ctx context.Context, client testClient, requirePaths, forbidPaths sets.Set[string]) error { return wait.PollUntilContextTimeout(ctx, 250*time.Millisecond, maxTimeout, true, func(ctx context.Context) (done bool, err error) { statusContent, err := client.Discovery().RESTClient().Get().AbsPath("/").SetHeader("Accept", "application/json").DoRaw(ctx) if err != nil { return false, err } rootPaths := metav1.RootPaths{} if err := json.Unmarshal(statusContent, &rootPaths); err != nil { return false, err } paths := sets.New(rootPaths.Paths...) if missing := requirePaths.Difference(paths); len(missing) > 0 { t.Logf("missing required root paths %v", sets.List(missing)) return false, nil } if present := forbidPaths.Intersection(paths); len(present) > 0 { t.Logf("present forbidden root paths %v", sets.List(present)) return false, nil } return true, nil }) } </ins> func WaitForGroups(ctx context.Context, client testClient, groups ...apidiscoveryv2beta1.APIGroupDiscovery) error { return WaitForResultWithCondition(ctx, client, func(groupList apidiscoveryv2beta1.APIGroupDiscoveryList) bool { for _, searchGroup := range groups {

Related discussion:
text: adding to milestone and making release blocking ... we should not release 1.29 in a state that the most commonly used conformance tool will fail
IMO we should clarify the state and go fix the tool, but we should keep the test and we should NOT add skip. Conformance docs are not a release artifact, and sonobuoy release lifecycle is not a conformance artifact. If we can't get sonobuoy working in a reasonable time, we already know the tests can be run by scheduling the image (which is ours) directly with kubectl instead and we can document that in the upstream conformance docs. We need to fix the for this image: ,%20master%20(dev) And it should probably be to release-blocking in the future.
it looks like the e2e change made in was not consciously made part of conformance, it was just trying to functionally test the bugfix being made. For 1.29, I think we should revert the change to and make an integration test that validates the fix instead. We can revisit whether content of is part of conformance or not for 1.30
I opened to revert the e2e change and move the functional test to an integration test
Filed to try to get the image job passing again.
We probably need another issue to track 1.30 follow-up?
If no one has done it, I could create another issue and follow up with the items that have been discussed here: outdated privilege requirement in commonly used conformance tools like sonobuoy aware of the required priviledge to avoid similar situations in the future. permission of GET "/" to authorized users via the role for root discovery.
Code change:
fakeDocker.VersionInfo = []string{} kubeClient.ReactFn = testclient.NewSimpleFake(&api.NodeList{Items: []api.Node{ <del> {ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}}, </del> <ins> {ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}}, </ins> }}).ReactFn mockCadvisor := testKubelet.fakeCadvisor machineInfo := &cadvisorApi.MachineInfo{

Related discussion:
text: Looks like some kubelet unit tests perform a lookupIp which ends up depending on the health of the local resolver. Specifically has this failing: Which leads to the test in: And we recently swapped out the node names from testnode to 127.0.0.1, so probably the culprit?
/ref
Interesting. I did swap out the node names in the unit test as part of to be 127.0.0.1, party because I thought that it wouldn't have any problems resolving during unit tests.
/cc
Code change:
fakeDocker.VersionInfo = []string{} kubeClient.ReactFn = testclient.NewSimpleFake(&api.NodeList{Items: []api.Node{ <del> {ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}}, </del> <ins> {ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}}, </ins> }}).ReactFn mockCadvisor := testKubelet.fakeCadvisor machineInfo := &cadvisorApi.MachineInfo{

Related discussion:
text: cc
What is the environment where you are running the tests? They pass for me on my mac, a linux box, and shippable seems to be passing them as well.
I couldn 't reproduce the failure either.
I'm out right now; i'll try them when i get back. For the record, I run hack/test- Paul Morie wrote:
Ahem FTR This acronym is going to go quad-plat Paul Morie wrote:
I ran on both platforms and saw the tests pass.
I'm still seeing it. Won't have bandwidth to dig in personally until tomorrow.
seeing a similar failure after a git pull
I knew I wasn't crazy
environment: fedora 21, golang: 1.4, docker: 1.6
I think I see the problem. When I set the node name to in the test I was assuming it would parse as a valid ip address and not end up at the LookupIP call. But only the node name (and not the kubelet hostname) is being set (except in my new test where I ) so it's trying to do a lookup. Do you think you could test just changing the default hostname from "testnode" to "127.0.0.1" and see if that fixes it for you?
Trying now
I still get failures but no panic / threaddump.
gist coming
this looks like again?
Good that it isn't panicing any longer. I can dig into the test failures tomorrow (I think it should be relatively simple to resolve).
cool, thanks for looking
Passes in master now for me. Closing.
FYI, they will break again once goes in, and then we can merge my PR to see if that fixes it for you.
:+1:
is in -- are these tests breaking again?
I just rebased which I think will fix them.
They are breaking again.
is on it.
Code change:
// a map[family]ip for the caller to release when everything else has // executed successfully func (rs *REST) handleClusterIPsForUpdatedService(oldService *api.Service, service *api.Service) (allocated map[api.IPFamily]string, toRelease map[api.IPFamily]string, err error) { <ins> // We don't want to upgrade (add an IP) or downgrade (remove an IP) // following a cluster downgrade/upgrade to/from dual-stackness // a PreferDualStack service following principle of least surprise // That means: PreferDualStack service will only be upgraded // if: // - changes type to RequireDualStack // - manually adding or removing ClusterIP (secondary) // - manually adding or removing IPFamily (secondary) if isMatchingPreferDualStackClusterIPFields(oldService, service) { return allocated, toRelease, nil // nothing more to do. } </ins> // use cases: // A: service changing types from ExternalName TO ClusterIP types ==> allocate all new // B: service changing types from ClusterIP types TO ExternalName ==> release all allocated

Related discussion:
text: 1) start with a single stack cluster 2) create a PreferDualStack service 3) configure API server with a 2nd service CIDR 4) Confirm the above service is not dual-stack 5) Add a label to the above service 6) Observe that the service becomes dual-stack I think the correct answer is to have a "repair loop" that upgrades (and downgrades!) PreferDualStack Services to match the config. I think this is a GA blocker.
/triage accepted The current thought is: of type with are special new breed of api objects that I don't think we have any similar precedent in the apis, i could be wrong. Their behavior is tied to how the cluster is configured. Thus their behavior must change as the cluster behavior change. a way the user is stating . If it is cluster then i am good with two ClusterIPs. If it is single stack cluster then I am also good with a single ClusterIP. One can assume that if the user was ok was with getting a additional IP when possible then they must be ok with taking it away when it is no longer possible. Based on the above I believe best course of action is create a new repair loop or piggyback on any of the existing (or better: fix it in rest stack during list and get calls). The new logic takes care of matching to the .
reproducer and regression test if somebody takes this, feel free to add it
ack - Scheduled to tackle 3rd week of May. Will definitely include that patch.