jiebi/CodeConvo huggingface dataset:
Code change:
isReady, err := testutils.PodRunningReady(p) framework.ExpectNoError(err) framework.ExpectEqual(isReady, true, "pod should be ready") <del> readyIn := readyTime.Sub(startedTime) framework.Logf("Container started at %v, pod became ready at %v, %v after startupProbe succeeded", startedTime, readyTime, readyIn) if readyIn < 0 { framework.Failf("Pod became ready before startupProbe succeeded") } if readyIn > 5*time.Second { framework.Failf("Pod became ready in %v, more than 5s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.", readyIn) } </del> }) /*

Related discussion:
text: For instance, I am not sure (from the name) that should include my latest fix...
In your PR the fix wasn't there:
you may send cherry-pick to 1.21 for which is using kubelet 1.21, not 1.22.
is using kubelet. Cherry-pick to 1.20 may be needed as well.
I have found this one that failed even with the fix: corresponding to:
I don't know what could explain this atm... but it seems we still have delay propagating container statuses.
In the job artifacts there are multiple places with this info including the clone records/logs, and In testgrid it's a row.
yeah afaict since the tests ran on my PR after your fix was merged into master, it showed the flake was still present :( Prow will always use the latest master when running CI.
Sorry for doubting what you said... I was probably in deny and hoped this annoyance was fixed once for all :(
Plan: we'll see if Clayton's pod lifecycle refactor and/or the jitter refactor for probes affects this.
Clayton's PR jitter refactor
since isn't (likely) coming to 1.22 we could cut off 90% of flakes by increasing to 15s delay:
it seems it fails even after merging so let's continue our discussion on slack
this is not a regression, our suggestion is not block release on it. feel free to remove the milestone
/milestone v1.23
In Kubelet maybe, but in the flakiness of the e2e tests it was. Unfortunately we can't easily explore the current state :/
In the release blocking jobs the frequency is not horrible now, so unblocking is probably fine, it is not as bad as it was. Outside of the release cycle this is still the flakiest test in some jobs, see e.g. where it's the only flake source we have on record currently.
if we end up needing it, we have a timing/period adjustment PR we can take that should help with the flakiness with
Code change:
// unset or false and it can be flipped later when storage // capacity information has been published. // <del> // This field is immutable. </del> <ins> // This field was immutable in Kubernetes <= 1.22 and now is mutable. </ins> // // This is a beta field and only available when the CSIStorageCapacity // feature is enabled. The default is false.

Related discussion:
text: Up to 1.21, that field is immutable. That means in practice, a CSIDriver object has to be deleted and then re-created with a different value instead. Mutation via delete/created is unnecessarily complex and as the only consumer of that field (the kube-scheduler) can handle mutation just fine (it'll always ask the informer cache for the current value), it's better to allow mutation.
/sig storage /assign
This is independent of allowing other fields to become mutable, but that might also make sense.
/triage accepted
Code change:
kubelet.kubeClient = fakeKubeClient kubelet.os = kubecontainer.FakeOS{} <del> kubelet.hostname = "testnode" </del> <ins> kubelet.hostname = testKubeletHostname </ins> kubelet.runtimeUpThreshold = maxWaitForContainerRuntime kubelet.networkPlugin, _ = network.InitNetworkPlugin([]network.NetworkPlugin{}, "", network.NewFakeHost(nil)) if tempDir, err := ioutil.TempDir("/tmp", "kubelet_test."); err != nil {

Related discussion:
text: Looks like some kubelet unit tests perform a lookupIp which ends up depending on the health of the local resolver. Specifically has this failing: Which leads to the test in: And we recently swapped out the node names from testnode to 127.0.0.1, so probably the culprit?
/ref
Interesting. I did swap out the node names in the unit test as part of to be 127.0.0.1, party because I thought that it wouldn't have any problems resolving during unit tests.
/cc
Code change:
kubelet.kubeClient = fakeKubeClient kubelet.os = kubecontainer.FakeOS{} <del> kubelet.hostname = "testnode" </del> <ins> kubelet.hostname = testKubeletHostname </ins> kubelet.runtimeUpThreshold = maxWaitForContainerRuntime kubelet.networkPlugin, _ = network.InitNetworkPlugin([]network.NetworkPlugin{}, "", network.NewFakeHost(nil)) if tempDir, err := ioutil.TempDir("/tmp", "kubelet_test."); err != nil {

Related discussion:
text: cc
What is the environment where you are running the tests? They pass for me on my mac, a linux box, and shippable seems to be passing them as well.
I couldn 't reproduce the failure either.
I'm out right now; i'll try them when i get back. For the record, I run hack/test- Paul Morie wrote:
Ahem FTR This acronym is going to go quad-plat Paul Morie wrote:
I ran on both platforms and saw the tests pass.
I'm still seeing it. Won't have bandwidth to dig in personally until tomorrow.
seeing a similar failure after a git pull
I knew I wasn't crazy
environment: fedora 21, golang: 1.4, docker: 1.6
I think I see the problem. When I set the node name to in the test I was assuming it would parse as a valid ip address and not end up at the LookupIP call. But only the node name (and not the kubelet hostname) is being set (except in my new test where I ) so it's trying to do a lookup. Do you think you could test just changing the default hostname from "testnode" to "127.0.0.1" and see if that fixes it for you?
Trying now
I still get failures but no panic / threaddump.
gist coming
this looks like again?
Good that it isn't panicing any longer. I can dig into the test failures tomorrow (I think it should be relatively simple to resolve).
cool, thanks for looking
Passes in master now for me. Closing.
FYI, they will break again once goes in, and then we can merge my PR to see if that fixes it for you.
:+1:
is in -- are these tests breaking again?
I just rebased which I think will fix them.
They are breaking again.
is on it.
Code change:
<ins> ntp: pkg: - installed ntp-service: service: - running - name: ntp - watch: - pkg: ntp </ins>

Related discussion:
text: At least on AWS, if the clock drifts 300 seconds (which it will), then we get the rather cryptic message: This is because AWS includes the timestamp in the request. Installing ntpd would stop this drifting from happening. If the drift does happen, the likely symptom is that EC2 calls would stop working: attach/detach volumes on the minions, instance sync and routing configuration on the master. I don't believe that GCE has the same problem.
Code change:
// InstanceExistsByProviderID returns true if the instance with the given provider id still exists and is running. // If false is returned with no error, the instance will be immediately deleted by the cloud controller manager. func (vs *VSphere) InstanceExistsByProviderID(ctx context.Context, providerID string) (bool, error) { <del> var nodeName string nodes, err := vs.nodeManager.GetNodeDetails() </del> <ins> nodeName, err := vs.GetNodeNameFromProviderID(providerID) </ins> if err != nil { <del> glog.Errorf("Error while obtaining Kubernetes node nodeVmDetail details. error : %+v", err) </del> <ins> glog.Errorf("Error while getting nodename for providerID %s", providerID) </ins> return false, err } <del> for _, node := range nodes { // ProviderID is UUID for nodes v1.9.3+ if node.VMUUID == GetUUIDFromProviderID(providerID) || node.NodeName == providerID { nodeName = node.NodeName break } } if nodeName == "" { msg := fmt.Sprintf("Error while obtaining Kubernetes nodename for providerID %s.", providerID) return false, errors.New(msg) } </del> _, err = vs.InstanceID(ctx, convertToK8sType(nodeName)) if err == nil { return true, nil

Related discussion:
text: Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/priority important-longterm
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/triage accepted /kind feature
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this can be closed now as we have non-graceful node shutdown.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
. "github.com/onsi/gomega" ) <del> func runLivenessTest(c *client.Client, podDescr *api.Pod) { </del> <ins> func runLivenessTest(c *client.Client, podDescr *api.Pod, expectRestart bool) { </ins> ns := "e2e-test-" + string(util.NewUUID()) By(fmt.Sprintf("Creating pod %s in namespace %s", podDescr.Name, ns))

Related discussion:
text: As far as I can tell, container liveness probes of type exec flat out don't work against clusters built at head. I had the kubelet always log the output of the exec'ed command, and it's always empty no matter what I exec, causing the probe to fail since it requires the output to be equal to the string "ok". If you want to test it, try adding a liveness probe of this form to any container: I also tried these lovely forms, to more closely match the liveness example under examples/: On the plus side, this may be an opportunistic time to switch over to
The reason that the e2e test for exec liveness probes hasn't caught this is that it only tests that containers are restarted when they fail liveness probes. It doesn't test that containers aren't restarted when their liveness checks should be succeeding. The e2e test's exec'ed commands also only ever produced empty output, rather than the expected "fail" output.
Is it possible to include this in a new pre-release, since it fixes a bug in health checking? I see it merged in master and I'm wondering what the policy is of releasing new code in a separate (0.16.*) release.
Code change:
import ( "fmt" <ins> "reflect" "sort" </ins> "strings" "testing"

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created several Workloads which require up to ten ports for both UDP and TCP protocol. Now my logs get spammed with . What you expected to happen: should consider -combination instead of only . How to reproduce it (as minimally and precisely as possible): Deploy any service with a node-port with both protocols, e.g.: Anything else we need to know?: This was already acknowledged by here: However, as this thread addresses a different bug and seems kind of stuck, i'd like to suggest that this might be addressed in a new context. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Deployed via Rancher v2.4.3
/sig api-machinery
/remove-sig api-machinery /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
How did you get into this situation? The allocator should not have granted the same node port twice?
Oh right, that issue. The only real way out is to de-allocate the NodePorts and then re-allocate them, I think
As this happens for all of my ports in the said Production Application (the one where my log gets spammed) as well as complete fresh installation (where i verified the bug), i think there's no "way out" except fixing the bug. As said, for me it's necessary to assign both protocols to the same port. This might be a niche-use-case, but checking the it's actually very likely for ports to allow tcp AND udp. I'm not sure about the history of this, but it might be that issue was the origin, which was adressed in PR ("Allow services which use same port, different protocol to use the same nodePort for both"). But this PR doesn't contain the adjustment of the said , so still considers the behaviour as invalid, while it's actually allowed and supported.
Code change:
import ( "fmt" <ins> "reflect" "sort" </ins> "strings" "testing"

Related discussion:
text: /assign
Ahh, I thought you meant that you had accidentally been allocated the same nodePorts and I was very confused when you said "all of them". I have reproduced this issue, and I can see it by inspection. The obvious fix would be to collect ports split by protocol, but I think that's not right since we don't store it split. An even simpler fix would be for to build a . Then for each protocol, for each port, build that contains all the port numbers, but remove any duplicates ONLY IF they are duplicated across protocols. Duplicates in the same protocol should be retained (to report an error later). I don't have time to tackle this tonight (and write a test), but it seems like a good bug.
perhaps I can fix this problem /assign
things may be a little more complicated. Right now the is not aware of the protocol of the port so I guess simply rewriting function to return is not enough, the port allocator will still return on same node port even with different protocol. perhaps we need to refactor the to make it protocol-aware.
I don't think we need to do make the allocator be protocol aware. IIRC we allow you to use the same nodeport on different protocols ONLY within a single service. This means (I think) that we can do hyper-local deduplication as we collect the nodeports for each service. If we find the same nodeport number with different protocols, just remove the duplicates because it is allowed. I.e. a service that looks like: return - deduped. if ANY OTHER service tried to use it would be an error at the "rebuilt" allocator Likewise if a single Service claims to use the same nodeport number more than once on the same protocol, it should be an error: return - NOT deduped.
obviously, we want a great unit test here :P
ah got it! thanks for your detailed explanation!
Hi! I have filed a PR to resolve this issue, could you take a look?
Code change:
} trap cleanup EXIT <ins> goarch=$(go env GOARCH) </ins> kind build node-image --image "$tag" "$(pwd)" <del> curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/$containerd-linux-amd64.tar.gz | tar -C "$tmpdir" -vzxf - curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/runc.amd64 >"$tmpdir/runc" </del> <ins> curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/$containerd-linux-"$goarch".tar.gz | tar -C "$tmpdir" -vzxf - curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/runc."$goarch" >"$tmpdir/runc" </ins> cat >"$tmpdir/Dockerfile" <<EOF FROM $tag

Related discussion:
text: I am trying to bring up a kind cluster with containerd in order to run e2e test on mac + kind. Not sure whether this is a bug or I missed some prerequisites. Any help would be greatly appreciated :) building the image dra/node:latest works as expected:bringing up the kind cluster failed (seems that kubelet is unhealthy and healthchecked failed) below the control plane logs: below the logs of one worker: successfully bring up the kind cluster with containerd Tried on M2 macbook pro I followed NONE Darwin MacBook-Pro-de- 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct 9 20:19:12 PDT 2022; root:xnu-8792.41.9~3/RELEASEARM64T6020 arm64 arm Darwin No response No response
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what type of mac you're using? for m1/m2 mac, you need to replace the linux-amd64 with linux-arm64(your log seems like using m1/m2 mac which same as the problem I've met before)
thanks I am going to try that
ping me if you can run e2e on mac. I setup the cluster but failed to run the e2e test..
can you please share the exact command you tried and the failure ?
If you use a newer version of kind you might not need to clobber the included containerd, not sure what version is required And yes, on M1/M2 you need to use arm binaries/images.
this is the command that I tried !
Code change:
} trap cleanup EXIT <ins> goarch=$(go env GOARCH) </ins> kind build node-image --image "$tag" "$(pwd)" <del> curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/$containerd-linux-amd64.tar.gz | tar -C "$tmpdir" -vzxf - curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/runc.amd64 >"$tmpdir/runc" </del> <ins> curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/$containerd-linux-"$goarch".tar.gz | tar -C "$tmpdir" -vzxf - curl -L --silent https://github.com/kind-ci/containerd-nightlies/releases/download/$containerd/runc."$goarch" >"$tmpdir/runc" </ins> cat >"$tmpdir/Dockerfile" <<EOF FROM $tag

Related discussion:
text: It worked for me just fine on M2 machine with this obvious fix for the build script: Here is a test output:
I'll submit a PR with the fix. Updating the documentation would be also nice to have. I hope and can do that.
DRA tests fail on kind 0.18.0. Which Containerd version is used there?
It was the latest 1.6.x at release time. 1.6 is their LTS branch before 2.0 comes which has some tricky changes for us to deal with (notably the CRI hosts config deprecation which will break all our local registry users :/)
Unfortunately 1.6 doesn't have CDI support, which DRA depends on. The version used in the build script is 1.6.0-830-g34d078e99, but it was probably built from containerd master after 1.6 was released.
thanks for providing the details. you could run the e2e as shown by and find the result
Code change:
VolumeMounts:  volumeMounts, LivenessProbe: componentProbe(10252, "/healthz"), Resources:     componentResources("200m"), <ins> Env:           getProxyEnvVars(), </ins> }, volumes...), kubeScheduler: componentPod(api.Container{ Name:          kubeScheduler,

Related discussion:
text: Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? http proxy, httpproxy, httpsproxy, AWS, kubeadm, kubectl, cloud-provider Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): v1.4.4 Environment: Cloud provider or hardware configuration: kubeadm on EC2 instances on AWS OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Linux ip-10-0-133-99 4.4.0-45-generic -Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: httpproxy, httpsproxy and noproxy set in /etc/environment, Docker config as well on startup scripts that run What happened: won't complete and gets stuck on it's first deployment. When one looks through the logs it shows that the pods are downloading fine but we are stuck due to the kube-controller-manager continually crashing as it can't contact AWS. Elsewhere in kubernetes I see references to and if the AWS cloud provider could use that it would be able to connect as well. From the POD if I manually set httpproxy and httpsproxy and no_proxy I am able to use the AWS CLI without any issues. What you expected to happen: I would expect to successfully complete and use the environment proxies set up on the system so as not to depend on a default Internet Gateway being connected to the VPC just for AWS access. How to reproduce it (as minimally and precisely as possible): Set up a public VPC with Squid set up as a proxy Set up a private VPC peered to the public VPC Set up a master node and run Set up a regular node and run will never fully initialize. If you remove the --cloud-provider aws` then it will complete and nodes will register. Anything else do we need to know:
what do you have in proxy variables ? (env | grep -i proxy) it tries to connect to (tcp 176.32.118.53:443) directly. You're expecting to connect there via proxy ? then do you have something like in your no_proxy ?
Code change:
VolumeMounts:  volumeMounts, LivenessProbe: componentProbe(10252, "/healthz"), Resources:     componentResources("200m"), <ins> Env:           getProxyEnvVars(), </ins> }, volumes...), kubeScheduler: componentPod(api.Container{ Name:          kubeScheduler,

Related discussion:
text: Unfortunately on the pod itself for the the environment variables aren't passed through. As this is created using I don't have control over the pod definition. So basically none of the *_proxy variables are set inside the pod in the docker image, however if I and export the variables and try to wget that URL it works. Basically I'm hoping we can update to respect and pass the environment variables down to the system pods if they exist, eg, add: Without it I can't stand up a cluster with AWS integration on a completely private VPC and for such a small change it would make a rather big difference.
ok, so request is actually to pass proxy variables into containers. interesting. I'll check how it can be done. BTW, for noproxy value in your example: things like 10.0.0.0/8 would not work on 99% of software that respects noproxy variable. it must be always exact IP match.
you're right that what I'm actually asking for is to pass proxy variables through either from a commandline setting on kubeadm or pick it up from /etc/environment or shell environment that launches kubeadm. I didn't realize it till I dug in to try and think about how one might go about fixing this. At first I thought we would just pick up the variables from the system environment but then I realized kube-controller-manager is deployed in a container and so we'd need to update the yaml file that launches it to contain the environment variables. As I'd like to go to production with this soon hopefully this is a small change that can be .
This issue was moved to kubernetes/kubeadm