jiebi/CodeConvo huggingface dataset:
Code change:
We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used. FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted

Related discussion:
text: "fmt" "math/rand" "sync" <ins> "sync/atomic" </ins> "testing" "time"
Code change:
We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used. FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted

Related discussion:
text: return KeyError{item, err} } keys.Insert(key) <del> if err := f.queueActionLocked(action, item); err != nil { </del> <ins> if err := f.queueActionInternalLocked(action, Replaced, item); err != nil { </ins> return fmt.Errorf("couldn't enqueue object: %v", err) } }
Code change:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

Related discussion:
text: return fmt.Errorf("failed to try resolving symlinks in path %q: %v", path, err) } path = evaluated <del> f, err := os.Open(path) </del> <ins> f, err := openFileShareDelete(path) </ins> if err != nil { return fmt.Errorf("failed to open log file %q: %v", path, err) }
Code change:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

Related discussion:
text: return err } if recreated { <del> newF, err := os.Open(path) </del> <ins> newF, err := openFileShareDelete(path) </ins> if err != nil { if os.IsNotExist(err) { continue
Code change:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

Related discussion:
text: <ins> //go:build !windows // +build !windows /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package logs import ( "os" ) func openFileShareDelete(path string) (*os.File, error) { // Noop. Only relevant for Windows. return os.Open(path) } </ins>
Code change:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

Related discussion:
text: "io" "os" "path/filepath" <del> goruntime "runtime" </del> "testing" "time"
Code change:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

Related discussion:
text: } func TestReadRotatedLog(t *testing.T) { <del> if goruntime.GOOS == "windows" { // TODO: remove skip once the failing test has been fixed. t.Skip("Skip failing test on Windows.") } </del> tmpDir := t.TempDir() file, err := os.CreateTemp(tmpDir, "logfile") if err != nil {
Code change:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

Related discussion:
text: } } <ins> // Finished writing into the file, close it, so we can delete it later. err = file.Close() assert.NoErrorf(t, err, "could not close file.") </ins> time.Sleep(20 * time.Millisecond) // Make the function ReadLogs end. fakeRuntimeService.Lock()
Code change:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

Related discussion:
text: <ins> //go:build windows // +build windows /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package logs import ( "os" "syscall" ) // Based on Windows implementation of Windows' syscall.Open // https://cs.opensource.google/go/go/+/refs/tags/go1.22.2:src/syscall/syscall_windows.go;l=342 // In addition to syscall.Open, this function also adds the syscall.FILE_SHARE_DELETE flag to sharemode, // which will allow us to read from the file without blocking the file from being deleted or renamed. // This is essential for Log Rotation which is done by renaming the open file. Without this, the file rename would fail. func openFileShareDelete(path string) (*os.File, error) { pathp, err := syscall.UTF16PtrFromString(path) if err != nil { return nil, err } var access uint32 = syscall.GENERIC_READ var sharemode uint32 = syscall.FILE_SHARE_READ | syscall.FILE_SHARE_WRITE | syscall.FILE_SHARE_DELETE var createmode uint32 = syscall.OPEN_EXISTING var attrs uint32 = syscall.FILE_ATTRIBUTE_NORMAL handle, err := syscall.CreateFile(pathp, access, sharemode, nil, createmode, attrs, 0) if err != nil { return nil, err } return os.NewFile(uintptr(handle), path), nil } </ins>
Code change:
in This flaking test case was newly as part of the above PR Flake was reproducible with stress tool: [root kubernetes]# go version go version go1.22.2 linux/ppc64le [root kubernetes]# stress - ^TestLog /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Looks like in this test case is a concatenation of logs from Pod0 and Pod1, hence it is failing in some cases. I will add a PR to split it.

Related discussion:
text: o.Prefix = true return o }, <del> expectedOutSubstrings: []string{"[pod/test-sts-0/test-container] test log content for pod test-sts-0n[pod/test-sts-1/test-container] test log content for pod test-sts-1n"}, </del> <ins> expectedOutSubstrings: []string{ "[pod/test-sts-0/test-container] test log content for pod test-sts-0n", "[pod/test-sts-1/test-container] test log content for pod test-sts-1n", }, </ins> }, { name: "pod logs with prefix: init container",
Code change:
Currently, the scheduler honors Unschedulable(AndUnresolvable) or Pending only from PreFilter, Filter, Reserve, and Permit (WaitOnPermit). It doesn't expect other extension points to return those statuses and just regard all non-success status as errors, that is, immediately retrying scheduling of Pods. Actually returns Pending at PreBind, which is currently being ignored as described. So, we should expand UnschedulablePlugins/PendingPlugins to include PreBind plugins. (or maybe all extension points too?) /cc /sig scheduling /kind feature
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Both seems reasonable to me.
Because is not handled for , the following error is logged:
This patch gets rid of the error. I am uncertain whether it has other implications? Performance was the same with and without it.
What I imagined is like that. It shouldn't have any other impact because volumebinding, which is the only PreBind plugin apart from DRA in in-tree, only returns success or error status. We have to mention the change in the release note for custom plugin developers though.
I've turned this into a PR:

Related discussion:
text: // latency for a pod. InitialAttemptTimestamp *time.Time // UnschedulablePlugins records the plugin names that the Pod failed with Unschedulable or UnschedulableAndUnresolvable status. <del> // It's registered only when the Pod is rejected in PreFilter, Filter, Reserve, or Permit (WaitOnPermit). </del> <ins> // It's registered only when the Pod is rejected in PreFilter, Filter, Reserve, PreBind or Permit (WaitOnPermit). </ins> UnschedulablePlugins sets.Set[string] // PendingPlugins records the plugin names that the Pod failed with Pending status. PendingPlugins sets.Set[string]
Code change:
Currently, the scheduler honors Unschedulable(AndUnresolvable) or Pending only from PreFilter, Filter, Reserve, and Permit (WaitOnPermit). It doesn't expect other extension points to return those statuses and just regard all non-success status as errors, that is, immediately retrying scheduling of Pods. Actually returns Pending at PreBind, which is currently being ignored as described. So, we should expand UnschedulablePlugins/PendingPlugins to include PreBind plugins. (or maybe all extension points too?) /cc /sig scheduling /kind feature
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Both seems reasonable to me.
Because is not handled for , the following error is logged:
This patch gets rid of the error. I am uncertain whether it has other implications? Performance was the same with and without it.
What I imagined is like that. It shouldn't have any other impact because volumebinding, which is the only PreBind plugin apart from DRA in in-tree, only returns success or error status. We have to mention the change in the release note for custom plugin developers though.
I've turned this into a PR:

Related discussion:
text: // Run "prebind" plugins. if status := fwk.RunPreBindPlugins(ctx, state, assumedPod, scheduleResult.SuggestedHost); !status.IsSuccess() { <ins> if status.IsRejected() { fitErr := &framework.FitError{ NumAllNodes: 1, Pod:         assumedPodInfo.Pod, Diagnosis: framework.Diagnosis{ NodeToStatusMap:      framework.NodeToStatusMap{scheduleResult.SuggestedHost: status}, UnschedulablePlugins: sets.New(status.Plugin()), }, } return framework.NewStatus(status.Code()).WithError(fitErr) } </ins> return status }
Code change:
/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Failures look like started when landed

Related discussion:
text: ginkgo.It("should reject a pod with an AppArmor profile", func(ctx context.Context) { status := runAppArmorTest(ctx, f, false, v1.DeprecatedAppArmorBetaProfileRuntimeDefault) <del> expectSoftRejection(status) </del> <ins> expectRejection(status) </ins> }) }) }
Code change:
/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Failures look like started when landed

Related discussion:
text: return e2epod.NewPodClient(f).Create(ctx, pod) } <del> func expectSoftRejection(status v1.PodStatus) { </del> <ins> func expectRejection(status v1.PodStatus) { </ins> args := []interface{}{"PodStatus: %+v", status} <del> gomega.Expect(status.Phase).To(gomega.Equal(v1.PodPending), args...) </del> <ins> gomega.Expect(status.Phase).To(gomega.Equal(v1.PodFailed), args...) </ins> gomega.Expect(status.Reason).To(gomega.Equal("AppArmor"), args...) gomega.Expect(status.Message).To(gomega.ContainSubstring("AppArmor"), args...) <del> gomega.Expect(status.ContainerStatuses[0].State.Waiting.Reason).To(gomega.Equal("Blocked"), args...) </del> } func isAppArmorEnabled() bool {