Change:
Observed this on our internal CI Job from After addition of test with commit on 17th February Saturday (IST) Though the internal CI job runs on architecture, we have also observed this on architecture. /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
^^
/sig node
don't see anything on linux for sure (4 hits on windows):
I’m wondering how to reproduce this locally.
Steps to reproduce: clone kubernetes go test ./pkg/kubelet/userns/ -race
where did you run it? Works just fine for me on Ubuntu x86_64:
I can't repro locally, I don't think there is any arch specific issue either. Based on comment and link I guess for some reason this test is running on Windows and os.Chmod there works differently. This feature is linux-only, so I'll rename the test the include '_linux' and that should solve this issue.
Isn't there any other way to avoid windows picking this up? Because lot of things are linux only (not just in userns), it would reduce the noise considerably if windows was opt-in.
I don't see any signs of windows here: It's not a x86_64 Linux, but not a windows for sure.
the link shared points to windows failures.
The test grid this issue has in description points to the same failure on ppc.
Let's see if this fixes it, otherwise we can remove that test too:
I ran it in one of our local VMs with centos.
What I don't understand is why this never fail when the PR was open.
How this suppose to fix a failure on PPC linux? and on x86_64 linux (I guess) mentioned by above?
which hw architecture do you run it on? Can you show the output of the on that machine?
Ohh, I got it! The CI is running the "go test" as root, with all the capabilities. So all test that rely on not having permissions to write somewhere, fail, as they have capabilities to bypass that. It seems completely weird that this didn't fail before. This smells like: Not all of our CI runs on open PRs The CI was changed to run as root now Both things can cause lot of noise and failures. If it wasn't any of those, I don't know what could have caused it.
it is not arch nor distro, it is that just some are running as root. IMHO, if not all are running as root or unprivileged, it is a bug in our CI configuration.
I have opened PR to fix the issue, but the root cause seems to be a bug in our CI. How can we report that we started running tests on some architectures as root, with all the capabilities, and therefore tests that exercise a permission check failure can't never work in those setups?
you're right, the test fails when running as root. I'd suggest to fix the test then. In shouldn't fail even if it runs with root privileges.
with root AND all the capabilities, you bypass lot of permission checks. I open a PR to just remove this test, as it wasn't adding a LOT of value either. I'm most worried about the rest of the unit test (outside the userns package), as if this change was done recently, I expect several others to fail too. How can we give this feedback to the relevant people?
I think should know more, but suspect that it was not done recently.
If it wasn't done recently, those workers running as root didn't run when the PR was open. So something changed recently, otherwise the test would have failed (it is not flaky, it always passes as unprivileged and it always fails as root).
Yes, the majority of the node related tests are not run on open PRs. We usually delegate feature testing to the author of the PR by triggering presubmits. The node tests are much longer than the required jobs on the PR and they usually filter out alpha jobs.

Discussion:
assert.Error(t, err) assert.ErrorContains(t, err, "read pods from disk") } <del> func TestMakeUserNsManagerFailsPodRecord(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, pkgfeatures.UserNamespacesSupport, true)() testUserNsPodsManager := &testUserNsPodsManager{ podList: []types.UID{"pod-1", "pod-2"}, podDir:  t.TempDir(), } // Remove read/execute permissions from this directory. if err := os.Chmod(testUserNsPodsManager.podDir, 0222); err != nil { t.Fatal(err) } _, err := MakeUserNsManager(testUserNsPodsManager) assert.Error(t, err) assert.ErrorContains(t, err, "record pod mappings") } </del>