jiebi/CodeConvo huggingface dataset:
Code change:
if [[ "${master}" == "true" && "${MASTER_OS_DISTRIBUTION}" == "gci" ]] || [[ "${master}" == "false" && "${NODE_OS_DISTRIBUTION}" == "gci" ]]; then cat >>$file <<EOF <del> VOLUME_PLUGIN_DIR: $(yaml-quote ${VOLUME_PLUGIN_DIR:-/home/kubernetes/flexvolume}) </del> <ins> VOLUME_PLUGIN_DIR: $(yaml-quote ${VOLUME_PLUGIN_DIR:-/etc/srv/kubernetes/kubelet-plugins/volume/exec}) </ins> EOF fi

Related discussion:
text: 4 plugin-] flexVolume driver k8s/dummy-attachable-e2e-tests-flexvolume-j8962: using default GetVolumeName for volume 0x1c5df60 I0824 17:43:59. 3728 ] operationExecutor.VerifyControllerAttachedVolume started for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0") pod "flex-client" (UID: "-88f3-11e7-a3f9-") E0824 17:43:59. 3728 ] Operation for ""flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0"" failed. No retries permitted until 2017-08-24 17:44:00. +0000 UTC (durationBeforeRetry 500ms). Error: Volume has not been to the list of VolumesInUse in the node's volume status for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0") pod "flex-client" (UID: "-88f3-11e7-a3f9-") Seems there is something wrong with VerifyControllerAttachedVolume, still at a loss as to why it's showing up in GCI but not Debian edit: get node -o yaml shows: yet it errors because the volume hasn't been to volumesInUse??? Aug 24 20:31:42 mawong-e2e-minion-group-npjh kubelet[9431]: E0824 20:31:42. 9431 ] Operation for ""flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-z8xc4/flex-volume-0"" failed. No retries permitted until 2017-08-24 20:31:44. +0000 UTC (durationBeforeRetry 2s).
Code change:
if [[ "${master}" == "true" && "${MASTER_OS_DISTRIBUTION}" == "gci" ]] || [[ "${master}" == "false" && "${NODE_OS_DISTRIBUTION}" == "gci" ]]; then cat >>$file <<EOF <del> VOLUME_PLUGIN_DIR: $(yaml-quote ${VOLUME_PLUGIN_DIR:-/home/kubernetes/flexvolume}) </del> <ins> VOLUME_PLUGIN_DIR: $(yaml-quote ${VOLUME_PLUGIN_DIR:-/etc/srv/kubernetes/kubelet-plugins/volume/exec}) </ins> EOF fi

Related discussion:
text: Error: Volume has not been to the list of VolumesInUse in the node's volume status for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-z8xc4/flex-volume-0") pod "flex-client" (UID: "-890b-11e7-9392-")
Never mind the GCI vs Debian issue, our test fix would have worked but in between and , something else changed and is causing both GCI and Debian tests to fail.
might be the cause, so we might be looking at legitimate failures caught by the test at this point (yay?) and not faulty tests. investigating...
Code change:
readonly node_logfiles="kube-proxy" readonly aws_logfiles="cloud-init-output" readonly gce_logfiles="startupscript" <del> readonly common_logfiles="kern" </del> <ins> readonly kern_logfile="kern" </ins> readonly initd_logfiles="docker" readonly supervisord_logfiles="kubelet supervisor/supervisord supervisor/kubelet-stdout supervisor/kubelet-stderr supervisor/docker-stdout supervisor/docker-stderr"

Related discussion:
text: Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "LRUExpireCache" Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT What happened: requires write lock since needs to manipulate its list to track recently used item. Currently it uses read lock so it may introduce race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
This was fixed in
Code change:
} } } <ins> func TestGetRbdImageInfo(t *testing.T) { tmpDir, err := utiltesting.MkTmpdir("rbd_test") if err != nil { t.Fatalf("error creating temp dir: %v", err) } defer os.RemoveAll(tmpDir) for i, c := range []struct { DeviceMountPath    string TargetRbdImageInfo *rbdImageInfo }{ { DeviceMountPath:    fmt.Sprintf("%s/plugins/kubernetes.io/rbd/rbd/pool1-image-image1", tmpDir), TargetRbdImageInfo: &rbdImageInfo{pool: "pool1", name: "image1"}, }, { DeviceMountPath:    fmt.Sprintf("%s/plugins/kubernetes.io/rbd/mounts/pool2-image-image2", tmpDir), TargetRbdImageInfo: &rbdImageInfo{pool: "pool2", name: "image2"}, }, } { rbdImageInfo, err := getRbdImageInfo(c.DeviceMountPath) if err != nil { t.Errorf("Case %d: getRbdImageInfo failed: %v", i, err) continue } if !reflect.DeepEqual(rbdImageInfo, c.TargetRbdImageInfo) { t.Errorf("Case %d: unexpected RbdImageInfo, wanted %v, got %v", i, c.TargetRbdImageInfo, rbdImageInfo) } } } </ins>

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pod was deleted successfully but there is still a lock on the ceph image. I find two logs,one for that lock not removed and another for that lock removed. I find that there are little difference between them. the log that lock not removed: and the log that lock removed: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.6.7 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): centos7 Kernel (e.g. ): 3.10.0-327.el7.x86_64 Install tools: Others:
/sig storage
Error 16 is EBUSY. similar to
were you running kubelet with cadvisor on?
/assign
yes, every kubelet run with cadvisor,and i know what code 16 represent for,but i still don't know the reason, is this a bug? How can i avoid this kind of situation from happening,thank you
It is probably related to
We see some other case may cause , kubelet run with nodeexporter and nodeexporter is deployed by k8s daemonset with -v /:/rootfs:ro (mount kubelet host rootfs into nodeexporter container). rbd device is mounted before nodeexporter started, because k8s 1.6.7 not support mount propagation, when the rbd plugin unmount /dev/rbd0 on the host namespace, the mount info will still leave over in node_exporter container namespace, may cause . hope this would help
thanks, does the problem persist if node_exporter stops collecting filesystem metrics?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .
Code change:
} } } <ins> func TestGetRbdImageInfo(t *testing.T) { tmpDir, err := utiltesting.MkTmpdir("rbd_test") if err != nil { t.Fatalf("error creating temp dir: %v", err) } defer os.RemoveAll(tmpDir) for i, c := range []struct { DeviceMountPath    string TargetRbdImageInfo *rbdImageInfo }{ { DeviceMountPath:    fmt.Sprintf("%s/plugins/kubernetes.io/rbd/rbd/pool1-image-image1", tmpDir), TargetRbdImageInfo: &rbdImageInfo{pool: "pool1", name: "image1"}, }, { DeviceMountPath:    fmt.Sprintf("%s/plugins/kubernetes.io/rbd/mounts/pool2-image-image2", tmpDir), TargetRbdImageInfo: &rbdImageInfo{pool: "pool2", name: "image2"}, }, } { rbdImageInfo, err := getRbdImageInfo(c.DeviceMountPath) if err != nil { t.Errorf("Case %d: getRbdImageInfo failed: %v", i, err) continue } if !reflect.DeepEqual(rbdImageInfo, c.TargetRbdImageInfo) { t.Errorf("Case %d: unexpected RbdImageInfo, wanted %v, got %v", i, c.TargetRbdImageInfo, rbdImageInfo) } } } </ins>

Related discussion:
text: Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
I'm experiencing this issue. From what I can read above, if the rbd mount happens before the node_exporter starts; we'll always experience status 16?
We're getting this issue. This is because the mounted volume is mounted on Cadvisor container too. So the /dev/rbdX file is still in use. But still didn't find any solution to fix.
rbd volume will stuck when failed as will always returen
My solution: container use a volumeMount. make a script file to be executed every a minute:
Code change:
"k8s.io/apimachinery/pkg/util/strategicpatch" core "k8s.io/client-go/testing" <ins> "time" </ins> ) func TestRealHistory_ListControllerRevisions(t *testing.T) {

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Statefulsets are sometimes continually rolling pods and I don't know why statefulset created: storageclass referenced: kubectl describe statefulset statefulset-test: As you can see from the above describe, the pod in the statefulset has been restarted a lot of times. Observing the created pod, 1) The pod comes up ready 2) immediately transitions to terminating 3) Is terminated 4) A new pod is created 5) Go back to (1) (repeats forever) Despite no updates to the statefulset spec ocurring. Making a change to the statefulset template (for example a trivial change adding a nonsense annotation) stops the rolling and fixes the issue When this statefulset was created, there was a mutating webhook enabled. After the observed pod rolling. I disabled this by making its configuration invalid and making it non required, and the rolling persisted. The status of the statefulset: : You can see that there is a discrepency between currentRevision and updateRevision. currentRevision: statefulset-test- updateRevision: statefulset-test- A short while later the status is changed to contain currentRevision: statefulset-test- updateRevision: statefulset-test- This change happened at Sun Apr 1 19:48:48 2018 Presumably after the rollout of statefulset-test- is complete.
Code change:
"k8s.io/apimachinery/pkg/util/strategicpatch" core "k8s.io/client-go/testing" <ins> "time" </ins> ) func TestRealHistory_ListControllerRevisions(t *testing.T) {

Related discussion:
text: Here are all the controllerrevisions from above: kubectl get controllerrevision statefulset-test- -o yaml kubectl get controllerrevision statefulset-test- -o yaml: kubectl get controllerrevision statefulset-test- -o yaml: The differences between the above controllerrevisions are insignificant (the hash, the creation timestamp, the name, the resource version, the uid, the self link) Interestingly all these controllerrevisions were created very close together in time shortly after I created the statefulset The current controllerrevisions are After watching the state change for a while, it actually seems like the statefulset is just cycling through the updateRevision being one of the above 10 controllerrevisions Leaders during this time From kube-controller-manager logs during this time, here is one excerpt: The error sync in there is suspicous, and as you can see the controllerrevisions are changing in the logs I have the full logs from a 5 minute window on either side of the above if necessary. The reason I'm inclined to think this is a kubernetes bug is that any update to the statefulset template fixes the problem. How to reproduce it (as minimally and precisely as possible): Not really sure. Worth just trying to create the above statefulset (requires creating the I tried deleting and recreating the above statefulset and it didn't happen, so something flaky is going on Anything else we need to know?: kubernetes 1.9.3 (client and server) cloud provider: aws
/sig apps
I realize this will probably need some further debugging, happy to get some direction
Consistent repro: Create a statefulset that creates a controllerrevision whose hash is between max int32 and max uint32 (this shouldn't be that hard to do, possibly a 50/50 chance of this). Manually create a controllerrevision identical to the first (same revision), incrementing both the name and hash by one. See that the statefulset controller will continually switch which controller revision should be the target of the statefulset.
Due to the issue fixed here: , under some circumstances controllerrevisions whose revision is the same were being considered equal.
Code change:
"k8s.io/apimachinery/pkg/util/strategicpatch" core "k8s.io/client-go/testing" <ins> "time" </ins> ) func TestRealHistory_ListControllerRevisions(t *testing.T) {

Related discussion:
text: This ends up causing this bug to manifest as follows: ListRevisions finds many controllerrevisions with the same revision (unclear how this state happens, but it can happen) The sort here randomly sorts them Due to the linked bug , FindEqualRevisions incorrectly finds that all these controller revisions are equal to the proposed udpate. The updateRevision selects the last revision found here Therefore, updateRevision, is randomly selected and causes the bug. This fixes the immediate issue, but the sort should probably be made stable (i.e not just look at the revision number, perhaps just consider the hash for tie breakers)
As an update, fixing the equality check will not be sufficient because it is possible for multiple controller revisions to be created that are actually equal, including the same hash, that have the same revision number. I believe the fix will need to be equal controllerrevisions with the same revision number from being created sort the controllerrevisions so that the controllerrevision that is the target of the statefulset is stable
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
ping I use the K8S which version is 1.11.4 and encounter the same problem. I want to reproduce the problem and find your way: Manually create a controllerrevision identical to the first (same revision), incrementing both the name and hash by one. Could you tell me how to mannually create a same controllerrevision identical to the first? Thanks in anvance.
Code change:
### Hook Handler Implementations Hook handlers are the way that hooks are surfaced to containers.  Containers can select the type of hook handler they would like to implement.  Kubernetes currently supports two different hook handler types: <del> * Exec - Executes a specific command (e.g. pre-stop.sh) inside the cgroup and namespaces of the container.  Resources consumed by the command are counted against the container.  Commands which return non-zero values are treated as container failures (and will cause kubelet to forcibly restart the container).  Parameters are passed to the command as traditional linux command line flags (e.g. pre-stop.sh --reason=HEALTH) </del> <ins> * Exec - Executes a specific command (e.g. pre-stop.sh) inside the cgroup and namespaces of the container.  Resources consumed by the command are counted against the container.  Commands which print "ok" to standard out (stdout) are treated as healthy, any other output is treated as container failures (and will cause kubelet to forcibly restart the container).  Parameters are passed to the command as traditional linux command line flags (e.g. pre-stop.sh --reason=HEALTH) </ins> * HTTP - Executes an HTTP request against a specific endpoint on the container.  HTTP error codes (5xx) and non-response/failure to connect are treated as container failures. Parameters are passed to the http endpoint as query args (e.g. http://some.server.com/some/path?reason=HEALTH)

Related discussion:
text: It seems that exec probes must emit "ok" to be considered success, rather than return 0. If this is what we really want, we should document it somewhere.
This is (was?) a limitation of docker exec, which didn't return the code. Brendan On Jan 28, 2015 8:38 AM, "Tim Hockin" wrote:
The doc says it looks at exit status, so we need to fix one or the other.
for kubelet
for exec
could you take a look at this? Feel free to re-assign it back to me.
FWIW the current behavior isn't SO bad, and it makes clear a failure in the thing that probed vs a failed probe. Just saying, we don't NECESSARILY need to change it. On Jan 28, 2015 5:26 PM, "Dawn Chen" wrote:
With docker 1.5 we can go back to using the exitCode instead of "ok" string. OOM kills will also have special exitCodes in docker 1.5. , Tim Hockin wrote:
ok, I'm going to change the docs for now, we can change back when we standardize on 1.5...
Code change:
func rcByNameContainer(name string, replicas int, image string, labels map[string]string, c api.Container) *api.ReplicationController { // Add "name": name to the labels, overwriting if it exists. labels["name"] = name <ins> gracePeriod := int64(0) </ins> return &api.ReplicationController{ TypeMeta: unversioned.TypeMeta{ Kind:       "ReplicationController",

Related discussion:
text: Nodes Network when a minion node becomes unreachable [replication controller] recreates pods scheduled on the unreachable minion node AND allows scheduling of pods on a minion after it rejoins the cluster(Failed 17 times in the last 30 runs. Stability: 43 %)Failed with 0 retries
for triage
I'll look into this.
I talked to about this and he had a pretty good guess in about two seconds about what the problem is. ReplicationController sync method does this: but DaemonSet controller doesn't. (I guess when DaemonSet was copy-pasted from ReplicationController, this was accidentally removed.) Pod reflector and syncer are running in separate goroutines, and if pod reflector doesn't run then syncer will think there are no pods on the node.
To be clear: this is about the test Controller Manager should not create/delete replicas across restart which is the one that is actually failing.
I'm confused you're saying that this is about , but you're updating . I thought they are about two different tests...
Yes sorry, I put my comment in the wrong issue. Reassigning this one to you, and I'll copy my comment into the other one.
First failure: We check if all namespace are deleted. If other tests failed to delete their namespace this test will fail. I'm going to delete this logic as we already check if we have successfully deleted our namespace. See also for some more details.
Second failure: We disable network by running the following command: I checked and it seems that it results in ing master to hang. I suspect that Kubelet hangs on updating node status and depending when it breaks the connection (by timeout) it will fit in test timeouts or not.
What is the default timeout for connections using client libraries to apiserver? I'm checking the code, but can't find it.
I'm not sure if we set one, but we should. I guess I would expect to see that in the config, w/ default set by this function: , Filip Grzadkowski wrote:
So: let's add a timeout, and revert .
I think that adding a timeout and reverting are orthogonal. I we want to simulate network problems with node-master communication then using REJECT is better anyway. At the same time adding timeout makes sense for many other reasons.
Code change:
func rcByNameContainer(name string, replicas int, image string, labels map[string]string, c api.Container) *api.ReplicationController { // Add "name": name to the labels, overwriting if it exists. labels["name"] = name <ins> gracePeriod := int64(0) </ins> return &api.ReplicationController{ TypeMeta: unversioned.TypeMeta{ Kind:       "ReplicationController",

Related discussion:
text: OK then, let's add a timeout and test both DROP and REJECT-- no sort of network problem should hang a component.
I think it didn't hang. It just that the kubelet was retrying too slow and sometimes it fit test timeout, sometime it didn't.
Could we reduce the timeout to make it always fit the test timeout? , Filip Grzadkowski
I guess we can. I'll try tomorrow.
It seems we have a new type of failure: It seems that kubelet on a machine where network is blocked doesn't notice new pod. After reading logs I assume that kubelet doesn't notice that the watch is broken and doesn't try to relist everything. It notices new pod only after watch timeout.
Ugh, I hope we don't have to start sending watch heartbeats.
It seems that all recent failures are with the following error: which means the reason is here: - can you please take a look or delegate?
I did some more debugging of it. I was looking into this failure: There is something strange happening here: run performTemporaryNetworkFailure: succesfully block the load from kubelet: the succesfully wait for node to become not ready: some time nodecontroller successfully removes the pod: replication controller creates a new pod: here is something I don't undertand - because of some reason, the test doesn't observe that the pod actually disappeared. - any thoughts on it?
BTW - this is exactly the same situation in:
OK - I think this is related to default 30s deletion grace period.
Yes - it seems that this is exactly the case. I will send out PR fixing this out for review.
I'm reopening it for tracking. If there won't be any failures within next few days - I will move the test back from flaky and close this issue.
Thanks for fixing, .
It seems that the main issue is fixed. However, there was one failure in the last 30 runs: in which one pod simply didn't started (and it was "preparation" for the test, not the test itself).
During next 2 days there was exactly one more failure of this test: it is exactly the same as the previous one. However, those are duplicates of So I'm going to move this test out of flaky suite.