jiebi/CodeConvo huggingface dataset:
Locally image version 1.1 does produce the cuda/GPU error shown above by :
More info is needed to find the RC for this test failure. I've put in a PR to get the test framework to preserve the container logs when the pod fails.
The pod logs show that it is timing out on an http connection, and it looks like it's the one where it tries to download the test data set for the tensorflow run. You can see it . Search down to . The data_download script in the image attempts to pull down a dataset from: . That url is live and healthy, so they might be blocking the gcloud.
This test depending on that external data doesn't look quite right :/
I'm discussing this test with sig-testing. I'll post an update later.
Talked to from sig-testing: he's contacting the remote to see if they are blocking us, because it looks like they are is pulling the script from a that is So I see three choices: upgrade this thing to a more recent edition delete it grab everything that's needed to run this and keep it in-house <- clear winner here what do you think: Maybe the folks over at sig-testing have a place to store the artifacts in-house? They are not massive (about 10MB see below) Size of data about 10MB: the tools and libs for the tensorflow run: the training and eval files for the testrun:
Here's a relevant issue in a related k8s repo about using/not using docker registries: Registries used in k/k CI should be on Kubernetes Community infra
looks good for the subject of putting up a new docker image containing the deps. This seems feasible and effective. I'd prefer it to deleting the test. I'll discuss it a bit more with a few others, but this seems like the way to go forward with this. TBS, I plan to take the above listed test requirements, and build a docker image containing all that is needed, and reset the test to pull the new docker image.
I've got some Dockerfile changes to fix this. I've tested it locally with no network and it does the full 300 epochs. PR coming soon.
Is it worth pushing up 182 files to get this green? It is about 6M of python, and 5M of a test dataset.
Its all stashed away in the to get the docker image to contain the prereqs. It is selected subdirectories of the quite old archived version (v1.9.0) of the code from here: current version v2.7.2 and two small test data sets from here:
OK I got it down to 43 files, including the 2 datasets. That's the drivers and the datasets and the libs they need. It all gets packed into the docker image.
quick question - is there another repository that is under sig-node that can host this image?
To be clear, the test's docker image is currently in a repo, the : This PR adds artifacts to an existing docker image in the above repo.
I've cancelled the above PR. I'll try to get something else going.
There has been a decent amount of discussion in slack relating to handling the test's required artifacts. The solution that seems best is to get some practical version of the test requirements into locally owned and controlled storage, so the endpoints that provide access to the assets will at least be under internal control. This is not code, and is not under development. It is a static dataset, that is now stored in an offsite archive, and will not require maintenance. The datasets are published with the "": (see bottom right) "This allows for the sharing and adaptation of the datasets for any purpose, provided that the appropriate credit is given." Here are some links to the dataset and where its hosted: https://archive-
I'd like to get the dataset that holds the test data (about 650k) hosted in GCS where I can get it via http. With that change it is expected that the test will go green. The failures show it is failing when it tries to download the artifact from the uncontrolled remote endpoint. Somebody, please help me get this artifact into an internally controlled server. Thanks
Was the idea of downloading this only on image creation in dockerfile rejected?
Thanks for the reminder I did not fairly evaluate that proposal. I just jumped to my earlier proposal. Thinking about this proposal I see that it would mean downloading the test data for inclusion into the docker image, while the image's test driver tooling is built to pull down the dataset itself from a remote.
That is a simple change to handle, while simply re-hosting the artifact on GCS would require only changing the endpoint. Also building the docker would probably still require network access to the artifact, which would probably require us to put it somewhere anyway. I assume, putting it into the docker image would best be done by downloading the test data artifact from somewhere. Would we then be storing the artifact twice - once in the docker image, and once in a locally controlled endpoint? If the docker build server has unrestricted access to the artifact, this proposal would be good. I don't have access to the servers to run tests. However, I could put up an RB for the Dockerfile to try it out. This is probably worth the effort. I'll get an PR going for trying to pull down the remote test-data artifact from the image build server, and bunde it into the docker image, and I'll link it to this ticket.
thank you. We do not change the image for a long time, so downloading it once would likely be enough. Once it is self-contained, it is even better for the continuous perf runs in tests.
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Currently the image builds for the first arch (amd64) but fails when it enters the cross compiling phase (arm64) and specifically when it tries to do apt installs from within the qemu-*-static emulator: The other images in node-perf fail to build in the same way. The jessie-dnsutils image build fails to build in the same way. I have tried pulling in newer base images up to buster, but it did not fix the problem. -Start focusing on the version of qemu. - qemu is version locked in the Makefile and has not changed in two years. There was an and the images were building at that time.
Try to find clues over there.
I tried pinning many relevant versions of from but none helped. Here's a clue: This ugly hack gets it past the observed error:
Note the previous PR was missing an important change, but I do not see any feasible way that could explain the current problem with failing, so I have not felt any urgency to put up a PR for it. FTR I need to bump the VERSION. I did not bump the version in I thought maybe the CI would bump it automatically.
There's an image version in . Its still pulling the old version and failing. Bump it from 1.2 to 1.3. So one more commit is needed.
The new image is present and runs perfectly. Tested it on local linux and in google cloud: It ran to completion. Note the new image URL. Its NOT the same as the old one:
podspec references the image Here's the . tf-wide-deep's url is built The above has staging in it. . Its already built and pushed to staging. It must be promoted when this new version number goes out.
FTR: My "testing" above did not catch the current problem because I did not run the full e2e_test binary. I only tested the image via a . AND I should have read way back at the beginning.
Local run of : has to change too.
Local test using the published 1.3 image run the standard way was successful with the above: ran the test successfully. So promote the image per bump the image version in the e2e test manifest change the
After the image promotion is merged, there's STILL one more commit coming: bump the image version in the e2e test manifest change the
its promoted and is found at the expected url local containerd cleared out local test was rerun pulling from the new prod registry local test using that new registry passes
its fixed. the test is green again.
What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.
Can be cleaned up with:
/sig node
the spam cycle is this loop: "pod create, pod schedule, worker reject, pod failed" -- we don't have a lot of protection in higher level workload controllers to handle pods that fail at kubelet admission time versus pods that just dont create due to quota/admission. in general, is the spam causing any other stability issues? is pod gc pruning the pods?
I wonder if we can do a trick in node lister to mitigate any race condition here...
we could maybe do something similar to to ensure node lister has synced...
hacking on option to ensure if kubelet has a node lister with a valid kube client, we wait for it to sync at least once here which should mitigate this issue. see:
We noticed the problem on the OLM pod which is managed by the OLM operator. Right or wrong, the operator does not like seeing the pod go to status and no longer reconciles the pod. The end result is that manual intervention (i.e. pod deletion) is required to fix the problem.
sure... i would have thought the pod is failed in that instance and reached a terminal state. either way, i am looking at what i can do to mitigate this.
The problem persists in our environment as well, and the workaround cited above will not do anymore. Is there something in the works for this one?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Hey Guys , I try to use the option --override with kubectl expose . First i did this and it work : Now i want to add another service with kubectl expose and this command does not work I have this error If i remove --overrides the command work . `
/sig cli
expose tries to convert version from "v1" to "_internal" which is not registered. This is the reason that the function doesn't works. == "" 316 if inline := cmdutil.GetFlagString(cmd, "overrides"); len(inline) 0 { 317 object, err = cmdutil.Merge(scheme.Codecs.LegacyCodec(scheme.Scheme.PrioritizedVersionsAllGroups()...), object, inline) 318 if err != nil { 319 return err 320 } 321 } === 156 // LegacyCodec encodes output to a given API versions, and decodes output into the internal form from 157 // any recognized source. The returned codec will always encode output to JSON. If a type is not 158 // found in the list of versions an error will be returned. 159 // 160 // This method is deprecated - clients and servers should negotiate a serializer by mime-type and 161 // invoke CodecForVersions. Callers that need only to read data should use UniversalDecoder(). 162 // 163 // TODO: make this call exist only in pkg/api, and initialize it with the set of default versions. 164 // All other callers will be forced to request a Codec directly. 165 func (f CodecFactory) LegacyCodec(version ) runtime.Codec { 166 return versioning.NewDefaultingCodecForScheme(f.scheme, f.legacySerializer, f.universal, schema.GroupVersions(version), runtime.InternalGroupVersioner) 167 }
kubectl help expose Expose a resource as a new Kubernetes service. Possible resources include (case insensitive): pod (po), service (svc), replicationcontroller (rc), deployment (deploy), replicaset (rs) In order to control the scope, I only register them as un-versioned type in the fix.
xref I think this may be one of the few cases where we can actually change the behavior in a "breaking" way. User creates service with , is empty Controller or user updates the status subresource, sets (not wiped or validated, should be) User updates in any way, is wiped It's clear that the is not meant to be set when is not "LoadBalancer". The question is: 1) Fail validation? 2) Silently discard the value? 3) Warn and then discard the value? 4) Just warn and leave it. Because we allowed it in the past, but it is so volatile, I lean towards 3. and - the people most familair with this type. Edit: PrepareForUpdate() runs before warnings. So: (1) is easy (2) is easy (4) is slightly less easy than (2) (3) is slightly less easy than (4)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
OMG change it to clusterIP as expected Status is wiped update status as ClusterIP ... and if I edit the service back to loadbalancer it keeps the status??
/kind bug
/retitle Service can be set when !=
We talked about this one at the SIG Network meeting today. suggested we use the feature gate for deprecation option to phase the current (problematic) functionality out.
/triage accept
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
A CSI plugin that the 1.10 controller manager can't detect (which could occur in a number of scenarios, such as when a rollout is being performed or a node is evacuated) seems to retry without backoff. In this case, I have a pod referencing a PVC -PV that references my CSI plugin, but no controller plugin running. The controller is probing every 100ms continuously for a single volume attachment. The controller should backoff. Have not tested with higher numbers of pods, but in a busy system this could cause the controller manager to burn CPU. Also, there isn't actually a great error telling the user why the probing is happening (I knew in this case, but an admin likely has no idea).
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /milestone v1.14 adding it to next milestone as per if it doesn't make it then it can be further moved
as KEP has been moved, unless an exception is made, /milestone v1.15
the linked kep above is unrelated to this issue. This is a bug and I don't think it needs an associated enhancement
sorry, got diverted by the comments on that KEP. /milestone v1.14
Hello from Bug Triage Shadow . I'd like to remind that code freeze is coming in 3 days (Friday, 8th March). Is this still planned to be fixed for 1.14?
We have not been tracking this for 1.14. /assign can you help triage and see if this is still an issue and if there is a quick fix?
I can't reproduce the issue, do you remember what you did? From the logs it seems that Watch channel consistently returns so new is called recursively (beware, 1.10 code, but it's very similar in master): Can it produce fairly stable 100ms delay between calls? Anyway, this is not usual code path, watcher does not return in most cases.
if I read the code correctly, such recursive must end up with timeout - caller does not care about return value and caller's watch probably won't return anything useful. Is there a reason why we don't return error? This will kick in exponential backoff in A/D controller.
clearing the milestone for tracking purposes to let the fix land in its own time /milestone clear
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I found the error would happen when a pv is attahing(volumeattachment has created) and the csi-attacher is deleted at the same time. Then the log would print so frequently (because of recursion).
is this fixed by ?
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: ! What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.20.1 Cloud provider or hardware configuration: OS (e.g: ): 7 Kernel (e.g. ): Linux zsm-test-2 3.10.0-1160.11.1.el7.x8664 SMP Fri Dec 18 16:34:56 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted /priority backlog
Hi , I followed the instructions in examples/logging-demo, and there was no "external IP" field for elasticsearch-logging or kibana-logging, as shown below: I start up my cluster via cluster/kube- Did I miss something or is it a bug? Thanks.
Correct. This demo does not create an ELB so there is no external IP address. Access is via service proxy at the master. Type: and you should see the URLs for Elasticsearch and Kibana.
please let us know whether: a. Satnam's comment resolves your issue b. the demo should be updated to prevent this misunderstanding c. the issue isn't resolved Thank you!
Yes, Satnam's comment resolve the issue. I updated the demo in .
Awesome, thanks for sending the patch! We'll discuss there. Always fantastic to have docs updated regularly by folks trying them.
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-slow [sig-storage] Dynamic Provisioning DynamicProvisioner [Slow] should provision storage with different parameters 5/13 https://k8s- v1beta1.StorageClass got removed in We should remove the v1beta1 test.
/assign can you help take a look?
/triage accepted
/sig storage
I run the test with the latest from master without removing the beta test and it failed with a different error before getting to the beta test part: Do you have any idea what would've happened in my setup? I redeployed k/k using master HEAD
For this testcase: I saw that the provisioned PD has a size of 1.5Gi according to kubernetes: And in gcloud It's really interesting that the test was expecting 2Gi to be created, is this something that was changed recently?
The logs you posted are showing pd csi as the provisioner. Were you testing with the intree driver or csi driver? The CSI driver should to the nearest Gb. I believe intree does the .
I think I did intree because I didn't send as an argument in my command: It's interesting that the PVC has this annotation too:
mmm something interesting about my setup is that I created the cluster with CSIMigration turned on Do you think that might be forcing the usage of ?
Yes likely. Try with CSIMigration off and see if that helps. If so, then this potentially seems like a rounding bug with the csi driver
Thanks I tried with and it did indeed round up to 2Gi.
In kubectl v1.8.0+, schema info is cached in the location specified by (defaulting to ), written with world-writeable permissions (rw-rw-rw-). If is specified and pointed at a different location accessible to other users/groups, the written files may be modified by other users/groups and disrupt the kubectl invocation. CVSS score: CVSS:3.0/AV:L/AC:H/PR:L/UI:R/S:U/C:L/I:L/A:N (3.3, low) What versions are affected? kubectl v1.8.0+ What configurations are affected? Invocations that point at world-writeable locations Impact Malformed responses written to the cache directory can disrupt the kubectl invocation Workaround Use the default location in the $HOME directory or point it at a directory that is only accessible to desired users/groups. (original description follows) ==== What happened: The files inside of ".kube/http-cache" are world writeable (rw-rw-rw-). While the default for these files appears to be the home directory, using the "--cache-dir" flag could put these files into a place where world writeable files would allow any user / process to modify the cache files. Modification of the cache files could influence the kubectl utility in a negative way for other users. What you expected to happen: Apply stricter file permissions to the http-cache files. How to reproduce it (as minimally and precisely as possible): Run any generic kubectl command which is successful and then list the cache directory $ kubectl get pods --all-namespaces $ ls -la Anything else we need to know?