Code change:
<ins> # Rollback workflow Build it in this directory. Make sure you have etcd dependency ready. Last time we use etcd v3.0.7. ``` $ go build . ``` Run it: ``` $ ./rollback2 --data-dir $ETCD_DATA_DIR --ttl 1h ``` This will rollback KV pairs from v3 into v2. If a key was attached to a lease before, it will be created with given TTL (default to 1h). On success, it will print at the end: ``` Finished successfully ``` Repeat this on all etcd members. You can do simple check on keys (if any exists): ``` etcdctl ls / ``` Important Note ------ This tool isn't recommended to use if any problem comes up in etcd3 backend. Please report bugs and we will fix it soon. If it's still preferred to run this tool, please backup all your data beforehand. This tool will also back up datadir to same path with ".rollback.backup" suffix. Caveats: - The tool doesn't preserve versions of keys. - If any v2 data exists before rollback, they will be wiped out. - v3 data only exists in the backup after successful rollback. [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/cluster/images/etcd/rollback/README.md?pixel)]() </ins>  No newline at end of file

Related discussion:
text: ci-kubernetes-e2e-gce-device-plugin-gpu.Overall ci-kubernetes-e2e-gce-device-plugin-gpu.Pod kubetest.Up Intermittently since 02/23 No response No response /sig testing /sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /sig node BTW, would you help to confirm if this is a release cut blocker? We plan to release 1.30.0-alpha.3 on Tuesday 27th February 2024.
It passed! Maybe it was some infra issues?
Failed again. /sig k8s-infra
Many recreating of nodes in log.
This is caused by infra flaky.
The problem is
This job only runs one test and GPU instances are hard to find we don’t need many of those instances for this job, can we please change this job to only use 2 workers?
/cc
tests failed in a recent run. It failed the first time. The Job failure now changed to test case failing today. It failed 4 times in a row.
/assign
I launched a test run, logged into the master node and took the kubeconfig and found the following: Relevant bit
Link some related updates recently on this job.
The error above in is weird to me. It said No firmware image found . In successful run before, like (02-27), there are no such failure logs. failure in only kubelet log shows the pod crashed and restarted for many times. No pod log or containerd log.
After , the CI is not always failing now and 4 in a row.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen as keeps flaking
Reopened this issue. <detailsIn response to : /reopen as keeps flaking Instructions for interacting with me using PR comments are available .