Change:
Since 8th JUL, the test cases don't show up on testgrid: It's not clear from testgrid, which runs pass and which failed atm 2022-07-08 Since 8th JUL, the build-log and junit artifacts looks different. Before, the junit file was split into parts: and it contains only a data (name, time, skipped*, ...), E.g: From , there's a single file, which is and contains all the and informations, which is a duplication of build logs. E.g.: Looking at the difference in kubernetes/kubernetes source code - there were number of changes to: ginkgo, klog and traces - all could be related: cc - I see you've made number of commits; would you be able to help? No response No response
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/milestone 1.25
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.25 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.25
I think that this is related to the ginkgov2 migration, the reporters in ginkgo has changed. Do you have a link to the code where the scale test run the e2e commands?
The test is defined here: We use kubekins. My guess is that the old "junit shards" we see were generated by different ginkgo processes/threads (number of junit files matches set in our test) and mentions that we the old parameter "--parallel" no longer works. Probably we need to migrate kubekins to use instead of "--parallel".
hmm, that seems a kubetest flag ,
I just checked kubetest code and it looks like the kubetest's --ginkgo-parallel flag gets propagated to --nodes ginkgo flag and it seems to work fine: The most recent test () has in build log.
it seems is using --nodes for parallel I think that now it aggregates the reports, also running in parallel
besides that, all the other jobs are reporting a single junit xml file, is this the real problem? do you have more ideas on why this file is not being interpreted by testgrid?
My guess is that 600MiB of junit exceeds some size limit in used in testgrid and for that reason it isn't interpreted by testgrid.
re: I think this envvar needs to be changed from: to Ref:
that doesn't look related, AFAIK the migration is about an API and that is a kubetest variable, we are passing the flag correctly. I think that the junit generation is a better lead, in k/k there is tooling to strip the file to avoid issues with large sizes it seems kubetest dumps it entirely someone is up to adapt kubetest to either split the xml generated or prune the large messages in k/k?
sorry, I just notice this, I am trying to answer some questions here if possible. as has said, Ginkgo V2 merged all results into a single junit file, the file is named as "", pls see:
it's only for golang's parallel flag, we shouldn't use that as it's not accepted by Ginkgov2, here is an example, is fine.
I don't think so, pls see:
+1 on the pruning and adapt with new junit file.
In deck's logs I see: So at least for deck the reason for not showing junit is too large file. I think testgrid implements similar logic probably.
is there any way to keep a simplified junit report instead of dump everything in that file? as what Ginkgo V1 did?
Deck is using limit of 200M:
the solution seems to adapt kubetest I can try to take a shot after code freeze
This is really something that we shouldn't have to do in kubetest :/ (consider this also impacts tools like sonobuoy), and we're generally not accepting more complexity in that tool
Attached v1 junit report and v2 junit report as a reference, v2 has embedded into each testcases even the testcase is skipped, this make the file for v2 is larger than v1. v1, v2 Just run one testcases, and v1 has 21145L, and v2 has 28258L
I don't know what's the below line bringing us anything when the testcase is skipped. is that by design?
might be easiest if we trim / split / ... in a custom reporter or on exit in , otherwise we have to consider kubetest, kubetest2, sonobuoy, etc.
how can we inject something like this then?
hey all, when running in parallel ginkgo v2 now merges the individual reports generated by each parallel process into one composite process. there is no way to turn this off (v1's behavior was a shortcut at the time and somewhat ugly and confusing). v2 also merges junit reports from multiple suites into a single file. this can be turned off with in v2 i also took a closer look at what few official-seeming junit specs exist and updated the reporter to match. currently there is no mechanism to control that default behavior. you can, however, build a custom reporter to do some filtering/processing in-suite and for a project of k8s scope that might make the most sense. for example - a reporter that modifies/filters the report object before manually calling manually. I'd be happy to share more detail/help if that approach is preferred.
How can we trigger this CI: ci-kubernetes-e2e-gce-scale-correctness?
I'm not sure if you can easily, as the project is constantly used by correctness and performance tests. Or at least I don't know how to run it - I've just wrote down in this issue to what I've observed.
FYI this is CI job that is being run every day: It requires 5k nodes cluster to run and we have 1 gcp project shared with other scalability tests that can handle it so there is no easy way to manually trigger it.
I have tried to split, but split the 600+ M xml file would be time consuming. I am not sure whether that job rely on the , if so, we might consider to add a custom reporter for that. For what I observed for each jobs triggered regularly for a PR, the size of report is less than 50M. The gap between 50 and 600 are huge, poor background for me to get more info on that job. Talking about the custom reporter, shall the testgrid or Spyglass smart enough to fetch the result from two different junit xml file? one is trimmed and the other one is original.
/cc for thoughts.
I think we should try to reduce the amount of information that gets stored in the single JUnit file. We also should ensure that we only store a single output stream (). That addresses the main problem ("too much information"). Splitting into different files feels like a poor workaround because eventually they need to be merged again in memory when dealing with the entire job run. I don't think we need the original, more complete JUnit file.
I am afraid someone need the original one, since the original one holds the single of truth if anyone wants to check the details on the test execution. +1 for couple of reasons, said that the split by the the number of parallelism is just a workaround at that time, v2 sound like did the right things, but this doesn't work for testgrid, . split is time consuming and not trivial to implement.
Here is the comparison after this pr: original one: - trimmed one: - more than 90% off. similarly, we will eventually get around 60 M for the job ?
Spyglass seems work, here is the , and is not impacted when the test failed. while the testcase passed or skipped, Spyglass doesn't provide the link to the or , so the Spyglass should be fine. Spyglass reads the trimmed xml file btw, not sure how it selects the file though.
checked all testcase status should be back. , the size now is about 2.5M. But one thing should be noted is that, name of the testcases has been changed a little bit which is caused by the new format junit xml. Here is one example, In V1, the testcase named as: "Kubernetes e2e suite.[sig-storage] Volumes NFSv4 should be mountable for NFSv4" now has been updated to: "Kubernetes e2e suite.[It] [sig-storage] Volumes NFSv4 should be mountable for NFSv4" pls note the string of "[It]" that is . So, you can only check the status of "Kubernetes e2e suite.[It] [sig-storage] Volumes NFSv4 should be mountable for NFSv4" now. We can somehow modify the report again to trim the string "[It]", this should be easy, but I am not quite sure whether we need to make everything the same as before. If you are okay with the new test name, I think we can close this issue, otherwise we can update the name as well. thoughts?
I personally think it's okay, k8s just deliver this file while how the downstream tools consume those data is not predictable. If the "IT" string to the testcase is not necessary then it should be a bug from Ginkgo, and should be fixed in Ginkgo, in this case, we can filter out the string as a temporary workaround.
Another diff I found, All skipped testcase won't shown in testgrid, e.g. "[sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node" the new format is <testcase name="[It] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node" classname="Kubernetes e2e suite" status="skipped" time="0."<skipped message="skipped - Driver nfs doesn't support Block -- skipping"</skipped</testcasepls note the attribute of "status="skipped" , it was, <testcase name="[sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node" classname="Kubernetes e2e suite" time="0"<skipped</skipped</testcaseCan testgrid or some tool grep the "status" to get the result instead? do you know someone familiar with the tool can help to update the tool a little bit?
I see , it seems the new reporting format breaks testgrid, can you help us to recover the functionality in testgrid?
the release 1.25 bug triage Lead. I'm reaching out to see the status of this issue and if we are still targeting the current release. Also, is it a release blocker? Can you please update the priority as well?
FWIW I think this is implemented here (junit in GCS =testgrid)
I don't think this is a release blocker, the change needed in Kubernetes has been merged, meanwhile, I pushed a pr in testgrid to close this
Thanks for replying back, I'll move it to 1.26 then
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.26 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.26
/close this issue has been resolved, I ca see that the test cases are showing up now, since the different fixes to the e2e framework. The remaining part is to show the skip test cases in testgrid, but that is another issue, better not to add more things here
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
cd "${KUBE_TARGET}" <del> if [ "$1" != "" ]; then go test -race $KUBE_TIMEOUT $KUBE_COVER -coverprofile=tmp.out "$KUBE_GO_PACKAGE/$1" "${@:2}" </del> <ins> while getopts "i:" opt ; do case $opt in i) iterations=$OPTARG ;; ?) echo "Invalid argument -$OPTARG" exit 1 ;; :) echo "Option -$OPTARG <value>" exit 1 ;; esac done shift $((OPTIND - 1)) if [[ -n "${iterations}" ]]; then echo "Running ${iterations} times" if [[ -n "$1" ]]; then pkg=$KUBE_GO_PACKAGE/$1 fi rm -f *.test # build a test binary echo "${pkg}" go test -c -race ${KUBE_TIMEOUT} "${pkg}" # keep going, even if there are failures pass=0 count=0 for i in $(seq 1 ${iterations}); do for test_binary in *.test; do if "./${test_binary}"; then ((pass++)) fi ((count++)) done done 2>&1 echo "${pass}" / "${count}" passing if [[ ${pass} != ${count} ]]; then exit 1 else exit 0 fi fi if [[ -n "$1" ]]; then go test -race ${KUBE_TIMEOUT} ${KUBE_COVER} -coverprofile=tmp.out "${KUBE_GO_PACKAGE}/$1" "${@:2}" </ins> exit 0 fi <del> find_test_dirs | xargs go test -race -timeout 30s $KUBE_COVER "${@:2}" </del> <ins> find_test_dirs | xargs go test -race -timeout 30s ${KUBE_COVER} "${@:2}" </ins>