Change:
export PROJECT="k8s-jkns-e2e-gke-prod" export CLOUDSDK_BUCKET="gs://cloud-sdk-testing/rc" export JENKINS_USE_SERVER_VERSION="y" <ins> export CLOUDSDK_API_ENDPOINT_OVERRIDES_CONTAINER="https://container.googleapis.com/" </ins> export ZONE="asia-east1-b" - 'gke-prod-parallel': description: 'Run E2E tests on GKE prod endpoint in parallel.'

Discussion:
With recent migration of Kubelet to structured logging introduced e2e tests that enable JSON in Kubelet. Organized migration of other components haven't yet started and is on best effort bases, yet some line were already migrated. As code-base will be slowly migrated, we need to expand the test to avoid errors like I propose to expand existing e2e test suite ( and ) to cover all control plane components (apiserver, controller-manager and scheduler). /cc
/sig instrumentation
/help
If all components have this flag do we just want to enable it on all components? We already increase the log level on all components in CI, it should be pretty straightforward to expand this to all: pick a new environment variable name for the script with a more appropriate name for all components instead of just kubelet set it on the job in addition to the old kubelet one, by sending a PR to test-infra send a PR to the kind script to set the value in all places where we set verbosity instead of just kubelet, switching to the new variable name
/triage accepted
/area logging
So currently the e2e tests do not actually read and test what the logs return the current job just makes sure that kubelet is running Would we want to write similar tests for the different control planes while we start migration or do you have something else in mind with respect to the automated testing ?
Yes, currently we just test if Kubelet is running to test that flag is working and if enabling json doesn't cause panics. This exact issue proposes to expand this to other control plane, but your idea would be a great follow up. Would be great to propose a test plan that will specify what we want to test and how. If you have ideas some ideas could you collect hem in some issue so we can discuss.
There's no portable way to get these logs. There are non portable ways but those cannot be used in a standard e2e test.
Oh so for the standard e2e tests it might be good enough to just test the startups of these control planes with the logging enabled.
I was thinking that we could get started with maybe writing similar to one for the remaining components and then maybe find a way of checking the logs generated as we have bigger discussions a) kube-apiserver --Completed b) etcd --- TODO c) kube-scheduler ---TODO d) kube-controller-manager --TODO
sound good.

FYI Etcd is not a kubernetes component as so it's out of scope of this issue.
/wg structured-logging As getting core components logs is not portable, we could test a container with same logging library and using Kubenetes logs API to access logs. Created to track it
it still might not be a terrible idea to write a simple test script to probe the exported logs from a cluster with json logging enabled for any non-json lines once we know that everything should be. I think this could probably be hacked up with jq or something, but I wouldn't position it like a normal e2e test.
Could you give a hint how to implement and run such script?
/priority important-soon This issue is based on feedback from Kubelet migration retrospection, it is required before organizing migration of next component. /cc
Would this follow the same flow as in where we check if json logging is enabled and then run jq separately or within the same flow. To be more specific: Seems to be getting the data and validating it so in that case would we be running after grabbing the logs ?
For now let's focus on the original tasks of having ( and ) test scenarios, also configure other components. We just need to update Kind script to also pass to other components. Example:
Would like to take this one! /cc
Great, let me know if you need any help? /remove-help
just created a PR:
Components to be picked: [x] kube-scheduler [ ] kube-apiserver [ ] controller-manager
Attempted to cover the remaining components in this
It would be great if we had e2e setup for all components before we start structured migration next week. will you be able to work on it this week? Please let me know if there is anything I can help with.
Thanks for help with control plane components, for 1.23 we are planning to migrate kube-proxy to structured logging. Could you also try to enable JSON logging on kube-proxy?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Confirmed that in last CI run Logs from kubelet and control plane components are written in JSON: Great Job Could you take a look into changing log format of kube-proxy?