Change:
<match reform.**> type record_reformer enable_ruby true <del> tag kubernetes.${tag_suffix[3].split('-')[0..-2].join('-')} </del> <ins> tag kubernetes.${tag_suffix[4].split('-')[0..-2].join('-')} </ins> </match> <source> type tail format none <del> path /varlog/salt/minion pos_file /varlog/gcp-salt.pos </del> <ins> path /var/log/salt/minion pos_file /var/log/gcp-salt.pos </ins> tag salt </source> <source> type tail format none <del> path /varlog/startupscript.log pos_file /varlog/gcp-startupscript.log.pos </del> <ins> path /var/log/startupscript.log pos_file /var/log/gcp-startupscript.log.pos </ins> tag startupscript </source> <source> type tail format none <del> path /varlog/docker.log pos_file /varlog/gcp-docker.log.pos </del> <ins> path /var/log/docker.log pos_file /var/log/gcp-docker.log.pos </ins> tag docker </source> <source> type tail format none <del> path /varlog/etcd.log pos_file /varlog/gcp-etcd.log.pos </del> <ins> path /var/log/etcd.log pos_file /var/log/gcp-etcd.log.pos </ins> tag etcd </source> <source> type tail format none <del> path /varlog/kubelet.log pos_file /varlog/gcp-kubelet.log.pos </del> <ins> path /var/log/kubelet.log pos_file /var/log/gcp-kubelet.log.pos </ins> tag kubelet </source> <source> type tail format none <del> path /varlog/kube-apiserver.log pos_file /varlog/gcp-kube-apiserver.log.pos </del> <ins> path /var/log/kube-apiserver.log pos_file /var/log/gcp-kube-apiserver.log.pos </ins> tag kube-apiserver </source> <source> type tail format none <del> path /varlog/kube-controller-manager.log pos_file /varlog/gcp-kube-controller-manager.log.pos </del> <ins> path /var/log/kube-controller-manager.log pos_file /var/log/gcp-kube-controller-manager.log.pos </ins> tag kube-controller-manager </source> <source> type tail format none <del> path /varlog/kube-scheduler.log pos_file /varlog/gcp-kube-scheduler.log.pos </del> <ins> path /var/log/kube-scheduler.log pos_file /var/log/gcp-kube-scheduler.log.pos </ins> tag kube-scheduler </source> <del> <match **> </del> <ins> # We use 2 output stanzas - one to handle the container logs and one to handle # the node daemon logs, the latter of which explicitly sends its logs to the # compute.googleapis.com service rather than container.googleapis.com to keep # them separate since most users don't care about the node logs. <match kubernetes.**> </ins> type google_cloud # Set the chunk limit conservatively to avoid exceeding the GCL limit # of 2MB per write request. buffer_chunk_limit 512K <del> # Cap buffer memory usage to 512KB/chunk * 128 chunks = 65 MB buffer_queue_limit 128 </del> <ins> # Cap the combined memory usage of this buffer and the one below to # 512KB/chunk * (96 + 32) chunks = 65 MB buffer_queue_limit 96 # Never wait more than 5 seconds before flushing logs in the non-error case. </ins> flush_interval 5s # Never wait longer than 5 minutes between retries. max_retry_wait 300 # Disable the limit on the number of retries (retry forever). disable_retry_limit </match> <ins> # Keep a smaller buffer here since these logs are less important than the user's # container logs. <match **> type google_cloud detect_subservice false buffer_chunk_limit 512K buffer_queue_limit 32 flush_interval 5s max_retry_wait 300 disable_retry_limit </match> </ins>

Discussion:
When I run Kubernetes cluster on GKE, it generates a lot of system logs coming from docker and kubelet. Even if my cluster runs just a few pods, I can see dozens or hundreds of log entries per second available in Google Cloud Logging in docker and kubelet logs. This makes the actual logs from my application get lost among them, if I do not set appropriate filter. Also, I am afraid that collecting massive amounts of logs imposes unnecessary workload on fluentd pod in my cluster. It would be good if I could just disable collection of kubelet/docker logs (and potentially other Kubernetes system logs) assuming that I do not want to have these logs exported.
/cc
since you tend to have a vested interest in node debug-ability, what's the minimum you'd want to have available here? From GKE's perspective, our users typically have no interest in these logs, so turning them off by default, even without an easy option to re-enable them, makes sense. Any user sophisticated enough to want them shouldn't have problems SSH'ing to the node for them. Is the story all that different in open-source kubernetes? Does anyone even care to have the kubelet and docker logs collected from the nodes? I've never used cloud logging to debug issues with them, it's always been easier to just SSH to the node. I'd lean towards not collecting them on the nodes in general. Also cc in case he has wisdom he'd like to impart
All I can say is that personally I have never accessed kubelet etc. logs "for real" when debugging -- as Alex says -- it is usually more direct just to SSH in. I would support not collecting these logs to improve cluster stability (but perhaps have a switch / technique to make it easy to collect them if the admin does care about this?).
One last call for to complain, because I'm starting on this now
Also
We want to avoid storing logs onto local disk in the future. In-order to achieve that, we need the existing cluster level logging infrastructure to work. Can the GKE problem be solved by moving system logs to a separate namespace? , Alex Robinson wrote:
How far in the future? Unless that's intended to happen soon, collecting the kubelet and docker logs in the meantime offers little value with a nontrivial resource (cpu+memory+egress) cost.

But yes, we can handle the system logs separately if we'd like. I think the most natural way would be to report them as compute engine logs, which could be done by adding an extra config option into the gcp output plugin.
, Alex Robinson wrote:
-- just to confirm, do you think this fix will be sufficient for you? In GKE, logs from containers will be listed under Container Engine, while logs from Docker and the Kubelet will be listed under Compute Engine, with the logs for each effectively being entirely seperated in the log viewer.
Yes, this will be sufficient in terms of not polluting Container Engine logs with system logs. I am still concerned about unnecessary load on fluentd caused by exporting Docker and Kubelet logs so ideally it would be great not to export these system logs if a user is not interested in them. But this is of slightly lower priority.
Hi, What's the status from this request? Is this available in 1.7.x or higher? Thanks.