Change:
When I run Kubernetes cluster on GKE, it generates a lot of system logs coming from docker and kubelet. Even if my cluster runs just a few pods, I can see dozens or hundreds of log entries per second available in Google Cloud Logging in docker and kubelet logs. This makes the actual logs from my application get lost among them, if I do not set appropriate filter. Also, I am afraid that collecting massive amounts of logs imposes unnecessary workload on fluentd pod in my cluster. It would be good if I could just disable collection of kubelet/docker logs (and potentially other Kubernetes system logs) assuming that I do not want to have these logs exported.
/cc
since you tend to have a vested interest in node debug-ability, what's the minimum you'd want to have available here? From GKE's perspective, our users typically have no interest in these logs, so turning them off by default, even without an easy option to re-enable them, makes sense. Any user sophisticated enough to want them shouldn't have problems SSH'ing to the node for them. Is the story all that different in open-source kubernetes? Does anyone even care to have the kubelet and docker logs collected from the nodes? I've never used cloud logging to debug issues with them, it's always been easier to just SSH to the node. I'd lean towards not collecting them on the nodes in general. Also cc in case he has wisdom he'd like to impart
All I can say is that personally I have never accessed kubelet etc. logs "for real" when debugging -- as Alex says -- it is usually more direct just to SSH in. I would support not collecting these logs to improve cluster stability (but perhaps have a switch / technique to make it easy to collect them if the admin does care about this?).
One last call for to complain, because I'm starting on this now
Also
We want to avoid storing logs onto local disk in the future. In-order to achieve that, we need the existing cluster level logging infrastructure to work. Can the GKE problem be solved by moving system logs to a separate namespace? , Alex Robinson wrote:
How far in the future? Unless that's intended to happen soon, collecting the kubelet and docker logs in the meantime offers little value with a nontrivial resource (cpu+memory+egress) cost. But yes, we can handle the system logs separately if we'd like. I think the most natural way would be to report them as compute engine logs, which could be done by adding an extra config option into the gcp output plugin.
, Alex Robinson wrote:
-- just to confirm, do you think this fix will be sufficient for you? In GKE, logs from containers will be listed under Container Engine, while logs from Docker and the Kubelet will be listed under Compute Engine, with the logs for each effectively being entirely seperated in the log viewer.
Yes, this will be sufficient in terms of not polluting Container Engine logs with system logs. I am still concerned about unnecessary load on fluentd caused by exporting Docker and Kubelet logs so ideally it would be great not to export these system logs if a user is not interested in them. But this is of slightly lower priority.
Hi, What's the status from this request? Is this available in 1.7.x or higher? Thanks.

Discussion:
# Disable prompts from apt. ENV DEBIAN_FRONTEND noninteractive <ins> # Keeps unneeded configs from being installed along with fluentd. ENV DO_NOT_INSTALL_CATCH_ALL_CONFIG true </ins> RUN apt-get -q update &&  apt-get install -y curl && 

.PHONY:	kbuild kpush <del> TAG = 1.12 </del> <ins> TAG = 1.13 </ins> # Rules for building the test image for deployment to Dockerhub with user kubernetes.

# This configuration file for Fluentd / td-agent is used # to watch changes to Docker log files that live in the # directory /var/lib/docker/containers/ and are symbolically <del> # linked to from the /varlog directory using names that capture the </del> <ins> # linked to from the /var/log directory using names that capture the </ins> # pod name and container name. These logs are then submitted to # Google Cloud Logging which assumes the installation of the cloud-logging plug-in. #

#    synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log #    -> #    /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log <del> # The /var/log directory on the host is mapped to the /varlog directory in the container </del> <ins> # The /var/log directory on the host is mapped to the /var/log directory in the container </ins> # running this instance of Fluentd and we end up collecting the file: <del> #   /varlog/containers/synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log </del> <ins> #   /var/log/containers/synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log </ins> # This results in the tag: <del> #  varlog.containers.synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log # The record reformer is used is discard the varlog.containers prefix and </del> <ins> #  var.log.containers.synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log # The record reformer is used is discard the var.log.containers prefix and </ins> # the Docker container ID suffix and "kubernetes." is pre-pended giving the # final tag which is ingested into Elasticsearch: #   kubernetes.synthetic-logger-0.25lps-pod_default-synth-lgr # This makes it easier for users to search for logs by pod name or by # the name of the Kubernetes container regardless of how many times the # Kubernetes pod has been restarted (resulting in a several Docker container IDs). <del> # TODO: Propagate the labels associated with a container along with its logs # so users can query logs using labels as well as or instead of the pod name # and container name. </del> <source> type tail format json time_key time <del> path /varlog/containers/*.log pos_file /varlog/gcp-containers.log.pos </del> <ins> path /var/log/containers/*.log pos_file /var/log/gcp-containers.log.pos </ins> time_format %Y-%m-%dT%H:%M:%S tag reform.* read_from_head true

<match reform.**> type record_reformer enable_ruby true <del> tag kubernetes.${tag_suffix[3].split('-')[0..-2].join('-')} </del> <ins> tag kubernetes.${tag_suffix[4].split('-')[0..-2].join('-')} </ins> </match> <source> type tail format none <del> path /varlog/salt/minion pos_file /varlog/gcp-salt.pos </del> <ins> path /var/log/salt/minion pos_file /var/log/gcp-salt.pos </ins> tag salt </source> <source> type tail format none <del> path /varlog/startupscript.log pos_file /varlog/gcp-startupscript.log.pos </del> <ins> path /var/log/startupscript.log pos_file /var/log/gcp-startupscript.log.pos </ins> tag startupscript </source> <source> type tail format none <del> path /varlog/docker.log pos_file /varlog/gcp-docker.log.pos </del> <ins> path /var/log/docker.log pos_file /var/log/gcp-docker.log.pos </ins> tag docker </source> <source> type tail format none <del> path /varlog/etcd.log pos_file /varlog/gcp-etcd.log.pos </del> <ins> path /var/log/etcd.log pos_file /var/log/gcp-etcd.log.pos </ins> tag etcd </source> <source> type tail format none <del> path /varlog/kubelet.log pos_file /varlog/gcp-kubelet.log.pos </del> <ins> path /var/log/kubelet.log pos_file /var/log/gcp-kubelet.log.pos </ins> tag kubelet </source> <source> type tail format none <del> path /varlog/kube-apiserver.log pos_file /varlog/gcp-kube-apiserver.log.pos </del> <ins> path /var/log/kube-apiserver.log pos_file /var/log/gcp-kube-apiserver.log.pos </ins> tag kube-apiserver </source> <source> type tail format none <del> path /varlog/kube-controller-manager.log pos_file /varlog/gcp-kube-controller-manager.log.pos </del> <ins> path /var/log/kube-controller-manager.log pos_file /var/log/gcp-kube-controller-manager.log.pos </ins> tag kube-controller-manager </source> <source> type tail format none <del> path /varlog/kube-scheduler.log pos_file /varlog/gcp-kube-scheduler.log.pos </del> <ins> path /var/log/kube-scheduler.log pos_file /var/log/gcp-kube-scheduler.log.pos </ins> tag kube-scheduler </source> <del> <match **> </del> <ins> # We use 2 output stanzas - one to handle the container logs and one to handle # the node daemon logs, the latter of which explicitly sends its logs to the # compute.googleapis.com service rather than container.googleapis.com to keep # them separate since most users don't care about the node logs. <match kubernetes.**> </ins> type google_cloud # Set the chunk limit conservatively to avoid exceeding the GCL limit # of 2MB per write request. buffer_chunk_limit 512K <del> # Cap buffer memory usage to 512KB/chunk * 128 chunks = 65 MB buffer_queue_limit 128 </del> <ins> # Cap the combined memory usage of this buffer and the one below to # 512KB/chunk * (96 + 32) chunks = 65 MB buffer_queue_limit 96 # Never wait more than 5 seconds before flushing logs in the non-error case. </ins> flush_interval 5s # Never wait longer than 5 minutes between retries. max_retry_wait 300 # Disable the limit on the number of retries (retry forever). disable_retry_limit </match> <ins> # Keep a smaller buffer here since these logs are less important than the user's # container logs. <match **> type google_cloud detect_subservice false buffer_chunk_limit 512K buffer_queue_limit 32 flush_interval 5s max_retry_wait 300 disable_retry_limit </match> </ins>

value: -q volumeMounts: - name: varlog <del> mountPath: /varlog - name: containers </del> <ins> mountPath: /var/log - name: varlibdockercontainers </ins> mountPath: /var/lib/docker/containers terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log <del> - name: containers </del> <ins> - name: varlibdockercontainers </ins> hostPath: path: /var/lib/docker/containers

spec: containers: - name: fluentd-cloud-logging <del> image: gcr.io/google_containers/fluentd-gcp:1.12 </del> <ins> image: gcr.io/google_containers/fluentd-gcp:1.13 </ins> resources: limits: cpu: 100m

value: -q volumeMounts: - name: varlog <del> mountPath: /varlog - name: containers </del> <ins> mountPath: /var/log - name: varlibdockercontainers </ins> mountPath: /var/lib/docker/containers terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log <del> - name: containers </del> <ins> - name: varlibdockercontainers </ins> hostPath: path: /var/lib/docker/containers ```