Code change:
Follow-up of Docker 1.13 introduced a change into the logging mechanism (), that split the log lines longer than 16K into chunks of no more than that. Current default logging integrations (fluentd-gcp and fluentd-es addons) do not support that and this might result in broken ingestion, i.e. when JSON parsing is broken for entries longer than 16K. One possible solution is introducing a fluentd plugin to the configuration to concat such entries in the similar way does it (assuming there is one out there, cc for that) /cc
Presumably this should be done in the Docker fluentd logging driver... The discussion on seems to be leaning that way...
In fluentd logging driver side, journald driver approach is better: I'm not sure buffering partial message in logging driver is good or not... In fluentd side, implementing or is needed.
do we need to block 1.8 release? can we please move this out of 1.8 if it's not?
Moved to 1.9, since it's not specific for 1.8
Is there any way this can be fixed by running a custom fluentd? Or is there a fluentd docker driver github issue I can follow for updates on when they expect to have this fixed? Txh
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Problem is that there's no fluentd plugin to fix the problem, so running a custom fluentd won't help, there's nothing to add to the configuration to make it work (unless you want to write your own solution) They don't think it's a docker problem and don't plan to fix it. However, it will be fixed in GKE
That seems fair from Docker's perspective. I'd rather have stable containers as well :) So you're thinking of switching from the current log drive - at least that's what my Google Container Optimized image in GKE 1.8.1 uses - to the journald driver to circumvent the problem? And then use a plugin like to recombine the partial messages?
No, that's not the case. Solution is TBD, but if you export your logs to Stackdriver (that has a limit of 100KB per entry) you won't see the difference
I think there's still a difference in that the 100KB stackdriver logging limit is larger than the 16KB docker limit at which the partial messaging starts, so without proper recombination of those chunked messages you'll lose the ability to send log lines larger than 16KB into stackdriver. However we're also running our own fluentd that sends messages straight into pub/sub, which has a limit of 10MB per message. And although we're trying to keep the messages much smaller than 16KB it would be nice to be able to send bigger messages without instrumenting our applications to push the messages into pub/sub themselves. I hope you find a nice solution that will make this possible :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten Still relevant, hasn't been fixed
I've been reading up on this issue, it's really too bad docker would not allow the configurable log-max-size option, this would be an easy temporary workaround to the issue until fully resolved. Regardless, I also wanted to point out a 2nd issue that the log splitting causes which might help increase the priority of this issue. If you are using json logging and have MCD or Structured Arguements attributes in your log messages, what ends up happening unfortunately is the attributes are lost on the split messages. This is a huge drawback right now. In my case I am using spring cloud sleuth which adds span and traceid's to all log messages allowing you to easily filter and correlate all related logging messages. When the messages are split, the traceid's are not part of the message in elasticsearch making filtering by traceid impossible. My application logging is setup as follows: is it still the plan to have fluentd put the logs back together into a single message? can anyone update if any progress is being made? thanks for your help.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Still relevant, hasn't been fixed
Would using the "concat" fluentd plugin be relevant? It seems to have explicit support for our specific use case. Edit: For EFK, Adding fluentd-plugin-concat to fluentd-es image (, ) and adding to the Works well. Would a PR with those changes be acceptable?
trying to solve this as well. Having the same issue as where long lines are being truncated and MDC gone.
Please see The big problem here is concurrency. Just concatenating all partial log messages as they arrive may be wrong. Let's say your app logs two events, called and here. Then the Docker log messages may be written to the JSON log file (or whatever Docker logging driver is used, e.g. in plain Docker or Docker Swarm) in this order: The fluentd concat plugin cannot decide what to do. What records belong together? The good message is: This can be solved (similar to ) as Docker logs contain all the needed information here: So you just have to extend these methods and the struct by the partial log meta data: Then you have to fix Mind the ID and ordinal of the partial log metadata so that you concatenate the right records in the correct order.
I just edited my above comment because I am not sure how the case of linebreaks (i.e. those written by the application) in log messages relate to this issue. However, often you can control the logging of your application and let it write JSON to STDOUT (e.g. for Java / Spring Boot applications) so you do not have linebreaks. Back to the original issue: I have serious doubts that will do it under load with a lot of concurrent partial messages. I observed this in a Docker Swarm setup with fluentd and the concat plugin configured analogously.
Could you please reopen this issue? The concurrency is the problem here: Let your container simultaneously produce two log messages of 16KB (let's say 20 KB). Then fluentd will concatenate all four messages to one large (16+16+4+4 KB = 40 KB) message. The fluentd (or whatever you use) needs to know the ID of the partial log metadata to concatenate only those messages with identical IDs. So this needs to be implemented in the (default) local JSON file log driver in Docker (moby).
Problem is that ID comes with Docker 18.06 if I'm not wrong, this is why I didn't use it: do we want to require a docker version which is 2 months old?
is no longer much involved with the project. if you would like to open this back up to alter the implementation, please file a new issue
If you do so, please ping me as well.
I have the concat plugin but still it is not concatenating, Any Idea?
No idea why it does not work. For up-to-date information, I have inserted: Just after the kubernetes_metadate injection.

Related discussion:
text: apiVersion: apps/v1 kind: DaemonSet metadata: <del> name: fluentd-es-v2.2.0 </del> <ins> name: fluentd-es-v2.2.1 </ins> namespace: kube-system labels: k8s-app: fluentd-es <del> version: v2.2.0 </del> <ins> version: v2.2.1 </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec: selector: matchLabels: k8s-app: fluentd-es <del> version: v2.2.0 </del> <ins> version: v2.2.1 </ins> template: metadata: labels: k8s-app: fluentd-es kubernetes.io/cluster-service: "true" <del> version: v2.2.0 </del> <ins> version: v2.2.1 </ins> # This annotation ensures that fluentd does not get evicted if the node # supports critical pod annotation based priority scheme. # Note that this does not guarantee admission on the nodes (#40573).