Change:
I was running a kubectl logs -f, not for long, that broke unexpectedly with Doesn't seem normal.
I'm seeing this on 1.6.2
do you use docker, if so, what log driver do you use? If you use the default log driver and have access to the node that the specific container ran on, it'd be great if you can provide content of the raw log file (located in ).
I saw this in a GKE cluster so I don't have access to the node....
Whose gke cluster? If you are the owner of the gke cluster, you do have access to the node VMs. You can into them.
So I have access after all :) docker 11.2 is used and the log driver is json-file. Unfortunately, I have scaled the deployment down since I had the issue and now I cannot find the container log under /var/log/containers. Noted for the next time tho
docker 11.2.6 using json-file I'll grab the log off the host next time i see it. I assume just like 20 lines above and below where kubectl logs fails would be sufficient?
Yes, that'd be great.
In case it helps someone else, here's how I grabbed the logs in question as is installed on the host The client-side experience the output from and the from that container's (mildly redacted, and I know you asked for about 20 lines, but the lines above those are some sensitive bits that I would prefer not to have to go through and redact; but I will if you think it's necessary) I have the contents of , too, but they are md5-level the exact same content Environmental information regrettably_ we are using the based from CoreOS, invoked as such:
Given , I really don't understand why this happens. The only possibility is that kubelet is reading and parsing a partial line. However, if the json log line is in-complete, should return error and kubelet will discard that line; Or else, it means that json is complete, and I don't understand why is empty when the actual log line contains everything. Could you check kubelet log to see whether there is anything suspicious, although unlikely.
sorry for the delay in getting back to you, I finally caught something interesting in the kubelet log I elided the info level lines in between those error level lines, as they were "MountVolume.Setup succeeded" and didn't seem relevant I also came here to see if there was another issue regarding intermingled log output, but searching for that in the issues is extremely hard given how many ways there are in english to describe the behavior: where other-pod-line-1 is very clearly the log output from a separate Pod on that same Node
I think this is a concurrency issue. I've not been able to reliably reproduce it but anecdotally have noticed that the logging is much less stable when multiple people are tailing at once. I can tail logs off-hours when no one is around and it will be fine for hours without this error. But during the work day when multiple people are tailing the same log we all will all see the error quite regularly. Pretty much every time the error happens, if I ask around it turns out someone else had just started tailing the same log.
In my case, I doubt that it is multiple people tailing simultaneously, but I will try to remember to start several and see if that impacts things I experience it pretty regularly, but I haven't tried reproducing it in any self-contained system -- there are so many variables to consider :( do you also experience that log lines are intermixed with other Pod's output?
I've never seen that. We have a few hundred pods, and haven't witnessed it once or seen anyone else complain about it. But if somehow the streams are being crossed for you, that could also point to a concurrency issue possibly. In your case it may not be multiple readers, but possibly multiple writers. I've not looked at the code though, so just a guess.
Thanks for your suggestion, Bryan, as I am able to blow up the kubelet at will now; I get one of these explosions for each kubectl that attaches: I'll see if I can get it to explode in a more isolated environment, since it doesn't appear to be related to the pods that are running
I see this bug frequently when calling many times in parallel on my kubeadm Kubernetes 1.7.4 cluster. I was using multiple threads (from a Python3 process) to get many logs at once using kubectl log, and that would cause me to hit this bug on about 25% of the requests. Switching to grabbing the logs one after the other (rather than doing many kubectl log shell commands at once) works around the problem, though is much an order of magnitude slower. To be clear: I am not using the follow option. I am using the tail option, typically .
Also reproduced the problem by fetching container logs in multiple processes. should fix the problem.
I am able to reproduce this with minikube v0.22.1 (which runs kubernetes 1.7.5). When following logs for separate pods concurrently (using ), I very quickly see . Funny thing is there's always one random pod for which the follow doesn't crash with this error (at least not for a long time) and keeps running successfully.
Seeing this error message with a 1.7.3 bare metal cluster.
FYI, the fix is in 1.7.6.

Discussion:
return nil } <del> // dockerJSONLog is the JSON log buffer used in parseDockerJSONLog. var dockerJSONLog = &jsonlog.JSONLog{} </del> // parseDockerJSONLog parses logs in Docker JSON log format. Docker JSON log format // example: //   {"log":"content 1","stream":"stdout","time":"2016-10-20T18:39:20.57606443Z"} //   {"log":"content 2","stream":"stderr","time":"2016-10-20T18:39:20.57606444Z"} func parseDockerJSONLog(log []byte, msg *logMessage) error { <del> dockerJSONLog.Reset() l := dockerJSONLog </del> <ins> var l = &jsonlog.JSONLog{} l.Reset() </ins> // TODO: JSON decoding is fairly expensive, we should evaluate this. if err := json.Unmarshal(log, l); err != nil { return fmt.Errorf("failed with %v to unmarshal log %q", err, l)