Change:
When working on I have found a regression in JSON logging format that happend in 1.23. In v1.23 JSON logging introduced two new features, log stream splitting and info log buffering. When I started measuring throughput it looked very promising, for example for info logs showed 10x improvement when using JSON split stream and 100Mi info log buffer. However when I introduced mixed logs (ever X info logs write 1 error logs), I noticed a large regression. For 10% of error logs written (apiserver during upgrade can write even 30% errors), I have found that just switching json on, results in log throughput being reduced from 25 MiB/s to 0.7MiB. I have bisected git repo and found that was the root cause. I expect that this regression would be noticed if we run scalability tests with JSON enabled. Loader code: /wg structured-logging /cc /sig scalability JSON should generate similar or greater log throughput when compared to text format. Run loader as described in from No response N/A N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
So to reproduce the regression, I need to do (default for error rate is 10%, 300 for message length, 10s duration) for old and new code, right?
Yes, let me give my results. Before text: 30.7MiB/s, json:39.6MiB/s After, text: 30.6MiB/s, json: 6.5MiB/s
should we try to get this in 1.23.1?
Yes, we just moved JSON and Structured Logging to Beta in v1.23. This bug should cause performance issues when there is non trivial amount of errors (1%) written by control plane. I expect that this has really bad impact on upgrades which I have seen can reach 30% of error logs from apiserver.
It's worth pointing out that this regression only occurs when writing to a disk-based file (see ). That also provides a very likely explanation for the huge performance difference: I bet zap not only flushes its buffered info messages, it also syncs the file. That is fast for tmpfs, but slow for real disks.
do you know how the Kubernetes components are run in those cases where logging performance matters? Do they directly write to a real disk or does it all go through pipes? If it is the latter, then this issue matters a lot less in practice. you mentioned at some point that you had scalability issues with logging that went away when using go-runner as wrapper instead of using klog's internal output redirection. I bet the root cause is the same: fast vs. slow sync of the output fd. That's because go-runner always runs output through one or more pipes. I don't know how realistic the scalability setup was, but it must have written log files to some real disk.

Discussion:
encoder := zapcore.NewJSONEncoder(encoderConfig) var core zapcore.Core if errorStream == nil { <del> core = zapcore.NewCore(encoder, zapcore.AddSync(infoStream), zapcore.Level(-127)) </del> <ins> core = zapcore.NewCore(encoder, infoStream, zapcore.Level(-127)) </ins> } else { <del> // Set up writing of error messages to stderr and info messages // to stdout. Info messages get optionally buffered and flushed // - through klog.FlushLogs -> zapr Flush -> zap Sync // - when an error gets logged // // The later is important when both streams get merged into a single // stream by the consumer (same console for a command line tool, pod // log for a container) because without it, messages get reordered. flushError := writeWithFlushing{ WriteSyncer: errorStream, other:       infoStream, } </del> highPriority := zap.LevelEnablerFunc(func(lvl zapcore.Level) bool { return lvl >= zapcore.ErrorLevel })

return lvl < zapcore.ErrorLevel }) core = zapcore.NewTee( <del> zapcore.NewCore(encoder, flushError, highPriority), </del> <ins> zapcore.NewCore(encoder, errorStream, highPriority), </ins> zapcore.NewCore(encoder, infoStream, lowPriority), ) }

var _ registry.LogFormatFactory = Factory{} func (f Factory) Create(options config.FormatOptions) (logr.Logger, func()) { <ins> stderr := zapcore.Lock(os.Stderr) </ins> if options.JSON.SplitStream { <del> // stdout for info messages, stderr for errors. infoStream := zapcore.Lock(os.Stdout) </del> <ins> stdout := zapcore.Lock(os.Stdout) </ins> size := options.JSON.InfoBufferSize.Value() if size > 0 { // Prevent integer overflow. if size > 2*1024*1024*1024 { size = 2 * 1024 * 1024 * 1024 } <del> infoStream = &zapcore.BufferedWriteSyncer{ WS:   infoStream, </del> <ins> stdout = &zapcore.BufferedWriteSyncer{ WS:   stdout, </ins> Size: int(size), } } <del> return NewJSONLogger(infoStream, zapcore.Lock(os.Stderr)) </del> <ins> // stdout for info messages, stderr for errors. return NewJSONLogger(stdout, stderr) </ins> } <del> // The default is to write to stderr (same as in klog's text output, // doesn't get mixed with normal program output). out := zapcore.Lock(os.Stderr) return NewJSONLogger(out, out) } // writeWithFlushing is a wrapper around an output stream which flushes another // output stream before each write. type writeWithFlushing struct { zapcore.WriteSyncer other zapcore.WriteSyncer } func (f writeWithFlushing) Write(bs []byte) (int, error) { f.other.Sync() return f.WriteSyncer.Write(bs) </del> <ins> // Write info messages and errors to stderr to prevent mixing with normal program output. return NewJSONLogger(stderr, nil) </ins> }