Code change:
We should introduce a buffer (similar to what we have in the webhook output backend) to the log file output backend. Audit events should be queued on a relatively large buffer chanel, and an output go routine should grab events off the queue and write them to disk. If an audit event is output when the channel is full, it should be dropped (this should be counted by a prometheus metric, and an e2e test should ensure that the count is below some threshold). This behavior should be optional (configurable) /cc is tracking the regression
Naturally os will cache , these cached data will be written to disk either by fsync or pdflush periodically. What do you think is the most time-costing procedure in audit?
The problem is not with the time consumed. The problem is that currently write is performed synchronously with processing the request, which slows the request down, no matter how well caching works
The problem is that currently write is performed synchronously with processing the request Sure synchronously write does cost time serializing. Introduce buffer mechanism can really reduce api request latency time.
It doesn't matter that much if it happens asynchronously Indeed
Yes, this is a good point to improve performance.
Now I totally comprehend this. I am pleased to implement this buffer.
I'll be happy to review that
audit log is important sensitive information which we should try to make sure it reaches disk at all cost. Does this effort compromises that in favor of optimizing for latency ?
I'll make sure it's configurable and if the security is critical, you can setup synchronous write in your deployment. Having said that, it's critical to be able to accommodate the case where having 100% calls written does not compensate for increased latencies. Does it make sense?
thanks my preference would be to keep the default as no buffering when making it configurable. I didnt understand how does one setup "synchronous write in your deployment" and how that will affect audit logs reaching disk. My worry is mainly around a crash where the buffered audit logs dont reach the disk and are lost
if use buffer, audit events lost is inevitable when crash. As like webhook backend, still exist this risk. But to improve apiserver qps, something should comprise. By the way, we still reserve the synchronous write mode, which is configurable.
To make it simple, when offered trade-off, people are more likely to give up audit logs reliability than performance. Makes sense?
i am not against this change. i think what would be best is to make the default as no buffering and enable buffering through a parameter. Also for Enterprises, the trade off is not correct, they would prefer to NOT give up audit logs
I understand what you're suggesting and I don't agree :) Switching to buffering will for sure increase the latency, but will only increase the probability of losing logs, which will never be zero. I'm pretty sure we need to have buffered mode by default. However, since this argument is about making a trade-off and since neither of us represents quorum, let's ask more people for their opinion What do you think?
The default is slightly less important here since audit logging as a whole is not enabled by default. If we do turn it on by default, it becomes more important. That said, my hunch is that we should default to the mode with lower performance impact, but I don't have strong feelings either way.
As long as there are options to turn it on of off (the synchronous write) I'm fine with either default. Still, I'd like to see due diligence towards ensuring low lost rate when using buffered write, and sufficient documentation around caveats of using one or the other. Audit, from my personal experience, is crucial for majority of enterprises and they won't tolerate not even single lost event.
That is not possible, there will always be a situation when an audit event can be lost: sudden power loss, disk/network failure, random bit flip, etc. The questions is whether 10^(-6) is low enough loss probability, or we need to get 10^(-14) by sacrificing performance (arbitrary numbers, just to illustrate my point). Note, that reduced performance can increase the number of timeouts and retries, increasing load and leading to failures and even more lost events.
I think this is an important bug that we're not testing audit logging at scale
Marking it as urgent, since it's blocking
If split into 3 prs, I must do it one by one . Now first one is up .
SGTM Status of this issue: ETA: 1/12/2017 Risks: Complicated fix required
has been reviewed for almost 2 months and has not done. If I raise 3 prs to replace it, I am still worried whether all of them can be merged. They act as a whole.
Now I'm going to devote my time to it, we'll do it
OK.
In progress, reviewed first PR, waiting for applying changes ETA: Dec 01
Per Exception discussion, this portion of the feature is being removed from the milestone. As such, changing the priority. /priority important-soon /remove-priority critical-urgent
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details

Related discussion:
text: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package buffered import ( "testing" "time" "github.com/stretchr/testify/require" "k8s.io/apimachinery/pkg/util/wait" auditinternal "k8s.io/apiserver/pkg/apis/audit" "k8s.io/apiserver/pkg/audit" ) var ( closedStopCh = func() <-chan struct{} { ch := make(chan struct{}) close(ch) return ch }() infiniteTimeCh <-chan time.Time = make(chan time.Time) closedTimeCh                    = func() <-chan time.Time { ch := make(chan time.Time) close(ch) return ch }() ) func newEvents(number int) []*auditinternal.Event { events := make([]*auditinternal.Event, number) for i := range events { events[i] = &auditinternal.Event{} } return events } func TestBufferedBackendCollectEvents(t *testing.T) { config := NewDefaultBatchConfig() testCases := []struct { desc          string timer         <-chan time.Time stopCh        <-chan struct{} numEvents     int wantBatchSize int }{ { desc:          "max batch size encountered", timer:         infiniteTimeCh, stopCh:        wait.NeverStop, numEvents:     config.MaxBatchSize + 1, wantBatchSize: config.MaxBatchSize, }, { desc:   "timer expired", timer:  closedTimeCh, stopCh: wait.NeverStop, }, { desc:   "chanel closed", timer:  infiniteTimeCh, stopCh: closedStopCh, }, } for _, tc := range testCases { tc := tc t.Run(tc.desc, func(t *testing.T) { t.Parallel() backend := NewBackend(&fakeBackend{}, config).(*bufferedBackend) backend.ProcessEvents(newEvents(tc.numEvents)...) batch := backend.collectEvents(tc.timer, tc.stopCh) require.Equal(t, tc.wantBatchSize, len(batch), "unexpected batch size") }) } } func TestBufferedBackendProcessEventsAfterStop(t *testing.T) { t.Parallel() backend := NewBackend(&fakeBackend{}, NewDefaultBatchConfig()).(*bufferedBackend) backend.Run(closedStopCh) backend.Shutdown() backend.ProcessEvents(newEvents(1)...) batch := backend.collectEvents(infiniteTimeCh, wait.NeverStop) require.Equal(t, 0, len(batch), "processed events after the backed has been stopped") } func TestBufferedBackendProcessEventsBufferFull(t *testing.T) { t.Parallel() config := NewDefaultBatchConfig() config.BufferSize = 1 backend := NewBackend(&fakeBackend{}, config).(*bufferedBackend) backend.ProcessEvents(newEvents(2)...) require.Equal(t, 1, len(backend.buffer), "buffed contains more elements than it should") } func TestBufferedBackendShutdownWaitsForDelegatedCalls(t *testing.T) { t.Parallel() delegatedCallStartCh := make(chan struct{}) delegatedCallEndCh := make(chan struct{}) delegateBackend := &fakeBackend{ onRequest: func(_ []*auditinternal.Event) { close(delegatedCallStartCh) <-delegatedCallEndCh }, } config := NewDefaultBatchConfig() backend := NewBackend(delegateBackend, config) // Run backend, process events, wait for them to be batched and for delegated call to start. stopCh := make(chan struct{}) backend.Run(stopCh) backend.ProcessEvents(newEvents(config.MaxBatchSize)...) <-delegatedCallStartCh // Start shutdown procedure. shutdownEndCh := make(chan struct{}) go func() { close(stopCh) backend.Shutdown() close(shutdownEndCh) }() // Wait for some time and then check whether Shutdown has exited. Can give false positive, // but never false negative. time.Sleep(100 * time.Millisecond) select { case <-shutdownEndCh: t.Fatalf("Shutdown exited before delegated call ended") default: } // Wait for Shutdown to exit after delegated call has exited. close(delegatedCallEndCh) <-shutdownEndCh } type fakeBackend struct { onRequest func(events []*auditinternal.Event) } var _ audit.Backend = &fakeBackend{} func (b *fakeBackend) Run(stopCh <-chan struct{}) error { return nil } func (b *fakeBackend) Shutdown() { return } func (b *fakeBackend) ProcessEvents(ev ...*auditinternal.Event) { if b.onRequest != nil { b.onRequest(ev) } } </ins>