Change:
We use a lot of CRDs (frequently more than 1,000) in the project. This has historically caused a bunch of problems with client-side API discovery, among other things. At the time of writing the latest releases of kubectl use a discovery client rate limiter with a generous QPS and burst (50 qps, 300 qps burst), so client-side rate limiting doesn't become a problem until the API server is serving more than 300 API groups. Despite rate-limiting no longer coming into effect, we noticed that discovery was very slow - taking 10-20 seconds to complete. After having a few folks at run a little test (see and following comments) we noticed that discovery specifically seems slow on MacOS - it's dramatically faster on Linux. Here's discovery happening on Linux with 1,577 CRDs across 343 API groups in around a second: ` Here's the same test run from MacOS against the exact same EKS cluster, taking 12+ seconds. I expected kubectl discovery on MacOS to be as fast as it was on Linux. an EKS cluster. ~1,500 CRDs across ~350 API groups. the discovery cache in how long the first command you run against the EKS cluster takes to complete Refer to and subsequent comments for more examples of folks reproducing this. <detailsNote that the version of kubectl was the same on Mac and Linux for my tests. </details<detailsAWS - EKS </details<detailsMacOS 12.4, M1 Max Linux as below </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
will show you how many milliseconds was the GET request taken. There are 37 GET requests after was removed. Is there any difference?
I'll add more details ASAP, but we're confident the slow-down is not in the network (i.e. the many GET requests). Initial profiling indicates it's something to do with writing what is fetched to the discovery cache on disk.
Watch out for anything where writes trigger : there was an issue on macOS where the only way for an app to be confident that a write had completed was to trigger a whole-filesystem sync. Libraries that like data integrity opted for the slower-but-effective approach. I've stopped using macOS so this hint is from a while back, but it might relevant.
Thanks for the tip The issue definitely seems to be in the . When I apply the diff below to switch over to a the slowness disappears and MacOS performance matches that of Linux - about a second for to perform discovery:
It seems to be a problem that would occur only once when establishing the . Should we add it to known issues? Some explain about fsync in MacOs in
Hi, I have no strong opinion here.
You're right, and this happens every 6 hours in the latest versions of kubectl when the cache is invalidated. It's still not an ideal user experience though - you can imagine kubectl 'hanging' for 10-20 seconds the first time you used it each day.
Also note that this should fix a lot of that problem:
has anyone been able to profile kubectl to see what it's doing?
Also note that your CPU is mostly doing nothing, so it's most likely stuck on IO
I think is correct - my bet is we're blocking on , which per uses on MacOS. Per : A mutex profile shows a lot of time waiting for inside , which you can see sets the argument to . I believe this means we're doing an for every HTTP response body we encounter while doing discovery per .
I've confirmed the slowdown is coming from the . Using the following diff: I see MacOS is down to around one second, similar to Linux:
How do folks feel about not explicitly calling for each file in the HTTP cache? Given that it is indeed only a cache I lean toward optimizing for write performance rather than data safety. If we did want to stop calling it seems like the best option would be to update the package of that we use today to make whether it calls with configurable. I'm skeptical we could get this change merged upstream though - the latest change to was three years ago. Another option might be to switch the the package. I just took a brief shot at this, but it seems like it would require some refactoring to the to handle calling on the underlying database.
I don't care a ton if we drop a cache file coherently. I care a lot if a write fails in a way that leaves a partially written file in a way that pollutes the cache and makes the user have to delete the file from the cache manually. I don't know which of those we're opening ourselves up to by writing without sync. Can we find out?
Presuming we do find that not fsyncing can result in leaving the cache in a broken state (as opposed to a partial state) would falling back to invaliding the cache and recreating it be acceptable?
Little more insight at and Notably the second link mentions:
I dug into this a bit - dumping state here before I sign off for the day. First, note that the discovery client uses two layers of cache - a "discovery cache" and an "HTTP cache". The former appears to be backed by the latter. My -fu is not strong enough to figure out why we do this. The best I could find is , which refers to "restoring the old cached client behavior" and appears to (re?)introduce the caching HTTP transport. Perhaps or know why this is? Only the HTTP cache is explicitly calling . The discovery cache just writes the files and calls , so it's already subject to the risks of not using . Per that risk is that despite the write having been flushed from the OS to the disk controller (e.g. by successfully calling ): My understanding of the and code that powers the HTTP cache is: If a cache file read returns an error we just act as if the value isn't in cache. If a cache file read returns garbled data that can't parse as an http response would fall back to its underlying (i.e. make a call to the API server) and will happily cache over the top of an existing (potentially garbled) file. The caches will be invalidated after six hours. So the failure mode that wouldn't be covered if we stopped calling is that if power were cut to the drive before its controller could sync down to persistent storage and_ this resulted in a file that was garbled just right such that it appeared to be a valid HTTP response but contained an invalid body (i.e. not real discovery data) someone would have to either manually remove their cache, or wait six hours for the cache to be automatically invalidated. It seems like we could probably make the discovery client force an uncached request through the caching HTTP transport if it encountered what appeared to be cached, invalid discovery data, but I do wonder how far we really need to go to ensure that this cache is never corrupted.
For either of these caches, there are techniques (checksums and the like) where we can allow the app to see whether or not it is looking at a coherent cache document or something that might not be correct. Whenever a cache entry doesn't look valid, the code can asset that the date is current and the entry checksum is OK. If the entry turns out to be invalid or stale, remove it and then act as if we got a read error. That should cover any kind of unsynchronised write failure, I think. For locking the cache we also don't need to ensure sync-to-disk, so long as all running processes see a consistent view of locking.
Agreed. My question is whether we need to add such techniques to this particular cache (or caches) before we'll feel comfortable removing the fsync. Removing the fsync from the HTTP cache would presumably bring it back down to the same level of guarantee as the discovery cache. My guess would be that the fact we were ever explicitly fsyncing each cache value in the HTTP cache was a happy accident rather than something that was explicitly a desirable quality for the HTTP cache.
It seemed like writing our own httpcache cache backend was the best way to go here, so I went ahead and one that uses checksums rather than syncs per the above PR.

Discussion:
package disk import ( <ins> "bytes" "crypto/sha256" "fmt" </ins> "net/http" "os" "path/filepath" "github.com/gregjones/httpcache" <del> "github.com/gregjones/httpcache/diskcache" </del> "github.com/peterbourgon/diskv" "k8s.io/klog/v2" )

BasePath: cacheDir, TempDir:  filepath.Join(cacheDir, ".diskv-temp"), }) <del> t := httpcache.NewTransport(diskcache.NewWithDiskv(d)) </del> <ins> t := httpcache.NewTransport(&sumDiskCache{disk: d}) </ins> t.Transport = rt return &cacheRoundTripper{rt: t}

} func (rt *cacheRoundTripper) WrappedRoundTripper() http.RoundTripper { return rt.rt.Transport } <ins> // A sumDiskCache is a cache backend for github.com/gregjones/httpcache. It is // similar to httpcache's diskcache package, but uses SHA256 sums to ensure // cache integrity at read time rather than fsyncing each cache entry to // increase the likelihood they will be persisted at write time. This avoids // significant performance degradation on MacOS. // // See https://github.com/kubernetes/kubernetes/issues/110753 for more. type sumDiskCache struct { disk *diskv.Diskv } // Get the requested key from the cache on disk. If Get encounters an error, or // the returned value is not a SHA256 sum followed by bytes with a matching // checksum it will return false to indicate a cache miss. func (c *sumDiskCache) Get(key string) ([]byte, bool) { b, err := c.disk.Read(sanitize(key)) if err != nil || len(b) < sha256.Size { return []byte{}, false } response := b[sha256.Size:] want := b[:sha256.Size] // The first 32 bytes of the file should be the SHA256 sum. got := sha256.Sum256(response) if !bytes.Equal(want, got[:]) { return []byte{}, false } return response, true } // Set writes the response to a file on disk. The filename will be the SHA256 // sum of the key. The file will contain a SHA256 sum of the response bytes, // followed by said response bytes. func (c *sumDiskCache) Set(key string, response []byte) { s := sha256.Sum256(response) _ = c.disk.Write(sanitize(key), append(s[:], response...)) // Nothing we can do with this error. } func (c *sumDiskCache) Delete(key string) { _ = c.disk.Erase(sanitize(key)) // Nothing we can do with this error. } // Sanitize an httpcache key such that it can be used as a diskv key, which must // be a valid filename. The httpcache key will either be the requested URL (if // the request method was GET) or "<method> <url>" for other methods, per the // httpcache.cacheKey function. func sanitize(key string) string { // These keys are not sensitive. We use sha256 to avoid a (potentially // malicious) collision causing the wrong cache data to be written or // accessed. return fmt.Sprintf("%x", sha256.Sum256([]byte(key))) } </ins>

import ( "bytes" <ins> "crypto/sha256" </ins> "io/ioutil" "net/http" "net/url"

"path/filepath" "testing" <ins> "github.com/peterbourgon/diskv" </ins> "github.com/stretchr/testify/assert" )

return rt.Response, rt.Err } <ins> func BenchmarkDiskCache(b *testing.B) { cacheDir, err := ioutil.TempDir("", "cache-rt") if err != nil { b.Fatal(err) } defer os.RemoveAll(cacheDir) d := diskv.New(diskv.Options{ PathPerm: os.FileMode(0750), FilePerm: os.FileMode(0660), BasePath: cacheDir, TempDir:  filepath.Join(cacheDir, ".diskv-temp"), }) k := "localhost:8080/apis/batch/v1.json" v, err := ioutil.ReadFile("../../testdata/apis/batch/v1.json") if err != nil { b.Fatal(err) } c := sumDiskCache{disk: d} for n := 0; n < b.N; n++ { c.Set(k, v) c.Get(k) c.Delete(k) } } </ins> func TestCacheRoundTripper(t *testing.T) { rt := &testRoundTripper{} cacheDir, err := ioutil.TempDir("", "cache-rt")

}) assert.NoError(err) } <ins> func TestSumDiskCache(t *testing.T) { assert := assert.New(t) // Ensure that we'll return a cache miss if the backing file doesn't exist. t.Run("NoSuchKey", func(t *testing.T) { cacheDir, err := ioutil.TempDir("", "cache-test") if err != nil { t.Fatal(err) } defer os.RemoveAll(cacheDir) d := diskv.New(diskv.Options{BasePath: cacheDir, TempDir: filepath.Join(cacheDir, ".diskv-temp")}) c := &sumDiskCache{disk: d} key := "testing" got, ok := c.Get(key) assert.False(ok) assert.Equal([]byte{}, got) }) // Ensure that we'll return a cache miss if the backing file is empty. t.Run("EmptyFile", func(t *testing.T) { cacheDir, err := ioutil.TempDir("", "cache-test") if err != nil { t.Fatal(err) } defer os.RemoveAll(cacheDir) d := diskv.New(diskv.Options{BasePath: cacheDir, TempDir: filepath.Join(cacheDir, ".diskv-temp")}) c := &sumDiskCache{disk: d} key := "testing" f, err := os.Create(filepath.Join(cacheDir, sanitize(key))) if err != nil { t.Fatal(err) } f.Close() got, ok := c.Get(key) assert.False(ok) assert.Equal([]byte{}, got) }) // Ensure that we'll return a cache miss if the backing has an invalid // checksum. t.Run("InvalidChecksum", func(t *testing.T) { cacheDir, err := ioutil.TempDir("", "cache-test") if err != nil { t.Fatal(err) } defer os.RemoveAll(cacheDir) d := diskv.New(diskv.Options{BasePath: cacheDir, TempDir: filepath.Join(cacheDir, ".diskv-temp")}) c := &sumDiskCache{disk: d} key := "testing" value := []byte("testing") mismatchedValue := []byte("testink") sum := sha256.Sum256(value) // Create a file with the sum of 'value' followed by the bytes of // 'mismatchedValue'. f, err := os.Create(filepath.Join(cacheDir, sanitize(key))) if err != nil { t.Fatal(err) } f.Write(sum[:]) f.Write(mismatchedValue) f.Close() // The mismatched checksum should result in a cache miss. got, ok := c.Get(key) assert.False(ok) assert.Equal([]byte{}, got) }) // Ensure that our disk cache will happily cache over the top of an existing // value. We depend on this behaviour to recover from corrupted cache // entries. When Get detects a bad checksum it will return a cache miss. // This should cause httpcache to fall back to its underlying transport and // to subsequently cache the new value, overwriting the corrupt one. t.Run("OverwriteExistingKey", func(t *testing.T) { cacheDir, err := ioutil.TempDir("", "cache-test") if err != nil { t.Fatal(err) } defer os.RemoveAll(cacheDir) d := diskv.New(diskv.Options{BasePath: cacheDir, TempDir: filepath.Join(cacheDir, ".diskv-temp")}) c := &sumDiskCache{disk: d} key := "testing" value := []byte("cool value!") // Write a value. c.Set(key, value) got, ok := c.Get(key) // Ensure we can read back what we wrote. assert.True(ok) assert.Equal(value, got) differentValue := []byte("I'm different!") // Write a different value. c.Set(key, differentValue) got, ok = c.Get(key) // Ensure we can read back the different value. assert.True(ok) assert.Equal(differentValue, got) }) // Ensure that deleting a key does in fact delete it. t.Run("DeleteKey", func(t *testing.T) { cacheDir, err := ioutil.TempDir("", "cache-test") if err != nil { t.Fatal(err) } defer os.RemoveAll(cacheDir) d := diskv.New(diskv.Options{BasePath: cacheDir, TempDir: filepath.Join(cacheDir, ".diskv-temp")}) c := &sumDiskCache{disk: d} key := "testing" value := []byte("coolValue") c.Set(key, value) // Ensure we successfully set the value. got, ok := c.Get(key) assert.True(ok) assert.Equal(value, got) c.Delete(key) // Ensure the value is gone. got, ok = c.Get(key) assert.False(ok) assert.Equal([]byte{}, got) // Ensure that deleting a non-existent value is a no-op. c.Delete(key) }) } </ins>

<del> // Package diskcache provides an implementation of httpcache.Cache that uses the diskv package // to supplement an in-memory map with persistent storage // package diskcache import ( "bytes" "crypto/md5" "encoding/hex" "github.com/peterbourgon/diskv" "io" ) // Cache is an implementation of httpcache.Cache that supplements the in-memory map with persistent storage type Cache struct { d *diskv.Diskv } // Get returns the response corresponding to key if present func (c *Cache) Get(key string) (resp []byte, ok bool) { key = keyToFilename(key) resp, err := c.d.Read(key) if err != nil { return []byte{}, false } return resp, true } // Set saves a response to the cache as key func (c *Cache) Set(key string, resp []byte) { key = keyToFilename(key) c.d.WriteStream(key, bytes.NewReader(resp), true) } // Delete removes the response with key from the cache func (c *Cache) Delete(key string) { key = keyToFilename(key) c.d.Erase(key) } func keyToFilename(key string) string { h := md5.New() io.WriteString(h, key) return hex.EncodeToString(h.Sum(nil)) } // New returns a new Cache that will store files in basePath func New(basePath string) *Cache { return &Cache{ d: diskv.New(diskv.Options{ BasePath:     basePath, CacheSizeMax: 100 * 1024 * 1024, // 100MB }), } } // NewWithDiskv returns a new Cache using the provided Diskv as underlying // storage. func NewWithDiskv(d *diskv.Diskv) *Cache { return &Cache{d} } </del>

# github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 => github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 ## explicit github.com/gregjones/httpcache <del> github.com/gregjones/httpcache/diskcache </del> # github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 => github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 ## explicit; go 1.14 github.com/grpc-ecosystem/go-grpc-middleware