Change:
Forked from From
/subscribe
I did some a bit deeper analysis and here are the most interesting findings: ~10% is spent on deferring the type of the object, basically: DataVersionAndKind() function: ~20% is spent on unmarshalling json of the remaining time (~70%) is spent in convertKV method: where 5-10% is spent on checkField() 5-10% is spent on tagOf() majority is spent on kvValue::value() However, I don't have any clear ideas how to improve it yet.
If you look at a memory allocation profile you'll see a huge amount of allocations of things like conversion.State and reflect.*, which will burn CPU in garbage collection as well.
It's just the second one. So basically, we spent ~30% of time on both unmarshallings.
In an ideal world, if we had more control over unmarshalling, we would be able to do both in a single pass (scan and buffer until we find the type/version keys, then reset and run on the known type). Also, it's worth remembering that the JSON unmarshal time is probably dominated by reflection and reflection related unmarshalling. A generated code approach to unmarshalling and conversion would target both of those. ----- Original Message -----
Yes - I completely agree that "type-specific-unmarshaling" would be much much more effective. Do you think it makes sense to do it now? (it seems to be a lot of work to do it, because it requires writing json parser, right?)
Improving encoding/decoding speed seems much better than adding caching. Caching always adds bugs which are hard to reproduce. There may be bugs with a new encoder/decoder, but they should be easy to repro and side by side testing and fuzz testing can work a lot of the bugs out of a new decoding scheme. Now that apiserver is the sole client of etcd, would it make sense to persist the data in etcd a more efficiently decodable form, such as a gob?
Anyone considered other libs? Also, 'perf top' under stress shows garbage collection easily dominates.
The investigation in pointed to ugorji which can generate a static unmarshaller for a known type (no one has tried it yet though). The dynamic type unmarshal is valuable in some places (clients, tools) but in the APIServer in many cases we know the exact type, so an unmarshal to the wrong type can be a type error (DecodeInto). We can always optimize the dynamic type unmarshal later (tokenizing scanner over the byte array that gets passed to decode) after optimizing the known type unmarshal. Another possible source of inefficiency in the client side of watch, which needs to break a stream into JSON chunks (the []byte passed to Decode()), and it reuses the json.Decoder logic to do that. I'm not sure that is zero copy, but it should be possible to get that down to zero copy (the tokenizer in encoding/json is pretty efficient, so essentially we just need to keep growing the buffer until we see a closed JSON object or get a decode error). Conversion is of course the dominant factor right now, so I don't want to imply that the unmarshalling is the most important problem right now. ----- Original Message -----
ugorji is better than ffjson for most practical considerations I've seen (without actually trying it) as referenced in the other thread. ----- Original Message -----
I agree, but if we have a dedicated per-type unmarshaling, we can unmarshal it directly to api objects (and omit json on this path) which should be much more efficient, right?
We still have to do a conversion when reading from etcd - what's in etcd has to be versioned as well.
garbage collection is due to the massive amounts of allocation done by decode/conversion. Unfortunately. +1000
Do you think that it's worth creating code-generators responsible for type-specific unmarshaling and conversion now? It seems to be a lot of work and it's hard to estimate the exact gains of it.
To be clear - I'm happy to work on that, but I want to ensure that we have agreement that it's worth doing now.
I did some quick experiment and few dedicated conversion functions for Pod, TypeMeta and ObjectMeta (to avoid using the reflect package). It seems that just this gives us more than 5% of gain, so I'm pretty sure that having a dedicate conversion functions for all types from api would have a huge impact. Do you think that adding them makes sense? [I mean adding some code generator that based on the given types, will generate such conversion functions]. I can start working on it tomorrow. [update] On the other hand, such script will not work if we don't have 1-1 corespondence between v1beta3 types and internal api types.
I'm ok with the approach as long as the generation is flexible and well factored (takes a runtime.Scheme with conversion funds and generates code). I know we have good tests around conversion, so I'm less worried about the impact.
Yes, I think it's worth doing. It is going to be a bit tricky to do correctly, though-- the caveat in your update isn't going to fly :) We can chat if you want to discuss some approaches to this. I think you may need to write a program that parses pkg/api/ and pkg/api/v1betaX, and outputs reflection-free code that does all the things in But there may be a few steps we want to do before that.
5% improvement doesn't seem like much. If you expand this to all (less common) types, do you expect like 10% or 15%? That still seems quite low for the complexity.
I expect this to be 40% (this is just some guess, but definitely more than 10-15%)
how should we validate "goodness", compare /metrics ?
In this case I'd accept an encode decode benchmark showing a few thousand cycles, as well as a demonstrated reduction in allocations (number and bytes).
I also think gating e2e on this would be good, as to detect performance regressions.
: I agree with - I think the easiest way is to have some decoding/conversion benchmarks (the is pretty good one). But exposing it in /metrics also sounds like a good idea.
Thank a lot for all kind of feedback. I'm starting to work on the code to automatically create conversion functions that doesn't use reflection.
I've started writing some code for it on Friday and spent some time today on it. I will try to send out some WIP PR tomorrow to start more technical discussion on it.
Looking forward to it :)
Slightly related to this, at least in terms of structure, is that I'm considering a different approach for unrecognized objects embedded in Lists and config formats that would allow better consumption by generic tools using swagger info. proposes using map[string]interface{} internally for any unknown types, which we can then manage opaquely. Whatever conversion code we evolve for this would be useful for that context as well.
This is mostly done - (at least at the v1.0 scope). The only remaining thing is to update the documentation. I will send a separate PR to fix it and close this issue later this week.

Discussion:
than the generic ones (which are based on reflections and thus are highly inefficient). <del> The conversion code resides with each versioned API - `pkg/api/<version>/conversion.go`.  To regenerate conversion functions: </del> <ins> The conversion code resides with each versioned API. There are two files: - `pkg/api/<version>/conversion.go` containing manually written conversion functions - `pkg/api/<version>/conversion_generated.go` containing auto-generated conversion functions Since auto-generated conversion functions are using manually written ones, those manually written should be named with a defined convention, i.e. a function converting type X in pkg a to type Y in pkg b, should be named: `convert_a_X_To_b_Y`. Also note that you can (and for efficiency reasons should) use auto-generated conversion functions when writing your conversion functions. Once all the necessary manually written conversions are added, you need to regenerate auto-generated ones. To regenerate them: </ins> - run ``` $ go run cmd/kube-conversion/conversion.go -v <version> -f <file1.txt> -n <file2.txt> ``` <del> - replace all conversion functions (convert* functions) in the above file with the contents of <file1.txt> - replace arguments of `newer.Scheme.AddGeneratedConversionFuncs` with the contents of <file2.txt> </del> <ins> - replace all conversion functions (convert* functions) in the `pkg/api/<version>/conversion_generated.go` with the contents of <file1.txt> - replace arguments of `newer.Scheme.AddGeneratedConversionFuncs` in the `pkg/api/<version>/conversion_generated.go` with the contents of <file2.txt> </ins> <del> Unsurprisingly, this also requires you to add tests to </del> <ins> Unsurprisingly, adding manually written conversion also requires you to add tests to </ins> `pkg/api/<version>/conversion_test.go`. ## Update the fuzzer