Deferring construction of key deserializers, trying to reduce startup overhead