Put back safer key manager factory caching #859