Don't walk whole setting symbol table on load.

This should help avoid some deserialization work, or at least get rid
of one more thing that forces a bunch of it.