Don't modify Objects in Builder

Redid part of #265 on 559fc95341aeee92c478a68cb01948a33415c48f. The rest will come in separate PRs.