We want to use MutationRef& whenever we aren't looking at an object that might disappear — copying a shared_ptr requires atomic ops that can actually show up as a cpu bottleneck.
Basically, we should only create new ones when passing off control and something else might (or will) remove the actual ref we're looking at.