Change the MutableObjectOperator to allow immutable objects and remove the Reference<T> hack

This is a big change, so we need several pairs of eyes to look over this.