Steal the dictionary only for refcount_ == 1

This brings back the optimal speed.