Add back unsafe_get and unsafe_mut, misc cleanups and fallout