Immediately rearrange memory on mode changes