Do proper cleanup after rename