Even more: you do NOT remove the video stream, but just put it "inactive" or, even if it gets really removed from the SDP, re-adding it  is fast since typically the video uses the same "UDP connection" (multiplexed with the audio track).

But forcing the remote to see a black video window seems really ugly IMHO. You don't have such a useless and confussing option in any existing VoIP software.