If you want to remove video while still keeping talking, then just remove the video (reINVITE and so on). Don't "mute" it.

BTW: not sure if currently we have an API for adding/removing stream tracks, but that should be the way IMHO.