The other side is not aware of every stream direction change.  It should interpret, emit a new event, and It does make the logic much more complex.

I would just support the local Audio and Video mute.