Fix issue with staying in observer mode even when no one is controlling
simulation