Prevent spectator interaction in some circumstances. Fixes #167