Squash another bug

If mouse exited and came back in, server would be confused