remove useless check when handling mouse presses

This saves a few branching