Limit click detection logic to mouse down events.