Check for right colors before clicking

Prevents accidental clicks in context where they don't make sense