You should size up if they are bent and stand away from your chest, or if you can see tugging around the button. It's far simpler to make things smaller than it is to attempt to fit into a blazer that's too tiny.