There's no hard and fast rule about it. It all depends on your personal style. You can keep your blazer unbuttoned in informal situations for a more relaxed feel. In social settings where you want a more formal look, buttoning up the blazer is a good idea.