Brogues are iconic shoes, which have long been a classic. Although they are super versatile and can be used in different casual styles, traditionally they are formal. So it's all right to wear a formal suit for your brogues.