Vests can be worn with formal shirts, button-down shirts, and even T-shirts. However, it is essential to ensure that the colors and patterns complement one another for a unified appearance.