The fabrics used in most sports bras aren’t made to withstand chlorine or saltwater. Breitwieser says swimming in a sports bra can cause the garment to break down much sooner. So while you can use one as a swim top, you're likely shortening its usable life.