It's fine, though it will make you more even more readily identifiable as a non-local than you already are so just make sure you have your wits about you while walking about. As much as I love yoga pants, I like to at least somewhat look the part of a local so I usually wear jeans and flats.