It is okay to wear wrinkled linen. Some casual garments actually look better with wrinkles because it makes them look more comfortable and gives them character.