While sweater dresses for ladies especially the oversized options are best worn during the cold season to provide better warmth and comfort, you can make them work for you in summer or spring. Hence, there is no specific time of the year to start wearing sweater dresses.