According to the Farmer's Almanac, the concept of not wearing white after Labor Day started in the 19th century as a way for the upper class to separate themselves from the working class.