During the time of Huangdi the spirit of earth prevailed and its colour was yellow hence the yellow earth and Yellow Emperor. During the time of King Wen of Zhou, the spirit of fire prevailed, presenting red colour. Fire would be replaced by water which would present black colour. The representative colours of the various seasons are the visual representations of the ecological environment. It makes sense to associate spring with green and wood as it is the season of growth of vegetation.
Summer is associated with red and fire, as it is the hottest season with maximum sunshine.
Associating autumn with white and metal needs a little more imagination. This probably had something to do with the withering of vegetation and transformation of a rich colourful world into whitish grey. Similarly the association of winter with water and black may be explained by the cold which forced people into hiding, thus bringing darkness. Besides this, black or other dark colours are used more often in winters. The historical reference in Lushi chungiu is also interesting.