It is a skill that would have been advantageous through mothers and children. I bet there are many more senses that will come to be known at a future time. This is a complex question because color affects our mood and the growth of plants, how insects thrive, many other life forms on the planet and it affects us too. Institutions like to paint their walls pale green which is supposed to be very soothing. Studies have shown that red is very exciting for people to see.
Some colors produce different effects.
Color has a cultural significance. The color that people wear for mourning varies enormously around the world, as does the color for wedding dresses. Color elicits different responses in many people. We don't like to play by the rules, and humans apply the different values of their cultures to what their senses record. I think I began to answer that in an earlier question.