Such people are called synesthetes. You probably did, you smart cookie. Like most creative people, I'm synesthetic, but only mildly. For me, numbers and letters don't have color, but color sometimes has flavor and music occasionally has color. Only about 1 in 2000 people are profoundly synesthetic.
Synesthetic interpretations are subjective and vary wildly between individuals.
These people haven't tapped into some unspoken language, it's just a biochemical brain glitch, an entertaining tangle in the wires.