Synesthesia is obviously something of a test case for the modularity hypothesis.
Others have suggested that the cross-talk is prior to V1, taking place in the thalamus. Cytowic probably still sticks to his limbic system story. The scanning evidence suggests it is mid-hierarchy areas like V4. The evidence Baron-Cohen cites is the fact that infant primary cortex responds to noise as well as light. At least scanning studies seem to show different regions lighting up when you read or listen to the same words.