Other sighted people could come along. We already have the concept of pain or red. We don't disagree here. The same lack of guarantuee holds for public meaning. Well, there are rules for application of public language, but there aren't rules for the application of those rules.
Explanations come to an end somewhere.
The rules are applied in our conventions and practices and because of that we have language. As a matter of fact, if I show you various samples of a colour and then ask you to go in the same way you will generally carry on picking out things of the same or similar colour. If you didn't, and if people generally didn't then we wouldn't have the concept of colour. They don't stand in need of justification. The practice doesn't stand in need of justification.