The sky will not be experienced by me as blue if I am on the dark hemisphere of the earth. What makes experience personal is that it is for this organism. One could say, by the same token, that my knowledge of physics is for my personal use and information, as is yours for you. In regard to both the inverted spectrum argument and the supposed privacy of pain, we could also consider scenarios where my brain is hooked up to inclusively more of the afferent pathways in your nervous system than just the pain receptors or the retina. The real difficulty posed by the apparent privateness of experience is not that it is a domain of privileged information, owing to hard connections between brain and body.
Information is rather a public domain.
What actually yields the sense of inviolable privacy is the possibility of deliberate misrepresentation that inheres in subjectivity, as well as in the ambiguities of language, so that the other is kept guessing as to one's inner state. Knowledge of other minds, like other types of knowledge, is a question of adequate models and of sensitivity to the information available. In poker, the cards are hidden, but not the faces and actions of the other players. The inside and outside perspectives are different modes of collecting the same information. The neurophysiologist's determination that my nerves and brain respond in certain patterns to light of a particular wavelength is the same information as my determination that I am seeing blue.