Another way to experience this inside-outside interference is to copy a portrait. Copy the photo. Outline the features and add shading if you like. Then turn the picture upside down and copy it again. You'll probably find the second version is more accurate.
Eyes are particularly difficult.
We all have a simple mental schema for an eye. It's almost impossible to refrain from imposing this on the image. Yet as viewers we perceive the slightest nuance in an eye. The tiniest change to a highlight can turn a knowing gleam into a manic glare. Passport photos make most of us look pretty undesirable in the first place.