You might think this is just because the inverted representation is so different from the way you would have stored the visual image in your mind. So whatever ability we're using to encode, memorize, and retrieve facial images, it's more specific than the equivalent visual ability when applied to other kinds of objects. Children and adults rely on different strategies to memorize and recognize faces. Young children use a piecemeal strategy because they notice and store information about individual features such as eyebrows or a mole. Gradually they switch to a configurational strategy which processes the overall patterns formed by the features and their positions.
Adults use both strategies, but the configurational strategy seems to be particularly important.
Until they reach puberty children rely primarily on the piecemeal strategy. When they are shown photos of different faces and two are expressing the same emotion, for example, they tend to identify them as the same person. Extraneous paraphernalia such as eyeglasses and earrings also throw them off. Eventually the adult strategy of relying on the overall facial pattern wins out because it works better. When we see a face rightside-up we can immediately spot distortions, such as when the area between the eyes is stretched so the eyes are too far apart.