Picture perception reveals mental geometry of 3D scene inferences
Authors: Qasim Zaidi
Famya Baig
Erin Koch
Journal: Proceedings of the National Academy of Sciences of the United States of America (2018)
Discipline: Multidisciplinary
Keywords: 3D scene understanding; mental geometry; picture perception; pose estimation; projective geometry.
Affiliations: Graduate Center for Vision Research, College of Optometry, State University of New York, New York, NY 10036., Graduate Center for Vision Research, College of Optometry, State University of New York, New York, NY 10036 .

Abstract:  Pose estimation of objects in real scenes is critically important for biological and machine visual systems, but little is known of how humans infer 3D poses from 2D retinal images. We show unexpectedly remarkable agreement in the 3D poses different observers estimate from pictures. We further show that all observers apply the same inferential rule from all viewpoints, utilizing the geometrically derived back-transform from retinal images to actual 3D scenes. Pose estimations are altered by a fronto-parallel bias, and by image distortions that appear to tilt the ground plane. We used pictures of single sticks or pairs of joined sticks taken from different camera angles. Observers viewed these from five directions, and matched the perceived pose of each stick by rotating an arrow on a horizontal touchscreen. The projection of each 3D stick to the 2D picture, and then onto the retina, is described by an invertible trigonometric expression. The inverted expression yields the back-projection for each object pose, camera elevation, and observer viewpoint. We show that a model that uses the back-projection, modulated by just two free parameters, explains 560 pose estimates per observer. By considering changes in retinal image orientations due to position and elevation of limbs, the model also explains perceived limb poses in a complex scene of two bodies lying on the ground. The inferential rules simply explain both perceptual invariance and dramatic distortions in poses of real and pictured objects, and show the benefits of incorporating projective geometry of light into mental inferences about 3D scenes. 

Retracted: No

PubPeer comments:
 I. Koch et al s proposal regarding a functional property of the visual system has, first and foremost, the flaw of being impossible. Specifically, they suggest that the brain applies a u201cback-projectionu201d function to retinal images in order to arrive at a representation of the orientations of objects relative to the viewer, whether we are dealing with images projected from slanted pictures or otherwise. The basic  back-projection function is described as follows: u201cFor a camera elevation of u03d5C, a stick lying at the center of the ground plane with a pose angle of u03a9T uniquely projects to the orientation, u03b8S, on the picture plane.u201d u03a9T=atan(tan(u03b8R) sin(u03d5c)). The authors suggest that observers perform such a u201cback-projection.u201d The problem with this suggestion is that the information required to apply this equation is simply not available to the visual system. Our perception of the location and orientation of the ground plane, and of our own orientation relative to it, are constructed based on the retinal projection. The famous Ames room illusion works partly because viewers cannot perceive that the floor of the room is actually tilted. As also demonstrated by the well-known tilted room illusion, our sense of the position of our own body in relation to the gravitational vertical is constructed on the basis of the retinal projection, regardless of the match between that construct and physical reality. Trompe l oeil effects by which we can manipulate people s impression of the degree to which the ground is receding are possible precisely because of our inability u2013 a physical impossibility u2013 to determine the actual angle between our eye and the ground plane or objects lying upon it. Koch et al s observers were shown pictorial photographic images that produced retinal images that produced perceptual representations that were more or less consistent with the physical reality underlying the photos. This allowed them to fudge, or overlook, the distinction between the information available in the retinal projection emanating from the images and the information available to outside observers like themselves. If the photos had been of the Ames room, then this distinction would have been more clear u2013 it would have been more clear, in other words, that the orientation of the ground plane relative to the viewer is constructed from the retinal image and is not independently available information to be used as a tool in that construction. In this case, the true, physical values for the various angles to be inputted into the formula would *not* produce a match to the percept. In other words, we would not observe u201cthe similarity between perceived poses and the back-projection functionu201d reported here. It s all rather confusing, but worth clarifying in your mind if you want to study it. The key point is that to understand the relationship between the retinal projection and the percept, you have to address the projection itself without bringing in other, unavailable, external information about the actual physical reality. The correlation between the physical reality and the percept holds under certain conditions and doesn t hold under others. The correlation between the retinal projection and the percept is what is stable and principled. II. Prior to introducing their back-propagation proposal the authors seem to laying claim to an idea that didn t originate with them. Their significance statement, in its entirety, reads (bold mine): u201cWe show that, in both 3D scene understanding and picture perception, observers mentally apply projective geometry to retinal images. Reliance on the same geometrical function is revealed by the surprisingly close agreement between observers in making judgments of 3D object poses. These judgments are in accordance with that predicted by a back-projection from retinal orientations to 3D poses, but are distorted by a bias to see poses as closer to fronto-parallel. **Reliance on retinal images explains distortions in perceptions of real scenes, and invariance in pictures**, including the classical conundrum of why certain image features always point at the observer regardless of viewpoint. These results have implications for investigating 3D scene inferences in biological systems, and designing machine vision systems.u201d It was never in any doubt that visual perceptual processes depend on retinal images. Retinal stimulation constitutes the proximal stimulus. There is no other possibility. So to claim to have discovered this fact at the conclusion of the present study seems to me quite bizarre. It was also not in question that the perception of pictorial scenes viewed at a slant is a largely if not perfectly a function of the retinal image produced at that slant. Why, I myself have published a very [striking demonstration](http: illusionoftheyear.com 2010 05 stretching-out-in-the-tub ) of this fact! I have also analyzed it quite straightforwardly, if not quantitatively, in a chapter on that illusion published in the Oxford Compendium of Illusions (2017, Ch. 24). Although I didn t actually measure anything, my analysis relies on relative angles and on a symmetry constraint applied to the changing retinal projection; symmetry is a precise and effectively quantitative term. In that chapter I say: u201cThe bathtub photo is a particularly striking example of how changes in the retinal projection produced by changing our viewpoint [relative to the picture plane] can cause changes in the shapes of the perceived objects.u201d So this is not a new idea. An author cited both by me and Koch et al, Erkelens (2013) has also shown a similar effect. As I say in the chapter: u201cErkelens (2013)...[estimated] the shape of the retinal projection with changing viewpoint [and] concluded that what is perceived when we view a picture from various angles is a direct function of the shape of its retinal projection.u201d Koch et al acknowledge that u201cIn relying on internalized rules of perspective geometry, [Erkelens ] proposal is somewhat similar to oursu201d but add that u201cour idea of using back-projections from retinal images per se simplifies explanations. For example, the u201cfollowingu201d behavior of fingers pointing out of the picture, studied by Koenderink et al. (8), is explained in terms of highly slanted surfaces, such as the left and right sides of a finger, being perceived as almost independent of the slant of the screen, whereas we explain it simply as due to back-projections of vertical retinal orientations that stay invariant with observer translations or viewpoints.u201d What their example actually shows is that the authors u201csimplifyingu201d proposals are too simple. In explaining the 2D to 3D transition, they are attempting to separate the factor of orientation from the factor of shape. Yet the stimuli projections used clearly create the impression of volumes with planar sides (specifically, parallelipipeds) As clearly shown in the bathtub illusion, the changes in the retinal projections of such forms are a function of the shapes of these planes, affecting both perceived 3D shape and perceived orientation u2013 these two features are not separable any more than the front and back of a surface. Instead of acknowledging this shaped reality and its consequences, the authors side-step it by referring to their figures as u201csticks,u201d and, as seen in the passage quoted just above, considering only their orientation in the plane. Koch et al then suggest that their model could be applied to the question of shape: u201cOur model is formulated only for object poses, but the same projective geometry methods can be used to formulate back-projection models for object sizes along different axes. Informally, the body parts in Fig. 1 B and D appear distorted in size and aspect ratio, in proportion to the retinal projections, so it is likely that a back-projection model could provide a good explanation for perceived distortions in object shapes.u201d The suggestion that the u201cback-projection modelu201d could deal with perception of shape is implausible, given what was discussed in I. But aside from this, it s not clear how a model that attempts to use projective geometry to reconstruct the 3D physical object underlying the projection, would differ in principle from Erkelens proposal, or from other proposals, such as those discussed by Z. Pizlo in his book, 3D Shape. Neither of these authors proposals have been shown to have general applicability. There is no simple formula for achieving u201cback projectionu201d from retinal projection to 3D percept because as has long been understood, this involves min max principles and inferences difficult to describe and model, not simple geometric algorithms. You cannot apply projective geometry to the retinal image to infer shape precisely because of the lack of external information discussed here in connection with the orientation claims. , 
  As shown by the graph of equation 1 in SI Appendix, Fig. S1A, sticks pointing directly at or away from the observer along the line of sight (u03a9T= 90u00b0 or 270u00b0) always project to vertical (u03b8R= 90u00b0 or 270u00b0) in the retinal plane, while sticks parallel to the observer project to horizontal in the retinal plane. For intermediate pose angles, there is a periodic modulation around the unit diagonal. The relationships described above only hold if the head isn t tilted left or right. It often is tilted (e.g.  cocked ) without perceptibly altering our perception of the shape or slant of things, in real life or in pictures. The real-world-object-to-retinal projection assumptions being made for the purposes of this model don t seem to apply to the visual system. , 
 It probably should not be overlooked here that the problem with this study isn t a problem with the procedure as such, isn t a problem with the statistics, isn t a problem with the analysis of the data, isn t a problem of transparency or of replicability - I don t see a reason not to expect this study to replicate. It is a failure to really challenge, experimentally, logically or on the basis of existing knowledge, the hypothesis to be tested - whose  success,: in other words, is not virtually assured a priori. Please see https: psyarxiv.com n5mbf for a more complete discussion of the issues and problems raised by the claims in this paper. ,