Author response: Image content is more important than Bouma’s Law for scene metamers
2019-02-27

Article Figures and data Abstract eLife digest Introduction Results Discussion Materials and methods Appendix 1 Appendix 2 Data availability References Decision letter Author response Article and author information Metrics Abstract We subjectively perceive our visual field with high fidelity, yet peripheral distortions can go unnoticed and peripheral objects can be difficult to identify (crowding). Prior work showed that humans could not discriminate images synthesised to match the responses of a mid-level ventral visual stream model when information was averaged in receptive fields with a scaling of about half their retinal eccentricity. This result implicated ventral visual area V2, approximated 'Bouma's Law' of crowding, and has subsequently been interpreted as a link between crowding zones, receptive field scaling, and our perceptual experience. However, this experiment never assessed natural images. We find that humans can easily discriminate real and model-generated images at V2 scaling, requiring scales at least as small as V1 receptive fields to generate metamers. We speculate that explaining why scenes look as they do may require incorporating segmentation and global organisational constraints in addition to local pooling. https://doi.org/10.7554/eLife.42512.001 eLife digest As you read this digest, your eyes move to follow the lines of text. But now try to hold your eyes in one position, while reading the text on either side and below: it soon becomes clear that peripheral vision is not as good as we tend to assume. It is not possible to read text far away from the center of your line of vision, but you can see 'something' out of the corner of your eye. You can see that there is text there, even if you cannot read it, and you can see where your screen or page ends. So how does the brain generate peripheral vision, and why does it differ from what you see when you look straight ahead? One idea is that the visual system averages information over areas of the peripheral visual field. This gives rise to texture-like patterns, as opposed to images made up of fine details. Imagine looking at an expanse of foliage, gravel or fur, for example. Your eyes cannot make out the individual leaves, pebbles or hairs. Instead, you perceive an overall pattern in the form of a texture. Our peripheral vision may also consist of such textures, created when the brain averages information over areas of space. Wallis, Funke et al. have now tested this idea using an existing computer model that averages visual input in this way. By giving the model a series of photographs to process, Wallis, Funke et al. obtained images that should in theory simulate peripheral vision. If the model mimics the mechanisms that generate peripheral vision, then healthy volunteers should be unable to distinguish the processed images from the original photographs. But in fact, the participants could easily discriminate the two sets of images. This suggests that the visual system does not solely use textures to represent information in the peripheral visual field. Wallis, Funke et al. propose that other factors, such as how the visual system separates and groups objects, may instead determine what we see in our peripheral vision. This knowledge could ultimately benefit patients with eye diseases such as macular degeneration, a condition that causes loss of vision in the center of the visual field and forces patients to rely on their peripheral vision. https://doi.org/10.7554/eLife.42512.002 Introduction Vision science seeks to understand why things look as they do (Koffka, 1935). Typically, our entire visual field looks subjectively crisp and clear. Yet our perception of the scene falling onto the peripheral retina is actually limited by at least three distinct sources: the optics of the eye, retinal sampling, and the mechanism(s) giving rise to crowding, in which our ability to identify and discriminate objects in the periphery is limited by the presence of nearby items (Bouma, 1970; Pelli and Tillman, 2008). Many other phenomena also demonstrate striking 'failures' of visual perception, for example change blindness (Rensink et al., 1997; O'Regan et al., 1999) and inattentional blindness (Mack and Rock, 1998), though there is some discussion as to what extent these are distinct from crowding (Rosenholtz, 2016). Whatever the case, it is clear that we can be insensitive to significant changes in the world despite our rich subjective experience. Visual crowding has been characterised as compulsory texture perception (Parkes et al., 2001; Lettvin, 1976) and compression (Balas et al., 2009; Rosenholtz et al., 2012a). This idea entails that we cannot perceive the precise structure of the visual world in the periphery. Rather, we are aware only of some set of summary statistics or ensemble properties of visual displays, such as the average size or orientation of a group of elements (Ariely, 2001; Dakin and Watt, 1997). One of the appeals of the summary statistic idea is that it can be directly motivated from the perspective of efficient coding as a form of compression. Image-computable texture summary statistics have been shown to be correlated with human performance in various tasks requiring the judgment of peripheral information, such as crowding and visual search (Rosenholtz et al., 2012a; Balas et al., 2009; Freeman and Simoncelli, 2011; Rosenholtz, 2016; Ehinger and Rosenholtz, 2016). Recently, it has even been suggested that summary statistics underlie our rich phenomenal experience itself—in the absence of focussed attention, we perceive only a texture-like visual world (Cohen et al., 2016). Across many tasks, summary statistic representations seem to capture aspects of peripheral vision when the scaling of their pooling regions corresponds to 'Bouma's Law' (Rosenholtz et al., 2012a; Balas et al., 2009; Freeman and Simoncelli, 2011; Wallis and Bex, 2012; Ehinger and Rosenholtz, 2016). Bouma's Law states that objects will crowd (correspondingly, statistics will be pooled) over spatial regions corresponding to approximately half the retinal eccentricity (Bouma, 1970; Pelli and Tillman, 2008; though see Rosen et al., 2014). While the precise value of Bouma's law can vary substantially even over different visual quadrants within an individual (see e.g. Petrov and Meleshkevich, 2011), we refer here to the broader notion that summary statistics are pooled over an area that increases linearly with eccentricity, rather than the exact factor of this increase (the exact factor becomes important in the paragraph below). If the visual system does indeed represent the periphery using summary statistics, then Bouma's scaling implies that as retinal eccentricity increases, increasingly large regions of space are texturised by the visual system. If a model captured these statistics and their pooling, and the model was amenable to being run in a generative mode, then images could be created that are indistinguishable from the original despite being physically different (metamers). These images would be equivalent to the model and to the human visual system (Freeman and Simoncelli, 2011; Wallis et al., 2016; Portilla and Simoncelli, 2000; Koenderink et al., 2017). Freeman and Simoncelli (2011) developed a model (hereafter, FS-model) in which texture-like summary statistics are pooled over spatial regions inspired by the receptive fields in primate visual cortex. The size of neural receptive fields in ventral visual stream areas increases as a function of retinal eccentricity, and as one moves downstream from V1 to V2 and V4 at a given eccentricity. Each visual area therefore has a signature scale factor, defined as the ratio of the receptive field diameter to retinal eccentricity (Freeman and Simoncelli, 2011). Similarly, the pooling regions of the FS-model also increase with retinal eccentricity with a definable scale factor. New images can be synthesised that match the summary statistics of original images at this scale factor. As scale factor increases, texture statistics are pooled over increasingly large regions of space, resulting in more distorted synthesised images relative to the original (that is, more information is discarded). The maximum scale factor for which the images remain indistinguishable (the critical scale) characterises perceptually-relevant compression in the visual system's representation. If the scale factor of the model corresponded to the scaling of the visual system in the responsible visual area, and information in upstream areas was irretrievably lost, then the images synthesised by the model should be indistinguishable while discarding as much information as possible. That is, we seek the maximum compression that is perceptually lossless: scrit⁢(I)=maxs:d⁢(I^s,I)=0⁡s, where scrit⁢(I) is the critical scale for an image I, I^s is a synthesised image at scale s and d is a perceptual distance. Larger scale factors discard more information than the relevant visual area and therefore the images should look different. Smaller scale factors preserve information that could be discarded without any perceptual effect. Crucially, it is the minimum critical scale over images that is important for the scaling theory. If the visual system computes summary statistics over fixed (image-independent) pooling regions in the same way as the model, then the model must be able to produce metamers for all images. While images may vary in their individual critical scales, the image with the smallest critical scale determines the maximum compression for appearance to be matched by the visual system in general, assuming an image-independent representation: ssystem=minI⁡scrit⁢(I) Freeman and Simoncelli showed that the largest scale factor for which two synthesised images could not be told apart was approximately 0.5, or pooling regions of about half the eccentricity. This scaling matched the signature of area V2, and also matched the approximate value of Bouma's Law. Subsequently, this result has been interpreted as demonstrating a link between receptive field scaling, crowding, and our rich phenomenal experience (e.g. Block, 2013; Cohen et al., 2016, Landy, 2013, Movshon and Simoncelli, 2014, Seth, 2014). These interpretations imply that the FS-model creates metamers for natural scenes. However, observers in Freeman and Simoncelli's experiment never saw the original scenes, but only compared synthesised images to each other. Showing that two model samples are indiscriminable from each other could yield trivial results. For example, two white noise samples matched to the mean and contrast of a natural scene would be easy to discriminate from the scene but hard to discriminate from each other. Furthemore, since synthesised images represent a specific subset of images, and the system critical scale ssystem is the minimum over all possible images, the ssystem estimated in Freeman and Simoncelli (2011) is likely to be an overestimate. No previous paper has estimated ssystem for the FS-model using natural images. Wallis et al., 2016 tested the related Portilla and Simoncelli (2000) model textures, and found that observers could easily discriminate these textures from original images in the periphery. However, the Portilla and Simoncelli model makes no explicit connection to neural receptive field scaling. In addition, relative to the textures tested by Wallis et al., 2016, the pooling region overlap used in the FS-model provides a strong constraint on the resulting syntheses, making the images much more similar to the originals. It is therefore still possible that the FS-model produces metamers for natural scenes for scale factors of 0.5. Results Measuring critical scale in the FS-model We tested whether the FS-model can produce metamers using an oddity design in which the observer had to pick the odd image out of three successively shown images (Figure 1E). In a three-alternative oddity paradigm, performance for metamerism would lie at 1/3 (dashed horizontal line, Figure 1F). We used two comparison conditions: either observers compared two model syntheses to each other (synth vs synth; as in Freeman and Simoncelli, 2011) or the original image to a model synthesis (orig vs synth). As in the original paper (Freeman and Simoncelli, 2011) we measured the performance of human observers for images synthesised with different scale factors (using Freeman and Simoncelli's code, see Materials and methods). To quantify the critical scale factor we fit the same nonlinear model as Freeman and Simoncelli, which parameterises sensitivity as a function of critical scale and gain, but using a mixed-effects model with random effects of participant and image (see Materials and methods). Figure 1 with 1 supplement see all Download asset Open asset Two texture pooling models fail to match arbitrary scene appearance. We selected ten scene-like (A) and ten texture-like (B) images from the MIT 1003 dataset (Judd et al., 2009, https://people.csail.mit.edu/tjudd/WherePeopleLook/index.html) and synthesised images to match them using the Freeman and Simoncelli model (FS scale 0.46 shown) or a model using CNN texture features (CNN 32; example scene and texture-like stimuli shown in (C) and (D) respectively). Images reproduced under a CC-BY license (https://creativecommons.org/licenses/by/3.0/) with changes as described in the Methods. (E): The oddity paradigm. Three images were presented in sequence, with two being physically-identical and one being the oddball. Participants indicated which image was the oddball (1, 2 or 3). On 'orig vs synth' trials participants compared real and synthesised images, whereas on 'synth vs synth' trials participants compared two images synthesised from the same model. (F): Performance as a function of scale factor (pooling region diameter divided by eccentricity) in the Freeman-Simoncelli model (circles) and for the CNN 32 model (triangles; arbitrary x-axis location). Points show grand mean ±2 SE over participants; faint lines link individual participant performance levels (FS-model) and faint triangles show individual CNN 32 performance. Solid curves and shaded regions show the fit of a nonlinear mixed-effects model estimating the critical scale and gain. Participants are still above chance for scene-like images in the original vs synth condition for the lowest scale factor of the FS-model we could generate, and for the CNN 32 model, indicating that neither model succeeds in producing metamers. (G): When comparing original and synthesised images, estimated critical scales (scale at which performance rises above chance) are lower for scene-like than for texture-like images. Points with error bars show population mean and 95% credible intervals. Triangles show posterior means for participants; diamonds show posterior means for images. Black squares show critical scale estimates of the four participants from Freeman and Simoncelli (2011) (x-position jittered to reduce overplotting); shaded regions denote the receptive field scaling of V1 and V2 estimated by Freeman and Simoncelli (2011). Data reproduced from Freeman and Simoncelli (2011) using WebPlotDigitizer v. 4.0.0 (Rohatgi, A., software under the GNU Affero General Public License v3, https://www.gnu.org/licenses/agpl-3.0.en.html). https://doi.org/10.7554/eLife.42512.003 We used 20 images to test the FS model. These images are split into two classes of ten images each, which we labelled 'scene-like' and 'texture-like'. The distinction of these two classes is based on the results of a pilot experiment with a model we developed, which is inspired by the FS model but based on a different set of image features (those extracted by a convolutional neural network; see Materials and methods and Appendix 2—figure 1). In this pilot experiment, we found that some images are easier to discriminate than others (Appendix 2—figure 7—figure 9). Easily-discriminable images tended to contain larger areas of inhomogenous structure, long edges, borders between different surfaces or objects, and angled edges providing perspective cues ('scene-like'). Difficult images tended to contain more visual textures: homogenous structure, patterned content, or materials ('texture-like'"). For example, images from the first class tended to contain more structure such as faces, text, skylines, buildings, and clearly segmented objects or people, whereas images from the second class tended to contain larger areas of visual texture such as grass, leaves, gravel, or fur. A similar distinction could also be made along the lines of 'human-made' versus 'natural' image structure, but we suspect the visual structure itself rather than its origin is of causal importance and so used that level of description. While our labelling of images in this way is debatable (for example, 'texture-like' regions contain some 'scene-like' content and vice versa) and to some degree based on subjective judgment, we hypothesised that this classification distinguishes the types of image content that are critical. If the visual system indeed created a texture-like summary in the periphery and the FS-model was a sufficient approximation of that process, then we should observe no difference in the average critical scale factor of images in each group (because image content would be irrelevant to the distribution of scrit⁢(I)). We start by considering the condition where participants compared synthesised images to each other—as in Freeman and Simoncelli (2011). Under this condition, there was little evidence that the critical scale depended on the image content (see curves in Figure 1F, synth vs synth). The critical scale (posterior mean with 95% credible interval quantiles) for scene-like images was 0.28, 95% CI [0.21, 0.36] and the critical scale for texture-like images was 0.37, 95% CI [0.27, 0.5] (Figure 1G). Though these critical scales are lower than those reported by Freeman and Simoncelli (2011), they are within the range of other reported critical scale factors (Freeman and Simoncelli, 2013). There was weak evidence for a difference in critical scale between texture-like and scene-like images, with the posterior distribution of scale differences being 0.09, 95% CI [−0.03, 0.24], p(β<0)=0.078 (where p(β<0) is the posterior probability of the difference being negative; symmetrical posterior distributions centered on zero would have p(β<0)=0.5). However, this evidence should be interpreted cautiously: because asymptotic performance never reaches high values, critical scale estimates are more uncertain than in the orig vs synth condition below (Figure 1G). This poor asymptotic performance may be because we used more images in our experiment than Freeman and Simoncelli, so participants were less familiar with the distortions that could appear. To make sure this difference did not arise due to different experimental paradigms (oddity vs. ABX), we repeated the experiment using the same ABX task as in Freeman and Simoncelli (Appendix 1—figure 4). This experiment again showed poor asymptotic performance, and furthermore demonstrated no evidence for a critical scale difference between the scene- and texture-like images. Taken together, our synth vs synth results are somewhat consistent with Freeman and Simoncelli, who reported no dependency of scrit⁢(I) on image. It seems likely that this is because comparing synthesised images to each other means that the model has removed higher-order structure that might allow images and the task becomes one of a specific the original image to model syntheses a different pattern of results. participants were able to discriminate the original images from their FS-model syntheses at scale factors of (Figure 1F). Performance above chance for all This result for scene-like and texture-like images. there was evidence that critical scale depended on the image syntheses matched the texture-like images on average with scale factors of 95% CI In the scene-like images were from their model syntheses even at the smallest scale we could generate The critical scale estimated for scene-like images was 95% CI images had critical scales than scene-like images on average (scale difference 95% CI This difference in critical scale was not to differences in the of the synthesis between scene-like and texture-like images. images had loss between the original and synthesised images in model than texture-like images on average (see Materials and methods). This is a of the importance of image since a texture summary model is a poor of scene-like content, the is also more likely to find local with high We that our result was not by this difference by a in which we the model the average loss in the two groups by images with loss the groups were matched in four scene-like images being see Materials and methods). The scene-like images had a critical scale of 95% CI in the orig vs synth condition, texture-like images again showed a critical scale of 95% CI and the difference distribution had a mean of 95% CI differences in synthesis loss do not our As the image with the minimum critical scale determines the largest compression that can be for the scaling model to hold For two images (Figure and the nonlinear mixed-effects model estimated critical scales of approximately (see Figure the minimum critical scale images in the reported above was However, the individual data for these images (Figure and that these critical scale estimates are by the of the mixed-effects model, not the data images were easy to discriminate from the original even for the lowest scale factor we could This suggests that the scale factor to generate metamers may be even lower than estimated by the mixed-effects model. Figure 2 with 1 supplement see all Download asset Open asset The two images with smallest critical scale estimates are even for the lowest scale factor we could (A) The original image. (B) example FS synthesis at scale factor (C) example FS synthesis at scale factor Images in and reproduced from the MIT 1003 (Judd et al., https://people.csail.mit.edu/tjudd/WherePeopleLook/index.html) under a CC-BY license (https://creativecommons.org/licenses/by/3.0/) with changes as described in the Methods. (D) The average data for this image. Points and error bars show grand mean and ±2 SE over and shaded area show posterior mean and 95% credible from the mixed-effects model. text posterior mean and 95% credible interval on the critical scale for this image. as for the image with the critical that in the model is likely to critical Our results show that pooling regions are to make metamers for scene-like images than for texture-like images. observers can small distortions by the FS-model at scale factors of in scene-like image content Figure and at scale and and at scale 0.46 to images A and syntheses at these scales are not metamers for natural scenes. image structure determines the of texture-like distortions In our first experiment we found that scene-like images lower critical scales than texture-like images. However, this is in photographs of natural scenes is a of local regions of the image rather than the image as a In addition, the classification of images above was based in on the of these images in a pilot We therefore a second experiment to test the importance of local image structure more directly Koenderink et al., et al., Wallis and Bex, using a set of images was not based on pilot results. Participants a texture-like by the texture model of et al., into either a scene-like or texture-like region (Figure These image regions were by author showed high with this Materials and methods). The were centered at an eccentricity of and we the of the (Figure This is to summary statistics in a pooling region et al., 2016). Participants between the original image and an image a local in a (Figure Figure Download asset Open asset to local texture distortions on image (A) A of an image was with a texture-like In different experimental the of the was (B) Two example images in which a 'scene-like' or inhomogenous region is distorted (C) Two example images in which a 'texture-like' or homogenous region is distorted (D) of an original image and the four used in the Images in reproduced from the MIT 1003 (Judd et al., https://people.csail.mit.edu/tjudd/WherePeopleLook/index.html) under a CC-BY license (https://creativecommons.org/licenses/by/3.0/) with changes as described in the Methods. of the in which the observer reported whether the first or second image the as a function of in scene-like and texture-like image link the performance of each observer based on a of Points show mean of observer error bars show ±2 The results showed that the of texture-like distortions depended on the image Participants were insensitive to even large texture-like distortions in texture-like image regions (Figure Performance for distortions of the was still to scene-like regions is for the three largest Discussion It is a idea that the appearance of scenes in the periphery is described by summary statistic textures captured at the scaling of V2 neural In here we show that humans are to the difference between original and images at this scale (Figure 1). A et al., a similar result in a set of images, and our results are also consistent with the made by Wallis et al. based on their with Portilla and Simoncelli textures et al., 2016). these results show that the pooling of texture-like features in the FS-model at the scaling of V2 receptive fields does not the appearance of natural images. One of Freeman and Simoncelli (2011) was the of a critical brain region a receptive field size from aspects of this have since the presence of texture-like features can discriminate V2 from V1 (Freeman et al., 2013; et al., 2016; see also et al., all higher-order structure not captured by the model by comparing syntheses to each only features that may be a way to distinguish the of in While texture-like representations may therefore be important for neural and Simoncelli, our results into the link between receptive field scaling and scene appearance. If the peripheral appearance of visual scenes is by image-independent pooling of texture-like then the pooling regions must be that participants in our experiment could easily discriminate the images in Figure and from those in Figure and images synthesised at a scaling must remain to the ssystem must be at least as small as V1 and even lower (Figure This may even be consistent with scaling in visual For example, the scaling of retinal receptive fields at the average eccentricity of our stimuli is approximately for the and and for the