Title: Filling in the details: Perceiving from low fidelity visual input
Authors: Farahnaz A. Wick
Michael L. Wick
Marc Pomplun
Venue: ICLR 2017
Keywords: Deep learning
Computer vision
Semi-Supervised Learning

Abstract: Humans perceive their surroundings in great detail even though most of our visual field is reduced to low-fidelity color-deprived (e.g., dichromatic) input by the retina. In contrast, most deep learning architectures deploy computational resources homogeneously to every part of the visual input. Is such a prodigal deployment of resources necessary? In this paper, we present a framework for investigating the extent to which connectionist architectures can perceive an image in full detail even when presented with low acuity, distorted input. Our goal is to initiate investigations that will be fruitful both for engineering better networks and also for eventually testing hypotheses on the neural mechanisms responsible for our own visual system's ability to perceive missing information. We find that networks can compensate for low acuity input by learning global feature functions that allow the network to fill in some of the missing details. For example, the networks accurately perceive shape and color in the periphery, even when 75\% of the input is achromatic and low resolution. On the other hand, the network is prone to similar mistakes as humans; for example, when presented with a fully grayscale landscape image, it perceives the sky as blue when the sky is actually a red sunset. 

TL;DR: Using generative models to create images from impoverished input similar to those received by our visual cortex

Review discussion:
Hi,
I have some questions regarding your paper:
- How does your model relate to state-of-the-art super resolution algorithms (e.g. https://arxiv.org/abs/1609.04802)? In your examples, you compare only to bilinear interpolation.
- What does the first row of Figure 3 show ("original")?
Thanks for the questions.  The paper you mention is indeed very impressive.
However, we do not compare to such systems as our goals are a bit different.
In particular, we're more interested in studying the behavior of the network under different types of degraded/foveated inputs in order to understand if and how the network copes with various stimuli. With this goal in mind, rather than employing a more complicated network engineered for super-resolution, we chose to study a simple network to facilitate interpretation of the results and the learnt mechanisms behind them.
Sorry for the confusion, "original" refers to the fact that the image is not foveated, that it is indeed the original image as it appears in the dataset.
Following up on the row "original" in Figure 3: Figure 3 does not show images at all, but weights of the autoencoder. Are these the weights of an autoencoder that you did train without any downsampling on the original images? If you use overcomplete representations, it seems surprising that this autoencoder did learn anything. Did you use some kind of regularization?
Yes, that is correct, the "original" row in Fig 3 corresponds to no downsampling.  We did not use any regularization (nor dropout).  The overcomplete AE could've learned the identify function and overfit, but perhaps the random weight initialization provides a bit of inductive bias to prevent this from occurring?  I don't think we have a bug, but will double check this evening when I have access to the code.
1. Why does the paper use AE in the framework?
2. What is the potential usage of the model? Can the model improve the performance of computer vision tasks, such as classification on ImageNet, Image retrieval, low-level image processing, etc? Is there any comparison on some specific tasks?
3. Is there any quantitative comparison with other attention-based models, such as Mnih 2014 https://papers.nips.cc/paper/5542-recurrent-models-of-visual-attention.pdf, Gregor 2015 https://arxiv.org/pdf/1502.04623.pdf, Zheng 2015 http://link.springer.com/article/10.1007/s11263-014-0765-x?
Thanks for your questions.
"Why does the paper use AE in the framework?"
Autoencoders (AEs) are able to take advantage of the immense amount of spatial structure in images, and each training image aggressively constrains the weights.
Further, the output of an AE can be visually inspected and interpreted.  Classification is a bit brutal in that it reduces a complex high-dimensional image to a single concept (label) and is thus unsuitable as a surrogate for "perceiving."  It is unclear whether similar global features would emerge in a classification setting (though it might be possible to transfer them from the AE).  We are not trying to disparage classification---it is a very useful task---rather we are merely pointing out that classification is not ideal for the problem we are studying.
"What is the potential usage of the model? Can the model improve the performance of computer vision tasks, such as classification on ImageNet, Image retrieval, low-level image processing, etc? Is there any comparison on some specific tasks?"
Good question.  We are interested in studying if and how the network copes with various input stimuli and what the nature of its solution is and how it behaves.
Thus, the primary use of the model is for scientific discovery/insight rather than engineering.
That said, one potential practical use is to save memory/bandwidth on the GPU.  The global features might allow us to transfer and store images with smaller memory footprints for different vision tasks.
However, we did not investigate this aspect in our paper.
"Is there any quantitative comparison with other attention-based models..."
Sorry for the confusion:  although we briefly discuss attention in the paper, it is outside the scope of this work.  In future work we would probably employ a mechanism similar to Mnih 2014 or one of the others you mention and study its behavior.
Can you provide more detail or comparisons on the significance of the different behaviors of the net under the different types and foveations?
At the moment, I feel this paper is lacking clear, concise takeaways.  MSE measurements seem to show mostly that (a) the auto-encoder is better than interpolation, and (b) larger input distortion amounts leads to higher reconstruction error.  What can we take away from the these experiments that the reader can use in their own work, for example?
For example, for a classification system (rather than autoencoder), what are the performance degradations under the different foveations?  Why?  How might this be used or taken advantage of for future systems?
The discussion on global features is the most interesting and relevant to a discussion like this, I feel, but even this could perhaps be studied in more detail:  Is it possible to do some sort of ablation to the images in addition to the foveations that illustrates the global features behavior, beyond a weight visualization?
Thanks for your engaging question, we are happy to elaborate on the results and conclusions. We agree that (a) and (b) are not the most interesting takeaways and are perhaps overemphasized due to their early appearance in the paper. A more important behavioral difference occurs between homogeneous (uniformly down-sampled) and heterogeneous (small fovea of high detail) foveations.  The difference is especially noticeable for color reconstruction.
Indeed, the network often struggles to reconstruct fully grayscale inputs (an example of a homogeneous foveation), producing mostly sepia tones (see Figure 6b "no color").
Yet, when given a small high-detail area of color, the network performs much better (see Figure 6b, "25% color").  Figure 6c quantifies (via MSE) the differences between these types of foveations:  the 25% color performs similarly to full color input, both of which performs twice as well as no color.  We find it interesting that a mixture of low and high detail input regions appears to be important for filling in details and it might be worth thinking about what type of inductive bias these heterogeneous inputs are imposing on the weights (something we have not yet explored).
A practical takeaway relevant to classification is that a heterogeneous down-sampling strategy might save space or bandwidth on the GPU (e.g., by discarding most, but not all of the color information).
While it is unclear whether a classification task provides enough constraints on the weights to learn the appropriate global features, it might be possible to reuse the features learnt by the autoencoder to save space on the GPU: although it requires more memory to store the weights for these features, the memory cost would be amortized over all the images in the dataset. Note however, that although our network is good at reconstructing color and shape information, it fails at texture.
Thus, if the classification task depends heavily on texture then the global features might be unreliable. We would expect classification degradation to depend on the type of classification problem.  Your suggestion on varying the input to evaluate global features is very intriguing, and we need to think a bit more about how to turn this into an evaluation.
We further hope that the phenomenon of the emergence of global features will eventually help hone in on theories of how the primate visual system performs perceptual filling-in.   Existing theories about the underlying neural mechanisms vary, but some, for example, rely on the existence of lateral connections between neurons.
Yet global features do not inherently require such connections and indeed our network lacks them.  It would be interesting to see if these simple networks exhibit similar behaviors as humans when presented controlled stimuli or optical illusions (see for example, www.uniformillusion.com for examples of filling-in illusions).
This paper is motivated by the ability that human's visual system can recognize contents of environment by from critical features, and tried to investigate whether neural networks can also have this kind of ability.
Specifically, the paper proposed to use Auto-Encoder (AE) as the network to reconstruct the low fidelity of visual input.
Moreover, similar to Mnih et al. (2014),  the paper also proposed to use a recurrent fashion to mimic the sequential behavior the  human visual system.
I think the paper is well motivated.
However, there are several concerns:
1. The baselines of the paper are too weak. Nearest neighbor, bilinear, bicubic and cubic interpolations without any learning procedure are of course performed worse than AE based models. The author should compare with the STOA methods such as https://arxiv.org/abs/1609.04802
2. Can the experiments based on AE support the idea that artificial neural networks can perceive an image from low fidelity? AE is only a kind of neural network, can the conclusion extend to other kind of networks? I think it would be much better if the authors can provide a more general conclusion.
Thanks for the comments.
1. It may be interesting to compare to a SOA super-resolution system on that particular task, but we would not expect our simple networks to compete with such systems (neural-network based or otherwise). We kept the network simple to more easily study their behavior (on various types of input degradations, not just low resolution) and the nature of the learnt solution.
2. Good question. We find AE to be a natural model for this problem because the output is easy to visually interpret. The idea could be easily extend to other kinds of autoencoders (with other types of layers or architectures), but it is less clear whether they could be directly extended to a classification architectures. The two problems here are (1) a lot of data would be required to provide enough constraints on the weights (2) how to know whether detail is being perceived or whether another signal is learned that happens to correlate with the classification problem.
I like the idea the paper is exploring.
Nevertheless I see some issues with the analysis:
- To get a better understanding of the quality of the results, I think at least some state-of-the-art comparisons should be included (e.g. by setting d times d pixel patches too their average and applying a denoising autoencoder). If they perform significantly better, then this indicates that the presented model is not yet taking all the information from the input image that could be used.
- SCT-R and FOV-R are supposed to test how much information can be restored from the Fovea alone as opposed to the Fovea together with low resolution periphery.
However, there is an additional difference between the two conditions: According to the paper, in SCT-R, part of the image was set to zero, while in FOV-R it was removed alltogether. With only one or two hidden layers, I could easily imagine this making a difference.
- On page 4, you compare the performance of FOV-R (1% error) with that of DS-D (1.5%) and attribute this to information about the periphery that the autoencoder extracts from the fovea.
While this might be the case, at least part of the reduced error will be due to the fact that the fovea is (hopefully) perfectly reconstructed. To answer the actual question "how much additional information about the periphery can be extracted from the fovea", you should consider calculating the error only in the periphery, i.e. the part of the image where DS-D and FOV-R got exactly the same input for.
Then any decreased error is only due to the additional fovea information.
Other issues:
- The images in Figure 2 (a) and (b) in the rows "factor 2", "factor 4", "factor 8" look very blurry. There seems some interpolation to be going on (although slighly different than the bilinear interpolation). This makes it hard to asses how much information is in these images. I think it would be much more insightfull to print them with "nearest" interpolation.
- Figure 3 caption too vague. Maybe add something like footnote 2?
- Often figures appear too early in paper which leads to lots of distance between text and figures.
Great suggestion about evaluating both the periphery and the fovea separately. As you suggest, this would provide more direct evidence that the high resolution areas are helping the low resolutions areas. And indeed, for some of our foveation functions it may be possible to tease apart the two regions and evaluate them separately.
Comparing structured noise to random noise (denoising AE) would be interesting as well.
RE: Other issues. We will integrate your suggestions, thanks.
This paper aims to characterize the perceptual ability of a neural network under different input conditions.  This is done by manipulating the input image x in various ways (e.g. downsamplig, foveating), and training an auto-encoder to reconstruct the original full-resolution image.  MSE and qualitative results are shown and compared for the different input conditions.
Unfortunately, this paper seems to lack focus, presenting a set of preliminary inspections with few concrete conclusions.
For example, at the end of sec 4.4, "This result is not surprising, given that FOV-R contains additional information .... These results suggests that a small number of foveations containing rich details might be all these neural networks need....".
But this hypothesis is left dangling:  What detailed regions are needed, and from where?  For what sort of tasks?
Secondly, it isn't clear to me what reconstruction behaviors are caused by a fundamental perception of the input, and what are artifacts of the autoencoder and pixelwise l2 loss?  A prime example is texture, which the autoencoder fails to recover.
But with a pixelwise loss, the network must predict high-frequency textures nearly pixel-for-pixel at training time; if this is impossible, then it will generate a pixelwise average of the training samples --- a flat region.
So then the network's inability to reconstruct textures is due to a problem generating them, specifically averaging from the training loss, not necessarily an issue in perceiving textures.  A network trained a different way (perhaps an adversarial network) may infer a texture is there, even if it wouldn't be able to generate it in a pixelwise l2 sense.
Similarly, the ability to perform color reconstruction given a color glimpse I think has much to do with disambiguating the color of an object/scene:  If there is an ambiguity, the network won't know which to "choose" (white flower or yellow flower?) and output an average, which is why there are so many sepia tones.
However, in its section on this, the paper only measures the reconstruction error for different amounts of color given, and does not drill very far into any hypotheses for why this behavior occurs.
There are some interesting measurements here, such as the amount of color needed in the foveation to reconstruct a color image, and the discussion on global features, which may start to get at a mechanism by which glimpses may propagate to an entire reconstruction.
But overall it's hard to know what to take away from this paper.  What are larger concrete conclusions that can be garnered from the details, and what mechanisms bring them about?  Can these be more thoroughly explored with more focus?
Thanks for your comments. We agree that there are some burning questions left unanswered, but at the same time hope that our initial data provides some interesting insights about a relatively unexplored area of research. As a first pass, we presented a breadth of problems to understand what a relatively simple network is capable of. It’s true that we did not explore each problem in depth but we wanted to gain an understanding on what happens under these three conditions: (1) foveations (2) missing information (scotomas) (3) presence or absence of color information in the fovea.
RE: MSE evaluation. Good point about texture.
Indeed MSE is not a perfect quantitative evaluation metric, but it would seem that nearly all other metrics have the same problems with texture. To help overcome MSE's deficiencies we also supply the actual output reconstructions so that they may also be qualitatively judged. Our conclusions regarding texture were not from an artifact of MSE, but from our own visual inspections of the images in which we determined much of the high frequency information in the periphery was clearly missing or inaccurate.
RE: ability to perform color reconstruction. We share your intuition about the sepia tones. There are some rare instances of the network learning to correlate shape and color for completely black and white foveation functions, but this was the exception not the norm. One case that is interesting is that the network colors a black&white sky as blue (actually this is incorrect because the sky happened to be a red sunset) by employing a learned global blue color feature. Again, this is a rare case and in general the network requires some color information, but we can only speculate on why.

Meta-review:
role: PC
data: comment: The program committee appreciates the authors' response to concerns raised in the reviews. Unfortunately, reviews are not leaning sufficiently towards acceptance. Reviewers find this direction of exploration to be interesting, but a bit preliminary at the moment. Authors are strongly encouraged to incorporate reviewer comments to make future iterations of the work stronger.

Decision: Reject