Venue: ICLR 2017 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The framework of Gatys et al. demonstrated that correlation statistics (empirical Gram matrices) of deep feature responses provide an excellent characterisation of visual textures. This paper investigates in detail which kind of deep or shallow networks may work well in this framework. One of the main findings is that that very shallow nets, consisting of a single filter bank with random weights, work surprisingly well, and for simple and regular textures may produce results which are visually superior to complex data-adapted filters such as the ones in networks like VGG-19. More broadly, the paper contains an interesting and informative discussion on the strength and limitations on such methods for texture synthesis.

Figure 4 shows that the optimisation of images with respect to shallow filter banks may result in texture images that have a lower VGG-19 loss than optimising the VGG-19 objective directly. This is imputed to the difficulty of optimising the highly non-linear VGG-19 cost function, which is a reasonable explanation. In the new supplementary material, the authors show that better optimisation results can be obtained by initialising the VGG-19-based optimisation with the shallow network optimisation results, which is a useful complement to the original experiments.

The main limitation of the paper is that it does not systematically compare different methods against a quantifiable objective. It is trivial to define image statistics that would allow to simply generate an exact copy of any reference texture, hence with very good visual quality. Such trivial statistics would also be very shallow. The aim is instead to capture a texture distribution, and measuring how well a method meets this challenge remains an open problem. Hence, while the empirical results seem to confirm the intuition that simple statistics are good enough for texture synthesis both in terms of quality and diversity (when compared to more complex statistics), it is difficult to conclusively confirm that this is the case.

The authors indicate that diversity could be measured in terms of entropy. This is reasonable, but, as they acknowledge in their answers to questions, difficult to do in practice. Furthermore, this would still not account for the other aspect of the problem, namely visual quality. They also suggest to perform a psychophysical assessment, which may be the only practical way of addressing this problem, but deem that to be material for future work.

Overall, since evaluation of image generation is such an hard problem, I think the paper still has sufficient strengths to warrant publication in ICLR. Still, some form of psychophysical assessment would be useful to confirm the intuitions that, at present, can only be obtained by inspecting the figure sin the paper and in the supplementary material.




review_title: A good paper with interesting empirical insights on texture synthesis

final_score: rating: null
review_content: I will post my thoughts in the final review, but other reviewers had already asked most of what I need for clarifications.
review_title: No further questions

final_score: rating: 7: Good paper, accept
review_content: This work proposed a simple but strong baseline for parametric texture synthesis. In empirical experiments, samples generated by the baseline composed by multi-scale and random filters sometime rival the VGG-based model which has multi-layer and pre-trained filters. The authors concluded that texture synthesis does not necessarily depend on deep hierarchical representations or the learned feature maps. 
This work is indeed interesting and insightful. However, the conclusions are needed to be further testified (especially for deep hierarchical representations). Firstly, all of generated samples by both VGG and single layer model are not perfect and much worse than the results from non-parametric methods.  Besides VGG-based model seems to do better in inpainting task in Figure 7. Last but not least, would a hierarchical model (instead of lots of filters with different size) handle multi-scale more efficiently? 
review_title: An interesting work with uncertain conclusions needed to be further testified.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper provides an interesting analysis of the conditions which enable generation of natural looking textures. The results is quite surprising, and analysis is quite thorough. 
I do think the evaluation methods require more work, but as other reviewers mentioned this could be an interesting line of work moving forwards and does not take too much from this current paper which, I think, should be accepted.
review_title: Important work which deserves 

final_score: rating: null
review_content: The framework of Gatys ea al. shows that correlation statistics (empirical gram matrices) of deep feature responses provides an excellent characterisations of visual texture. This paper investigates in detail which kind of deep (or shallow) features may work well in this framework. 

One of the main results is that that very shallow networks, consisting of a number of parallel one-layer filter banks with random weights, work surprisingly well, and for simple and regular textures tend to produce results which are visually better than using very complex data-adapted filters such as the ones in VGG-19.

The contribution is a nice one, interesting due to its fairly surprising findings, and possibly more interesting due to the caveats that it explicitly and implicitly raises about such methods for texture synthesis.

I have a few questions for the authors. Figure 4 show that direct optimisation of shallow filters results in texture images that have a lower VGG-19 loss. This is imputed to the difficulty of optimising the highly non-linear VGG-19, which is a reasonable explanation. In fact, if by optimising the simpler statistics we can obtain a copy of the input texture, obviously the VGG-19 loss would also be zero. However:

- The optimisation algorithm is local, so it is possible that, by restarting the VGG-19 optimisation, a result with a lower energy could be obtained. Is this possible?

- As discussed in detail in the last section of the paper, a texture method that just copies the reference texture image is useless; the aim is instead to draw "fair" samples from the texture intended as an image distribution. Is it possible that by fitting the energy function less, the texture sampling properties are actually better?

- Connected to the latter point, since it is clear that only reproducing an image statistics is meaningless (as the best possible result is the original image), can the authors discuss how this family of texture generation methods should, at least in principle, be tested?

- Several filter types are compared (Fourier, random, etc). However, the differences are not discussed very explicitly. The main conclusion is random filters are, sometimes, just as good as filters trained from data. What about e.g. the Fourier filters?




review_title: Clarifications and discussion about evaluation methodology

final_score: rating: null
review_content: The texture evaluation (described in eq. 6) could not measure the perceptual quality for me. Some generated textures looks broken (the bottom-right in Figure 1), which were seen as an successful example. Why not do comparison on texture inpainting like in [1] and you could use structural similarity index [2] to measure the perceptual quality more properly? 

[1] Heess, N., Williams, C. K. I. and Hinton, G. E.Learning generative texture models with extended Fields-of-Experts.
[2] Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE TRANSACTIONS ON IMAGE PROCESSING, 13(4), 600–612.
review_title: The used method of texture evaluation seems to be suspicious. Why not try compare different networks on texture inpainting task?

Meta-review: The authors agreed that the paper presented a solid contribution and interesting (and somewhat surprising findings). The experiments are thorough and convincing, and while some reviewers raised concerns about a lack of comparisons of methods on a clear quantifiable objective, this is unfortunately a common issue in this field. Overall, this paper is a solid contribution with findings that would be interesting to the ICLR audience.
Decision: Accept (Poster)