Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper investigates a simple extension of Gatys et al. CNN-based texture descriptors for image generation. Similar to Gatys et al., the method uses as texture descriptor the empirical intra-channel correlation matrix of the CNN feature response at some layer of a deep network. Differently from Gatys et al., longer range correlations are measured by introducing a shift between the correlated feature responses, which translates in a simple modification of the original architecture.

The idea is simple but has interesting effects on the generated textures and can be extended to transformations other than translation. While longer range correlations could be accounted for by considering the response of deeper CNN features in the original method by Gatys et al., the authors show that modelling them explicitly using shallower features is more effective, which is reasonable.

An important limitation that this work shares with most of its peers is the lack of a principled quantitative evaluation protocol, such that judging the effectiveness of the approach remains almost entirely a qualitative affair. While this should not be considered a significant drawback of the paper due to the objective difficulty of solving this open issue, nevertheless it is somewhat limiting that no principled evaluation method could be devised and implemented. The authors suggest that, as future work, a possible evaluation method could be based on a classification task -- this is a potentially interesting approach that merits some further investigation.

review_title: A good paper with some interesting insights in texture modelling by matching long-range statistics

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper introduces a variation to the CNN-based texture synthesis procedure of Gatys et al. that matches correlations between spatially shifted feature responses in addition to the correlations between feature responses at the same position in the feature maps. 
The paper claims that this 
a) improves texture synthesis for textures with long-range regular structures, that are not preserved with the Gatys et al. method
b) improves performance on texture inpainting tasks compared to the Gatys et al. method
c) improves results in season transfer when combined with the style transfer method by Gatys et al. 
Furthermore the paper shows that
d) by matching correlations between spatially flipped feature maps, symmetry properties around the flipping axis can be preserved.

I agree with claim a). However, the generated textures still have some issues such as greyish regions so the problem is not solved. Additionally, the procedure proposed is very costly which makes an already slow texture synthesis method substantially slower. For example, in comparison, the concurrent work by Liu et al. (http://arxiv.org/abs/1605.01141) that tackles the same problem by adding constraints to the Fourier spectrum of the synthesised texture shows comparable or better results while being far more efficient.

Also with b) the presented results constitute an improvement over the Gatys et al. method but again the results are not too exciting - one would not prefer this model to other inpainting algorithms.

With c) I don’t see a clear advantage of the proposed method to the existing Gatys et al. algorithm.

Finally, d) is a neat idea and the initial results look interesting but they don’t go much further than that. 

All in all I think it is decent work but neither its originality and technical complexity nor the quality of the results are convincing enough for acceptance.
That said, I could imagine this to be a nice contribution to the workshop track though.

review_title: Decent work but not good enough

final_score: rating: null
review_content: The paper investigates a simple extension of Gatys et al. CNN-based texture descriptors for image generation. The idea is to measure correlation statistics between shifted texture locations. This is obtained as a simple modification of Gatys et al. where a shift operation is introduced before the Gram matrices are computed.

The idea is a simple one but with interesting effects on the generated texture. It can also be extended to transformations other than translation, which is a nice bonus. However, I have a few questions that I would like to see discussed in the paper.

First, an indirect way of capturing longer range statistics is simply to compute the correlation between deeper CNN feature responses, as these would have automatically larger receptive fields. It would be nice to explain why measuring longer ranger correlations is preferable.

Second, there is a fundamental issue which makes evaluation of this and similar approaches difficult. Here the goal is to generate an image which resembles the reference. Crucially, however, the result must also be different from the input, as the trivial solution is to simply copy the reference image as is. In other words, the goal is to learn a generative model of the ``texture'', intended as a distribution over analogous images. This is clearly difficult, particularly when long range relationships are involved since the statistics of those is difficult to model from a single small texture sample. Can the authors describe what should be done, at lest in the ideal case, in order to say whether this and similar methods are good models of textures?



review_title: Significance of the method and assessing its quality

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a modification of the parametric texture synthesis model of Gatys et al. to take into account long-range correlations of textures. To this end the authors add the Gram matrices between spatially shifted feature vectors to the synthesis loss. Some of the synthesised textures are visually superior to the original Gatys et al. method, in particular on textures with very structured long-range correlations (such as bricks).

The paper is well written, the method and intuitions are clearly exposed and the authors perform quite a wide range of synthesis experiments on different textures.

My only concern, which is true for all methods including Gatys et al., is the variability of the samples. Clearly the global minimum of the proposed objective is the original image itself. This issue is partially circumvented by performing inpainting experiments, by which the synthesised paths needs to stay coherent with the borders (as the authors did). There are no additional insights into this problem in this paper, which would have been a plus.

All in all, this work is a simple and nice modification of Gatys at al. which is worth publishing but does not constitute a major breakthrough.



review_title: Simple modification of Gatys et al. that takes into account long-range correlations and gives visually pleasing synthesis results.

final_score: rating: null
review_content: The inpainting results with the Gatys et al. method look like there are some problems with the convergence of the optimisation. Could you please report the values of the Gatys et al. loss function for both, the result of your method as well as the result of the Gatys et al. method? 
I could imagine that the results of your method would also give a smaller loss for the loss function of the Gatys et al. method, which would indicate that a better optimiser for the Gatys et al. method might achieve comparable results.
review_title: Inpainting comparison with Gatys et al

Meta-review: The program committee appreciates the authors' response to concerns raised in the reviews. While there are some concerns about the computational speed of the approach as well as its advantage over existing methods for some textures, reviewers are excited by the ability of this work to produce structured texture that requires long-range interactions. Overall, the work has contributions that are worth presenting at ICLR.
Decision: Accept (Poster)