Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: After a first manuscript that needed majors edits, the revised version
offers an interesting GAN approach based the scattering transform.

Approach is well motivated with proper references to the recent literature.

Experiments are not state of the art but clearly demonstrate that the
proposed approach does provide meaningful results.
review_title: interesting and original insights on VAE/GAN with the scattering transform

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
The paper proposes a generative model for images that does no require to learn a discriminator (as in GAN’s) or learned embedding. The proposed generator is obtained by learning an inverse operator for a scattering transform.

The paper is well written and clear. The main contribution of the work is to show that one can design an embedding with some desirable properties and recover, to a good degree, most of the interesting aspects of generative models. However, the model doesn’t seem to be able to produce high quality samples. In my view, having a learned pseudo-inverse for scattering coefficients is interesting on its own right. The authors should show more clearly the generalization capabilities to test samples. Is the network able to invert images that follow the train distribution but are not in the training set?

As the authors point out, the representation is non-invertible. It seems that using an L2 loss in pixel space for training the generator would necessarily lead to blurred reconstructions (and samples) (as it produces a point estimate). Unless the generator overfits the training data, but then it would not generalize. The reason being that many images would lie in the level set for a given feature vector, and the generator cannot deterministically disambiguate which one to match. 

The sampling method described in Section 3.2 does not suffer from this problem, although as the authors point out, a good initialization is required. Would it make sense to combine the two? Use the generator network to produce a good initial condition and then refine it with the iterative procedure.

This property is exploited in the conditional generation setting in:

Bruna, J. et al "Super-resolution with deep convolutional sufficient statistics." arXiv preprint arXiv:1511.05666 (2015).

The samples produced by the model are of poorer quality than those obtained with GAN’s. Clearly the model is assigning mass to regions of the space where there are not valid images.  (similar effect that suffer models train with MLE). Could you please comment on this point?

The title is a bit misleading in my view. “Analyzing GANs” suggests analyzing the model in general, this is, its architecture and training method (e.g. loss functions etc). However the analysis concentrates in the structure of the generator and the particular case of inverting scattering coefficients.

However, I do find very interesting the analysis provided in Section 3.2. The idea of using meaningful intermediate (and stable) targets for the first two layers seems like a very good idea. Are there any practical differences in terms of quality of the results? This might show in more complex datasets.

Could you please provide details on what is the dimensionality of the scattering representation at different scales? Say, how many coefficients are in S_5?

In Figure 3, it would be good to show some interpolation results for test images as well, to have a visual reference.

The authors mention that considering the network as a memory storage would allow to better recover known faces from unknown faces. It seems that it would be known from unknown images. Meaning, it is not clear why this method would generalize to novel image from the same individuals. Also, the memory would be quite rigid, as adding a new image would require adapting the generator.

Other minor points:

Last paragraph of page 1, “Th inverse \Phi…” is missing the ‘e’.

Some references (to figures or citations) seem to be missing, e.g. at the end of page 4, at the beginning of page 5, before equation (6). 

Also, some citations should be corrected, for instance, at the end of the first paragraph of Section 3.1: 

“… wavelet filters Malat (2016).” 

Sould be:

“... wavelet filters (Malat, 2016).” 

First paragraph of Section 3.3. The word generator is repeated.

review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors introduce scattering transforms as image generative models in the context of Generative Adversarial Networks and suggest why they could be seen as Gaussianization transforms with controlled information loss and invertibility.
Writing is suggestive and experimental results are interesting, so I clearly recommend acceptation. 

I would appreciate more intuition on some claims (e.g. relation between Lipschitz continuity and wavelets) but they refer to the appropriate reference to Mallat, so this is not a major problem for the interested reader.

However, related to the above non-intuitive claim, here is a question on a related Gaussianization transform missed by the authors that (I feel) fulfils the conditions defined in the paper but it is not obviously related to wavelets. Authors cite Chen & Gopinath (2000) and critizise that their approach suffers from the curse of dimensionality because of the ICA stage. However, other people [Laparra et al. Iterative Gaussianization from ICA to random rotations IEEE Trans.Neural Nets 2011] proved that the ICA stage is not required (but only marginal operations followed by even random rotations). That transform seems to be Lipschitz continuous as well -since it is smooth and derivable-. In fact it has been also used for image synthesis. However, it is not obviously related to wavelets... Any comment?

Another relation to previous literature: in the end, the proposed analysis (or Gaussianization) transform is basically a wavelet transform where the different scale filters are applied in a cascade (fig 1). This is similar to Gaussian Scale Mixture  models for texture analysis [Portilla & Simoncelli Int. J. Comp. Vis. 2000] in which after wavelet transform, local division is performed to obtain Gaussian variables, and these can be used to synthesize the learned textures. That is similar to Divisive Normalization models of visual neuroscience that perform similar normalization alfter wavelets to factorize the PDF (e.g. [Lyu&Simoncelli Radial Gaussianization Neur.Comput. 2009], or [Malo et al. Neur.Comput. 2010]).

Minor notation issues: authors use a notation for functions that seems confusing (to me) since it looks like linear products. For instance: GZ for G(Z) [1st page] and phiX for phi(X) [2nd page] Sx for S(x) [in page 5]... 

review_title: Authors generate images from Gaussian noise using scattering networks. Interesting analysis of Gaussianization transforms constrained to be continuous to deformations -> Accept

Meta-review: The paper got mixed scores of 4 (R1), 6 (R3), 8 (R2). R1 initially gave up after a few pages of reading, due to clarity problems. But looking over the revised version was much happier, so raised their score to 7. R2, who is knowledge about the area, was very positive about the paper, feeling it is a very interesting idea. R3 was also cautiously positive. The authors have absorbed the comments by the reviewers to make significant changes to the paper. The AC feels the idea is interesting, even if the experimental results aren't that compelling, so feels the paper can be accepted.

Decision: Accept (Poster)