Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Fractal compression represents data (for example an image) as the fixed point of a parameterized operator. Encoding the image requires finding the parameters of the operator (typically done via an explicit search algorithm), while decoding the image requires iterating the operator to recover the fixed point.

This paper proposes neural collages, which as in fractal compression represent data as the fixed point of a parameterized map, but uses hypernetworks to amortize the cost of finding these parameters.

The paper introduces the method, and uses it as the decoder for deep generative models, for image compression, and to generate fractal art.
strengths_and_weaknesses: - The core idea is original, and an interesting way of leveraging neural networks for fractal image compression.
- The applications nicely demonstrate the wide applicability of the method.
- The presentation of the method could be clearer, with more intuition to accompany the idea, and some clearer distinction between what is prior background vs. new ideas.
questions: - What was the architecture used for the hypernetwork? Is there any intuition or heuristics for how to design that architecture? I don't have any intuition for what one might expect to work there.
- Could you add a little more detail about the auxiliary domains? How much do those affect the overall performance?
limitations: yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies using learning algorithms to improve fractal compression. The fractal compression leverages self-similarities to compress the data, which happens in many common signals, such as images. However, the expressiveness of the compression is constrained by the model families it uses. Traditional methods use affine transformations, which is limited. This paper consider a learning-based method, which they call Collage. They also propose a hypernetwork to tackle the potential optimization difficulties. 

strengths_and_weaknesses: # Strong 

A new learning- based method to improve a classic compression algorithm with significant performance improvements as shown numerically and visually in the experiments. 

# Weak

- The paper is not easy to follow to be honest as a paper in a machine learning venue. Although the authors spend quite an amount of effort describing fractal compression in Section 2, and covers some theoretical results. In the end, I have to read other papers first to understand the idea.  I would suggest the author to provide more illustrative examples to guide readers not in compression background to understand the task and shortens some mathematical properties by moving to the appendix, as they are proven by previous literature and not really mentioned in the main algorithm design.  I want to note that I’m criticizing the paper working on applications from other domains. It would be better if the authors make the problem more clear to the readers as it is in a ML venue not in the signal processing venue.   This comment might be biased as I’m not an expert in this domain. 

- It would be nice to provide more methodology details. For example, what is J here in the final equation which depends on d_H, also what’s the hypernetwork architecture and the output .. etc. Providing more algorithm details would greatly help reader to capture the idea of the proposed method.  

- The related works are shortened too much. It would be great if the authors provide a better overview of the recent progress of learning-based compression methods in the related works. For example, in addition to the very first NeRF paper, there are many great follow-up works worth mentioning.  A suggestion is the same as above by moving some non-necessary results from the prior works to the appendix to gain more space. 

questions: - It seems we need some conditions to ensure the convergence when solving the optimization problem. However, I don’t see it to be explicitly imposed in the algorithm design. Do I miss it in the paper anywhere?

- When solving the optimization problem below line 228, usually how many iterations do we need to get z*? And if the iteration is a lot, would that create any issues? (i.e. RNN with many time stamps.) 

- There are many algorithms proposed to use a single feedforward pass of a neural network to solve different optimization problems. However, many problems are not really solved well by neural networks. How about the case of the hypernetwork here? 


limitations: No. The authors mention in Appedix B.2, however, there is only Appendix B.1.  Also, it would be great to discuss the limitation too. As it's a new attempt on this compression method, I tend to believe there are still rooms to improve.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a collage function which is parameterized to be able to generate variable-resolution images according to a fractal pattern.  To be able to handle datasets/distributions, this collage function is parameterized using a hypernetwork which takes the entire example as its input.  This is used to achieve improved image compression and variable-resolution image generation.  

Notes: 

  -Natural patterns described using "self-similarity", self-referential transformation.  

  -Represent data with self-referential transform, in single forward pass.  

  -Collage VAE can generalize over multiple possible image resolutions.  

  -Given an image z, iterate fixed point map: z[t+1] = A(w)z[t] + b(w)

  -Constrain collage function to be a contraction.  

  -We can fit this separately for each image, or use hypernet based on a full-image encoder.  

strengths_and_weaknesses: Strengths: 

  -The ability to generate with dynamics resolutions is an interesting property.  

  -The improvements on image compression are nice but the baselines all use either implicit representations, fractals, or non-neural (if I read it correctly).  

  -The subject of the paper is fairly creative and seems unique.  

Weakness:   

-(Minor): In Figure 1, it might be more compelling to give a natural machine learning example on the top row.  Although I suppose the fact that the fractal structure is exact is perhaps worth something.  

  -(Minor): In Figure 5, the red/blue outlines around the sub-images should be much thicker.  Many readers will miss it currently.  

  -Constructing the parameters of the collage function F may be a lot to ask an encoder network to do.  If I understand it correctly, this means compressing the entire object's information into a single vector (or a small set of vectors, independent of the size of the object) which is used to parameterize F?  

  -The experimental strategy is a bit odd to me.  It seems like the best result to show would be that the method improves the generative modeling of data with a clear fractal structure (either hand-designed or natural).  Then, it would be interesting to show that other deep learning methods fail to generalize over this fractal structure (which I suppose they are limited due to having a fixed resolution, but could be interesting to study over some range of resolutions).  


questions: To what extent do datasets in machine learning really have a fractal structure?  I feel like this paper is somewhat split between completely artificial examples (like the snowflake made of snowflake images) and natural image datasets where the existence of fractal structure is non-obvious.  It would be nice if there were more datasets with a fractal structure which is somewhat organic.  Perhaps satellite images of weather patterns?  
limitations: Yes I don't think there is an ethical issue, and the limitations are fairly straightforward.
review_title: null

Meta-review: metareview: This paper introduces neural collages, which are operators that aim to represent an image via parameters to a self-referential transformation. The parameters for a given are predicted by a hypernetwork. The approach is mainly evaluated on image compression, but applications to generative modeling are considered. While reviewers had some trouble reviewing the submission because it falls out of their area of expertise, they generally agreed that the approach was novel, interesting, and had compelling applications.
confidence: Certain
Decision: Accept