Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents a method of alternately meta-learning the Implicit Neural Representations (INR) for a set of signals using the MAML method, and pruning the INR using the global magnitude-based pruning method for sparsifying the INR. The proposed method is tested on the 2D image regression task in three datasets (CeleA, Imagenette and 2D SDF). In comparisons, three baselines are used (MAML+w/ random pruning,  MAML+w/ dense-narrow, and w/o MAML + dense-narrow). The proposed method obtains better PSNR across different number of optimization steps, thus showing more efficient learning in terms of computation and memory footprint. 
main_review: Overall, the proposed method seems to apply existing techniques, a meta-learning method using MAML and a pruning strategy using global magnitude-based pruning, to sparsify the INRs.  The technical novelty thus seems to be limited.  The baselines also are limited. It will be better If different meta-learning methods and pruning strategies are tested.  So, I think this paper is marginally below the bar of NeurIPS. 

More specifically, 

Pros: 
+ The paper is well motivated and easy to follow. 
+ The observations of applying MAML+Magnitude-based Pruning in sparsifying INRs are informative. 
+ The experimental results are promising with good analyses. 

Cons: 
- Only one type of meta-learning methods, MAML and one type of pruning strategies, magnitude-based pruning are studied. More are entailed to gain more insights. 
- Vanilla INRs are used (SIREN and FFN). It will be better if some new designs are studied towards more effective and more efficient meta-learning and pruning. 
- The proposed method does not test the cross-dataset meta-learning and pruning, which seems to be more interesting for scaling up INRs in a computation and memory efficient way
limitations_and_societal_impact: The limitations of the proposed method are considered in the manuscript, but not sufficient as stated above. Potential ethical side-effects are addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper describes a method for representing sets of signals using coordinate-based neural representations (or INRs) by combining a meta learning approach for learning the initial weights with a network pruning approach for ensuring that this initialization is as sparse and memory-efficient as possible. The paper contributes an algorithm, Meta-SparseINR, which alternates between meta-learning weight initializations for coordinate-based networks and pruning the weights in the meta-learned initialization. The algorithm results in a sparse network initialization which can be then specialized in a few gradient steps to represent any signal in a dataset, and generalize to new signals taken from the same distribution. The paper demonstrates that this additional pruning step built into the meta learning process leads to improved results when compared to just meta-learning dense network initializations with a comparable number of parameters. The method is evaluated on an image-fitting task for multiple datasets, and it is shown that this approach does lead to a better performance versus memory trade-off when compared to just network pruning or meta-learning alone.

main_review: I will break down the review into an analysis of strengths and weaknesses of the paper. In my opinion, the strengths of the paper are:
1. The results show that applying the combination of meta learning and network pruning iteratively as in Meta-SparseINR result in noticeable improvement over performing either of these methods alone. This is a valuable contribution, since it justifies the papers’ claim that this method can bridge the gap between meta learning methods which speed up learning INRs, and quantization or pruning methods which decrease the memory required to store a dataset of INRs.
    - The experiments show that Meta-SparseINR outperforms just performing meta learning with a representation with the same number of parameters, especially when there are few parameters (Fig 2). Interestingly, the sparse representation also converges with less optimization steps (Fig 3a).
    - The experiments show that Meta-SparseINR retains the benefits of meta learning and converges significantly faster to a higher quality representation than training from scratch (and presumably pruning or quantization this scratch training, since this likely does not improve the quality of the representations trained) (Fig 3a).
2. The paper is well motivated - one of the largest challenges with coordinate-based networks is generalization to represent datasets of signals. This paper attempts to remedy this by considering both axes of computational requirements to do this - optimization time, and memory consumption. This could potentially enable representing large datasets with INRs, which would allow for easier studying of the characteristics of (and possibly generalization across) these types of signal representations.
3. The paper is written very clearly and concisely. The experiments shown directly justify the claims that are made in the paper, especially those in Fig 2, 3, 4. The methodology is described in detail, and justification for design choices is provided. This is valuable as it makes it much easier to understand what the method is doing, and thus more likely to be adopted into future research work.

The weaknesses of the paper are:
1. The main concern I have is related to the choice of learning a sparse initialization, instead of learning an initialization which can then be made sparse when fit to each dataset signal. I have a few questions and possible concerns:
    - The method only learns a sparse initial INR. Once this initialization is applied to represent a new signal, is the learned mask retained and the final represented signal also sparse? It must be, otherwise memory is only saved when storing the **initialization**, and not each of the dataset signal representations.
    - The most logical baseline to compare to would be one which first meta-learns an initialization (non-sparse), and then uses this initialization to fit each of the dataset signals independently (non-sparse), and then applies some pruning or quantization method to each of these final representations (sparse or quantized). In this case, the (non-sparse) meta-learned initialization would still provide the same speedup bonus on optimization, and the quantization or pruning on the (non-sparse) dataset signal representation could provide a similar memory benefit in storage by making it sparse or quantized. I understand that pruning is an iterative process, which requires fine tuning after making a representation sparse, so this type of baseline would likely still incur additional computational cost when fitting each dataset signal for this fine tuning. But, I still think this makes sense to compare to, in terms of computational requirement and representation quality, alongside just simply applying meta learning to a smaller INR architecture.
    - Without some comparison like this or explanation why this baseline doesn’t make sense, I think that it’s not super well justified why the meta learning and pruning steps need to be iteratively applied to the initialization, instead of sequentially applied to the initialization and then the dataset signals. Since this is the main originality of the contributed algorithm, I think some justification of this would make the paper significantly stronger.
2. I think there is a flaw in the conclusion of section 5.3. Figure 5 shows that pruning using the *Winning Ticket* method performs equivalently to just training a smaller representation, which is supposed to highlight some benefit of sparsity when trained with the meta-initialization. However, it is explained that these results are obtained by training each of these representations for 10k steps, while the comparable plot where the sparse initializations outperform the smaller networks is only benchmarked after training for 100 iterations (Fig 2). I suspect that meta-learning only enables improved performance for a limited number of iterations, after which the magnitude of contribution of the initialization likely doesn’t matter anymore. I.e., if Fig 2 was left to train for as long as Fig 5, it’s possible that we’d see the same trend with the meta-learned initializations as well.

Overall, the paper contributes a valuable algorithm for combining network pruning with meta-learning methods. The contribution of this algorithm is well-motivated for learning to represent classes of signals using INRs, and the paper shows that it does better than existing methods which apply just meta learning or pruning. Because of this, I think the paper is already quite good. I believe that the comparison to independently performing meta-learning **and** network pruning could remove all reasonable doubt that the formulation of Meta-SparseINR for learning the initialization, and then directly optimizing this sparse representation on dataset signals, is the right way to go about learning a dataset of INRs. This, along with some clarification on the conclusions drawn from Fig 5, would make this a much stronger paper.

Some other minor comments which could possibly be clarified to improve the quality of the paper:
- For Fig 5, the supplement mentions that this was obtained from a single image of a Fox? Does this trend hold for multiple images.
- Can the repeated application of sparsity throughout the meta learning process reduce the memory required to train the model? One big issue with MAML and other gradient-based meta learning algorithms is the memory requirement of differentiating through the optimization process. If making this network sparser can reduce this, this could be very impactful.

**Update after Author Response**:
The additional experiments provided by the authors have addressed the concerns brought up in my original review. I retain my score, and think that this is a good paper which should be accepted.

limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impacts of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Authors propose a method for learning sparse implicit functional representations for signal data. `Their method combines ideas from MAML to learn a good initialization from which a sparse neural network representing the signal can be efficiently estimated. Experiments on a few datasets suggest that the proposed method achieves better PSNR than more naive approaches for the same levels of sparsity. Overall, the paper is in a good shape but still could benefit from a few additions.
main_review: Novelty:
To my knowledge, both the application and the proposed method are novel.

Clarity:
Overall the paper is well-written and easy to follow.

A minor comment is that I would prefer more rigour in presenting eq. 4 and connecting it to algorithm 1, particularly, in how the iterative procedure is actually performing the inner-loop optimization explicitly writtein eq. 4. Does the algorithm at least provably improves on the objective? It is not completely obvious to me and I would like to know exactly what guarantees Algorithm 1 has.

Significance:

To me it's fairly interesting to be able to sparsely represent natural signals both from abstract point of view and also because it's opening certain applications such as more efficient image or audio compression algorithms.

I'm this very curious if there is any chance of comparing their method to standard image compression algorithms such as JPEG in the paper which should not be very difficult.

Quality:

Overall I find the proposal method technically sound and experimental evaluation adequate (also see my comment in the clarity section).

One of the baselines which I think is missing from the experimental evaluation is some kind of a functional/implicit-style auto-encoder with the varying dimensionality of the hidden variable. I understand that it won't be fully comparable to the model because even if the code itself is short, the decoder network still needs to be transmitted somehow, but arguably one can do this beforehand, so I still think it is a valid experiment (also see question 1).

I also don't fully understand if a more standard lottery-ticket method can applied for the task of simply learning a sparse representation of a single signal. Albeit, again, not being fully comparable to the proposed method, it would probably achieve even better PSNRs and thus provide an important, perhaps gold-standard data-point.

Other questions:
1) Have authors tried an alternative approach where the neural network representing the function is not sparse itself, but only a few parameters of it are allowed to be modified and supposed to contain information about a particular signal? 
An example of this approach be found in [1] (although there the setting was quite different), where certain weights in some of the fully-connected layers are modifyable. 
2) Why did authors use Imagenette and not the standard ImageNet? I would be more confident (perhaps, falsely) in the results if they were performed on a more diverse dataset. I don't think this would require much work compared to Imagenette experiments and I'm very curiuous to see the results.

References:
[1] Bartunov, S., Rae, J. W., Osindero, S., & Lillicrap, T. P. (2019). Meta-learning deep energy-based memory models. arXiv preprint arXiv:1910.02720.
limitations_and_societal_impact: Discussed adequately.
review_title: null

Meta-review: The paper proposes to learn functional representations of signals such as images using meta learning and network pruning, so that an initial sparse network can be perturbed to fit each training or testing image. The proposed method is novel, simple and clean, and it can be useful in applications based on functional representations. The experiments are solid and thorough, especially with the additional experiments performed during the rebuttal period. 

After rebuttal, this paper received three final ratings of 7, 7, 7 (the reviewer who initially assigned a rating of 5 upgraded the rating to 7, but was unable to make the change formally due to technical error). The rebuttals have addressed all the concerns of the reviewers. 

This paper makes a valuable contribution and can be accepted. 
Decision: Accept (Poster)