Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: (Implicit)^2 models bring implicit computation to bear on implicit representation. Deep equilibrium network editions of the SIREN and MFN implicit representation architectures are defined and experimentally compared and contrasted with their standard explicit counterparts. The (implicit)^2 editions are optimized with two proposed approximations: reuse of fixed points during forward and truncation during backward. When these approximates are used with the proposed implicit architectures the training time and memory alike are reduced to constant w.r.t. the effective "depth" of the model as counted in fixed point iterations. This differs from explicit models, where time and memory scale with the number of layers (L), and standard use of deep equilibrium nets, where time scales with fixed point iterations (M) and memory is constant. As a further point on optimization, regularization by spectral normalization is found to be useful for (implicit)^2 training. Analysis experiments justify each approximation, in that the effects are clear in Figures 3 & 4, but the conditions of these experiments are not specified. The quality of results for explicit and implicit^2 are compared for a single input each for the three modalities of image, audio, and video. (implicit)^2 improves PSNR in all cases, and even by 10 or more percent relative for a 512x512 grayscale image and 23 percent relative for a 7-second sound. Accuracy (PSNR) and computation (memory usage in GB and training step time in sec.) are measured across three architectures for one input image: accuracy is higher or equal across model sizes, the text claims 3x faster training speed (in total?) and the text specifies memory saving though the reported charts do not (see Figure 5). The text motivates the computational attractiveness of implicit computation for implicit representations, but does not cover alternatives for reducing time or memory with explicit computation for implicit representations (like sampling/mini-batching, gradient accumulation, reversibility, model decomposition like in KiloNerf or ACORN etc.).
main_review: Novelty:
- This is the first work to bring implicit computation to bear on implicit representation.
- Two implicit variations on implicit representation architectures are proposed, for SIREN and MFN, which are both recent and good approaches.
- The forward approximation of fixed point reuse and its experimental analysis are new, and the backward approximation of truncation is more related to other work (as discussed in the text), but the suitability of the chosen truncation (T = 1) is newly shown.

Significance:
- Implicit computation and implicit representation are both burgeoning topics that differ from the established mainstreams of explicit deep network architectures and grid/mesh/etc. representations respectively. This work joins both, and points out their potential complementarity, and so more work at this interface could follow.
- Improving accuracy (PSNR) on two strong architectures is impressive. However, the scope of the experiments is too narrow to know if the improvements are real (see next point). The quantification of the claimed computational advantages are also poorly measured/communicated: memory usage seems to be the same across explicit/implicit (Fig. 5) and differences if any in results w/ and w/o the proposed approximations are not covered.
- The results are too few to reliably measure quality w.r.t. the explicit counterparts of the proposed (implicit)^2 models. It is certainly a plus to consider three different data modalities, but that is a standard already established by prior work like SIREN, and comparisons need to be done on more than one input per modality. One example does not equal an evaluation.
- The experiments may not be properly tuned or controlled. For instance, batch size was chosen by the convenience of what would fit the largest model in memory (see supplement). Batch size and learning rate schedule can be sensitive hyperparameters, for example, so they should have been chosen by validation or used known good choices from existing works when comparing to them. Spectral normalization is applied to the (implicit)^2 models proposed, but not to the corresponding explicit models, and so results are potentially confounded by this difference.

Clarity:
- The overall organization is good, with clearly named sections, and the main conceptual figures (1 & 2) summarize the ideas.
- Key results, like the justification for the proposed forward/backward approximations, are not grounded in the concrete details of the model/data/optimization/and experimental conditions in general. None of these can be determined from the captions or text for Figures 3 and 4 (or at least I could not after several readings).
- The background on implicit computation is too vague: "designed to have no prescribed computation graph" is not quite right, as the computation graph of the iterated module is definite, but its number of iterations and the dynamics of its latents are not.
- There are a number of typos throughout, which while superficial, do nevertheless distract the reader. (See draft feedback.)

Decision:
This work's union of implicit computation and implicit representation is promising but preliminary. Trying to make the best of both is insightful, and the proposed approximations to make good on their combination are reasonable. The difference in approach is clear for implicit^2 vs. existing implicit and explicit computation, but the difference in experimental results is more potential than proven. Results are shown for a single image, a single sound, and a single video and the analysis for key properties underlying the proposed computational acceleration (fixed point reuse in Figure 3 and gradient truncation in Figure 4) are illustrative in effect but totally unexplained in experimental conditions (like architecture, data, etc.). Furthermore, it would be informative to check the use of this work's forward/backward approximations on DEQs for other purposes than implicit representations: it is fine if they work or not, but we would learn more about implicit vs. explicit. The quality of results, computational costs, and justification of the acceleration schemes are the main points of the paper and so they deserve more complete and systematic evidence.

For rebuttal:
- Please explain the actual effects on memory usage, since Figure 5 seems to indicate no memory is spared, but the text claims reductions.
- Please elaborate on the computation during training vs. testing of (implicit)^2 and explicit computation. Is it more expensive to query (implicit)^2 than explicit computations of implicit representations? Explicit could use less memory during testing in not needing the optimizer, right?
- Please report on the accuracy/computation of (implicit)^2 against the simple baseline of explicit computation with mini-batch sampling (that is, SGD on subsampled batches of input coordinates and outputs) or gradient accumulation. Does (implicit)^2 strictly dominate?

Draft Feedback
- typos: l.91 missing references with empty "[]", l. 99 "paramterized", l.111 "infintesimal", l. 195 "diemsnional", l. 203 "ponits", l. 211 "an simple", l. 211 "trianing"

**Post-Response**: The clarifications and additional results in the response have convinced me to raise my score to a 6. As the exposition and experiments still seem somewhat premature, and it seems some editing may be needed to address the nuances of the computational requirements (which are a main contribution of this submission), I cannot raise the score higher. Nevertheless the combination of implicit computation and implicit representation deserves more attention, and so this work can be considered for pointing in this direction with the first results.
limitations_and_societal_impact: A computational limitation specific to this work is addressed by Section 5. However, the section is exclusively about this one computational limitation for storing past fixed points (state used for optimization). Discussion of potential limitations with regard to data fidelity or modality is missing, especially with respect to explicit computation for implicit representations. Regarding potential societal impacts, this work is about a generic method to improve the training of implicit representations, and as such does not have specific concerns.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Summary
------------------
This manuscript proposes the use of implicit differentiation (i.e. using the implicit function theorem)
to train variants of so-called "implicit" representations (also known as "coordinate based networks") 
the authors develop.
Specifically the manuscript extends the formulations of SIRENs and MFNs in the form of Deep Equilibrium 
Networks (DEQ) that can thus be trained through implicit differentiation. To make training of those 
architectures efficient, the manuscript suggests two approximations. The first approximation lies in the 
forward pass and circumvents the use of an expensive Fixed Point Solver. The second approximation
lies in the backward pass and circumvents the expensive inversion of typically "large" Jacobian matrices.
The authors evaluate their new architectures and training schemes on image, audio and video fitting and
show that smaller Implicit^2 networks can fit signals better than SIREN or MFNs.
main_review: Main review
------------------
I find the A+B idea of mixing two seemingly orthogonal concepts (implicit representations and implicit
representations) interesting. The writing is globally clear, even though the style is rather 
heterogeneous through the manuscript. Some sections would definitely deserve some more love (and clarity).

The motivation of the paper is also clear to me: we want better and more memory friendly "implicit 
representations" and the use of implicit differentiation techniques that are typically cheaper 
(e.g. in Neural ODEs with the method of the adjoint) seems promising!

Despite those interesting ideas, and the promise of more memory efficient implicit representations (at the
cost of more time-consuming forward and backward passes), the paper fails at convincing me this is the case.
In fact, if I understand correctly Figure 5, a glance at it shows immediately that there is no gain in the 
memory consumption between the implicit differentiation method proposed by the authors and the "explicit"
differentiations of conventional SIRENs and MFN. Furthermore, implicit differentiation is indeed slower. All 
in all, we end up with a slower method that is as memory consuming. Therefore, I would absolutely requalify
the claims in the introduction.

As a matter of fact, the authors end up twisting those initial claims (that turn out to be misleading) 
through the paper, and we arrive at Section 4 where essentially the main claim is that with a smaller 
number of parameters (a smaller architecture) the architecture and implicit differentiation scheme
proposed by the authors seem to perform better than a SIREN or MFN. As a summary, Implicit^2 has the 
same memory consumption for an equivalent architecture as a conventional SIREN or MFN, runs slower,
but seems to perform better! Consequently, if the paper is accepted, the storyline must be reworked, 
the motivation on why implicit differentiation helps to do so too, and hints or experiments on why 
those architectures perform better should be provided.

Originality and significance
----------------------------------------
I find this work interesting, I am not completely sure about its level of significance but believe significance 
would be increased if code for implementing and training this architecture was distributed. In terms of
originality, this work falls in the category of A+B papers, combining two existing ideas, it seems relatively
well executed: results despite being on toy problems seem promising.

Questions
------------
- Can the authors explain why the implicit^2 architecture ends up consuming as much memory as
SIREN or MFN for a same architecture contrary to what was expected and motivated the whole paper?
- Can the authors explain/hint why this is the case that the SIREN or MFN perform worse than 
Implicit^2 on the tasks shown in the paper? Is there some inductive bias in Implicit^2 that helps
for the task presented? Is it simply that those optimize better?


More specific comments and typos
-------------------------------------
- I would suggest to improve the caption of all the figures so that they can be self-contained,
at the moment, many of this figures cannot be understood at a glance.
- I do not think the choice of Figure~5 is very good: again it does not convey the main claim 
of the paper: that implicit^2 is helping in terms of memory consumption. Now, if the point to 
convey is that PSNR with a smaller Implicit^2 architecture can be better than with a bigger
conventional SIREN or MFN architecture, other plots or tables would be more appropriate.

Typos
----------
- l.181 correspsonding network -> corresponding network
- l.195 diemsnional -> dimensional
- l.210 this do not -> does not
- l.211 at trianing iteration -> at training iteration
- l.238 to ex scale -> ??
- l.272 achieve -> achieves
limitations_and_societal_impact: I am not sure this is applicable to this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposed $($Implicit$)^2$, which replaces the internal layer of implicit neural representation (INR) with implicit layers. The proposed method is efficient in terms of memory and training time. For the fast training, the author proposed the re-usage of fixed-point and also used truncated backpropagation (approximated gradient). The method is sensible and also empirically demonstrates its effectiveness.
main_review: -----------------------------------------------------------------
**Strength**
1. The motivation of using an implicit layer into implicit neural representation (INR) makes sense (as it significantly reduces memory utilization).

2. The performance gains (for both efficiency and training time) are significant.

3. Although there are no theoretical guarantees, the authors empirically showed that the re-usage of fixed-point and truncated back gradients are sensible.  

4. The writing and presentation are clear.
-----------------------------------------------------------------
**Weakness**

**1. Some scalability concerns: the experiments are demonstrated under somewhat small-scale datasets**
- If possible, can the author show experiments requiring a large number of INR layers, i.e., a complex signal? For instance, 3D shapeNet needs eight layers to be represented with a Fourier feature network [2]. Demonstrating experiments under such conditions will be interesting for $($Implicit$)^2$.
- Alternatively, datasets that are used in Neural Radiance Fields [1] will be significant.
I believe such experiments are the mainstream task of INR.

**2. (Minor) Comparison of inference time**
- I recognize that direct comparison of inference time is somewhat difficult (due to CPU utilization for the solver). However, still believe the paper should include such part. 
- If possible, can the author compare the inference time of $($Implicit$)^2$ with the baselines?

**3. (Minor) The variance seems to be big in Table 1** 
- Can the author evaluate the images for a large number than 16, e.g., 250 samples, by following [3]? (Of course, the author can use a smaller dataset, e.g., CelebA)
-----------------------------------------------------------------
**Justification of the rating**\
I recommend accept. I believe the idea of the implicit layer is well suited for INR and also shows significant efficiency. I do enjoy reading the manuscript.

I still have some concerns about the scalability issues; therefore, I politely request the authors to share the experimental results mentioned in the weakness part.

-----------------------------------------------------------------
**Reference**\
[1] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Mildenhall et al., ECCV 2020\
[2] Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. Tanick et al., NeurIPS 2020\
[3] Learned Initializations for Optimizing Coordinate-Based Neural Representations. Tanick et al., CVPR 2021



limitations_and_societal_impact: The authors have addressed the limitations.\
For societal impacts, I agree that this work may not have negative impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work suggests that combining implicit-depth models (e.g. DEQs) with implicit representation approaches (e.g. SIREN) can bring performance improvements. The method is evaluated on image, audio and video representation.
main_review: This work aims to develop several ideas simultaneously and does not always achieve the prescribed goal. At its core, this paper is about combining implicit-depth models with implicit representations to improve empirical performance in data representation tasks. The second major component of this paper is an attempt at introducing methodological improvements to implicit-depth models (particularly DEQs [1]) with root finding solution reuse and truncated gradients. 

The work does a really poor job at motivating the combination of these two models (beyond the similarity in nomenclature), or performing an analysis on the reasons why performance improvements should even be expected. The novel material (page 5 onwards) introduces a few changes that can be applied to all DEQ models. These have nothing to do with implicit representations, yet in this work they are sold as a core component of the model. See for example Figure 1, where (Implicit)$^2$ is exactly a DEQ  with the proposed improvements.

The proposed improvements to implicit models are reasonable. Root reutilization is definitely useful and is simultaneously being proposed in other recent papers on the topic. I would also like to point out that the original DEQ implementation also features root solution reuse (see for example [2]), so this section not particularly surprising. Truncated gradients and spectral regularization are similarly natural improvements over the DEQ formulation which I was glad to see here. I would suggest the authors decouple these improvements with their combination of models, perhaps introducing them beforehand.

To me, the greatest limitation of this work are the experiments. Given that the work does not appear motivated by theory or particular insights, I expect strong results to carry the work. I'm seeing marginal improvements, with the most impressive being SNR of smaller implicit Gabor/Fourier-MFN compared to larger explicit variants. Implicit SIRENs actually seem to perform worse in several tasks, see for example audio reconstruction in the supplementary (Figure 3). Image and 3D model representations do appear slightly sharper. The code submission is organized well.

Overall, I believe this paper is borderline, and I am definitely willing to modify my score, particularly if the authors can provide a stronger motivation behind the merging of these two modeling frameworks. As it stands, I see this work only as a preliminary exploration which might spur more definitive follow-up work on better implicit representation models.


* (1) Deep Equilibrium Models
* (2) https://github.com/locuslab/deq/blob/beta/DEQModel/models/transformers/deq_transformer.py
* (3) Stabilizing Equilibrium Models by Jacobian Regularization

-----------------

Minor:

> While the modification seems simple, the addition of implicitness in fact yields a much richer set of functions than the ones can be represented by a forward pass of F alone

This is wrong as a general statement and is not proven anywhere in the paper. Hand-wavy general comments such as this one have no place in the paper.

> Therefore, the forward and backward passes of implicit models can rely on independent black-box solvers, and these models consume only a constant training memory (as they do not need to store anything)

Incorrect and misleading. Implicit models (both Neural ODEs and ) require storage of various quantities to ensure low errors. Backsolve continuous adjoints for Neural ODEs can give you gradients at no additional memory cost, but can also introduce unbounded errors depending on the solver. Similarly, DEQs require storage of root solutions for faster training. Repeating the fixed point iteration at each training step, while free of memory overheads, is often too slow. There are also other costs discussed in [3] (3.4)
limitations_and_societal_impact: The authors provide a satisfactory discussion on limitations.
review_title: null

Meta-review: After the discussion and rebuttal process, all reviewers now recommend acceptance, albeit with several borderline ratings. Several reviewers flagged limitations of the experimental evaluation, which the authors have responded to by running additional experiments. The discussion also cleared up some confusion about memory savings, which seems to have stemmed from a misinterpretation of Figure 5.

Overall, reviewers seem to agree that the combination of implicit computation and implicit representations is interesting and creates some nice synergies, and also that this idea deserves the attention of the research community. I am therefore inclined to recommend acceptance, though this is conditional on the authors adding the new results to the paper and addressing the various other remarks and recommendations made by the reviewers (e.g. with respect to Figure 5). 
Decision: Accept (Poster)