$(\textrm{Implicit})^2$: Implicit Layers for Implicit Representations
Authors: Zhichun Huang
Shaojie Bai
J Zico Kolter
Venue: NeurIPS 2021 Poster (2021)
Keywords: implicit neural representation
implicit deep learning
deep equilibrium model

Abstract: Recent research in deep learning has investigated two very different forms of ''implicitness'': implicit representations model high-frequency data such as images or 3D shapes directly via a low-dimensional neural network (often using e.g., sinusoidal bases or nonlinearities); implicit layers, in contrast, refer to techniques where the forward pass of a network is computed via non-linear dynamical systems, such as fixed-point or differential equation solutions, with the backward pass computed via the implicit function theorem.  In this work, we demonstrate that these two seemingly orthogonal concepts are remarkably well-suited for each other. In particular, we show that by exploiting fixed-point implicit layer to model implicit representations, we can substantially improve upon the performance of the conventional explicit-layer-based approach. Additionally, as implicit representation networks are typically trained in large-batch settings, we propose to leverage the property of implicit layers to amortize the cost of fixed-point forward/backward passes over training steps -- thereby addressing one of the primary challenges with implicit layers (that many iterations are required for the black-box fixed-point solvers). We empirically evaluated our method on learning multiple implicit representations for images, videos and audios, showing that our $(\textrm{Implicit})^2$ approach substantially improve upon existing models while being both faster to train and much more memory efficient.

(Implicit) 2 Implicit Layers for Implicit Representations
Zhichun Huang Carnegie Mellon University Pittsburgh, PA 15213 chichun@css cmu, edu
Shaojie Bai Carnegie Mellon University Pittsburgh, PA 15213 shaojieb@cs cmu. edu
J. Zico Kolter
Carnegie Mellon University Pittsburgh, PA 15213 zkolterècs cmu edu
Abstract
Recent research in deep learning has investigated two very different forms "implicimesss: implicit representations model high-frequency data such images or 3D shapes directly via a low-dimensional neural network (often using e.g. sinusoidal bases or nonlinearities); implicit layers, in contrast, refer to techniques where the forward pass of a network is computed via non-linear dynamical systems, such fixed-point or differential equation solutions, with the backward pass computed via the implicit function theorem. In this work, we demonstrate that these two seemingly orthogonal concepts are remarkably well-suited for each other. In particular, we show that by exploiting exeedpooint implicit layer to model implicit representations, we can substantially improve upon the performance of the conventional sxplicitlayee layer-baedd approach. Additionally, as implicit representation networks are typically trained in large-batch settings, we propose to leverage the property of implicit layers to amortize the cost of fixed-point forward/backwanl passes over training steps thereby addressing one of the primary challenges with implicit layers (that many iterations are required for the black-box fixed- point solvers). empirically evaluated our method on learning multiple implicit representations for images, audios, videos, and 3D models, showing that our (Implicit)² approach substantially improve upon existing models while being both faster to train and much more memory efficient.
1 Introduction
The concept of "implicitness" has recently been applied to various contexts of machine learning research. One particular thread is implicit representations [19] 22 28 (7], which aims to learn continuous representation high-frequency. often discretely-measune Formally, given spatial-temporal coordinate input E Rd implicit representation of it is function & d - C, where C is the space of desired quantity (e.g. color, volume density, distance, etc.) and & is usually parameterized with neural network. Such implicitness has multiple benefits over the conventional discrete (e.g., grid-based) representations, e.g., as & is defined on a continuous domain, an implicitly represented input consumes much less storage than the original input; as another example, Φ is differentiable, hence allowing for the computation of higher-order derivatives. However, the training of these & networks themselves are usually quite memory-consumine consuming, since we usually deal with very high-resolution images and videos, and training is typically done in full-bate batch mode (e.g., when training on batch size of 512 x 512, a simple 4-layer MLP with 1024 hidden units already requires > 16 GB memory just to store the intermediate activations).
signals such as large images.
recent (orthogonal) usage of implicitness in deep learning comes from the notion of implicit layers/models, where the word is used to characterize model architectures (rather than input repre- sentations). The nomenclature comes from the concept of implicit vs. explicit functions: instead of
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

representing model as an explicit stacking of layers, an implicit model solves a non-linear dynamical system [T]] (e.g.. ODEs 7 or fixed-points [2)).. Importandl, these implicit models decouple the forward and backward passes and analytically differentiate via the implicit function theorem [14]. This enables these models to use constant training memory, irrespective of the forward pass trajectory. Moreover, recent works on deep equilibrium models [ have been shown to be able to achieve results on par with the tate--of-the-at explicit models (e.g.. Transformers (3I] on very large-scale vision or NLP tasks 2 However, despite training memory efficiency, these implicit layers are also slower to compute (e.g., by 2-3x), since the solvers are usually iterative in nature and ultimately require many function evaluations.
Despite the success of these "implicit" methods in their respective areas, the two concepts so far have rarely intersected: existing models used for implicit representations are neither implicit in nature (since they are still stacked multi-layer explicit operators trained end-to-end in the feedforward manner), nor do they exploit any properties of the implicit functions during training (e.g., implicit differentiation). In this paper, we argue that implicit representations and implicit layers are remarkably well-suited to each other, and their use together can compensate for their aforementioned drawbacks.
Specifically,, we propose combine the best of both worlds by replacing the explicit models used in conventional implicit representation learning with implicit layers, whose output is defined by a fixed point condition, thus dubbed (Implicit)² networks. Formally, let x be the input and F be an (often shallow) layer, the (Implicit) approach learns the following implicit representation Φ:
(xx) = W=* + b, where =² = #=)
Importantly, we show that these two "implicitness" complement each other well, especially in two important aspects. First, the unique large-batch training scheme of implicit representations as well as the forward/backward decoupling properties of implicit models permit to amortize the cost of the iterative solver that would otherwise make implicit models slow to train. We show that by fixed-point reuse (in the forward pass) and Jacobian approximation (in the backward pass), training an implicit model on implicit representation tasks requires almost no computation overhead. Second, implicit models allow us train these high-frequene input tasks with much lower memory consumption while not losing speed or performance, thus significantly lowering the hardware requirement and computation budget of training implicit representations. We evaluate (Implicit) on multiple implicit representations for images, videos, and audios, showing substantial improvement upon existing competitive explicit-model methods like SIREN 27 or MFN 11 using equivalent-sized networks.
2
Background and Preliminaries
2.1 Implicit Neural Representations (INR)
While high-frequency data such as images and scene geometry have been traditionally represented discretely (e.g., pixel/voxel grids mesh points), recent work has demonstrated the possibility of
replacing them with continuous functions parameterized by multi-layer perceptrons (MLPs) [27] 28
These deep networks have been used to learn differentiable representations of various forms, such as continuous spatial-temporal image representations 5 |. high-resolution scene representations [26
22], . and volumetric rendering [19 22], and have been shown to be significantly more compact than
the conventional grid-based approaches.
However, training these INR models is not casy. For example, the training of implicit representations is usually conducted in very large batch sizes (c.g., 512 x 512), which renders these (albeit simple) during training. As another example, recent works have shown that typical deep networks with ReLU/tanhssigmoid non-linearitier fail to capture the fine details of input signal, and resorted to periodic non-linearities [27], Fourier feature encodings (28), or repeated application of non- linear filters (e.g., Gabor wavelets) to the input and then multiply with the features [1I]. We introduce below the two latest, highly competitive models that have been developed on this task, which our work will build on (see $3.1)
MLPs very memory-consumee
Sine-activated Network for Implicit Representations (SIREN). To overcome the detrimental effect of traditional non-linearisies like ReLU/ttan on modeling fine details and higher-order derivative of the input signals. Sitzmann et al. 27 proposes to use sinusoidal activation functions that allow explicit supervision on any derivatives of the input signal. The resulting MIPss, though simple, have been shown to achieve state-of-the-arr performance in representing images, videos, complex
2

---

geometries, and more (27). Formally, given an input z, SIREN learns an implicit representation by the following L-layer MLP:
(xx) = ( 00 (z) +
(2)
where 9i is the i-th layer of the network that computes gi gi(h!t) = sin(W,hl + b,) with sine non-linearity, and h10 denotes the hidden state at i-th layer.
Multiplicative Filter Networks (MFN). More recently, Fathony et al. [1] proposed to forego the SIREN-like design (i.c., MLPs with periodic activation functions) in favor of multiplicative operations that are proven to be similarly expressive. Specifically, at cach layer of the network, an
MFN applies learnable non-linear filter kernel (parameterized by (1)) g(2; 0(0) the original input x IR", and elementwise multiply it with a linear transformation the features; i.e., formally,
(xx = Whl4 + b, where h | = (W,, 1 + b,) g(2; (()) and A h0 = g(z; e())
(3)
Fathony al. M in particular proposed two instantiations that admit sinusoids Gabor wavelet as the filter g. which are called FourierNet and GaborNet, respectively:
Drourier (I; Fourner) = sin(w ()())
(4)
g [Gaabor (1;; Gabor) = sin(w x + (0))) exp (- (-(x()/2) (x (x- (1))))))11)))))
o where Gahor (w R" ((0) E R. 7(0) μ¹ E R"} (similar for Founser) and (n(() , ((0
denotes the mean and scale of the Gabor filter. These models have shown performance on par with or better than the cenioddi-oooninartty--oooe
networks like SIREN on range of tasks like image and
video representations.
2.2 Implicit Models
Whereas explicit models hierarchically stack a sequence of L non-linear operators to build a compu- tation graph for their forward/backward passes (where L is often referred to as the depth of model), implicit model instead define the output as a solution to a non-linear dynamical system which in most cases can't 't be expressed as fixed computation graph. For example, the Neural ODE model [ solves an initial value problem ODE that intuitively corresponds to infinitesimal residual layers [3] Another example is the recent deep equilibrium (DEQ) models [ whose forward pass involves computing the fixed point of a shallow layer F.
Importantly, unlike explicit models, these implicit models do not need to store any intermediate activations during training: to produce gradients in the backward pass, we can directly differentiate through their final outputs (e.g.. by adjoint method for Neural ODEs 7 and implicit differentiation for equilibrium models []), regardless of how the forward pass was computed. Therefore, the forward and backward passes of implicit models can rely independent black-box solvers, and these models consume only a constant training memory. Our work is primarily built on the formulation of deep equilibrium models, which introduce in detail below.
Deep Equilibrium (DEQ) Models.
Given an input x and a layer F(2; .>) (e-g., a residual block, or a multi-head self-attention layer [31),, DEQ model aims to approximate an infinite-level stacking of this layer (i.e. 2[++1| (2 2)) with i = 1,..., L and L + oo). To achieve this purpose, Bai al. advocated for directly solving a fixed- point condition defined by F and r::
2" = F(=;;>
(5)
where z* is called the equilibrium since, intuitively, stacking another F(-,2) layer on top does not change the activation. To solve for the equilibrium point, prior works have had success with various kinds of advanced solvers, such as Broyden's 's method I [7], and Peaceman-Rachfor splitting (23 32 24). Then, in the backward pass, we can produce the gradient by the implicit function theorem directly applied on this equilibrium point 2², with a loss t:
dL (I (1- JF) 0F(2";)
Vi d=
(6)
a(.)
where Jr denotes the Jacobian of F evaluated at =². As the inverse of matrix is expensive to compute, in practice the backward pass is usually formulated another linear fixed-point condition
3

---

Explicit
DEQ
(Implicit 2
01A)
Time OLM) Menway (11)
Thme 0.1)
Manoyy 0(1)
Forward
Backward
Forward
Bockward
Forward
Backward
XL
Truncated Baplwarrd Graadient
Black-box Foxed Point Salver
poid-poiint iterations
E
Zg
Former z"
(a)
(b)
(c)
Figure 1: (a) Explicit networks incur OL) complexity in both time and memory during training. (b) Original DEQ models requires constant memory. yet they take (M) time in forward/backwward passes, where M is the number of fixed axed-point solver steps. (c) In contrast, (Implicit) Network consumes constant amortized memory and time in both forward and backward passes.
or in y: y. JF 3z" = =y which again can be solved via any of the advanced black-box solvers. Thus theoretically, the model only needs store x, 2" and F(2";z) in memory training time, which is equivalent to the memory consumption of merely one F layer. Recent works [1 have demonstrated that DEQ models can scale high-dimensional tasks and be as competitive as explicit structures like ResNet 13 and Transformers [3I] on large-scale computer vision and NLP tasks. However, these implicit models are also slower (typically by x3 train, because both forward and backward passes involve numerous iterative steps.
Our work studies the novel combinations of implicit models and implicit represenlation learning. Specifically, we show that these two "implicitness" are surprisingly well-suited to each other, as they almost perfectly compensate for each others' drawbacks. We demonstrate that implicit modeling of a simple layer F substantially improves the training speed, memory, and performance on implicit representation tasks when compared the aforementioned state-of-the-an deep explicit networks.
3 (Implicit) Networks
As discussed in $2.1 learning implicit representation for complex signals is often more effective when projecting the input coordinates onto high high-frequency bases (c.g. sinusoidal functions). making the designs of these models typically different from deep networks used in more common applications, like image classifications (c.g., ResNets [33]. In this work, we propose to directly build on the designs of these aforementioned state-of-the-art layer architectures by only making minimal modifications, but modeling them as shallow implicit models. We briefly introduce below the two instantiations of our proposed (Implicit)? approach, which are based SIREN (27] and MFN (11, respectively, followed by discussion of how implicit representation tasks can enable us to amortize the cost of training these implicit networks significantly in practice.
3.1 (Implicit)² Network Architctures
Implicit Sine-activated Networks (iSIREN). Motivated by the success of SIREN, we first propose an input-injected variant of SIREN (denoted as SIREN (input inj.)) suitable for implicit models, i.c.,
FSRER (2; .)) sin(W (2 + sin (V=)) + Ux + b)
where, following prior works on deep equilibrium models 11 32 24), the transformed input æ is added to the hidden representation = (see Fig. 2). Just like canonical SIREN layer, any order of derivative of this input-injected SIREN variant is also an input-injected SIREN of the same form. Therefore, the proposed variant inherits the same high-frequency and bigher-order differentiability properties SIREN, and we name the implicit model that solves its fixed-point iSIREN.

---

Linear Layer
Sine activation
Filter function
ISIREN
iMFN
Figure 2: The architecture of one implicit layer F(2;z) a)) in iMFN and iSIREN. Note that when 0,
evaluating F(2; x)) once is equivalent to the output the respective explicit layer.
Implicit Multiplicative Filter Networks (iMFN). Additionally, we introduce variant based on the recent Multiplicative Filter Networks (MFN) [Л]. As MINs only perform linear transformations on hidden features, which are multiplied with a non-linear filter function on the original input æ, slightly modify the layer design as follows to introduce additional input injection:
FMPN(2; æ) (W + g 02)) + 6) g( 01)
where y denotes the filter function of choice, such as Gabor or Fourier (gGahor and MFourier in Eq. 4 filters 11. We call the implicit models defined by this layer iMFN. Moreover, just like the original MFN [T. we show that the output of iMFN (which is the fixed-poii of FMEN) is also linear combination of the non-linear filter functions (see Appendix ?? for proof).
We provide visualizations of these two architectural instantiations in Fig. With these variants, we formally introduce the corresponding (Implicit) approaches to implicit representation tasks as follows:
((x) = W'2 + where 2" = Fx);;),, and {FMEN: FSIREN)
3.2 Accelerated Training of (Implicit)² Networks
Compared explicit networks, training with implicit models significantly lowers the memory budget required (thanks to implicit differentiation). This is especially compelling for training implicit representation tasks, which are often memory-botllenecke one can exploit 1) the large-batch nature of implicit representation training, and 2) the nature of fixed-point optimizations (in both forward and backward), to significantly improve the speed of these models as well, making them eventually both more memory-effiiiet and time-efficient than explicit models (while achieving performance as competitive, or better; see
this subsection, we describe how
Suppose fix training step t and let the corresponding network param- eter (i.e., weights of the network) be given by 9[z)- Typically, a deep equilibrium model solves for the fixed-point z(a) given an input with
Fixed-poing Reuse.
10
# of fixed-poing point iterations
Zero initialization
Reusing fixed point
black-box fixed-point solver starting with ini- tial guess |( (which is usually O random vector ())
6
4
FixedPointSalve(( (1), x, (0))
(10)
where we note that =(t) does not depend at all on this initial guess z!0 of the optimization. De- pending on the quality of the initial guess, the optimization process itself can take different number of steps. For example, in the unlikely
0
200
400
600
800
1000
Training step
Figure 3: Zero initialization v.s. fixed-point reuse in terms of required steps to converge. Further details of this experiment can be found in $4.1
circumstance where we guessed 20 to be =[e)> the fixed-point solvers will directly converge at iteration However, in practice, this property rarely used, as given a complex high-dimensional input (c.g., high-resolution image), it is hard to know (or learn) what makes good initial guess.
5

---

Therefore, these implicit models' aforementioned memory efficiency comes at a cost: without good initial guesses, we must spend more iterative updates order to secure convergence to a good enough estimate of the fixed point in practice, rendering a slower model. However, we argue that the training on many implicit representation tasks, which uses the entire batch of data (e.g.. to fit an RGB-color image implicit representation, each pixel of this image is a "sample" of this full batch), can compensate for this issue by providing us very reasonable initial guesses. In particular, training in the full-batch mode allows us to have direct access to all of the fixed-points =(e) for all inputs in the training set immediately after step t. Assuming the model updates between iterations t and + 1 are small (which is true in practice, such as via SGD with a small learning rate 77), one can safely assume the fixed- sreddppoints of the updated layer F(r; x., ((141)) do not deviate much from its current estimate F(:: 2, (())) as learning rate ll O, which implies *((+1) + Z We empirically verify this these full-batch training settings (see Fig- 3) where majority of the fixed-point convergences (to a target residual level æ = 0.01) finish in exactly step.
z Formally, F(;;r, Q(-1))
However, we note that this does not imply all training iterations will gain a similar level of convergence boost (c.g., see the first 100 steps in Fig. 3 In particular, at training iteration t = O, we do not have a "previous estimate" at all, which means we still have to solve for z" by running the solver for several steps with a neutral initial guess. Thus, we highlight that such fixed-point reuse amortizes the cost of later training iterations, and indeed, the majority of the training of the implicit models. Empirically, we found that performing one step of the fixed-point iteration is sufficient for training the iSIREN and iMFN networks; i.e., at training step t 1, we simply perform
(111) F(z(o): xr, (()))
(11)
Truncated Backward Gradient. As mentioned in 22. the backward gradient of implicit models (Eq. (6) can be alternatively obtained by solving a linear exintpoint system
JJ
y JF
(12)
Oz
to avoid expliciting inverting the matrix. How- ever, recent works have suggested that assuming suffi- cient stability in the DEQ model layer F, we can also 8t J by an expansion of
approximate 3. it as a Neumann series (15)
40
PSNI
30
T di
3 0z*
Σ (JF)
(13)
T
20
T T==
For example, Fung et al. [12 propose to the 0- th order approximation and perform a Jacobian-free backward gradient estimation (assuming Lipschitz dt F); i.e., ~ a=*. Although this approximated gradient was empirically shown to effective im- age classification tasks, we found it to substantially hurt the training-time error convergence in implicit representation learning. As result, we propose to use a truncated backward gradient such that the Neu- mann series is unrolled for T > 0 steps to better approximate the exact implicit differentiation. Formally, the proposed gradient has the form
T
10
1000
2000
3000
4000
5000
Training step
Figure 4: Training PSNR comparison be- tween different levels of gradient approxima- tion. T' denotes the approximation order as in Eq. [4] where T' = oo indicates convergence to the solution.
T Vape=
Oe
OF(=";))
Σ (JP) = 02 1=0
(14)
0(-)
In practice, we found T = to be already sufficient for implicit representation tasks (see Fig. which requires only one evaluation of the vector-Jacobian product 85 JF.
d6
Spectral Normalization. We additionally apply spectral normalization on the layers F we use for the implicit models, which ensures that (for forward passes) and Jr (for backward passes) are contractive mappings, and is able to guarantee unique and stable fixed fixed-points due to the Banach Fixed

---

Point Theorem 33. Specifically, we adopt the nower-iteration in [20] 4 to scale the spectral norms of W in Eq. 7 and 8 after each training iteration if become 1, which incurs little additional computation cost 20 However, we note that contractivity is not always necessary for fixed-poin convergence (especially in the presence of stronger fixed-point solvers, as shown in large-scale DEQ models I], and weaker regularizers such weight normalization [25] may also suffice in practice.
4 Experiments
We evaluate our (Implicit)² networks on several representation tasks and compare the difference in performance between the implicit and explicit modeling of implicit representations across a variety of configurations. For both explicit networks and implicit networks, we use L to denote the number of layers and D to denote the hidden dimensionality of the features. Unless stated otherwise, our implicit models have exactly one F layer (i.e., L 1), in the exact form we described in $3.1 Our set of experiments drawn from prior works where competitive implicit representation networks are evaluated on [1] 27
Overall, our results of learning implicit representation on various domains (including images, videos, audios) suggest that the (Implicit) approach offers clear improvements over existing explicit models used for these purposes, where we are able achieve the same or better level of performance while being up x more memory-effiiient and 3x more time-efficient. We provide our implementatior and the descriptions the tasks and datasets can be found the appendix.
4.1 Image Representation
Gebor-MFN
Founee-MFN
STREN (input inj
We first evaluate the difference between explicit net- works and (Implicit) networks on representing a high-resolution 512 512 grayscale image, which a commonly used goalpost for evaluating implicit representation models from the scikit-image pack- age [30]. In particular, we fit an implicit representa- tion function & (Ir, - C, where C is the desired color space (and in this case C = R). We train each model for 5000 iterations (under the same setting) using all pixels in the image (i.c., batch size 262,144), and demonstrate the final peak signal-to-noise ratio (PSNR), memory consumption, and average training Figure 6: Learned 512 x 512 grayscale image step time in Fig. 5 The results convey two interesting facts about using (Implicit) networks: Com- pared to standard l-yeyrr explicit networks (4L-256D) even small implicit network (1L-256D)
Explict.
Implicit
Official implementation can be found at
Gabor- MFN
Fourier-MFR
SIREN (input inj.)
44
16.
PSNRA (1)
002
45.2
12
Meroeey/tm (2)
Stapy timee/ro (.).
Meroory/en (1)
Stepp time//m (1)
Xeemoryy/Cnn (1))
Step times/me (U)
6.5
300
4.5
120
3.5
130
2.8
2.3
10C
10C
36
Explitt 4L-256D
Implicit 1I:256D (umns)
Impllicit 1Z-5:2D (oun))
Figure S: Comparison of step time, memory consumption and PSNR between explicit and implicit models trained on the 512 x 512 grayscale image.
7

---

with approximately 75% fewer parameters achieves comparable performance, while requiring only roughly 1/3 of the training time and memory footprint. When given a higher parameter budget, an implicit network with similar model size (i.e., 1l-512D) outperforms the best explicit network by a large margin.
We also demonstrate the effect of tixed-poin reuse and truncated backward gradient using IL-512D Fourier MFN on fitting the same 512x512 image, with results shown in Fig. 3 and 4 Specifically, by reusing fixed-points in implicit layers, the solver takes significantly fewer steps to converge for the majority of the training period, making the forward evaluation of implicit layers drastically more efficient. Meanwhile, Fig. 4 shows that the Oth-order gradient approximation in [12] (i.e. T = 0), albeit efficient, could substantially hurt model performance. (i.e. = 1) greatly reduces the performance gap with only one additional vector-Jacobian product evaluation, thus achieving better balance between model efficiency and effectiveness.
contrast, our proposed approximation
4.2 Image Generalization
In number of applications 19 2 implicit representations are used to infer unobserved parts of data. In order to demonstrate the ability for (Implicit)² networks to generalize well, we train the network only 25% of the pixels from each image the Natural and Text dataset, following [1I] and evaluate PSNR on an unobserved 25% portion of the image. The average PSNR over all 16 high-resolution images in Natural and Text is reported in Table and visualization of the improvement by (Implicit)* models is shown in Fig.
be seen that (Implicit)² networks improve substantially over the explicit baselines, where the best-performing implicit model outperforms the explicit counterpart by > 0.7 in PSNR using the same architecture. Visually, (Implicit) networks produce sharper and shape-consistent representation compared with the best performing explicit network. We observe similar improvements in training speed and memory consumption in the image representation task (about 3x). For completeness,
Gabor-MFN
Fourier-MFN
SIREN
Explicit
Implicit
Figure 7: Samples of best performing explicit/implici models learned on Natural
Natural
Tea
1L-256D
1L-512D
4L-256D
1L-256D
1L-512D
4L-256
Fourier MFN
23.27 3.18
23.30 1. 3,05
24.57 3.35
24.64 2.11
24.84 1. 2.10
26.67 d: 2.06
Foarier-iMPN (ours)
24.88 44
25.19 3.64
24.52 3.33
2.14 26.90
27.19 1: 1.83
26.48 = 2.04
Gabor-MEN
24.16 + 3.35
24.68 3,46
24.65 + 3,38
27.19 + 2.18
27.74 + 2.13
27.57 = 2.10
Gabor-iMI-N (ours)
24.91 +: 3.41
25.42 +: 3.76
24.53 3.33
2.18 27.53
28.07 :: 2.00
27.40 = 2.10
SIREN (input inj.)
22.88 3.0
24.52 4: 3.28
24.10 3.34
24.54 1: 2.19
25.69 1: 2.18
26.21 = 2.19
ISIREN (ours)
24.28 + 3.37
24.92 + 3,58
24.05 3.39
26.06
2.18
26.81 + 2.09
26.31 = 2.20
Table 1: PSNR (in dB) for all models image generalization. The reported mean + std is taken over the individual PSNR of the 16 images.
8

---

we also evaluate an (Implicit) network where F also consists of a stack of 4 layers (rather than 1), and observe that the implicit modeling of an already deep structure does not yield much improvement.
4.3 Audio Representation
1L-256D
1L-512D
4L-256D
We further show that the proposed (Implicit) network outperforms ex- plicit networks on representing very- high-frequency audio signals. Follow- ing [27], we train the models to fit a 7- second music piece. We observe that in all cases, the implicit modeling sig- nificantly outperforms the respective explicit models.
Explict
PSNI:
PSNR $1.37
PSNK 30.42
Implicit
In Fig. g we show the reconstruction results and the respective errors when
PSNR: 35.64
PSNR: 43.79
PSNR: 47.29
Figure Audio signals represented using Fourier-MENs. Lower error magnitudes (in light blue) are better.
using Fourier-MIN to fit the audio sig- nal, with more results available the appendix. Although an (Implicit) model only defines one layer, it is able to outperform explicit models in terms of PSNR by more than 23% (43.79 vs. 35.42). while still retaining the same level of memory and speed improvements as before. We also find that, in this case, having deeper architecture in an implicit layer (implicit 4T.2556D leads to improved performance.
4.4 Video Representation
We may add an extra dimension to the input of implicit representation func- tion and try to learn a model for video sequences Φ (1, y) - C, where I
IL-1024D
LL-2048
4L-1024D
Fourier-MFN
24.97 1.08
26.9 + 0.97
27.64 + 0.90
Fourier-IMEEN (ours)
25,85 1.00
27.7 + 0.90
28.03 + 0.95
is the space of time. We aim to rep-
Gabor-MFN
26.15 1.02
0.78
29.64 0.81
resent a 300-frame 512 512 video using each model. The PSNR results
Gabor-iMPN (ours)
26.45 0.98
28.79 0.77
29.20 + 1.04
are shown Table 2
SIREN (input inj.)
25.12 0.96
26.04 0.99
26.52 0.86
iSIREN (ours)
26.03 0.92
27.08
0.93
27.12 0.89
Unlike in the image representation set- ting, we observe that, due to the rel- atively few parameters compared to the size of the video data (the smallest model only has as many parameters as 1.3% of all pixel values), model sizes have a more significant impact on the overall performance. But still, in most configurations, we found (Implicit) models lead to consistently non-trivial improvement in performance when compared to their explicit counterparts, while doing so with equivalent or less memory budget.
Table 2: PSNR for the video representation task. The re- ported mean + std is taken over all frames of the video.
4.5 3D Geometry Representation
To further demonstrate the benefit of (Implicit)? networks over their explicit counterparts, we choose three formulations of Fourier-MFN with similar parameter count (i.e. explicit 1 L-512D, explicit 4L- 256D, and implicit L-512D) and fitted them on several 3D object meshes with the point occupancy prediction objective similar to [8] given input coordinate = (x, y. z). the model tb : IR3 + IR is trained to predict a binary label indicating whether the point corresponding to the coordinate is located inside the target object (0 if the point is outside the object, and 1 if inside). The visualized normal maps of the learned representations on one of the objects are shown in Fig. [ We further evaluated the prediction IoU over test points densely sampled near the mesh and present them in the same figure. Experimental details and results on additional 3D models are available in the appendix.
The results show that, with similar model size, the lne-layer network modeled with the (Implicit) formulation is able to capture more details than an explicit model with an identical (1L-512D) or a deeper (4 (4T.-266D) D) structure, and at the same time having a training time and memory advantage similar to the ones in Fig. [5] Such an example demonstrates the superior parameter and memory efficiency of (Implicit) networks.

---

Explicit 1L-512D
Explicit 4L 256D
Implicit LL-512D
Ground Truth
BoU:
lol:
Figure 9: Normal maps and loUs of fitted dragon object using Fourier-MFNs in Occupancy Net- work [8. The (Implicit) approach performs better than its deep explicit counterpart.
We note that in both the video and 3D geometry tasks, the signal is either of large dimension or defined continuously, where we may not fit the entire data into a single batch. Therefore, the implicit models usually entail more than one step to reach convergence due the challenge in fixed-point reuse with mini-batch training (see a more detailed analysis in appendix ??). Nevertheless, our proposed method still able boost the training efficiency by leveraging truncated gradient, while improving upon its explicit counterparts in most cases.
Limitations and Directions for Future Work
Our (Implicit) approach can be significantly more time-efficient and memory-friendly than explicit networks at training time: however, at inference time, the implicit models must resort the normal fixed-point solving. Furthermore, accelerated training in the forward pass by fixed point reuse (see $3.2) in practice can be bottlenecked by storage or hardware I/O bandwidth constraints; i.e., works best when we can fit all or a large portion of the fixed point into memory for quick retrieval. Although such assumption can be easily satisfied for implicit representation learning in many cases, for extremely large data spaces (e.g., in the continuous domain, like Park et al. [22] and Mildenhall al. 19, where samples are drawn continuously in RR3 or very long video frames like 4.4). the fixed point reuse may not always work as well and could require more solver steps.
Smarter caching strategies for the fixed points may be developed in the future, where we consider only subset of the input space for caching throughout the training. For example, one may consider a voxel-grid-basee sparse storage for the fixed points, similar to the ones used in [16], such that fixed points of any batch of inputs may be approximately obtained from interpolation of the sparse storage. We leave this direction for future work.
6 Conclusion
In this paper, we propose (Implicit) networks, an application implicit layers to implicit repre- sentation tasks. We show that these two concepts of "implicitness" complement each other well, allowing us take advantage their properties (e.g., fixed point reuse) to produce a significantly more efficient training routine and usually more parameter-efficien models for implicit representation learning. We demonstrate through our set experiments that the implicit modeling of implicit representations may in many cases be conveniently used as a drop-in replacement for existing state- of-the-art explicit models like SIREN and MFN further improve the performance and reduce the memory/computation budget required to train these tasks.
10

---

7 Acknowledgement
We thank Swaminathan Gurumurthy for reviewing the paper and for helpful discussions.
Funding Disclosure
Shaojie Bai was sponsored by grant from the Bosch Center for Artificial Intelligence.
References
[1] Bai, J. Kolter, and Koltun, Deep equilibrium models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[2] Bai, V. Koltun, and J. Z. Kolter. Multiscale deep equilibrium models. In Advances Neural Information Processing Systems (NeurIPS), 2020.
[3] Banach. Sur les opérations dans les ensembles abstraits leur application aux équations intégrales. Fund. math, 3(1 13):133-181, 13):
1922.
[4] Behrmann, W. Grathwohl, R. T. Chen, D. Duvenaud, and J.-H. Jacobsen. Invertible residual networks. In International Conference on Machine Learning, pages 573-582. PMLR, 2019.
[5] M. Bemana, K. Myszkowski, H.-P. Seidel, and T. Ritschel. X-fields: Implicit neural view-, light- and time-image interpolation. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2020), (9(6), 2020. doi:
[6] C. G. Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of Computation, 1965.
[7] T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems (NIPS), 2018.
[8] Y. Chen, S. Liu, and X. Wang. Learning continuous image representation with local implicit image function. arXiv preprint arXiv:2012.0916/ 2020.
[9] E. Dupont, A. Doucet, and Y. W. Teh. Augmented neural ODEs. Advances in Neural Information Processing Systems (NIPS), 2019.
[10] D. Duvenaud, J. Z. Kolter, and M. Johnson. Deep implicit layers tutorial neural ODEs, deep equilibirum models, and beyond. Neural Information Processing Systems Tutorial, 2020.
[11] R. Fathony, K. Sahu, D. Willmott, and Z. Kolter. Multiplicative filter networks. In International Conference on Learning Representations, 2021. URL httpps///peenrreirrr
[12] S. W. Fung, H. Heaton, Q. Li, D. McKenzie, S. Osher, and W. Yin. Fixed point networks: Implicit depth models with Jacobian-free backprop. arXiv:2103. 12803, 2021.
[13) K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[14] S. G. Krantz and H. R. Parks. The implicit function theorem: History, theory, and applications. Springer, 2012.
[15] R. Liao, Y. Xiong, E. Fetaya, L. Zhang, K. Yoon, X. Pitkow, R. Urtasun, and R. Zemel. Reviving and improving recurrent back poockpropagation. In International Conference on Machine Learning (ICML), 2018.
[16] I.. Liu, Gu, K. Z. Lin, T.S.. Chua, and Theobalt. Neural sparse voxel fields. NeurIPS, 2020.

---

[17] C. Lu, J. Chen, C. Li, Q. Wang, and J. Zhu. Implicit normalizing flows. In International Conference on Learning Representations (ICLR), 2021.
[18] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4460-444 2019.
[19] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference Computer Vision, pages 405-421. Springer, 2020.
[20] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. URL
httpp:///ppenevvev..eett/ wttp:///ww..eetttmmadddddddaaaad B1QRgziT-
[21] M. Niemeyer, I.. Mescheder, M. Oechsle, and A. Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3504-3515, 2020.
[22] Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[23] D. W. Peaceman and H. H. Rachford, Jr. The numerical solution of parabolic and elliptic differential equations. Journal of the Society for industrial and Applied Mathematics, 3(1): 28-41, 1955.
[24] M. Revay, R. Wang, and R. Manchester. Lipschitz bounded equilibrium networks. arXiv:2010.01732, 2020.
[25] T. Salimans and D. P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems (NIPS), 2016.
[26] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d-structure-ave awaree neural scene representations. In Advances in Neural Information Processing Systems, 2019.
[27] V. Sitzmann, J. N. Martel, A. W. Bergman, Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020.
[28) M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Kei,, N. Raghavan, U. Singhal, R. Ra- mamoorthi, Barron, and Ng. Fourier features networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.
[29] A. Tewari, O. Fried, J. Thies, V. Sitzmann. S. Lombardi, K. Sunkavalli, R. Simon, J. Saragih, M. Nicßner, et al. State of the art on neural rendering. In Computer Graphics Forum, volume 39, pages 701-727. Wiley Online Library, 2020.
[30] S. der Walt, J. L. Schönberger, J. Nunez-Iglesias, F. Boulogne, J. D. Warner, N. Yager, E. Gouillart, T. Yu, and the scikit-image contributors. scikit-image: image processing in Python. PeerJ, 2:e453, 6 2014. ISSN 2167-8359. doi: 10.7717/pperj.453. URL https
//doi.org//11 (7t1/pee7717/eerr 453
[31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances Neural Information Processing Systems (NIPS), 2017.
[32] E. Winston and J. Z. Kolter. Monotone operator equilibrium networks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
12


Reviews:
[{"review_id": "yQt5y1Xf9pM", "reviewer": "Reviewer_7RDY", "summary": "Summary\n------------------\nThis manuscript proposes the use of implicit differentiation (i.e. using the implicit function theorem)\nto train variants of so-called \"implicit\" representations (also known as \"coordinate based networks\") \nthe authors develop.\nSpecifically the manuscript extends the formulations of SIRENs and MFNs in the form of Deep Equilibrium \nNetworks (DEQ) that can thus be trained through implicit differentiation. To make training of those \narchitectures efficient, the manuscript suggests two approximations. The first approximation lies in the \nforward pass and circumvents the use of an expensive Fixed Point Solver. The second approximation\nlies in the backward pass and circumvents the expensive inversion of typically \"large\" Jacobian matrices.\nThe authors evaluate their new architectures and training schemes on image, audio and video fitting and\nshow that smaller Implicit^2 networks can fit signals better than SIREN or MFNs.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main review\n------------------\nI find the A+B idea of mixing two seemingly orthogonal concepts (implicit representations and implicit\nrepresentations) interesting. The writing is globally clear, even though the style is rather \nheterogeneous through the manuscript. Some sections would definitely deserve some more love (and clarity).\n\nThe motivation of the paper is also clear to me: we want better and more memory friendly \"implicit \nrepresentations\" and the use of implicit differentiation techniques that are typically cheaper \n(e.g. in Neural ODEs with the method of the adjoint) seems promising!\n\nDespite those interesting ideas, and the promise of more memory efficient implicit representations (at the\ncost of more time-consuming forward and backward passes), the paper fails at convincing me this is the case.\nIn fact, if I understand correctly Figure 5, a glance at it shows immediately that there is no gain in the \nmemory consumption between the implicit differentiation method proposed by the authors and the \"explicit\"\ndifferentiations of conventional SIRENs and MFN. Furthermore, implicit differentiation is indeed slower. All \nin all, we end up with a slower method that is as memory consuming. Therefore, I would absolutely requalify\nthe claims in the introduction.\n\nAs a matter of fact, the authors end up twisting those initial claims (that turn out to be misleading) \nthrough the paper, and we arrive at Section 4 where essentially the main claim is that with a smaller \nnumber of parameters (a smaller architecture) the architecture and implicit differentiation scheme\nproposed by the authors seem to perform better than a SIREN or MFN. As a summary, Implicit^2 has the \nsame memory consumption for an equivalent architecture as a conventional SIREN or MFN, runs slower,\nbut seems to perform better! Consequently, if the paper is accepted, the storyline must be reworked, \nthe motivation on why implicit differentiation helps to do so too, and hints or experiments on why \nthose architectures perform better should be provided.\n\nOriginality and significance\n----------------------------------------\nI find this work interesting, I am not completely sure about its level of significance but believe significance \nwould be increased if code for implementing and training this architecture was distributed. In terms of\noriginality, this work falls in the category of A+B papers, combining two existing ideas, it seems relatively\nwell executed: results despite being on toy problems seem promising.\n\nQuestions\n------------\n- Can the authors explain why the implicit^2 architecture ends up consuming as much memory as\nSIREN or MFN for a same architecture contrary to what was expected and motivated the whole paper?\n- Can the authors explain/hint why this is the case that the SIREN or MFN perform worse than \nImplicit^2 on the tasks shown in the paper? Is there some inductive bias in Implicit^2 that helps\nfor the task presented? Is it simply that those optimize better?\n\n\nMore specific comments and typos\n-------------------------------------\n- I would suggest to improve the caption of all the figures so that they can be self-contained,\nat the moment, many of this figures cannot be understood at a glance.\n- I do not think the choice of Figure~5 is very good: again it does not convey the main claim \nof the paper: that implicit^2 is helping in terms of memory consumption. Now, if the point to \nconvey is that PSNR with a smaller Implicit^2 architecture can be better than with a bigger\nconventional SIREN or MFN architecture, other plots or tables would be more appropriate.\n\nTypos\n----------\n- l.181 correspsonding network -> corresponding network\n- l.195 diemsnional -> dimensional\n- l.210 this do not -> does not\n- l.211 at trianing iteration -> at training iteration\n- l.238 to ex scale -> ??\n- l.272 achieve -> achieves", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I am not sure this is applicable to this paper.", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "qU9ZNWzrml2", "reviewer": "Reviewer_YkMx", "summary": "(Implicit)^2 models bring implicit computation to bear on implicit representation. Deep equilibrium network editions of the SIREN and MFN implicit representation architectures are defined and experimentally compared and contrasted with their standard explicit counterparts. The (implicit)^2 editions are optimized with two proposed approximations: reuse of fixed points during forward and truncation during backward. When these approximates are used with the proposed implicit architectures the training time and memory alike are reduced to constant w.r.t. the effective \"depth\" of the model as counted in fixed point iterations. This differs from explicit models, where time and memory scale with the number of layers (L), and standard use of deep equilibrium nets, where time scales with fixed point iterations (M) and memory is constant. As a further point on optimization, regularization by spectral normalization is found to be useful for (implicit)^2 training. Analysis experiments justify each approximation, in that the effects are clear in Figures 3 & 4, but the conditions of these experiments are not specified. The quality of results for explicit and implicit^2 are compared for a single input each for the three modalities of image, audio, and video. (implicit)^2 improves PSNR in all cases, and even by 10 or more percent relative for a 512x512 grayscale image and 23 percent relative for a 7-second sound. Accuracy (PSNR) and computation (memory usage in GB and training step time in sec.) are measured across three architectures for one input image: accuracy is higher or equal across model sizes, the text claims 3x faster training speed (in total?) and the text specifies memory saving though the reported charts do not (see Figure 5). The text motivates the computational attractiveness of implicit computation for implicit representations, but does not cover alternatives for reducing time or memory with explicit computation for implicit representations (like sampling/mini-batching, gradient accumulation, reversibility, model decomposition like in KiloNerf or ACORN etc.).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty:\n- This is the first work to bring implicit computation to bear on implicit representation.\n- Two implicit variations on implicit representation architectures are proposed, for SIREN and MFN, which are both recent and good approaches.\n- The forward approximation of fixed point reuse and its experimental analysis are new, and the backward approximation of truncation is more related to other work (as discussed in the text), but the suitability of the chosen truncation (T = 1) is newly shown.\n\nSignificance:\n- Implicit computation and implicit representation are both burgeoning topics that differ from the established mainstreams of explicit deep network architectures and grid/mesh/etc. representations respectively. This work joins both, and points out their potential complementarity, and so more work at this interface could follow.\n- Improving accuracy (PSNR) on two strong architectures is impressive. However, the scope of the experiments is too narrow to know if the improvements are real (see next point). The quantification of the claimed computational advantages are also poorly measured/communicated: memory usage seems to be the same across explicit/implicit (Fig. 5) and differences if any in results w/ and w/o the proposed approximations are not covered.\n- The results are too few to reliably measure quality w.r.t. the explicit counterparts of the proposed (implicit)^2 models. It is certainly a plus to consider three different data modalities, but that is a standard already established by prior work like SIREN, and comparisons need to be done on more than one input per modality. One example does not equal an evaluation.\n- The experiments may not be properly tuned or controlled. For instance, batch size was chosen by the convenience of what would fit the largest model in memory (see supplement). Batch size and learning rate schedule can be sensitive hyperparameters, for example, so they should have been chosen by validation or used known good choices from existing works when comparing to them. Spectral normalization is applied to the (implicit)^2 models proposed, but not to the corresponding explicit models, and so results are potentially confounded by this difference.\n\nClarity:\n- The overall organization is good, with clearly named sections, and the main conceptual figures (1 & 2) summarize the ideas.\n- Key results, like the justification for the proposed forward/backward approximations, are not grounded in the concrete details of the model/data/optimization/and experimental conditions in general. None of these can be determined from the captions or text for Figures 3 and 4 (or at least I could not after several readings).\n- The background on implicit computation is too vague: \"designed to have no prescribed computation graph\" is not quite right, as the computation graph of the iterated module is definite, but its number of iterations and the dynamics of its latents are not.\n- There are a number of typos throughout, which while superficial, do nevertheless distract the reader. (See draft feedback.)\n\nDecision:\nThis work's union of implicit computation and implicit representation is promising but preliminary. Trying to make the best of both is insightful, and the proposed approximations to make good on their combination are reasonable. The difference in approach is clear for implicit^2 vs. existing implicit and explicit computation, but the difference in experimental results is more potential than proven. Results are shown for a single image, a single sound, and a single video and the analysis for key properties underlying the proposed computational acceleration (fixed point reuse in Figure 3 and gradient truncation in Figure 4) are illustrative in effect but totally unexplained in experimental conditions (like architecture, data, etc.). Furthermore, it would be informative to check the use of this work's forward/backward approximations on DEQs for other purposes than implicit representations: it is fine if they work or not, but we would learn more about implicit vs. explicit. The quality of results, computational costs, and justification of the acceleration schemes are the main points of the paper and so they deserve more complete and systematic evidence.\n\nFor rebuttal:\n- Please explain the actual effects on memory usage, since Figure 5 seems to indicate no memory is spared, but the text claims reductions.\n- Please elaborate on the computation during training vs. testing of (implicit)^2 and explicit computation. Is it more expensive to query (implicit)^2 than explicit computations of implicit representations? Explicit could use less memory during testing in not needing the optimizer, right?\n- Please report on the accuracy/computation of (implicit)^2 against the simple baseline of explicit computation with mini-batch sampling (that is, SGD on subsampled batches of input coordinates and outputs) or gradient accumulation. Does (implicit)^2 strictly dominate?\n\nDraft Feedback\n- typos: l.91 missing references with empty \"[]\", l. 99 \"paramterized\", l.111 \"infintesimal\", l. 195 \"diemsnional\", l. 203 \"ponits\", l. 211 \"an simple\", l. 211 \"trianing\"\n\n**Post-Response**: The clarifications and additional results in the response have convinced me to raise my score to a 6. As the exposition and experiments still seem somewhat premature, and it seems some editing may be needed to address the nuances of the computational requirements (which are a main contribution of this submission), I cannot raise the score higher. Nevertheless the combination of implicit computation and implicit representation deserves more attention, and so this work can be considered for pointing in this direction with the first results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A computational limitation specific to this work is addressed by Section 5. However, the section is exclusively about this one computational limitation for storing past fixed points (state used for optimization). Discussion of potential limitations with regard to data fidelity or modality is missing, especially with respect to explicit computation for implicit representations. Regarding potential societal impacts, this work is about a generic method to improve the training of implicit representations, and as such does not have specific concerns.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "IhlFtuhpzY", "reviewer": "Reviewer_R9UM", "summary": "The paper proposed $($Implicit$)^2$, which replaces the internal layer of implicit neural representation (INR) with implicit layers. The proposed method is efficient in terms of memory and training time. For the fast training, the author proposed the re-usage of fixed-point and also used truncated backpropagation (approximated gradient). The method is sensible and also empirically demonstrates its effectiveness.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-----------------------------------------------------------------\n**Strength**\n1. The motivation of using an implicit layer into implicit neural representation (INR) makes sense (as it significantly reduces memory utilization).\n\n2. The performance gains (for both efficiency and training time) are significant.\n\n3. Although there are no theoretical guarantees, the authors empirically showed that the re-usage of fixed-point and truncated back gradients are sensible.  \n\n4. The writing and presentation are clear.\n-----------------------------------------------------------------\n**Weakness**\n\n**1. Some scalability concerns: the experiments are demonstrated under somewhat small-scale datasets**\n- If possible, can the author show experiments requiring a large number of INR layers, i.e., a complex signal? For instance, 3D shapeNet needs eight layers to be represented with a Fourier feature network [2]. Demonstrating experiments under such conditions will be interesting for $($Implicit$)^2$.\n- Alternatively, datasets that are used in Neural Radiance Fields [1] will be significant.\nI believe such experiments are the mainstream task of INR.\n\n**2. (Minor) Comparison of inference time**\n- I recognize that direct comparison of inference time is somewhat difficult (due to CPU utilization for the solver). However, still believe the paper should include such part. \n- If possible, can the author compare the inference time of $($Implicit$)^2$ with the baselines?\n\n**3. (Minor) The variance seems to be big in Table 1** \n- Can the author evaluate the images for a large number than 16, e.g., 250 samples, by following [3]? (Of course, the author can use a smaller dataset, e.g., CelebA)\n-----------------------------------------------------------------\n**Justification of the rating**\\\nI recommend accept. I believe the idea of the implicit layer is well suited for INR and also shows significant efficiency. I do enjoy reading the manuscript.\n\nI still have some concerns about the scalability issues; therefore, I politely request the authors to share the experimental results mentioned in the weakness part.\n\n-----------------------------------------------------------------\n**Reference**\\\n[1] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Mildenhall et al., ECCV 2020\\\n[2] Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. Tanick et al., NeurIPS 2020\\\n[3] Learned Initializations for Optimizing Coordinate-Based Neural Representations. Tanick et al., CVPR 2021\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations.\\\nFor societal impacts, I agree that this work may not have negative impacts.\n", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns found.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "HPkPl1Fd_M", "reviewer": "Reviewer_6HGN", "summary": "This work suggests that combining implicit-depth models (e.g. DEQs) with implicit representation approaches (e.g. SIREN) can bring performance improvements. The method is evaluated on image, audio and video representation.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work aims to develop several ideas simultaneously and does not always achieve the prescribed goal. At its core, this paper is about combining implicit-depth models with implicit representations to improve empirical performance in data representation tasks. The second major component of this paper is an attempt at introducing methodological improvements to implicit-depth models (particularly DEQs [1]) with root finding solution reuse and truncated gradients. \n\nThe work does a really poor job at motivating the combination of these two models (beyond the similarity in nomenclature), or performing an analysis on the reasons why performance improvements should even be expected. The novel material (page 5 onwards) introduces a few changes that can be applied to all DEQ models. These have nothing to do with implicit representations, yet in this work they are sold as a core component of the model. See for example Figure 1, where (Implicit)$^2$ is exactly a DEQ  with the proposed improvements.\n\nThe proposed improvements to implicit models are reasonable. Root reutilization is definitely useful and is simultaneously being proposed in other recent papers on the topic. I would also like to point out that the original DEQ implementation also features root solution reuse (see for example [2]), so this section not particularly surprising. Truncated gradients and spectral regularization are similarly natural improvements over the DEQ formulation which I was glad to see here. I would suggest the authors decouple these improvements with their combination of models, perhaps introducing them beforehand.\n\nTo me, the greatest limitation of this work are the experiments. Given that the work does not appear motivated by theory or particular insights, I expect strong results to carry the work. I'm seeing marginal improvements, with the most impressive being SNR of smaller implicit Gabor/Fourier-MFN compared to larger explicit variants. Implicit SIRENs actually seem to perform worse in several tasks, see for example audio reconstruction in the supplementary (Figure 3). Image and 3D model representations do appear slightly sharper. The code submission is organized well.\n\nOverall, I believe this paper is borderline, and I am definitely willing to modify my score, particularly if the authors can provide a stronger motivation behind the merging of these two modeling frameworks. As it stands, I see this work only as a preliminary exploration which might spur more definitive follow-up work on better implicit representation models.\n\n\n* (1) Deep Equilibrium Models\n* (2) https://github.com/locuslab/deq/blob/beta/DEQModel/models/transformers/deq_transformer.py\n* (3) Stabilizing Equilibrium Models by Jacobian Regularization\n\n-----------------\n\nMinor:\n\n> While the modification seems simple, the addition of implicitness in fact yields a much richer set of functions than the ones can be represented by a forward pass of F alone\n\nThis is wrong as a general statement and is not proven anywhere in the paper. Hand-wavy general comments such as this one have no place in the paper.\n\n> Therefore, the forward and backward passes of implicit models can rely on independent black-box solvers, and these models consume only a constant training memory (as they do not need to store anything)\n\nIncorrect and misleading. Implicit models (both Neural ODEs and ) require storage of various quantities to ensure low errors. Backsolve continuous adjoints for Neural ODEs can give you gradients at no additional memory cost, but can also introduce unbounded errors depending on the solver. Similarly, DEQs require storage of root solutions for faster training. Repeating the fixed point iteration at each training step, while free of memory overheads, is often too slow. There are also other costs discussed in [3] (3.4)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide a satisfactory discussion on limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the discussion and rebuttal process, all reviewers now recommend acceptance, albeit with several borderline ratings. Several reviewers flagged limitations of the experimental evaluation, which the authors have responded to by running additional experiments. The discussion also cleared up some confusion about memory savings, which seems to have stemmed from a misinterpretation of Figure 5.

Overall, reviewers seem to agree that the combination of implicit computation and implicit representations is interesting and creates some nice synergies, and also that this idea deserves the attention of the research community. I am therefore inclined to recommend acceptance, though this is conditional on the authors adding the new results to the paper and addressing the various other remarks and recommendations made by the reviewers (e.g. with respect to Figure 5). 
Decision: Accept (Poster)