Multiplicative Filter Networks
Authors: Rizal Fathony
Anit Kumar Sahu
Devin Willmott
J Zico Kolter
Venue: ICLR 2021 Poster (2021)
Keywords: Deep Architectures
Implicit Neural Representations
Fourier Features

Abstract: Although deep networks are typically used to approximate functions over high dimensional inputs, recent work has increased interest in neural networks as function approximators for low-dimensional-but-complex functions, such as representing images as a function of pixel coordinates, solving differential equations, or representing signed distance fields or neural radiance fields.  Key to these recent successes has been the use of new elements such as sinusoidal nonlinearities, or Fourier features in positional encodings, which vastly outperform simple ReLU networks.  In this paper, we propose and empirically demonstrate that an arguably simpler class of function approximators can work just as well for such problems: multiplicative filter networks.  In these networks, we avoid traditional compositional depth altogether, and simply multiply together (linear functions of) sinusoidal or Gabor wavelet functions applied to the input.  This representation has the notable advantage that the entire function can simply be viewed as a linear function approximator over an exponential number of Fourier or Gabor basis functions, respectively.  Despite this simplicity, when compared to recent approaches that use Fourier features with ReLU networks or sinusoidal activation networks, we show that these multiplicative filter networks largely outperform or match the performance of these recent approaches on the domains highlighted in these past works.

Published as a conference paper at ICLR 2021
MULTIPLICATIVE FILTER NETWORKS
Rizal Fathony
Anit Kumar Sahu
Bosch Center for Artificial Intelligence
Amazon Alexa Al Seattle, WA anit. www.tgaalhegge.m....... com
Pittsbrgh, PA
rizal. fazhonyathoyeeus bosch. com
Devin Willmott"
J. Zico Kolter
Bosch Center for Artificial Intelligence Pittsuurgh, PA devin, wi 1 mott @us. bosch. com
Bosch Center for Artificial Intelligence Carnegie Mellon University Pittsburgh, PA zkolterêcs cmp. edu
ABSTRACT
Although deep networks typically used to approximate functions over high dimensional inputs, recent work has increased interest in neural networks as func- tion approximators for www-dimensional-but.c comtpllpe functions, such as represent- ing images as a function of pixel coordinates, solving differential equations, or representing signed distance functions or neural radiance fields. Key these re- cent successes has been the use of new elements such as sinusoidal nonlineari- ties or Fourier features in positional encodings, which vastly outperform simple ReLU networks. In this paper, we propose and empirically demonstrate that an arguably simpler class of function approximators can work just as well for such problems: multiplicative filter networks. In these networks, we avoid traditional compositional depth altogether, and simply multiply together (linear functions of) sinusoidal Gabor wavelet functions applied to the input. This representation has the notable advantage that the entire function can simply be viewed as a linear function approximator over exponential number of Fourier Gabor basis fune- tions, respectively. Despite this simplicity, when compared to recent approaches that use Fourier features with ReLU networks sinusoidal activation networks, we show that these multiplicative filter networks largely outperform or match the performance of these approaches the domains highlighted in these past works.
INTRODUCTION
Neural networks most commonly used to approximate functions over high-dimensional input spaces, such as functions that operate on images or long text sequences. However, there has been a func-
recent growing interest in neural networks used to approximate now-dimeensoonal-but-ooomp tions: for example, one could represent continuous image as function f R²2 - R where the input to this function specifies (ar, y) coordinates of a location in the image, and the output specifies the RGB value of the pixel at that location. However, two recent papers in particular have argued that specific architectural changes are required make (fully-connected) deep networks suitable to this task: [Sitzmann et al. (2020) employ sinusoidal activation functions within a multi-layer net- works (called the SIREN architecture); and Tancik et al.] (2020) propose random Fourier features input to traditional ReLU-based network. Both papers show that the resulting networks can ap- proximate these wiw-dimensional functions much better than simple feedforward ReLU networks, and achieve striking results in representing fairly complex functions (e.g. 3D signed distance fields or neural radiance fields) with a high degree of fidelity. However, the precise benefit of sinusoidal bases first layer of Fourier features such representations work well for these tasks.
difficult characterize, and remains unclear why
'The first three authors contributed equally, listed in alphabetical order by last name.
Work done while Bosch Center for Artificial Intelligence.

---

Published as a conference paper at ICLR 2021
In this paper, however, we argue and empirically demonstrate that arguably simpler class of fune- tions can work as well or better than these peeviously-rooposee networks on this task. Specifically, we propose architecture multi-layer network that achieves representation power through compositional depth, the MFN in- stead simply repeatedly applies nonlinear filters (such as a sinusoid or a Gabor wavelet function) to the network's ' imput, then multiplies together linear functions of these features. The notable advan- tage of this representation that, owing to the multiplicative properties of Fourier and Gabor filters, the entire function is ultimately just a linear function of (an exponential number of) these Fourier or Gabor features of the input. Indeed, we can express the exact linear form of these MFNs, which can make their analysis considerably simpler than that for deep networks, where compositions of nonlinear activation's 's make the entire function difficult to characterize.
call the multiplicatiie filter network (MFN). Unlike traditional
In this work, we show that despite this simplicity, the proposed networks often perform as well or better than the previously proposed SIREN or Fourier feature networks. Specifically, we compare our approach posed in the SIREN and Fourier features papers. We show that MFNs achieve better performance deltas when increasing the depth width of the networks. Despite this, we do emphasize that SIREN networks, in particular, appear to retain some notable advantages over MFNs, such as a bias towards smoother regions in the represented function and its gradients. However, especially given the fact that MFNs ultimately just correspond to a linear Fourier or Wavelet representation of a low- dimensional function, we believe they should be considered a standard benchmark for future work on such problems, to indicate where the compositional depth of typical deep networks can propose a substantial benefit.
networks with comparable numbers of parameters to the exact benchmarks pro-
2 BACKGROUND AND RELATED WORK
Our approach is related many previous works in Fourier and Wavelet transforms, random Fourier features, and implicit neural representations. We explore the connection among the areas below.
Fourier and Wavelet transforms. Transforming time or space domain signals to frequency domain using transforms such as Fourier and Wavelet transforms have been at the heart of many develop- ments in image processing, signal processing, and computer vision. In particular, the Fourier trans- form (Bracewell & Bracewell, 1986 Vetterli et al. 2014) and its various forms have found usage in myriad applications, such as spectroscopy, quantum mechanies, signal processing. Wavelet trans- forms, which in particular aid in multi-scale analysis, have been found to be particularly useful in data compression, JPFG2000 (Rabbani 2002) being one example.
Random Fourier features. A seminal work by Rahimi & Recht (2008) demonstrates the power of Fourier transform machine learning applications. They show that simply projecting the original dataset into random Fourier bases vastly improves the expressiveness of models as it approximates kernel computations, Many subsequent works apply the Fourier features and variations (Rahimi & Recht 2009 Le et al.) [2013] Yu et al., (016)) to improve machine learning algorithm performance in many domain areas, including classification (Sun et al. [2018] 8: Rawat et al. (2019), regression (Avron
et al.
2017 Brault et al.) 2016), clustering (Chitta et al. 2012 Liu et al. (2019), online learning (Lin
et al.
2014 Hu et al. 2015), and deep learning (Xue et al.
2019 Mehrkanoon & Suykens 2018
Rick Chang et al. 2016 Mairal et al. 2014 Jacot al. 2018 8 Tancik et 2020).
Implicit neural representations. A recent line of work in representing signals as a continuous function parameterized by neural network (instead of using the traditional discrete representation) is gaining popularity. This strategy has been used to represent different objects such as images
(Nguyen et al.] [2015] Stanley| 2007). shapes (Park et al., 2019; Genova et al. 2019 Chen & Zhang
2019]Chabra et al. 2020). scenes (Mildenhall et al. 20200 Sitzmann et al.
2019 Doing et al. 2020
Niemeyer et al., (2000, and textures (Oechsle et al.) 2019 Henzler et al., 2020p. most these
applications, the standard neural networks architecture with multi-layer perceptrons and ReLU ac- tivation function is often used. Recently, motivated by the success of Fourier transform in machine learning, a few papers have suggested architectural changes that integrate periodic nonlinearities into the network. Mildenhall et al. (2020); Zhong et al. (2020): Tancik et al. (2020) proposed the use of sinusoidal mapping of the input features (Rahimi & Recht 2008) that uses positional encoding and Gaussian random distribution in the mapping. Others (Klocek et al., 2019; SSizzmann et al., (020) have proposed the use of sinusoidal activation function within multi-layer perceptron architecture.
2

---

Published as a conference paper at ICLR 2021
Both of these strategies are demonstrated to vastly improve the results many object representation tasks.
MULTIPLICATIVE FILTER NETWORKS
A traditional k-layer deep network f R" + R is typically defined by recurrence such z(1)
((41) -o (w(() + 6(0) i I,, -....,,--- --
(1)
f(x) = w(() (1)) + (b)
where a denotes nonlinearity applied elementwise, W E Rd411xd; and b(() E R444 denote the weight and bias of the ith layer, and 2(*) Rd denotes the hidden unit at layer i. We refer to these networks as compositional depth networks, because each nonlinearity is applied compositionally to outputs of the previous nonlinearity in order to achieve its representational complexity.
The SIREN or Fourier feature networks (Sitzmann et al., 2020) and Tancik al. (2020) respec- tively can be viewed simple specializations of this structure. In SIREN network, one uses the sinusoid ar (x) = sin(x) as the nonlinearity, plus proper initialization of the weights and scaling of the input. In Fourier features network, one replaces the input layer with
(1) sin [2x + 6) cos(S2> + σ)
(2)
where Ω E R xn is matrix of random (0, r2) variables (r being a hyperparameter of the method), but with the typical ReLU nonlinearities (xx) ReLU(xx).
Our proposed multiplicatiie filter network, in contrast, uses a different recursion that never results in composition of nonlinear functions. Specifically, an MFN is defined via the following recursion
z(1) 9
(x; (2;0'/ 0(1))
(++1) (w(())((0 +b()) b(0)) 9 (2: (x:00(1000(( :
(3)
f(x) (k) -1) ((K)
where o denotes elementwise multiplication, W(),, ((0), 200 are all defined as above, but where T - R.d, is parameterized by parameters g( (the size of g(o) can vary to implicitly define the output dimensions d,) and denotes nonlinear filter applied to the input directly. Of immediate importance here is that in such network, we never apply nonlinearity to the output of a previous nonlinearity. All the nonlinearity of the network occurs within the functions; layers 2("), after passing through linear function, are simply multiplied by new filters of the input. This results in a considerably different type of function that is currently employed by most multi-layer networks, and indeed it is largely only by convention that refer such function "network" at all.
We now present two instantiations of the MFN, using sinusoids or a Gabor wavelet as the filter gi we call these two networks the FOURIERNET and GABORNET respectively. As we show, the crucial property of a function f represented by FOURIERNET or GABORNET is that the entire function f can also be written as a linear combination of sinusoids and Gabor wavelets of the input respectively (albeit an exponentially large number of such features, but of course also with highly reduced space of allowable coefficients on this exponential number of terms, since there are only a polynomial number of parameters that define the MFN). Thus, we would claim that the MFN really looks more like a (rich) Fourier or Wavelet representation of the underlying signal, just one that happens have similar parameterization deep networks (and which can be tuned by typical gradient descent methods).
3.1 MULTTPLICATIIEE FOURIER NETWORKS
As our first instantiation of the MFN, we consider using a simple sinusoidal filter (x;; (()) sin(w (0)
3

---

Published as a conference paper at ICLR 2021
Rd). We term such a network the FOURIERNET, as the sinusoidal activation (with arbitrary phase shifts, to represent sine or cosine functions equally) corresponds naturally to Fourier random feature representation of the entire function.
with parameters ((0 (w)() E Rd.x . o0
An immediate and compelling feature of the FOURIERNET, compared to networks based upon com- position, is that its output be directly viewed linear function of (an exponential number of) Fourier bases, with low-rank set of coefficients determined by the parameters of the network. This conveyed by the following theorem.
Theorem 1. The output of a Fourier Network is given by a linear combination of sinusoidal bases,
T
[5(1) = Σ
Oi sin( + σx) + b,
(5)
for some coesficients o frequencies WL.T, phase offsets ØuR, and bias term b.
In other words, the FOURIERNET represents its final function as a linear combination of traditional Fourier bases, just do "classical" random Fourier features, for instance. The key element of the proof (given in the appendix) is the fact that for two Fourier filters, with parameters w, ø and т, e respectively, their elementwise product can be transformed to a sum of the same type of filters
1
sin(wz + Φ) sin(Tx ()) cos ((w - T)x
- cos ((w + 7)x + +
(6)
(note that the cosine can be expressed as sine with a separate phase offset). Moreover, an inspection of the proof also lets us compute the exact coefficients of the linear expansion, as function of the network parameters. This is shown the following corollary.
Corollary 1.
Let i1, .2,...,, range over all I[ d, possible indices of each hidden unit of each layer of an MFN, and let s2,...;8k E
+[-1,+1} range over all 2 possible binary signs; then the expansion of ig from (2) is given by all the terms
(k)
(k-1)
seit ... ( w
á
2
13.82 12.81
B skW (k) +... + s2 + Wii
(2)
Sx (K) + +... $20 (2)
(1)
π
5k
with similar form for terms that begin at the i layer. multiplied by the corresponding 65) term.
This corollary follows simply by inspection of the proof in the appendix, noting that each additional multiplicative layer creates both positive and negative combination frequencies in the sinusoid terms, and a multiplication of the corresponding entries of W. In other words, the multiplicative represent an exponential number sinusoidal functions.
"depth" of the FOURIERNET allows but with the constraint that the actual number of coefficients on these features is given by "low- rank" tensor consisting mainly the coefficients the W matrices. This expansion also suggests method for initializing parameters specific this network in manner that scales appropriately with the network size. Specifically, however W()s are initialized (typically random uniform or Gaussian, though here with an additional scaling factor that depends on the relative seale of the input), one should divide these terms by Vk. to ensure that the variance of the final frequency wi is independent of the number of layers.
3.2 MULTIPLICATIVE GABOR NETWORKS
well well-known deficiency of the pure Fourier bases is that they have global support, and thus may have difficulty representing more local features. A common alternative to these bases is the of Gabor filter to capture both a frequency and spatial locality component. Specifically, we consider a Gabor filter of the form
gz(1;0(1) = exp (-2g)
(3)
r 11 (0
sin (w.50 + og())
(8)
2
12
4

---

Published as a conference paper at ICLR 2021
Ground Truth
Ff best
rr Pootion
SRtW
fountrtee
Cabsrittt
Imaee Represeniation. first 1000 Training Iteraaioons
Basic
Pax tonal
SIREN
FourierNet
Sabortiet
ngining Roretions
Figure 1: Left: Performance of various models on an image representation task (top row) and three frames from a video representation task (remaining rows). Leftmost column shows ground truth. Right: PSNR of each model in the image reconstruction task for the first 1000 training iterations.
with parameters (00)
(6) E RR, E R. 1 E RD, Pi:d, E TR. (here 14 denotes the 7i:d,
(6) denotes the scale term), and where for simplicity we spec- ify the functional form of each .....,d, .. coordinates of the function g R Rd. We call the MFN using this filter the GaborNet.
mean of the jth Gabor filter and 7/
As with the FourierNet, a compelling feature of the Gabor network is that the final function f can be represented as a linear combination of Gabor filters. This is captured by the following theorem:
Theorem 2. The oulput of Gabor Network is given by linear combination of Gabor bases,
T
f5(x) = Σ ãi exp (-2 null²) sin(@r + σ.) + b,
for coefficients or.,, scales "nr means Pı.r, frequences Wi.r, phase offsets o1: and bias term b.
The proof is given in the appendix, but the basic procedure is the same as the above: using the fact that just like Fourier filters, the product of Gabor filters is also linear combination of (a different set of) Gabor filters. Likewise, we can also compute the explicit form of the coefficients and for this linear basis expansion, with the explicit form again given in the appendix. One relevant point, though, is how we choose initializations for the y and parameters. Since effectively acts as an inverse covariance term of a Gaussian, Gamma(o, .3) random variable (the conjugate prior of the Gaussian inverse covariance), is reasonable choice for this parameter. And since the functions in the final linear expansion end up being a sum of the individual random 70()) terms at each layer, we scale each layer's a term by 1/k effectively control this parameter the final layer. We also simply choose each ((() be uniformly distributed over the range of the allowable input space x.
4 EXPERIMENTAL RESULTS
We test MFNs on broad range of representation tasks, showing that the relative simplicity of MFNs improves upon the performance of existing neural representation methods. Our set of ex- periments draws from those presented in Sitzmann et al. (2020) alongside SIREN (image repre- sentation, shape representation, and differential equation experiments) and in Tancik et al. (2020) alongside Fourier feature networks with Gaussian random features, which we call FF Gaussian (image generalization and 3D inverse rendering experiments). In each case, we compare against the set of models tested in the original experiment (generally, either SIREN or FF Gaussian, along with basic ReLU MLP). A PyTorch implementation of MFN is available at https: 7/qithub. com/ http://osccreeeacchchcooco....ce htt.mmttttpiaaaaemmmmemmmmmm -et.ettvvv----eeeeeeeewwxs and full de- tails on hyperparameters and training specifications available in the appendix.
5

---

Published as a conference paper at ICLR 2021
Diaust Burn
Neduoss
Noate Tar
Wiluio gVUJ
gyUggXUU
gyUJgXJU
gyUlgXJU
gvUggXJU
rIn
Figure 2: Image generalization samples from the Natural and Text datasets.
4.1 IMAGE REPRESENTATION & GENERALIZATION
We first examine the ability of several networks ar- chitectures in the task of image representation de- scribed in Section 1, where we fit the network to a func- tion f _ E using a dataset where input coor- dinates (I, 3) corresponding to the output pixel value those coordinates (with or RGB images, respectively). To demonstrate this, we construct such dataset from a 256 256 pixel grayscale image and fit various models using sim- ple mean squared error (MSE) loss, including SIREN, GABORNET, FOURIERNET, and ReLU MLPs with and without positional encoding (PE). Visual results and plot of PSNR early in training are shown in (Fig- ure D. In particular, both FOURIERNET and GABOR- NET show quicker initial convergence than other archi- tectures. PSNRs after training (Table IT show that SIREN eventually outperiorms FOURIERNET, while GABORNET remains the best model throughout the training. Indeed, after only 1000 training iterations, GABORNET performs reconstruction better than all other models trained for 10 times longer.
Table I: PSNR of each model's recon- struction in image and video representa- tion tasks after 10,000 training iterations. For video representation, mean + stan- dard deviation over all frames is reported.
1 3 for grayscale
Method
PSNR (in dB) Video
Image
FF Basic
20.13
24.09 + 1.03
FF Positional
40.09
27.90 + 0.99
SIREN
56.54
30.58 + 0.93
FOURIERNET
43.32
27.93 + 0.91
GABORNET
73.98
29.83 + 0.71
We can broaden the task above to represent video by appending a third dimension to the input: the output corresponding to the input (ar, y. is the pixel value (I, y) frame We aim to represent 300 frame color video with 512 x 512 resolution in this manner, testing all of the architectures used in the previous experiment. As shown in Figure SIREN and GABORNET are most capable of reproducing fine details of the original video, such whiskers and and eye color. This is reflected in the PSNR of these reconstructions (Table I); SIREN performs best with a PSNR over 30 dB, while the GABOR NET reconstruction comes within 1 dB of SIREN and exhibits much lower variation across frames than any other model.
In addition representing images, we demon- strate that the MFNs are able to generalize the representation unseen pixels. We train the net- works using only 25% of the image pixels (ev- ery other pixel in the width and height dimen- sions) and evaluate using the complete images. We compare the results of our methods with the Fourier feature networks on two datasets (natural and text images) presented in Tancik et al. (2020). The peak signal-to-noise ratio (PSNR) metric is used to evaluate the performance. As we can see from Table 2 both FOURIERNET and GABOR- NET outperform all versions of the Fourier feature networks that uses basic, positional encoding, and random Gaussian features. Some examples of the generated images are presented in Figure
Table 2: Image generalization results (mean + standard deviation of PSNR).
Method
Natural
Text
FF Basic
21.61 + 2.62
20.50 + 2.13
FF Positional
25.13 + 4.01
26.49 3.11
FF Gaussian
25.57 + 4.18
30.46 + 1.97
FOURIERNET
26.03 + 2.77
31.02 2.04
GABORNET
26.18
2.95
31.19 + 2.00

---

Published as a conference paper at ICLR 2021
Total training loss after training for differential equations, where the iterations column refers to the number of iterations for which the networks were trained. PLI and PGI refer to Poisson equation based image reconstruction using Laplacians and gradients respectively. Helmholtz and wave refer to the single source inversion task.
Table 3:
Problem
Iters
SIREN
FOURIERNET
GABORNET
PGI
10000
0.55
1.16
0.46
PLI
10000
3432.56
76663.52
107.129
Helmholtz
50000
84949.92
46486.0
18845.75
Wave
10000
2090.15
691.34
441.13
Ground Truth
FourierNat
Siren
GaborNet
ReLU
Ground Truth
FourierNet
Siren
GaborNel
ReLU
Image
Image
Gradiemt
Laplocian
(a) Poisson Image Reconstruction: Gradients
(b) Poisson Image Reconstraction: Laplacians
Figure 3: Poisson Image Reconstruction: In the left and right figures, image the left is recon- structed using gradients and Laplacians respectively; the top row depicts the reconstructed images, while the bottom row indicates the fitted gradients and the fitted Laplacians.
with additional images in Appendix B.2. Visually, the MFNs generated images also have bet- ter quality over the baselines, particularly in Text datasets. Some parts of the text are missing in the baselines images (highlighted with red rectangles in Figure 2 whereas the MFNs completely generate all parts of the text in the images.
4.2 DIFFERENTIAL EQUATIONS
In this section, aim solve boundary value problems which are supervised by different forms of gradient information from the functional at hand. We first focus on the Poisson equation, where we demonstrate image reconstruction in two settings where the supervision for the model is brought about by gradients and Laplacians respectively. It is worth noting that the model never pre- sented with real function values. We then focus two 2nd order differential equations, namely, the Helmholtz equation and the wave equation, where we solve for the wave field, where the network is supervised by known source function.
We demonstrate image reconstruction using gradients and compare the performance of FOURIER- NET and GABORNET with SIREN and ReLU MLP for the Poisson equation. We use the same loss function as in (Sizzmann et al. (2200) in (5). Figures 3a and 3b show that the image is reconstructed successfully when the networks are trained while being supervised by gradients and Laplacians re- spectively, while ReLU fails spectacularly. Table B depicts the losses of each method after 10000 iterations, where can be seen that GABORNET beats other baselines in terms of performance.
The Helmholtz and wave equations are related the physical modeling of diffusion and waves and are closely related to a Fourier transform. Hence, we focus our attention on describing the Helmholtz equation. We aim to solve for the wave field and compare the performance of FOURIERNET and GABORNET with SIREN and ReLU MLP. To accommodate for complex-valued solutions, the net- work is configured to output two values which can be interpreted as the real and imaginary parts. We use the same loss function as used in (Sitzmann et al., (220)) (see Section 4.3 of (Sitzmann et al.) 2020) for details). Figure showw the magnitude and the phase of the reconstructed wave front for single Gaussian source placed at the center of medium with uniform wave propagation velocity. Table 3 depicts the losses of each method after 50000 iterations, and shows that GABORNET beats FOURIERNET and SIREN in terms of performance, while, previously shown in (Sizzmann al.) 2020). ReLU MLP fails miserably. Details pertaining to the network and training are relegated to Appendix B.3.
7

---

Published as a conference paper at ICLR 2021
SIREN
Relu MLP
Fourieriet
Geboriict
(a)
Ground
F! Bast
FF Post ona
Gwsssn
Poureriett
Gabe
(b)
Figure 5: Shape representations from fitting signed distance functions (a). 2D rendered photographs from view synthesis experiments (b).
4.3 SHAPE REPRESENTATION VIA SIGNED DISTANCE FIELDS
Recent work Park et al. 2019) has shape
explored the problem representation with neural architec- tures, often with surprisingly effee- tive results. This is done by training on raw geometric data: given an ori- ented point cloud, seek to learn a function f IR3 - R that takes points as input such that the zero level set {x f(x) 0} of the network ac- curately represents the surfaces the shape. Effective training objectives for this task has been explored in con-
Ground Trub
FourieNett
Siren
Gaborfitt
RoLU
Maggituse
Figure 4: Solving Helmholtz equation for single point
siderable depth; we use the training loss presented by Park et al. (2019)
source placed the center of a medium with uniform wave propagation velocity. The top row presents the magnitude, while the bottom row presents the phase.
and used in Sitzmann et al. (2020),
which includes terms involving the network's output (penalties encourage SDF values and away from for surface and off- surface points, respectively) as well as its gradients (a term encouraging the gradient to match the surface points' normals, and gradient norm penalty throughout the entire 3D space). complete description of the loss function is left to the appendix.
Figure 55 shows the results of shape representation task SIREN, standard ReLU MLP, and both variants of MFN. As can be seen, the ReLU network's fails to represent some features of the scene entirely, such as doorways, picture frames, and pillows. Both MFN architectures far outperform this baseline, and are able to reconstruct the room and objects within it to recognizable degree. However, likely owing to its strong ability produce smooth outputs and gradients, SIREN is largely able to avoid the visual artifacts on flat surfaces like walls that remain in reconstructions by FOU and, to a lesser extent, GABORNET.
4.4 3D INVERSE RENDERING FOR VIEW SYNTHESIS
In this view synthesis task, we aim to reconstruct 3D representation from the observed 2D pho- tographs. Using the reconstructed 3D representation, we then render 2D images from new view- points. We use the "simplified Neural Radiance Fields (NeRF)" task on Lego dataset presented in Tancik et al. (2020). The networks are trained to predict the color (in RGB format) and the volume density at given 3D location of the viewpoint. Volumetric rendering is then used to re-render the 2D image photograph the viewpoint. The training loss is computed the mean squared error between the rendered photograph and the actual image observations. The results in Table 4 show that both MENs perform competitively in the task. GABOR has a slight advantage
8

---

Published as a conference paper at ICLR 2021
the baselines and FOURIERNET in terms of the overall PSNR metric. It also arguably produces slightly better image rendering quality as shown in Figure Pp and other images in Appendix B.5.
CONCLUSION
We have introduced multiplicative filter networks (MFNs), class of neural representation architectures that forego the usual com- positional notion of network depth in favor of a similarly expres- sive multiplicative operation. They also admit natural signal processing interpretation, as in the two instantiations of MFNs, FOURIERNET and GABORNET, which are proven to be exactly equivalent to linear combination of sinusoidal or Gabor wavelet bases, respectively. In experiments, simplicity relative to other deep architectures designed for im- plicit representation, MFNs stand up surpass the previous state the art on a battery of representation tasks.
Table 4: View synthesis results (mean st.d. of PSNR).
Method
PSNR
FF Basic
23.37 = 0,96
FF Positional
25.76 = 0.79
show that, despite their
FF Gaussian
25.76 = 0.92
FOURIERNET
25.20 = 0.72
GABORNET
25.81 = 0.76
REFERENCES
Haim Avron, Michael Kapralov, Cameron Musco, Christopher Musco, Ameya Velingker, and Amir Zandich. Random fourier features for kemnel ridge regression: Approximation bounds and statis- tical guarantees. In International Conference on Machine Learning, pp. 253-262, 2017.
Ronald Newbold Bracewell and Ronald N Bracewell. The Fourier transform and its applications, volume 31999. McGraw-Hill New York, 1986.
Romain Brault, Markus Heinonen, and Florence Buc. Random fourier features for operator-valued kernels. In Asian Conference on Machine Learning, pp. 110-125, 2016.
Rohan Chabra, Jan Eric Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub. Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. arXiv preprint arXiv:2003. 10983, 2020.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 5939-5948, 2019.
Radha Chitta, Rong Jin, and Anil K Jain. Efficient kernel clustering using random fourier features. In 2012 IEEE 12th International Conference Data Mining, pp. 161-170. IEEE, 2012.
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7154-7164, 2019.
Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Learning neural 3d texture space from 2d exemplars. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recog- nilion, pp.
2020.
Zhen Hu, Ming Lin, and Changshui Zhang. Dependent online kernel learning with constant number of random fourier features. IEEE transactions on neural networks and learning systems, 26(10): 2464-2476, 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- cralization in neural networks. Advances in neural information processing systems, pp. 8571- 8580, 2018.
Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang. Matthias Nießner, and Thomas Funkhouser. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6001-6010, 2020.
Sylwester Klocek, Lukasz Maziarka, Maciej Wolczyk, Jacek Tabor, Jakub Nowak, and Marek Smieja. Hypernewoor functional image representation. International Conference on Arti- ficial Neural Networks, pp. 496-510. Springer, 2019.

---

Published as a conference paper at ICLR 2021
Quoc Le, Tamás Sarlós, and Alex Smola. Fastfood Fastfood-approxinne approximating kemnel expansions in loglinear time. In Proceedings of the international conference on machine learning, volume 85, 2013.
Ming Lin, Shifeng Weng, and Changshui Zhang. On the sample complexity of random fourier features for online learning: How many random fourier features do we need? ACM Transactions on Knowledge Discovery from Data (TKDD), 8(3):1-19, 2014.
Jingshu Liu, Li Wang, and Jinglei Liu. Efficient preference clustering via random fourier features. Big Data Mining and Analytics, 2(3):195- 2019.
Julien Mairal, Piotr Koniusz, 7aid Harchaoui, and Cordelia Schmid. Convolutional kernel networks. In Advances in neural information processing systems, pp. 2627-2635, 2014.
Siamak Mehrkanoon and Johan AK Suykens. Deep hybrid keural-keree networks using random fourier features. Neurocomputing. 298:46-54, 2018.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv preprint arXiv:2003.08934, 2020.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3504-3515 2020.
Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 4531-4540, 2019.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 165-174, 2019.
Majid Rabbani. Jpeg2000: Image compression fundamentals, standards and practice. Journal of Electronic Imaging, 11(2):286, 2002.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in neural information processing systems, pp. 1177-1184 184, 2008
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization learning. In Advances in neural information processing systems, pp. 1313- 1320, 2009.
Ankit Singh Rawat, Jiecao Chen, Felix Xinnan X Yu, Ananda Theertha Suresh, and Sanjiv Kumar. Sampled softmax with random fourier features. In Advances in Neural Information Processing Systems, pp. 13857-13867, 2019.
Jen-Hao Rick Chang, Aswin C Sankaranarayanan, and BVK Vijaya Kumar. Random features for sparse signal classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5404-5412, 2016.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-awwrce neural scene representations. In Advances in Neural Information Pro- cessing Systems, pp. 1121-1132, 2019.
Vincent Sizzmann, Julien NP Martel, Alexander Bergman, David B Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. arXiv preprint 2.Xiv:2006009661. 2020.
Kenneth O Stanley.
Compositional pattern producing networks: novel abstraction of development. Genetic programming and evolvable machines, 8(2):131-162, 2007.
10

---

Published as a conference paper at ICLR 2021
Yitong Sun, Anna Gilbert, and Ambuj Tewari. But how does it work in theory? linear svm with random features. In Advances in Neural Information Processing Systems, pp. -379-3388, 2018.
Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv:2006.10733 2020.
arXiv preprint
Martin Vetterli, Jelena Kovačević, and Vivek K Goyal, Foundations of signal processing. Cambridge University Press, 2014.
Hui Xue, Zhe Wu, and Wei-Xiang Sun. Deep spectral kernel learning. In Proceedings the 28th International Joint Conference on Artificial Intelligence, pp. 4019 AAAI Press, 2019.
Felix Xinnan X Yu, Ananda Theertha Suresh, Krzysztof M Choromanski, Daniel N Holtmann- Rice, and Sanjiv Kumar. Orthogonal random features. In Advances in Neural Information Processing Systems, pp. 1975-1983, 2016.
Ellen D Zhong, Tristan Bepler, Joseph H Davis, and Bonnie Berger. Reconstructing continuous distributions of 3d protein structure from cryo-em images. In ICLR, 2020.


Reviews:
[{"review_id": "QoIdDs83A7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper proposes two schemes to learn better representation, one based on Fourier features and another on Gabor filters. The idea is to place nonlinearity into the feature encoding rather than network layers like ReLU, and the goal is to learn better representation with more compact models. Preliminary results outperform existing techniques on several image and video representation tasks. One generalization is shown for image completion.\n\n**Strength**\nThe idea is simple but seems effective. It's interesting to leverage Gabor filters' capability in spatial and frequency support to enable better feature encoding. Results show a better reconstruction of high frequency details and experiments are comprehensive covering multiple aspects of image and video tasks.\n\n**Weakness**\n\n    - I do not see how MFN 'largely outperforms' existing baseline methods. It is difficult to identify the quality difference between output from the proposed method and SIREN -- shape representation seems to even prefer SIREN's results (what is the ground truth for Figure 5a). The paper is based on the idea of replacing compositional models with recursive, multiplicative ones, though neither the theory nor the results are convincing to prove this linear approximation is better. I have a hard time getting the intuition of the advantages of the proposed method.\n    - this paper, and like other baselines (e.g. SIREN) do not comment much on the generalization power of these encoding schemes. Apart from image completion, are there other experiments showing the non-overfitting results, for example, on shape representation or 3D tasks?\n    - the proposed model has shown to be more efficient in training, and I assume it is also more compact in size, but there is no analysis or comments on that?\n\n**Suggestions**\n\n    - Result figures are hard to spot differences against baselines. It's recommended to use a zoom or plot the difference image to show the difference.\n    - typo in Corollary 2 -- do you mean linear combination of Gabor bases?\n    - It's recommended to add reference next to baseline names in tables (e.g. place citation next to 'FF Positional' if that refers a paper method)\n    - In Corollary 1, $\\Omega$ is not explicitly defined (though it's not hard to infer what it means).", "extra_scores": {}, "extra_text": {}}, {"review_id": "E398ALWJmAd", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:   This paper complements a class of recent work on function approximation for image function representation that leverages random fourier functions and sinusiodal activation functions (SIREN).  The main insight in the paper an alternative scheme that just repeatedly applies nonlinear filters (sinusoids, gabor wavelet functions) to the networks input and multiplies together linear functions of these features.  The authors show that due to multiplicative properties of fourier/ gabor filters the end resulting mapping is also represented by a linear combination of fourier/gabor filters. Essentially the entire function is a linear function approximator over an exponential number of fourier/gabor basis functions with a low-rank, polynomial number of coefficients of the MFN network.  Experiments are done to compare the proposed FourierNet, GaborNets against past work (FF, FF with positional encoding, FF Gaussian SIREN).  The results are competitive with other methods and in certain cases GaborNets outperform in PSNR benchmarks for various problems.\n\nPros:  I like the paper very much, it is very well written and clear. The idea is novel.  \n\nCons: The empirical benchmarks are the same as used in the SIREN paper.  I would liked to see a discussion or analysis on tradeoffs between using SIREN or GaborNet (i.e. in what settings they are appropriate and perform perturbation analysis). \n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "8NYeov8Y21G", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary.\nThe paper proposes a novel neural architecture for representing complex functions over on a low-dimensional domain (e.g. images on coordinates). Instead of following standard deep-learning ideas, the method stacks linear layers, but modulates each layer with a non-linear function (e.g. Fourier or Gabor functions). It is shown that this results in a linear combination of exponentially many Fourier/Gabor functions. Extensive experiments are performed, following the complete suite of experiments in SIREN [Sitzmann 2020], and it is shown that the proposed method outperforms available neural methods, except on video representation and signed-distance functions.\n\nReasons for score.\nThe paper proposes a novel method and shows superior performance on some tasks. However, generalization performance comparison for standard signal processing is missing. Hence, I suggest a weak accept.\n\nPro.\n- simple and novel idea that fits nicely into standard frameworks\n- experiments demonstrate its superiority on several tasks, even generalization compared to Fourier Features\n- well written and easy to understand\n\nCon. \n- although theoretically shown, it remains empirically unclear if the method is able to leverage the exponential number of functions. I.e. an experiment to compare with standard Fourier and Gabor transformation should be performed. As a Fourier basis can represent any finitely sampled signal perfectly, a generalization analysis should be performed akin to pg.6 bottom. Put another way: when is the neuronal representation useful as opposed to just useing a Fourier/Gabor transform?\n- it should be worked out more carfully what's novel throughout the paper. E.g. in section 4.2. \"while ReLU fails spectacularly\" - this has already been shown in the SIREN paper, and needs emphasis for readers not aware of this.\n\n\nMinor.\n- pg. 3 top: Wˆ{(i)} the dimensionality should be transposed, no?\n- do all the methods in Figure 1 have the same numbers of free parameters? \n- Table 3: is the relative performance stable wrt. to the number of iters? would be good to see the whole plot of performance vs. iters for all methods in the appendix.\n- Figure 4: Ground -> Ground-truth; why is there a square box for SIREN and GaborNet? (it's not in the SIREN paper)\n\n\nUdpate: with the additional experiments and corrections in the paper, I believe the paper is in good shape and contributes to the literature in the field. Hence it should be accepted.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2au4wdFcrLP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose a new network architecture, multiplicative filter network (MFN),  for implicit function reconstruction. The proposed network incorporate elementwise multiplication of nonlinear filters to replace to conventional nonlinear functions, e.g. ReLU, for better signal reconstruction. Two different multiplicative filter network is studied, FourierNet and GaborNet, which used Fourier and Gabor functions as the nonlinear functions in the MFN. Experiments have shown that the proposed method can outperform SIREN under the similar level of computational complexity.\n\nPositive:\nThe paper is easy to follow and the proposed method seems to be working well especially for the GaborNet. \n\nNegative:\nOn top of SIREN, this paper looks incremental. Although it makes some theoretical study that the proposed FourierNet and GaborNet can favorably reconstruct original signals, there is no theoretical comparisons with this work and SIREN. There is only empirical study to show that the results of the proposed method is better than SIREN (GaborNet only, the FourierNet performs worse than SIREN).  \n\nOpen Question:\nI have checked the SIREN and its source codes. Although SIREN can reconstruct original image favorably, its network structure is unfriendly for image editing and/or other image processing tasks. Have the authors of this paper encounter similar problem in MFN?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes multiplicative filter networks (GaborNet and FourierNet) as functional approximations of deepnets. The proposed networks are a sequence of multiplications linear functions of sinusoidal or Gabor filters. The authors show that in some cases the performance of proposed networks outperforms the existing deepnets using ReLu activations. This representation is notably simpler as well. Moreover, compared to classical Fourier approach, the proposed method scales to higher dimensions in practice as well.
The downside of the paper is that it is not clear how to empirically use exponentially many Fourier functions. Moreover, proposed methods have more parameters, and the additional parameters are linear in size of the hidden layer.

The paper is clearly written and the authors improved the quality of the paper and added additional experiments to support their claim through the review process and I appreciate that.
Decision: Accept (Poster)