Multiplicative Interactions and Where to Find Them
Authors: Siddhant M. Jayakumar
Wojciech M. Czarnecki
Jacob Menick
Jonathan Schwarz
Jack Rae
Simon Osindero
Yee Whye Teh
Tim Harley
Razvan Pascanu
Venue:  (2020)
Keywords: multiplicative interactions
hypernetworks
attention

Abstract: We explore the role of multiplicative interaction as a unifying framework to describe a range of classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst others.
Multiplicative interaction layers as primitive operations have a long-established presence in the literature, though this often not emphasized and thus under-appreciated. We begin by showing that such layers strictly enrich the representable function classes of neural networks. We conjecture that multiplicative interactions offer a particularly powerful inductive bias when fusing multiple streams of information or when conditional computation is required. We therefore argue that they should be considered in many situation where multiple compute or information paths need to be combined, in place of the simple and oft-used concatenation operation. Finally, we back up our claims and demonstrate the potential of multiplicative interactions by applying them in large-scale complex RL and sequence modelling tasks, where their use allows us to deliver state-of-the-art results, and thereby provides new evidence in support of multiplicative interactions playing a more prominent role when designing new neural network architectures.

Published as a conference paper at ICLR 2020
MULTIPLICATIVE INTERACTIONS AND WHERE TO FIND THEM
Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osidnero, Yee Whye Teh, Tim Harley, Razvan Pascanu DeepMind
(sidmj, lejlot, Smenick, schwarzjn, jwrae, os osindero, ywteh, tharley, razp) @google.ccoooooooo
ABSTRACT
We explore the role of multiplicative interaction as a unifying framework to describe range of classical and modern neural network architectural motifs, such as gating, attention layers, hypernetworks, and dynamic convolutions amongst others. Multi- plicative interaction layers as primitive operations have a eongebstablished presence in the literature, though this often not emphasized and thus under-appreciated. We begin by showing that such layers strictly enrich the representable function classes of neural networks. We conjecture that multiplicative interactions offer a particularly powerful inductive bias when fusing multiple streams of information or when conditional computation is required. We therefore argue that they should be considered in many situation where multiple compute or information paths need to be combined, in place of the simple and oft-used concatenation operation. Finally, we back up our claims and demonstrate the potential of multiplicative interactions by applying them in large-scale complex RL and sequence modelling tasks, where their use allows us to deliver state-of-te oore--fftheeaaee results, and thereby provides new evidence support of multiplicative interactions playing more prominent role when designing new neural network architectures.
INTRODUCTION
Much attention has recently turned toward the design of custom neural network architectures and components in order to increase efficiency, maximise performance, or otherwise introduce desirable inductive biases. While there have been plethora of newer, intricate architectures proposed, in this work we train our sights instead on an older staple of the deep learning toolkit: multiplicative interactions.
Although the term itself has fallen somewhat out of favour, multiplicative interactions have reap- peared in range of modern architectural designs. We start this work by considering multiplicative interactions as an object of study in their own right. We describe various formulations and how they relate each other well as connect more recent architectural developments (c.g. hypernetworks Ha et al. (2017). dynamic convolutions Wu et al. (2019) to the rich and longer-standing literature on multiplicative interactions.
We hypothesise that multiplicative interactions are suitable for representing certain meaningful classes of functions needed build algorithmic operations such as conditional statements or similarity metrics, and more generally as an effective way of integrating contextual information in network a way that generalizes effectively. We show this empirically controlled synthetic scenarios, and also demonstrate significant performance improvement variety of challenging, large-scale reinforcement learning (RL) and sequence modelling tasks when conceptually simple multiplicative interaction module is incorporated.
Such improvements are consistent with our hypothesis that the use of appropriately applied mul- tiplicative interactions can provide more data-efficient learning, better generalization, and stronger performance. We argue that these operations should feature more widely in neural networks in and of themselves, especially in the
suitable inductive bias over function classes leading to

---

Published as a conference paper at ICLR 2020
of mutttplicative
Wz2
Multiplicative Interactions
Palyou Perand
Gatling
Metric Leaming
Attention
Scaling
Hadamard product
General bilinezr form
Canviation
Lower
No Beeelroong
Hyper
reab cluss
Scator
Diagonal
Unconstranee
exbed
Scalor
Vector
Murric
Figure 1: (Left) Venn diagrams of multiplicative interactions with respect other model classes commonly used in ML. (Right) Comparison of various orders of multiplicative interactions and their relation other perspectives.
increasingly important setting of integrating multiple streams of information (including endogenously created streams in branching architectures).
Our contributions are thus: (i) re-explore multiplicative interactions and their design principles: (ii) to aid the community's ss understanding of other models (hypernetworks, gating, multiplicative RNNs) through them; (iii) show their efficacy representing certain solutions; and (iv) empirically apply them to large scale sequence modeling and reinforcement learning problems, where we demonstrate state-of -fee--of-hheee the-art results.
MULTIPLICATIVE INTERACTIONS
start by introducing notation and formalising the concept of multiplicative interactions. The underlying question we are trying answer is how to combine two different streams of information. Specifically, given x E R" and z E RT our goal is to model an unknown function ftarges (x,z) IR that entails some interaction between the two variables. In practice and might be arbitary hidden activations, different input modalities (e.g. vision and language), or conditioning information and inputs.
The standard approach is to approximate ftarget by a neural network f. If f restricted to employ single layer of weights, we typically [(x, z) W [x: z] b, where [x; z] represents the concatenation of and z, and W [R(+++)>* and b IR& are learned parameters. The interaction between x and z is only additive given this formulation. However through stacking multiple similar layers (with element-wise nonlinearities inbetween), can approximate any function frargtt given
sufficient data (and capacity).
In contrast, single layer with maltiplicative interactions would impose the functional form
f(x, z) ==0=="'a ='' + z z Vx b
where W is 3D weight tensor, U. V are regular weight matrices and b is vecto We posit that this specific form, while more costly, is more flexible, providing the right inductive bias to learn certain families of functions that of interest in practice. Additionally, many existing techniques can be shown to rely on variations of the above bilinear form as detailed below.
Hypernetworks Multiplicative Interactions. network that is used to generate the weights of another neural network given some context or input vector z. Particularly f(x: ff) becomes f(x: g(z; Φ)). . In the case where of and g are affine (as in the original work), such a network is exactly equivalent to the multiplicative form described above.
Hypernetwork Ha et al. (0117) is neural
Specifically, we can decompose equation (1) and set W -TW + V and b' zTU + We can now see W as the generated 2D weight matrix and b' as the generated bias from some hypernetwork.
We have the 3D tensor W mexxnk and (z Wx), Eng W.
2

---

Published as a conference paper at ICLR 2020
This allows us to have an input-conditional weight matrix and bias vector that then used to generate output = W'x + b'. We can also consider the more general case of any affine transformation being generated by some arbitrary neural network, which can also be viewed as a multiplicative interaction where we first embed the context z and then use it in the equation above. This provides a basis for thinking about hypernetworks themselves as variations potentially accounting for a considerable amount of their efficacy.
the theme of multiplicative interactions,
Diagonal Forms and Gating Mechanisms. Let us consider a diagonal approximation the pro- jected W' This is given by a particular parametrization of W = z W + V above (see Figure right). Multiplying with w diag(a1..... an) can be implemented efficiently as f = a @ x where @ represents elementwise multiplication or the Hadamard product (similarly for the bias). This form now resembles commonly used gating methods, albeit they are often used with additional non-linearities (e.g. sigmoid units Dauphin et al. (2017); Van den Oord et al.] (2016)). It can be viewed as hypernetwork as well, where z W represents the function generating parameters.
Attention and Multiplicative Interactions. While not the focus of this work, we note that attention systems in sequence modelling (Vaswani et al. 2017; Bahdanau et al., 2014) similarly use multi- plicative interactions to effectively scale different parts of the input. This is typically done using the diagonal form above with m = f(, z). m @ x where m is often a bounded mask. Attention systems are typically used with different aims to those we describe here: they can suppress amplify certain inputs and allow long-range dependencies by combining inputs across time-steps (when masking above is followed by a pooling layer, for example). We use these insights to posit that while more expensive, considering higher order interaction (generating a vector mask) might prove more beneficial to such systems but we do not specifically consider attention in this paper and leave it to future work.
Scales and Biases. Further, we can make another low-rank approximation to the diagonal form and generate instead a scalar matrix - i.c. the hypernetwork outputs single scalar scale (and/or bias) parameter per channel or feature vector we are considering, instead of vector. We can again write this as f = z W @ x where z ' = a.. This is common in methods such as FiLM (Perez et al. 2018 Dumoulin et al. 2018 2017).
Multiplicative Interaction and Metric Learning, Another highly related field of active research is that of metric learning, where one tries to find the most suitable metric to measure similarity between objects in some parametrised space of metrics. One of the most commonly used classes is that of Mahalanobis distances dx(x, z) = ||x - z||c = (x z) cx z), which again maps onto multiplicative interaction units dc (x, z) (x z)
z) CTC- 2x CCCZ-Z 2 - z. In metric learning, however, one usually explicitly defines losses over tuples (or higher order n-tulles) with direct supervision, while here we consider building blocks that can learn metric internally, without direct supervision.
The Taxonomy of Multiplicative Interactions. Finally, we summarise these relationships in figure [ We can think of multiplicative interactions equivalently in terms of either: (a) the approximation (c) the
to the 3D tensor made; (b) the output of the "projected" context by the hypernetwork; operation used to combine the generated weights/context and the input. For example, the general bilinear form is equivalent to a vanilla hypernetwork that generates weight matrix for a matrix multiplication. Similarly, a diagonal 3D tensor is equivalent to hypernetwork that generates a vector and is combined with hadamard product.
3 EXPRESSIVITY OF THE MODEL
Vanilla MIPs are universal approximators that is, for every continuous function [0, 1] - IR (con- sidered our target) and every approximation error e >> there exist hidden units H and corresponding parameter values 0 such that the distance in function space between the MLP output and the target function is smaller than e. Consequently adding new modules/building blocks does not affect the approximation power of neural nets, however such modifications can change the hypotheses space- the set of functions that can be represented exactly (with 0 error), and the compactness of a good estimator (how many parameters are needed), as well as learnability.
We first show that multiplicative interactions strictly enlarge the hypotheses space of vanilla MLPs that is, we add new functions which multi-layer multiplicative models can now represent perfectly,
3

---

Published as a conference paper at ICLR 2020
<Wy
54- (Ir
MLP
Pruned MLP
No. of params
1200
Pruned MI
Input Dimensicnallity 1-)
Input Dimensionality (d)
Input Dimensiona 6d)
Input Dimensionality (o)
Figure 2: Number of parameters needed for a regular, single layer MLP (blue line) to represent the function up to 0.1 MSE over the domain of standard d-dimensional Gaussian compared to the same quantity for a multiplicative model (green line). a denotes sigmoid. Dotted lines represent pruned models where all weights below absolute value of 0.001 were dropped. Note that for MLP all parameters are actually used, while for MI module some of these functions (summation and dot product) can be compactly represented with pruning.
while also preserving our ability to represent those in the existing set modeled perfectly by vanilla MLPs (full proof appendix (A).
Theorem 1. Let Hmip denote the hypotheses space of standard MLPs with ReLU activation function, and let Hym denote the hypotheses space of analogous networks, but with each linear layer replaced with multiplicative layer, then we have Hmip Hmme
While valuable, such a result can also be trivially obtained by adding somewhat exotic activation functions (e.g. Weierstrass function o (zx) >> n= 0.5" cos(7">r) which is continuous function but nowhere differentiable (Weierstrass) 1895); see appendix for the proof) to the pool of typically used ones. While increasing the hypothesis space its is not of great significance, the crucial point here is that the set Nms \ Hmlp helps extend our coverage to the set of basic functions that would expect need in composing solutions that mimic systems of interest such as logical, physical, or biological ones.
Figure 2 shows the learnability (up to certain error) of some simple input functions against the number of parameters needed. We consider summation, gating, and dot products which are basic buildings blocks of operations such as conditional statements or similarity metrics, and fundamental for implementing rich behaviours when combining different sources of information. For the gating and dot- product function classes, the complexity of MLPs required to learn them seems grow exponentially, while the growth for multiplicative models is quadratic. On the other hand summation trivially easier for an MIP. Thus we do not argue that multiplicative interactions are a silver bullet but that such interactions add an important class of functions to the hypothesis set that are often the right inductive bias, or algorithmic building block, for many kinds of problems. subsequent sections we show empirically that using them context-integration layers leads to good performance gains across a range of tasks.
RELATED WORK
There vast body of literature surrounding multiplicative interactions, and these ideas have long history, for example being discussed in the foundational era of connectionism (Rumelhart et al. 1986). Below highlight some of the key works developed in the community over the last few decades and aim to show how these relate to each other.
Some of the earliest models leveraging multiplicative interactions were higher-order Boltzmann machines or autoencoders (Sejnowski] 1986 Memisevic & Hinton 2007 Taylor & Hinton 2009).
Currently, the most common usage of multiplicative interactions seen in models that enjoy widespread adoption is via factorised or diagonal representation of the necessary 3D weight tensor. The LSTM cell (Hochreiter Schmidhuber 1997) (and its descendents such as the GRU (Cho et al. 2014)) employ multiplicative interactions of this form in the gating units that are crucial for the long-term sta- bility of memories. Enhanced multiplicative versions of LSTMs have also been formulated (Sutskever et al. 2011: Wu et al.) 2016) Krause et al.] 2016): these approaches essentially combine the pre- vious hidden state and current input via an element- wise or hadamard product between projected representations.
4

---

Published as a conference paper at ICLR 2020
Similarly, bilinear layers (often low-rank factorizations) have appeared extensively in the computer vision literature (Gao et al. 2016 Kim et al.) 2016) and beyond (Dumoulin et al. 2018). Squeeze- and-excitation networks, for example, can be seen as an instantiation of this idea (Hu et al., 2018).
Specifically in visual-question answering systems, models like FiL (Perez et al. 2018) or class- conditional batch norm (Brock al., 2019 Perez al. 2017) use such diagonal forms generate per-channel scales and biases as function of some context. This has been shown to be effective at capturing relationships between the two different modalities (text and vision), as well as providing powerful mechanism to allow single network to conditionally specialize on multiple different tasks. Further, multimodal domains such as VQA have also seen such bilinear models used in combination
with attention systems Yuu et al. 2017 Lu et al. 2016 Xu & Saenko 2016: Schwartz et al. (017).
Further, there are many additional works using gating mechanisms which can thought of as such diagonal approximations used in conjunction with additional point-wise non-linearities or softmaxes. Recent examples of such include pixelCNNs (Van den Oord et al. 2016) and Highway Networks (Srivastava et al.) 2015 Zilly et al. 2017), among others (Dauphin al. 2017), and earlier examples can be seen in works such as Mixtures of Experts (Jacobs et al. 1991) and successors.
Multiplicative interactions in the non-factorised sense can also be thought of as restricted class of Hypernetworks (Ha et al. 2017):: models that generate the weights of one network from another. While the original presentation (Ha et al., 2017) considered their use for model compression in feed-forward nets (i.e. using layer IDs to generate weights), they also investigate HyperL.STTe STMMs in which per timestep multiplicative biases generated. A similar approach has also been applied to generating parameters in convolutional nets via "dynamic convolutions" where the size of the generated parameters is controlled by tying filters (Wu et al., 2019). Further, these ideas have been extended to Bayesian forms (Krueger et al. 2017) and also used for example, in architecture search (Brock et al. 2017).
Lastly, multiplicative interactions used to scale contributions from different spatial or temporal components play a key role in attention mechanisms (Bahdanau et al. 20044) Vaswani et al. (017). They have also been used in some RL works to better condition information, e.g. in Feudal Networks (Vezhnevets et al.] (017)) as a way for manager and worker units to interact, and better action- conditioning (Oh al. 2015).
5 EXPERIMENTAL SETUP
We aim to demonstrate that the incorporation of multiplicative interactions can boost performance across a wide range of problems and domains, and conjecture that this is because they effectively allow for better routing and integration of different kinds of information. Specifically we will show that multiplicative interactions allow better integration of (a) latent variables in decoder models, (b) task or contextual information in multitask learning, (c) recurrent state in sequence models. We neural process regression, multitask RL and language modelling as exemplar domains. Further details of architectures and syper-parametee can be found in the appendix.
Note on Implementation and Terminology. We use M(x, z) below to mean the function f(x, z) = z z''x= + z" U + Bx + b (or referred to as MI in the legends). In all cases we im- plement this using series of standard linear layers with a reshape operation in between to form the intermediate matrix (equivalently this can be done with einsum or tensor product notation; we provide a simple implementation in the appendix). The quantity fi(z) z" z'w + B, (where as above W is 3D and B 2D bias) represents the 2D output of projecting the contextual information. We refer to this interchangeably as the 2D-contextull projection or "generated weights" using the hypernet terminology. Similarly the "generated bias" is the ID projection of the context z, that is, fa(z)= = u (and thus f(x; z) = fi (z)x fz(z)).
While we aim to draw connections to other models and approximations in the literature above, our aim is not to advertise one specific instantiation or approximation over the other As such, we use the same form above in all experiments (unless specified otherwise) and control parameter count by controlling the size of z. Undoubtedly, practitioners will find that task specific tuning of the above form might yield comparable or better results with fewer parameters, given the right approximations.
5

---

Published as a conference paper at ICLR 2020
Tay Multhask Regression (10 Tasks)
Tay Multitask Regression (20 Tasks)
Tay Multiiask Regression (30 Tasks)
200
300
000
801
1050
Figure 3: Averaged learning curves for different models while varying the number of tasks in the toy multitask regression domain. Shaded regions represent standard error of mean estimation.
6 LEARNING CONTEXT DEPENDENT LAYERS FOR MULTITASK LEARNING
We start by considering the general paradigm of multitask- multitask-leamingg, wherein the goal is to train one model to solve K different tasks. We show that we can boost performance here by learning context task-conditional layers with multiplicative interactions. There is generally trade-off between the transfer from similar tasks and the negative interference between those with very different solutions. Our claim is thus that contiiiioon layers provide rest-o-botthwoorddd approach, with an inductive bias that allows transfer (as opposed to a multiheaded architecture) while also limiting interference.
We first demonstrate this with a toy example, where we attempt to regress two different classes of functions, affine and sines, with one model. Specifically, y a,z + by and y a, sin(10r) + 6, where aj and by are sampled per task from uniform distribution. In Figure 3 we show results averaged over multiple runs, as we vary the number of tasks. We train both standard MI.P with multiple heads and one with that is given task ID additional input and can see that the neither is able to use task information to do any better. On the other hand, the results show that a task-conditioned M(x, t) layer allows the model to learn both tasks better with less interference and also increased data efficiency. We see that the gains with using an MM layer are more pronounced as we increase the number of tasks. More details are provided in appendix D
6.1 MULTITASK RL ON DMLAB-30
Next, we consider larger scale problem: multitask RL on the DeepMind Lab-30 domain (Beattie et al.) 2016). This is suite of 30 tasks in partially-observable.
mist-person-perspeetii
3D
environment, encompassing a range of laser-tag, navigation, and memory levels. We use a typical actor-critic RL setup within the Impala framework of Esppholt et al. (2018), with multiple actors and a single learner with off-policy correction (further details provided in appendix section E}. We use exactly the architecture as the original works (Espeholt al., 2018 Hessel al., 2019): a stack of convolutional layers followed by an LSTM, with output h, at each timestep. Normally these are then projected policies ı and value functions V that are shared across all tasks. At test time agents typically not allowed to access ground truth task ID (e.g. this means value functions at training time could use this information).
Multi-head Policy and Value Layers We first show that a multi-headed agent architecture with one policy and value head per level does in fact boost performance. While this does use privileged information (task ID), this does show that there is some degree of interference between levels from sharing policy and value layers.
Multiplicative Policy and Value Layers We can instead consider using multiplicative layers here to integrate task information (one-hot task ID I to modulate compute-paths. We learn a task embedding as below, and use it in multiplicative layer that projects to policy and value functions. That is, now have с = relu relu(MLP(LI) as a context, and Te, V2 = M(h, c).
We show results in the same figure and find that such layer provides further boost in performance. This can be viewed as generating policy layers from some learnt embedding of the task ID. Our hypothesis is that while multiheaded architectures reduce interference, they also remove the ability of policy-transter between the different layers. Further, each head only gets 1/ K of the number of gradient updates (when training K tasks).

---

Published as a conference paper at ICLR 2020
Normelized Score (With Tesk (D)
Normalised Score (No Task ID)..
1-SNE Cormenated Weights
WATERMAZE
Human Narmallion Score
Human Normalised Score
PSYORAAS
5
%.
5
LINGUNGE
CoUeOTs
02
Frames
Frames
Baseline Agent
(Learnt Context)
(Level ID)
Multiheaded
Figure 4: (a) A t-SNE plot of the generated weights from an M layer. (b) Human normalised
performance (capped at 100) when using task ID context to an M layer. (c) Using learnt context instead.
Multiplicative Policies with Learnt Contexts We find (somewhat surprisingly) that we can get similar or greater performance gains without using any task information, replacing task ID I instead with a learnt non-linear projection of the LSTM output. We now have, c = relu relu(MMPP(h.) and Te, Vi = M(h,, c).
We show a t-SNE plot of the 2D projection of the state of the LSTM in Figure This coloured by level name and the transparency value indicates the timestep. We see at timestep O, all weights are the same, and as the levels progress, we find that can naturally detect or cluster task information, showing that the model can readily detect the current task type. We posit however that the affine policy- value decoders may not have the right inductive bias to use this information well. We conjecture that the multiplicative interaction, via the learnt embedding, c, provides the right inductive bias: allowing the model to integrate information and providing additional densely-gated conditional-comput paths that leverage task similarity where appropriate, whilst guarding against interference where tasks differ and thus more effectively learning task conditioned behaviour.
We note that we match state-of tiite-of-the-ar the-art performance on this domain which previously used the PopArt Method (Hessel et al.) 2019) (both achieve 73% normalised human score), with a simpler method. PopArt involves having task-specific value functions and using adaptive gradient normalisation (to limit interference between different reward scales). Our method also reduces the number of extra syper-parameters needed to zero. As such, PopArt solves an orthogonal problem to the one considered here and we posit that these methods can be combined in future work. We also leave as future work the analysis of whether such hyper-value functions are able to implicitly learn the different reward scales that explictly parameterised in methods like PopArt.
7 LATENT VARIABLE MODELS WITH MULTIPLICATIVE DECODERS
We previously demonstrated the ability of multiplicative interactions to extract and efficiently combine contextual information. We next explore the paradigm where the two streams of information being combined refer to semantically different features. Specifically, we investigate how contextual latent variables can be better integrated into neural decoders. We consider neural processes for few-shot regression. Briefly, neural processes (Garneoo et al. [018b] a) are a neural analogue to Gaussian Processes. For few shot regression, they work by predicting a function value y at new observations x" having observed previous values (x,, (referred to contexts) of the same function. As opposed to training a single predictor on the (x, ys). Neural Processes learn to infer a distribution over functions that are consistent with the observations collected so far.
This is achieved by embedding context points y4) individually with an encoder network, and then taking the mean of these embeddings. This gives latent variables that a representation of the function that maps x to y, i.e. y = f(x, z). A new data point x" is mapped to y" by passing [2:x"] through decoder network.
We aim to increase the expressivity of the decoder by improving the conditioning on z. The standard approach is to concatenate and (denoted (I.(((x2))) leading a purely additive relationship.
7

---

Published as a conference paper at ICLR 2020
Instead, we replace the final layer of the MLP decoder with the multiplicative form M(x, z). As an additional baseline. we consider skip connections between the latent variable and each layer of the decoder (denoted Skip MLP((x: z]) Dieng et al. (2018)),
means to avoid latent variable collapse. We apply all methods to the regression task on function draws from a GP prior Garnelo et al. (2018b) and summarize results in Figure(5 a). The results show that multiplicative forms able to better condition on latent information compared to the baseline MLPs. Further experimental details are provided in the appendix F
MULTIPLICATIVE EMBEDDINGS FOR LANGUAGE MODELLING
Finally, we consider word-level language modelling with recurrent models. At each time-step, the network outputs prediction about the word in the sequence, given the current generated word (ground truth at training) and its recurrent state. A standard architecture is to project one-hot word vectors x, input embeddings z Wx,, followed by LSTM with output h,. our predicted output embedding 4,+1 = WWh,x, + b and output y(+1 = softmax(2,, W + b2) where the embedding weights W are tied.
h, We then produce
We posit that computing embeddings with multiplicative interactions instead will allow the model better take its recurrent context into account. We thus compute the output embedding as follows: relu Wyh, + b) and 1 = .Mc²", h2). Instead of being quadratic in h, directly we use this context vector с defined above. This serves two purposes: firstly, we introduce an additional on- on-linear pathway in the network and secondly, we have (i) < dim(h,) which allows us to drastically cut down the parameter count (for example, we have an LSTM output size of 2048, but a context size of only 32). This is an alternative to having diagonal approximation.
We can similarly also have a multiplicative input embedding to the LSTM such that z also integrates recurrent information from h We could define z analogous to above: 1 = -M(z,, h;-1)
zi" ",hhh,- + zu'U + Vh-1 + b. This equation is in fact very similar to the expression used to generate the gates and candidate cell of the LSTM: the last three terms are identical. A diagonal form of the first term has been used in used inside multiplicative RNNs and LSTMs (Sutskever et al. 2011
Krause et al. 2016).
We report results (in Table 1) on running this model on Wikitext- 103. For multiplicative output embeddings we use the 3D form (with the low-rank bottleneck as described above) and we use a diagonal form for the input embeddings when combining the approaches. The rest of architectural choices and hyper-parameters are reported the appendix. We find that adding multiplicative decoder (output) embeddings provides a boost in performance and further adding input embeddings increases these gains. We ascribe this to the ability of the embeddings to now be markedly changed by context and allowing better integration of information by the inductive bias in these interactions.
Table 1: Word-level perplexity on WikiText-100 103
Model
Valid
Test
No. Params
LSTM Rac et al. (2018)
34.1
34.3
88M
Gated CNN Dauphin al.
(017)
37.2
RMC Santoro et al.
2018
30.8
31.6
Trellis Networks Bai al.
2019)
30.35
180M
TransformerXI.
Dai et al.
2018
17.7
18.3
257M
LSTM (ours)
34.7
36.7
88M
LSTM + MultDec
31.7
33.7
105M
LSTM + MultEncDec
28.9
30.3
110M
We note that we have competitive results using only a single-layer LSTM as our base model and far fewer parameters overall. Our intuition is that using such embeddings is orthognal to most of the other recent advances proposed and can thus be stacked on top of them. We leave as future work the integration of these ideas with Transformer based models.
8

---

Published as a conference paper at ICLR 2020
WikiTest-
VANTTert-105
M N.7txzl
Training Stops (XIP)
Num Trai Words
Num Train Words
1990
Figure 5: Results on Neural Processes and language modelling WikiText-103 103.
CONCLUSION AND FUTURE WORK
In this work considered multiplicative interactions and various formulations thereof, connecting them to a variety of architectures, both older and modern, such as Hypernetworks, multplicative LSTMs or gating methods. We hypothesise that the ability of such networks better represent broader range of algorithmic primitives (e.g. to better integrate contextual or caskiconditional information to fuse multiple stream of data. We first tested empirically this hypothesis in two controlled settings, in order to minimize the effect of confounding factors. We further show that we could match state-of-the-art methods on multiple domains with only LSTMs and multiplicative units. While we do not necessarily advocate for specific instance of the above methods, we hope that this work leads to a broader understanding and consideration of such methods by practitioners, and in some cases replacing the standard practice of concatenation when using conditioning, contextual inputs, or additional sources of information.
or inner products) allows them
We believe there are many ways to explore this space of ideas more broadly, for instance looking at: the role of various approximations to these methods; ways to make their implementations more efficient; and their application newer domains. Finally, while attention models use some of these multiplicative interactions, we hope that applying some of the lessons from this work (such as higher order interactions) will allow even greater integration of information in attention systems.
ACKNOWLEDGEMENTS
The authors would like to thank Karen Simonyan and Sander Dicleman for their inputs and comments on the experiments well as early drafts of the paper. We'd also like to thank Ali Razavi, Pablo Sprechmann, Alex Pritzel and Erich Elsen for insightful discussions around such multiplicative models and their applications.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint wrXiv:1409.0477. 2014.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. ICLR, 2019.
Charles Beattic, Joel Z Leibo, Denis Teplyashin. Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016.
Andrew Brock, Theodore Lim, J. M. Ritchie, and Nick Weston. Smash: One-shot model architecture search through hypernetworks, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ICLR, 2019.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406. 1078, 2014.

---

Published as a conference paper at ICLR 2020
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoe V Le, and Ruslan Salakhutdinov. Transformer-xl xl: Language modeling with longer-tem terer-termmoo dependency. 2018.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Pruceedings of the 34th International Conference on Machine Learning- Volume 70. pp. 933-92 1339911.22 JMLR. org, 2017.
Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. Avoiding latent variable collapse with generative skip models. arXiv preprini arXiv: FeXiv::807.0444 0076004663 2018
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. learned representation for artistic style. Proc. of ICLR, 2, 2017.
Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wist transformations. Distill, 2018. doi: 10.33915/distil.000111 https://disilll.pb/008f///eee 8/fat0/rreeatteeiiieeeee wiwteueewwii-ttanneeeeee wwterauurreiiiittttaanfmaaattss
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. ICML, 2018.
Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. Compact bilinear pooling. In Proceedings of the IEEE conference on computer vision and pattern recognilion, pp. 317-326, 2016.
Marta Gamelo, Dan Rosenbaum, Chris J Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo J Rezende, and SM Eslami. Conditional neural processes. ICML, 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint a8Xiv: 01662, 2018b.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. ICLR, 2017.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czareecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. AAAI, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, Geoffrey E Hinton, et al. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint xrXiv:1711.09846 2017.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv: arXiv:/901 1901.05711, 2019.
Jin-Hwa Kim, Kyoung-
Syoung-WWoon Woon On, Woosang Lim, Jeonghee Kim, Jung- Woo Ha, and Byoung- Tak Zhang. Hadamard product for low-rank bilinear pooling. arXiv preprint arXiv: 161000433 201 6.
Ben Krause, Liang Lu, lain Murray, and Steve Renals. Multiplicative 1stm for sequence modelling, 2016.
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint arXiv: 171.004799 2017.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-iman imaae co-attention for visual question answering. In Advances In Neural Information Processing Systems, pp. 289-297, 2016.
10

---

Published as a conference paper at ICLR 2020
Roland Memisevic and Geoffrey Hinton. Unsupervised leaning of image transformations. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. IEEE, 2007.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action- eation-conditional video prediction using deep networks atari games. In Advances in neural information processing systems, pp. 2863-2871, 2015.
Ethan Perez, Harm De Vries, Florian Strub, Vincent Dumoulin, and Aaron Courville. Learning visual reasoning without strong priors. arXiv preprint
2017.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Jack W Rae, Chris Dyer, Peter Dayan, and Timothy P Lillicrap. Fast parametric learning with activation memorization. arXiv preprint arXiv: 10049, 2018.
Malcolm Reynolds, Gabriel Barth-Maron, Frederic Besse, Diego de Las Casas, Andreas Fidjeland, Tim Green, Andria Puigdomenech, Sébastien Racanière, Jack Rae, and Fabio Viola. Open sourcing Sonnet library for constructing neural networks. https:/ com open-sourcing sonnet/] 2017.
D. E. Rumelhart, G. E. Hinton, and J. L. McClelland. Parallel distributed processing: Explorations in the microstructure cognition, vol. chapter A General Framework for Parallel Distributed Processing, pp. 45-76. MIT Press, Cambridge, MA, USA, 1986. ISBN 0-262-68053-X. URL http:://l. acm. 2ttp://aaa.....co...2........... 10426
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 7299-7310, 2018.
Idan Schwartz, Alexander Schwing, and Tamir Hazan. High-order attention models for visual question answering. In Advances in Neural Information Processing Systems, pp. 3664-3674, 2017.
Terrence J Sejnowski. Higher-order boltzmann machines. In AIP Conference Proceedings, volume 151, pp. 398-3888- 403. AIP, 1986.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint 00387, 2015.
Ilya Sutskever, James Martens, and Geoffrey Hinton. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 1017-1024, 201
Graham W Taylor and Geoffrey E Hinton. Factored conditional restricted boltzmann machines for modeling motion style. Proceedings of the 26th annual international conference on machine learning, pp. 1035-1032. ACM, 2009.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. In Advances in neural information processing systems, pp. 4798. 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3540-3549, JMLR. org, 2017.
Karl Weierstrass. On continuous functions of a real argument which possess a definite derivative for value of the argument. Königlich Preussichen Akademie der Wissenschaften, 2:71-74, 1895.
11

---

Published as a conference paper at ICLR 2020
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. ICLR, 2019.
Yuhuai Wu, Saizheng Zhang, Ying Zhang, Yoshua Bengio, and Ruslan R Salakhutdinov. On multiplicative integration with recurrent neural networks. In Advances in neural information processing systems, pp. 2856-2864, 2016.
Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. European Conference on Computer Vision, pp. 451 551-466. Springer, 2016.
Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In Proceedings of the IEEE international conference computer vision, pp.
2017.
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 4189 4198. JMLR. org, 2017
12

---

Published as a conference paper at ICLR 2020
A EXPRESSIVITY OF THE MODEL
Theorem 1. Let Hmip denote the hypotheses space of standard MLPs with ReLU activation function, and let Hmm denote the hypotheses space of analogous networks, but with each linear layer replaced with multiplicative layer, then we have Hmmpp & Hmm-
Proof. Inclusion comes directly from the fact that we split input into arbitrary parts [x;z] we get:
x Wz B V2 c= x² Wz+ [B:; V [x: + c x" Wz A +
which proves that Hmlp C. Hmm-- Thus, the only aspect of the theorm left prove that the inclusion strict. Let us consider ID function - x²² and for simplicity let
(a domain where context equals input). A single layer MLP with single multiplicative unit can represent this function exactly, by using A and W I, as then we obtain ""Wx= x" = ||x|| Since our function is positive, it is not affecting the multiplicative network output. For a regular MLP, let us first notice that we need least one hidden layer, as otherwise MLP a linear function and f is not. Lets denote by V, c and w, b weights and biases of second, and first layers respectively. Then we have to satisfy
f(x) g(V" max we + b) c) x²
where g is transformation represented by all higher layers of the MLP (in particular if there are just 2 layers, then g(x) = x). Note that RHS is differentiable everywhere, while LHS is differentiable iff for each and for each x we have W.I - b, + O (or is independent from x, which x² does not satisfy). However, this is impossible, if wi j then we can always pick æ -b./w, and if all w; = 0, then f(x) = g(c) + x², leading to a contradiction.
Theorem 2. Let Hy denote the hypotheses space of standard MLPs with a activation function, and Hyr analogous set, where some activations replaced with Weiestrass function fw. Then have Hretu & Hon and Ho & Hw:for any a that is differentiable everywhere.
Proof. Inclusion comes directly from the fact that only some activations are replaced, and in particular we can always replace none, thus leading to equality of hypotheses classes. To show that the inclusion strict lets consider Weierstrass function itself f(x) = (w)) We definitely have f E Hy as can define 1 hidden layer network, with one hidden neuron and all the weights set to 1, and all biases to O. Now, relu networks are piece-wise linear while the Weierstrass function is nowhere differentiable Weierstrass (1895) and thus not piece- pisce-wise linear. Similarly, network with activation that is differentiable everywhere (e.g. sigmoid or tanh) is everywhere differentiable wrt. inputs, while Weierstrass function nowhere Weierstrass (1995).
B SIMPLE IMPLEMENTATION OF MI LAYER
We use Tensorflow and Sonnet Reynolds et al. (201)) for all our model implementations. The example below a simple code snippet for adding a multiplicative layer to any model, using the Sonnet framework example.
Simple python code for MI Layers import sonnet snt import tensorflow as if
A standard linear layer
B is the batch size E is the input size
C is the context size
# input of size [B, E] # context of size [B, C]
XZ = if. concat ([x,)] 1)
y = snt. Linear coutput_size (xz)
Instead, we generate a W and b This defines implicit 3D weight tensor
13

---

used Published as a conference paper at ICLR 2020
W snt Linear (output_siz) input_size (z)
= snt Linear ooupputssize (z)
Reshape to the correct shape Note we have B weight matrices
i.c. one per batch clement
W if reshape (W, [B, inzut_size, output_size] ))
Output
if matmul W) +
C DETAILS FOR SIMPLE FUNCTION EXPERIMENTS
For the experiments modeling the simple two-input functions, we consider MLP and MI models and plot number of parameters against the input hidden size. The two models are
MLP: linear(size), relu, linear(output_sia) _ize
MI Network: MI(size), linear(oulput_sizze
Here size is the largest value such that the network has not more than variables and we sweep over [1, 10, 15, 20, 30, 40, 50, 60, 80, 100, 120, 140, 160, 180, 200]. We sweep over learning rates 0.1, 0,001, 0.0001 and pick the best result. Models are trained using Adam optimiser for 6,000 steps using Mean Squared Error loss (MSE) on mini-batches of size 100 sampled from a standard Gaussian. The reported error is based on 10,000 samples from the same distribution to minimize the estimation variance.
D DETAILS FOR Toy REGRESSION EXPERIMENTS
In multitask toy regression we coinsider MLP, MI and multiheaded models. For E IR (input to the task) and z (represented as one-hot encoding of a task ID) we use:
Conditional MLP: concat(x, linear(20)(2), linear(30), relu, linear(20), relu, linear(1)
Conditional MI: MI( (linear(30), relu, linear(20), relu](x), linear(20)(z)
Multiheaded MLP: linear(20), relu, linear(30), relu, linear( 1); where the last linear is separate per task
Each of these models is trained in multitask setup, where we first sample T tasks (20, 40 or 60). Half of these tasks involve fitting an affine function ax + where both a and b are sampled from uniform distribution over [0. 1]; and half represent scaled sine waves, a sin( (10x) + with and b sampled in the same way. Training performed with Adam optimiser (with learning rate - 3), on mini batches of size 50 per task (so the total batch size is 50T). Models are trained for 10,000 steps with Mean Squared Error loss (MSE), and the logarithm of the training MSE is reported for analysis. Each training is repeated 60 times to minimise the variance coming from the stochasticity.
E DETAILS OF MULTITASK DEEPMIND LAB TRAINING
We train multi-task on 30 DeepMind lab levels Beattie et al. (2016) concurrently using 5 actors per task and multi-gpu learner with 4 GPUs. We follow exactly the model and the Impala architecture/ Espeholt et al. (2018). Our models are all trained with population based training (PBT) Jaderberg et al. (2017) 7) and we show below the average 24 independent learners each for both the baseline and our best method. We train all models for 10 billion frames of data across all levels. The human normalised score is calculated independently for each level and capped at 100 (i.e score = min( min(score, 100))
three populations with
The architecture is follows:
14

---

Published as a conference paper at ICLR 2020
Conv2D: 16ch, 8x8 kernels, stride=4
ReLU
Conv2D: 32ch 4x4 kemdel.trrie=22
ReLU
Linear layer with output size 256
ReLU
Concatenation with one hot encoded last action and last reward and language instruction
LSTM (256 hidden units)
Policy = Linear layer or multiplicative interaction followed by softmax
Value function Linear layer or multiplicative intereaction
Normallsed Score (No ID)
Human Normalised Score
70
80
40
30
10
0.0
1.2
0.4
0.5
0.5
Frames
1e10
F DETAILS OF NEURAL PROCESS EXPERIMENTS
Inspired by the experiments in Garnelo et al. 2018b), apply Neural Processes to functions drawn for a Gaussian Process prior with an exponentiated quadratic kernel b(xx,²) = o
2²)² /12) with fixed of = 1.0 and, importantly, a random / ~ U[0.1, 10.0) for each draw, resulting in a broad distribution of functions. We also add Gaussian observation noise, i.c. y NN provide deterministic transformation / of the context
and set a 0.02. In all experiments, in addition to the latent variable z, using separate encoder networks for each. For both SKIP MLP and the proposed MI, concatenate and first, i.e. MI(x,concatt (2, h)) writing concat() to denote concatenation. A more detailed discussion on adding an the benefits of adding additional deterministic path in provided in Kim et al. (019).
SKIP MLP[2; concat h)] and
The determinsitic encoder used to obtain h is a deep MLP with 6 layers of 128 units each. The latent encoder used for consists of 3 hidden layers of 128 units, parameterising mean and standard deviation of a 64-dimensioonl Gaussian distributed latent variable. The decoder network used to predict on new target inputs x consists of layers of 128 units. We use relu activiatons throughout all components of the network. The network is trained until convergence with Adam, using learning rate of 0.0001 and absolute value gradient clipping with a threshold of 10.0.
G LANGUAGE MODELLING
We train single layer LSTM of hidden size 2048 hidden units. The input to the LSTM is an embedding of size 256 and then output the LSTM is projected down to 256 with a single linear layer for the baseline. The input and output embedding word matrices are tied. We use training sequence length of 128. For the multiplicative model, the output embedding is calculated with an MI layer who's context input is generated by linear layer with output size 32 followed by relu. We use dropout rate of 0.3 and a learning rate of 0.001 for all models. All models are trained with the Adam optimiser.
15

---

Published as a conference paper at ICLR 2020
For the multiplicative encoder a diagonal form of the M(.) layer and for the multiplicative deocder we use the full MM.) form (with the bottleneck described above). This amounts to adding about 20M parameters which is the same adding 1000 hidden units. We get 6 perplexity improvement in performance whereas naively adding 1000 hidden units only gave us an improvement of perplexity. Further the parameter count could be drastically reduced by considering diagonal low rank approximations, but we do not specifically optimise for this in this work.
16


Reviews:
[{"review_id": "HkgyJKGZor", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper takes a component of neural networks that is sometimes, but not always used — multiplicative interactions — and goes into depth in analyzing and discussing the different ways in which computed features may interact multiplicatively. The analysis and math are clear, and the main points are made with neither too many nor too few equations. Authors cast several other approaches (gating, attention, Hypernets) within the same formulation, which while not being groundbreaking or overly novel is nice to see compiled into this particular form.\n\nDecision: weak accept.\n\nPros:\n - I can imagine any researcher getting into the field and interested in this sort of thing reading this paper as a canonical, concise, and moderately thorough intro to the concept.\n - Beautiful drawing\n\nCons:\n - Experiments are a little light. To make the paper more thorough, it would be nice to see cases where multiplicative interactions are *not* needed, e.g. if one sticks multiplicative interactions in a network trained to classify MNIST or CIFAR-10, do they help at all? If not, why not?\n - Further ablation studies would also benefit the paper. For example, in the language modeling section, how would the results change with varying bottleneck size? With a completely diagonal approximation? Finally, LSTMs already have multiplicative interactions in them (via gating, as mentioned in the paper). Would the proposed form of multiplicative interaction obviate the need for LSTM gating all together?\n\nWith additional experiments this paper could be a strong accept and recommended reading for many first year grad students. As is, it’s probably still worth accepting.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlQXYsl5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores different types of multiplicative interactions. This allows understanding of both efficacy of multiplicative interaction (i.e., MI vs MLP), and the common between multiplicative-type models (e.g., hypernetworks, FiLM, gating,  attention etc). The authors also find MI models able to achieve a state-of-the-art performance on language modeling (WikiText-103) and reinforcement learning problems(DMLAB-30), along with several toy examples. \n\nStrengths: \n*The discussed MI subject is important research area, the paper presents the vast related work, proven by the fact MI techniques appears in many recent models.\n* The authors  did comprehensive evaluation to show importance and usefulness of multiplicative interactions. A  toy regression experiment to show superiority of MI vs MLP . A Reinforcement learning task, i.e DMLAB-30 on par with the state-of-the-art model, but with simple model (i.e., less parameters). A sequence prediction with an alternative embedding technique that improves both accuracy and number of needed parameters. \n* The paper is of high quality: well organized, clear, with good supporting figures.  \n\nWeaknesses:\n* I suggest a better explanation how the suggested models compare to state-of-the-art models.  Currently, it is hard to assess the impact. For instance, proposed model alternate existing baselines, such as PopArt, or is completely novel?  \n* Although mentioned, it's not the focus of this work, the paper should have discussed attention models more.  Specifically recent years multimodal attention relied on multiplicative interactions. Therefore, at least in this domain multiplicative interactions are not \"under-appreciated\". Relevant papers: [1, 2, 3, 4] - advancement of multiplicative interactions over the years, [5] - introduce co-attention, [6] - introduce higher-order interactions (between three vectors), [7] - introduce bilinear attention .\n\nTo conclude, multiplicative interactions are extremely important, and I find the paper exploration useful. In addition,  the paper is of high quality, and with satisfied experiments to prove their claims. I do suggest a better discussion about multimodal attention networks, which are relevant examples.\n\n[1] - Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering; Xu et al.\n[2] - Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding; Fukui et al.\n[3] - Hadamard Product for Low-rank Bilinear Pooling; Kim et al.\n[4] - Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering; Yu et al.\n[5] - Hierarchical Question-Image Co-Attention for Visual Question Answering; Lu et al.\n[6] - High-Order Attention Models for Visual Question Answering; Schwartz et al.\n[7]  - Bilinear Attention Networks; Kim et al. \n ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJl4_UpTYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents multiplicative interaction as a unified characterization for representing commonly used model architecture design components (e.g. gating, attention layers and hypernetworks). Multiplicative interactions can be viewed as an effective way of integrating contextual information in a network. Through a series of thorough empirical experiments, this paper demonstrates superior performance on a variety of tasks (RL, sequence modeling) when a such multiplicative interaction module is incorporated.\n\nThe framework seems applicable for learning tasks where a latent variable or context embedding presents. And the paper hypothesizes that multiplicative interactions can help introduce desirable inductive biases, therefore leading to improved generalization performance. However, the theoretical intuition and justification for such hypothesis is missing, which weakens the contribution of the paper. \n\nIn Table 1, the model LSTM with Multiplicative Decoder has more parameters (105M) than the vanilla LSTM in comparison (88M). It’d be good to provide a more fair comparison by slightly increasing the capacity of baseline LSTM model (e.g., using larger output dimension to match the capacity). This will rule out the cofounding factor that the improved performance is due to the algorithm instead of increased model capacity. \n\nIn Section 8, it’d be convincing if the authors can also report results for replacing default LSTM cell with multiplicative interactions. In paragraph 3, the authors only discussed the feasibility conceptually, without providing experimental results to support this argument. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a unifying perspective regarding a variety of popular DNN architectures in terms of the inclusion of multiplicative interaction layers.  Such layers increase the representational power of conventional linear layers, which the paper argues can induce a useful inductive bias in practical scenarios such as when multiple streams of information are fused.  Empirical support is provided to validate these claims and showcase the potential of multiplicative interactions in occupying broader practical roles.

All reviewers agreed to accept this paper, although some concerns were raised in terms of novelty, clarity, and the relationship with state-of-the-art models.  However, the author rebuttal and updated revision are adequate, and I believe that this paper should be accepted.
Decision: Accept (Poster)