Neural Arithmetic Units
Authors: Andreas Madsen
Alexander Rosenberg Johansen
Venue:  (2020)

Abstract: Neural networks can approximate complex functions, but they struggle to perform exact arithmetic operations over real numbers. The lack of inductive bias for arithmetic operations leaves neural networks without the underlying logic necessary to extrapolate on tasks such as addition, subtraction, and multiplication. We present two new neural network components: the Neural Addition Unit (NAU), which can learn exact addition and subtraction; and the Neural Multiplication Unit (NMU) that can multiply subsets of a vector. The NMU is, to our knowledge, the first arithmetic neural network component that can learn to multiply elements from a vector, when the hidden size is large. The two new components draw inspiration from a theoretical analysis of recently proposed arithmetic components. We find that careful initialization, restricting parameter space, and regularizing for sparsity is important when optimizing the NAU and NMU. Our proposed units NAU and NMU, compared with previous neural units, converge more consistently, have fewer parameters, learn faster, can converge for larger hidden sizes, obtain sparse and meaningful weights, and can extrapolate to negative and small values.

Published a conference paper at ICLR 2020
NEURAL ARITHMETIC UNITS
Andreas Madsen
Alexander Rosenberg Johansen
Computationally Demanding anwebdk & gma il. com
Technical University of Denmark aler@dtu. .kk
ABSTRACT
Neural networks can approximate complex functions, but they struggle to perform exact arithmetic operations over real numbers. The lack of inductive bias for arithmetic operations leaves neural networks without the underlying logic necessary to extrapolate on tasks such as addition, subtraction, and multiplication. We present new neural network components: the Neural Addition Unit (NAU), which can learn exact addition and subtraction; and the Neural Multiplication Unit (NMU) that can multiply subsets of vector. The NMU is, to our knowledge, the first arithmetic neural network component that can learn to multiply elements from vector, when the hidden size is large. The two new components draw inspiration from a theoretical analysis of recently proposed arithmetic components. We find that careful initialization, restricting parameter space, and regularizing for sparsity is important when optimizing the NAU and NMU. Our proposed units NAU and NMU, compared with previous neural units. converge more consistently, have fewer parameters, learn faster, can converge for larger hidden sizes, obtain sparse and meaningful weights, and can extrapolate to negative and small values
INTRODUCTION
When studying intelligence, insects, reptiles, and humans have been found to possess neurons with the capacity to hold integers. real numbers, and perform arithmetic operations [Nieder] 2016)) Rugani et al.) 2009; Gallistel 2018). In our quest to mimic intelligence, we have put much faith in neural networks, which in turn has provided unparalleled and often superhuman performance in tasks requiring high cognitive abilities (Silver et al. 2016 Devlin et al. 2018 OpenAl et al. 2018). However, when using neural networks to solve simple arithmetic problems, such as counting, multiplication, or comparison, they systematically fail to extrapolate onto unseen ranges (Lake & Baroni 2018 (Suzgun et al. [0099] Trask et al. 2018). The absence inductive bias makes it difficult for neural networks to extrapolate well on arithmetic tasks as they lack the underlying logic to represent the required operations.
A neural component that can solve arithmetic problems should be able to: take an arbitrary hidden input, learn select the appropriate elements, and apply the desired arithmetic operation. A recent attempt to achieve this goal is the Neural Arithmetic Logic Unit (NALU) by Trask et al. (2018).)
The NALU models the inductive bias explicitly via two sub-units: the NAC, for addition//ubtraction and the NAC, for multiplicatiion/diiiioon The sub-units are softly gated between, using sigmoid function, to exclusively select one of the sub-units. However, we find that the soft gating-ma mechanism and the NAC, are fragile and hard to learn.
In this paper, we analyze and improve upon the NAC+ and NAC. with respect to addition, subtraction. and multiplication. Our proposed improvements, namely the Neural Addition Unit (NAU) and Neural Multiplication Unit (NMU), are more theoretically founded and improve performance regarding stability, speed of convergence, and interpretability of weights. Most importantly, the NMU supports both negative and small numbers and large hidden input-size, which is paramount as neural networks are overparameterized and hidden values are often unbounded.
The improvements, which are based on theoretical analysis of the NALU and its components, are achieved by a simplification of the parameter matrix for a better gradient signal, sparsity regularizer, and a new multiplication unit that can be optimally initialized. The NMU does not support division.
Implementation available GitHuh:
https: thub, hwtee///eeeeeeeedeee//eeeee st.bb e-nalu

---

Published conference paper at ICLR 2020
21
W
21,2 22
Wi, Fn
Figure 1: Visualization of the NMU, where the weights (Wij) controls gating between 1 (identity) 24, each intermediate result then multiplied explicitly form 2g-
However, we find that the NAC. in practice also only supports multiplication and cannot learn division (thcoretical analysis division discussed in section 2.3
To analyze the impact of each improvement, introduce several variants of the NAC.. We find that allowing division makes optimization for multiplication harder, linear and regularized weights improve convergence, and the NMU way of multiplying is critical when increasing the hidden size.
Furthermore, we improve upon existing benchmarks in Trask et al. (2018) by expanding the "simple function task", expanding "MNIST Counting and Arithmetic Tasks" with a multiplicative task, and using an improved success- success-criterion Madsen & Johansen (2019). This success- srrees-crittrion is important because the arithmetic units solving logical problem. We propose the MNIST multiplication variant want test the NMU's and NAC, 's ability to learn from real data and extrapolate.
1.1 LEARNING A 10 PARAMETER FUNCTION
Consider the static function = (0) + 222 (x1 + + + .44 for I E R24, To illustrate the ability of NAC., NALU, and our proposed NMU, we conduct 100 experiments for each model to learn this function. Table [ shows that the NMU has a higher success rate and converges faster.
Table 1: Comparison of the success-rate, when the model converged, and the sparsity error for all weight matrices, with 95% confidence interval the (I) + x2) (x1 22 + va + + task. Each value a summary of 100 different seeds.
Op
Model
Success
Solved at iteration step
Sparsity emror
Rate
Median
Mean
Mean
NAC.
13%
+8%
5.5 10a
5.9 10a +7.8-10
7.5 10
+220-10
-5%
-2.0-10-9
NALU
26% +9%
7.0 10
7,8 7,8 10
46.2-10²
9.2 10
+1.7-10-6
x
-8%
-5.6-103
-1.7-10-6
NMU
94% +3%
1.4 10
1.4 104 +2.2-102
2.6 10 +6.4-10-2
-2,1-102
-6.4-10-9
2 INTRODUCING DIFFERENTIABLE BINARY ARITHMETIC OPERATIONS
We define our problem learning set of static arithmetic operations between selected elements a vector. E.g. for a vector x leamn the function (I; + x1) .zy. The approach taking in this paper is to develop unit for addition/suubrratiion and unit for multiplication, and then let each unit decide which inputs to include using hackpropagation.
We develop these units by taking inspiration from a theoretical analysis of Neural Arithmetic Logic Unit (NALU) by Trask et al. (2018).
2.1 INTRODUCING NALU
The Neural Arithmetic Logic Unit (NALU) consists of two sub-units: the NAC., and NAC,, The sub-units represent either the - or the [x, operations. The NALU then assumes that either NAC+ NAC. will be selected exclusively, using sigmoid gating-mechane mechanism.

---

Published conference paper at ICLR 2020
The NAC+ and NAC, are defined accordingly,
Wheiher = tanh (Wh, he- Jo (Mherrer))
(1)
HL-1
N 2hy =
Σ
Wo Flae1
(2)
he-1=1
He
NAC. 2h, = cxp
Σ
Whiders log( +()
(3)
where W. M& E RR are weight matrices and _hz-; is the input. The matrices are combined using tanh-sigmoid transformation bias the parameters towards {-1, solution. Having {-1.0, }} allows NAC+ to compute exact {+, -} operations between elements of a vector. The NAC. uses an exponential- transformation for the {x, operations, which works within e precision and for positive inputs only.
The NALU combines these units with gating mechanism z = g NAC+ + (1-g) g) o NAC. given g = o GG)) Thus allowing NALU to decide between all of (+, + using backpropagation.
2.2 WEIGHT MATRIX CONSTRUCTION AND THE NEURAL ADDITION UNIT
Glorot & Bengio (010) show that E[a,,] 0 at initialization is a desired property, as it pre- vents an explosion of both the output and the gradients. To satisfy this property with Wh, i.
tanh(Wi, bbe ) (Mrr ,he)., ) initialization must satisfy E[tanh(W::
initialization unbiased as it samples evenly between + and (..).. = 0. In NALU, this and +. Unfortunately, this initialization also causes the expectation of the gradient to become zero, as shown in (4).
OC =E
Oc OW he
tatahh(Wn ['' =0
(Min-
(4)
Besides the issue of initialization, our empirical analysis (table shows that this weight construction (T) do not create the desired bias for (, }- This bias is desired as it restricts the solution space exact addition, and in section 2.5 also exact multiplication, which is an intrinsic property of an underlying arithmetic function. However, this bias does not necessarily restrict the output space as a plain linear transformation will always be able to scale values accordingly.
To solve these issues, we a sparsifying regularizer to the function (C 2+ Asparee Rt.spa and use a simple linear construction, where Wh is clamped to [-1, 1] in each iteration.
Whe-nike min(max (ee- -1), 1);
(5)
Ht
He-1
1
Re
min (Wehe - W Whel
(6)
He He-1 11=1 11-1=1
Ha-1 Σ
NAU The
Wheheer Phr-1
he-1=1
2.3 CHALLENGES OF DIVISION
The NAC., as formulated in equation 3 has the capability to compute exact multiplication and division, or more precisely multiplication of the inverse of elements from a vector, when a weight in W Whz-iah by -I

---

Published conference paper at ICLR 2020
However, this flexibility creates critical optimization challenges. transformation, NAC. can be expressed as
By expanding the exp-log-
He-a
NAC 2hg
II hz-1=1
(lzhe Whz
In equation ( if | is near zero [E]= 0 is desired property when initializing (Glorot
Bengio 2. Wh h, is negative, and is small, then the output explode. This issue present even for reasonably large value (such as = 0.1), and just slightly negative W hhe Also note that the curvature
visualized in figure 2
cause convergence
unstable area.
This singularity issue in the optimization space also makes multiplication challenging, which further suggests that supporting division is undesirable. These observations are also found empirically in Trask et al. 20188 table 1) and Appendix C.7
:00
100-
300-
300
312
200
200-
212
10-
100-
133
c5
02
(a) NAC. with 10-7
(b) NAC.
(c) NAC. with
Figure 2: RMS loss curvature for a NAC+ unit followed by a NAC.. The weight matrices are constrained to Wi =
us W2 = ueg w> J. The problem is (x) + +22) (x, + + + +4)
for = 11, 1.8, 2). The solution is w1 Wy = 1 in (a), with many unstable alternatives.
2.4 INITIALIZATION OF NAC.
Initialization is important for fast and consistent convergence. A desired property is that weights are initialized such that ][=,,, (Glorot & Bengio 2010 Using second order Taylor approximation assuming all 2 are uncorrelated; the expectation NAC, can estimated
E Blan,] R=
1 +
Vor [We 1 log( [[ .ll + e)² = E(sn) > 1.
(9)
As shown in equation 9 satisfying E[z,,] 0 for NAC. is likely impossible. The variance cannot input-independently initialized and is expected explode (proofs Appendix B.3
2.5 THE NEURAL MULTIPLICATION UNIT
To solve the the gradient and initialization challenges for NAC, propose unit for multipli- cation: the Neural Multiplication Unit (NMU)
WNe the min(max Whe be Jee 0),
(10)
H
Resparee
1
Σ
Σ min (Whrnnhes 1 Whe
(11)
He H1-1 he=1 he-1=1
He-1
NMU:
2he
II
(Whe
W. Whe-)
(12)
hr-1=1
The NMU is regularized similar to the NAU and has multiplicative identity when Wa, 1 0. The NMU does not support division by design. As opposed to the NAC.. the NMU can represent input of both negative and positive values and is not bounded, which allows the NMU to extrapolate Fhe that negative smaller than Its gradients derived in Appendix A.3

---

Published conference paper at ICLR 2020
2.6 MOMENTS AND INITIALIZATION
The NAU is a linear layer and can be initialized using Glorot & Bengio (010). The NAC- unit can also achieve ideal initialization, although it is less trivial (details Appendix B.2)
The NMU is initialized with ElW he- E| E[ = O, which is the case for most neural units (Glorot Bengio 2010), the expectation can be approximated to
1/2.
Assuming 2/y-1 are uncorrelated, and
1 2
He-1
El Ps
(13)
which approaches zero for H - oo (see Appendix B.4 The NMU can, assuming Var Phe-1 and He-1 is large, optimally initialized with Var W Wh myll = á (proof in Appendix B.4.3
2.7 REGULARIZER SCALING
We the regularizer scaling defined in 14 We motivate this by observing optimization consists of two parts: a warmup period, where Whe-i.he should get close to the solution, unhindered by the sparsity regularizer, followed by a period where the solution is made sparse.
1 -
Aatart
-1).))
Asparse
Asparse max
min
Aand Aastrtt
(14)
2.8 CHALLENGES OF GATING BETWEEN ADDITION AND MULTIPLICATION
eating-mechanism The purpose the gating mechanism is to select either NAC, or NAC, exclusively. This assumes that the correct sub-unit is selected by the NALU, since selecting the wrong sub-unit leaves no gradient signal for the correct sub-unit.
Empirically we find this assumption be problematic. We observe that both sub-units converge the beginning of training whereafter the gating-mechanise seemingly random, converge towards either the addition multiplication unit. Our study shows that gating behaves close to random for both NALU and a gated NMU/NAU variant. However, when gate correctly selects multiplication our NMU converges much more consistently. We provide empirical analysis in Appendix C..5 for both NALU and gated version of NAU/NMU.
As the output-size grows, randomly choosing the correct gating value becomes an exponential increasing problem. Because of these challenges we leave solving the issue of sparse gating for future work and focus on improving the sub-units NAC+ and NAC..
3 RELATED WORK
Pure neural models using convolutions, gating, differentiable memory, and/or attention architectures have attempted learn arithmetic tasks through backpropagation (Kaiser Sutskever 2016 Kalch brenner et al., 2016 Graves et al., [2014] Freivalds & Liepins) 2017). Some of these results have close to perfect extrapolation, However, the models are constrained to only work with well-defined arithmetic setups having input redundancy, single operation, and one-hot representations of numbers for input and output. Our proposed models does not have these restrictions.
The Neural Arithmetic Expression Calculator (Chen et al. 2018) can learn real number arithmetic by having neural network sub-components and repeatedly combine them through mnemory-encodee decoder architecture learned with hierarchical reinforcement learning. While this model has the ability to dynamically handle a larger variety of expressions compared to our solution they require an explicit definition of the operations, which we do not.
In our experiments, the NAU is used do subset-selection, which is then followed by either a summation or multiplication. An alternative, fully differentiable version, is to use a gumbel- softmax that can perform exact subset-selection (Xie & Ermon 2019). However, this is restricted to a predefined subset size, which is strong assumption that units limited by.

---

Published conference paper at ICLR 2020
4 EXPERIMENTAL RESULTS
4.1 ARITHMETIC DATASETS
The arithmetic dataset is a replica of the "simple function task" by Trask et al. (2018). The goal is to
sum tWO random contiguous subsets of vector and apply an arithmetic operation defined in (15)
f
Σ N2gattt
where x E R2, 24 ~ Uniform Flowwrr /upperr, E {+, - x (15)
where n (default 100), UFLowwer Fupper] (interpolation default U|,, 2] and extrapolation default
2 6). and other dataset parameters are used to assess learning capability (see details in Appendix
and the effect of varying the parameters in Appendix C.4)
4.1.1 MODEL EVALUATION
We define the success-criterion as solution that is acceptably close to perfect solution. To evaluate if model instance solves the task consistently, we compare the MSE a nearly- solution on the extrapolation range over many seeds. If W1. W2 defines the weights of the fitted model, Wi is nearly-perfect, and W; is perfect (example in equation (66), then the criteria for successful convergence Lw, W,
LW; measured on the extrapolation error, for 10-5, We report a 95% confidence interval using a binomial distribution (Wilson 1927).
< L,
1 1- 1- 1 0+e 0+ 1 1 1-c W W; = 1
Wi =
(16)
To measure the speed of convergence, we report the first iteration for which Lw,W, L.;.W. is satisfied, with a 95% confidence interval calculated using gamma distribution with maximum likelihood profiling. Only instances that solved the task are included.
We assume approximate discrete solution with parameters close to {-1, 0, is important for inferring exact arithmetic operations. measure the sparsity, we introduce a sparsity error (defined in equation[]] TT). Similar to the convergence metric, we only include model instances that did solve the task and report the 95% confidence interval, which calculated using beta distribution with maximum likelihood profiling.
max min([WWe-n.kee Exparsity
[Wh Whllll
(17)
b
4.1.2 ARITHMETIC OPERATION COMPARISON
We compare models on different arithmetic operations o E {+, x]- The multiplication models, NMU and NAC., have addition unit first, either NAU NAC4, followed by a multiplication unit. The additionnsubbttaaction models are two layers of the same unit. The NALU model consists of two NALU layers. See explicit definitions and regularization values in Appendix
Each experiment is trained for 5 10 iterations with early stopping by using the validation dataset, which is based on the interpolation range (details in Appendix C.2)) The results presented in table 2 For multiplication, the NMU succeeds more often and converges faster than the NAC. and NALU. For addition and substraction. the NAU and NAC+ has similar success-rate (100%), but the NAU is significantly faster solving both of the the task. Moreover, the NAU reaches significantly sparser solution than the NAC4. Interestingly, a linear model has hard time solving subtraction. more extensive comparison is included in Appendix C.7 and an ablation study is included in Appendix C.3
4.1.3 EVALUATING THEORETICAL CLAIMS
To validate our theoretical claim, that the NMU model works better than NAC, for a larger hidden input-size, increase the hidden size of the network thereby adding redundant units. Redundant units are very common in neural networks, which are often overparameterized.
Additionally, the NMU model is, unlike the NAC. model, capable of supporting inputs that are both negative and positive. To validate this empirically, the training and validation datasets are sampled for U|- 2, 2]. and then tested on U - U U[2, 6). The other ranges defined in Appendix C.4

---

Published conference paper at ICLR 2020
Table 2: Comparison of: success-rate, first iteration reaching success, and sparsity emor, all with 95% confidence interval on the "arithmetic datasets" task. Each value is summary of 100 different seeds.
Op
Model
Success
Solved at iteration step
Sparsity error
Rate
Median
Mean
Mean
NAC.
31% -10%
2.8 10
3.0 10 +2.9-10"
5.8 10 +-.8-10 -2.6-10
-2.4-10a
NALU
0%
+4%
0%
NMU
98%
+1%
1.4 10
1.5 10
+5.0-10"
4.2 10-10-
-2.9-10
-.6--10
NAC4
100%
+0%
2.5 10"
4.9 10.
+5.2-10*
2.3 13-10-11
+6.5-10-3
-4.5-104
Linear
100%
+0%%
6.1.101 104
6.3 104
+2.5-10"
15-10-
+3.6-10
4%
-3.3-10a
-3.6-10-4
+
NALU
14%
8%
1,5 106
1.6 10
+3.8-10"
1.7.10-1
+2.7-10
-2.5.10-2
NAU
100%
+0%%
1.8 10a
3.9 10 +4.5-10"
3.2 10
-1.3-10-3
4%%
-3.7-104
-1.3-10-3
NAC+
100%
+.%%
9.0 109
3.7 10 +3.8-10"
2.3 2.3110-1 +5.4.1033
4%
-3.5-104
-5.4-10-3
Linear
7%
+7%
3.3 3
1.4 10
47.0-10"
1.8 13-10-1
+7.2-10
-4%
NALU
14%
-8%
1.9 106
1.9
+4.4-10
2.1 10
+2:2-11
-5%
-4.5-109
-2,2-10-2
NAU
100% +00%
5.0 103
1.6 105 +1.7-10
6.6 10
-2.5-10
-1.6-10
-199-10-2
Finally, for a fair comparison we introduce two new units: A variant of NAC. denoted NAC. that only supports multiplication by constraining the weights with W oW). And a variant, named NAC. NMU. that uses clamped linear weights and sparsity regularization identically to the NMU.
Figure shows that the NMU can handle much larger hidden-size and negative inputs. Furthermore, results for NAC... and N NAC, NMM validate that removing division and adding bias improves the success-rate, but are not enough when the hidden-size is large, as there is no ideal initialization. Interestingly, no models can learn U[1.1, 1.2), suggesting certain input ranges might be troublesome.
Success
Solved iteration stap
Sparsity
1.00
5e-06
0.25
0.20
0.75
3e-05
0.15
0.50
2e-06
0.10
1e+06
0.05
0.00
0e-00
0.00
10
10
Hidden size
1.00
0.0045
0.75
3o+06-
0.0020
0.0015
0.50
2e+06
0.0010
1e+06
0.00
0e+00
0.0000
U1-2-11
U1-2.2)
[j0.1]
000.1.0.21
1...1.1.1
20f
U(-2.-11
U(-2.2)
Uto
2.0.1.22.
un 1.1.2
20]
U1-2-11
u[-2.2]
Uj0.1)
U.0.1..2f
Uf
UfL.1.1 1f1.1.12)
UT30.20
Uf
Interpolation range
model
NAC
NAC
Figure 3: Multiplication task results when varying the hidden input-size and when varying the input-range. Extrapolation ranges are defined in Appendix C.4

---

Published conference paper at ICLR 2020
PRODUCT OF SEQUENTIAL MNIST
To investigate deep neural network can be optimized when backpropagating through an arithmetic unit, the arithmetic units are used as recurrent-unit over sequence of MNIST digits, where the target is fit the cumulative product. This task is similar "MNIST Counting and Arithmetic Tasks" in Trask et al. (2018 but uses multiplication rather than addition (addition is in Appendix D.2 Each model trained on sequences of length 2 and tested on sequences of up to 20 MNIST digits.
We define the success-criterion by comparing the MSE of each model with baseline model that has a correct solution for the arithmetic unit. If the MSE of each model is less than the upper 1% MSSE-confidenceiittery of the baseline model, then the model is considered successfully converged.
Sparsity and "solved at iteration step" is determined as described in experiment 4.1 The validation set the last 5000 MNIST digits from the training set, which used for early stopping.
this experiment, we found that having an unconstrained "input- network" can cause the multiplication-unal units to learn an undesired solution, e.g. (0.1 81 + 0.1) = 9. Such network do solve the problem but not in the intended way. To prevent this solution, regularize the CNN output Eh. ) This regularizer is applied to the NMU and NAC. NNMU models. See Appendix D.4 for the results where this regularizer is not used.
with R. H. Liee Hi Dmy-1(1 (1 Whe ..h))
Figure 4 shows that the NMU does not hinder learning
complex neural network. Moreover, the NMU can extrapolate to much longer sequences than what is trained on.
S Success rate
S Salved at iteration step
5 parsity error
1.00
3e+05
0.3
0.75
Ze 105
0.2
0.50
1e+05
0.1
0.00
0e+00
0.0
10
12
14
16
18
20
12
10
10
12
14
16
18
20
Extrapolation length
model
NAC.
LSTM
NALU
NMU
Figure 4: MNIST sequential multiplication task. Each model is trained sequences of two digits. results are for extrapolating to longer sequences. Error-bars represent the 95% confidence interval.
5 CONCLUSION
By including theoretical considerations, such as initialization, gradients, and sparsity, we have developed the Neural Multiplication Unit (NMU) and the Neural Addition Unit (NAU), which outperforms state-of-the-art models established extrapolation and sequential tasks. Our models converge more consistently, faster, to an more interpretable solution, and supports input ranges.
A natural next step would be to extend the NMU to support division and add gating between the NMU and NAU, to be comparable in thcoretical features with NALU. However find, both experimentally and theoretically, that learning division is impractical, because of the singularity when dividing by zero, and that a sigmoid-gate choosing between two functions with vastly different convergences properties, such multiplication unit and addition unit, cannot be consistently learned.
Finally, when considering more than just inputs the multiplication unit, model performs significantly better than previously proposed methods and their variations. The ability for a neural unit to consider more than two inputs is critical in neural networks which are often overparameterized.
"The same CNN used,
thab. http:///////ttooooooooooo/orrrrr

---

Published conference paper at ICLR 2020
ACKNOWLEDGMENTS
We would like thank Andrew Trask and the other authors of the NALU paper, for highlighting the importance and challenges of extrapolation Neural Networks.
We would also like to thank the students Raja Shan Zaker Kreen and William Frisch Moller from The Technical University of Denmark, who initially showed that the NALU do not converge consistently.
Alexander R. Johansen and the computing resources from the Technical University of Denmark, where funded by the Innovation Foundation Denmark through the DABAI project.
REFERENCES
Kaiyu Chen, Yihan Dong. Xipeng Qiu, and Zitian Chen. Neural arithmetic expression calculator. CoRR, abs/1809.085%) 2018. U
08590
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: pre training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805 2018. URL
http:://ara.vv..rg//abbs.
04805
Karlis Freivalds and Renars Liepins. Improving the neural GPU architecture for algorithm learning.
CoRR, Abh/1702.088721 2017.
Charles R. Gallistel. Finding numbers in the brain. Philosophical Tronsactions of the Royal Society Biological Sciences. 373(1740:20170119 2018. doi: 10. 20.1099/ssbb2201..0111 URL RRLhtER tps
// roya seciet
ish www.g...ogoooooo..ooo...o doi abs / 10. 10
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In JMLR W&CP: Proceedings of the Thirleenth International Conference on Artificial Intelligence and Statistics (AISTATS 2010), volume 9, pp. 249-256, May 2010.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. CoRR, abs/1410.5401, 2014. httttp://axxi..or////aa/
Lukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. BttL:tttt////xi..ooo htagg//bb/l1110 09228
Nal Kalchbrenner, Ivo Danihelka, and Alex Graves, Grid long short-term memory, In 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL. http: /ttt:://arxiv.. http:///aawwv...//ab..ssss// org.abs/////////o /000 507 01526
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In The 3rd International Conference for Learning Representations, San Diego, 2015, pp. 1rX6v:1412.6980, Dec 2014.
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-toe sequueeece recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässso, Stockholm, Sweden, July 10-15, 15, 2018. pp. 2779-288,, 2018. URL http:://proceeeennggg wwwpingsssms.rr...mmm ml.. /tw.ppeee////////ee.ee.. htm
Andreas Madsen and Alexander R. Johansen. Measuring arithmetic extrapolation performance, In Science meets Engineering of Deep Learning at 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), volume abs/1910.01888, Vancouver, Canada, October 2019.
URRlhhttp///axxiv.oo -
Andreas Nieder. The neuronal code for number. Nature Reviews Neuroscience, 17:366 EP 05 2016 2016.40.
2016. URL. htt.:tttss///dii.ooott

---

Published conference paper at ICLR 2020
OpenAI, Marein Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub W. Pachocki, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng. and Wojciech Zaremba. Learning dexterous -n-hand manipulation. CoRR, abs/1808.00177, 2018. URL http
Rosa Rugani, Laura Fontanari, Eleonora Simoni, Lucia Regolin, and Giorgio Vallortigara. Arithmetic newborn chicks. Proceedings of the Royal Society B: Biological Sciences, 27 27616)))2666666)611626 2009. doi: 10. w0.1098/rssb..oo 200000000..00000004..0.....44444 0044. URL Sttps///boalloooeeeyyaalliiiii org/ dol/abs/ 1ww.:00098///0 2009. 0004
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George den Driess- che, Julian Schrittieseer, Ioannis Antonoglou, Vedavyas Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, llya Sutskever, Timothy P. Lillicrap. Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 52847587)44444889 2016. doi: 1..1038/nauurc1966
1038 1ttp:///0..000//aaueeeee5531
Mirac Suzgun, Yonatan Belinkov, and Stuart M. Shieber. On evaluating the generalization of LSTM models in formal languages. In Proceedings of the Society for Computation in Linguistics (SCiL), pp. 277-286, January 2019.
Andrew Trask, Felix Hill, Scott E Reed, Jack Rae, Chris Dyer, and Phil Blun- som. Neural arithmetic logic units. Advances Neural Information Process- ing Systems 31, pp. 8035 8044. 2018. URL http::/paper:: wtp.ses.ppppecccppppr.e..e unt.::iio.ooooo.on...........
Edwin B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Staristical Association, 22(158):209-22 1927. doi: 10. 10800012214599 1927. 10502953. URL https: http://wwwwwaaafnno/ll.ccc.. http://a....aaaoo/oo//////////// 1927.
Sang Michael Xie and Stefano Ermon. Differentiable subset sampling. CoRR, ahw/ 2019.
URL httt::://aaxw..og/////////// http///aa////..... /tt///991..11111 10517

---

Published conference paper at ICLR 2020
A GRADIENT DERIVATIVES
A. WEIGHT MATRIX CONSTRUCTION
For clarity the weight matrix construction defined using scalar notation
Wiy,he- (anh(WWr,hh-a )o (Mnn,ee-))
(18)
The of the loss with respect Wi and Magcher is then derived using backpropagation.
0C
OC
@Whe.he-1
@W @Wh
ki- ÖW ÖWhr.he-
0C
(1 tanh tanh?(Weedde Do 1
WWhcke-
(19)
0C
OC
OWh..r-
OC
- (ddhe
@Wh,.kr-
As seen from this result, one only needs consider We
for NAC+ and NAC., the gradient
with respect to Weeeee and Me is a multiplication on WW,
8C
A.2 GRADIENT OF NAC.
The NAC. is defined using scalar notation.
H Whithen log(1zne- +c)
Zhe exp
(20)
The gradient of the loss with respect to Whecee can the be derived using backpropagation.
H
Ozhe
Whenee log(lEN,,,1 log(l=m)_,
aW he-1
exp
(21)
= Zh, ( +e)
We wish to derive the backpropagation term on, becomes:
ac Sahe"
because 2h, affects { 1 this
Hees
dc
Hess
OC 0zhzt1
Homs
0zhett
8n,
Σ
Sherl
(22)
Thee
Dchess
Ozpe
0zee
hm+==1
h--1=1
03at1l Bane
To make it easier to derive
re-express the The Ehzts
H
2hrt1
exp
Σ
When log(Izno +0)
(23)

---

Published conference paper at ICLR 2020
The gradient of 85be41
is them:
Ushy
0=he+1
H1
Olog( E,,, (,,)+))
= exp
When ahe log((znel +e)
Wheieche
02he
Oane
abs' abs'(2he)
= exp
When log( te) Whe Whesscee |zhel +
(24)
abs'
(Zhe) |=
Zbems W hhe
abs' (zh,) is the gradient the absolute function. In the paper we denote this as sign(zN,) for brevity, However, depending on the exact definition used there may be a difference for 2h, O, as abs' (0) is undefined. In practicality this doesn't matter much though, although theoretically it does mean that the expectation this is theoretically undefined when M[
A.3 GRADIENT OF NMU
In scalar notation the NMU defined
He
Zhe
II h=-1=1
+1
(25)
The gradient of the loss with respect Wh fairly trivial. Note that every term but the one for hy-1, is just a constant with respect to Wh, i.be The product, except the term for hy-1 can be Using this fact, the gradient can be expressed as:
expressed as Wh Sha-1
OC
OC
Ozhs
0C
2hs
(2he-1 1) (26)
Owin.he-
Dzhr
Owhehhe-
8hhr Wh Zhz-1 + W
Why-
Similarly, the gradient
ac
which essential in backpropagation can equally easily derived as:
Bone
ac
H,
JL
Ozh.
Hr
2h.
W Whe-ide
(27)
02h
DaNe
Dehe
Whe Se Rhe
+1 Whe the

---

Published conference paper at ICLR 2020
MOMENTS
B.1 OVERVIEW
B.1.1 MOMENTS AND INITIALIZATION FOR ADDITION
The desired properties for initialization are according to Glorot et al. (Glorot & Bengio 2010):
UC
E Ele = 0
02h1-1
(28)
0C
0C
Vorlzhel Var [Zhe-a]
Var
Var
82h1-1
82ks
B.1.2 INITIALIZATION FOR ADDITION
Glorot initialization can not be used NAC+ as Wh is not sampled directly. Assuming that Wheche- ~ Uniform -r, and Maekees ~ Uniform[-r, r]. then the variance can be derived (see proof in Appendix B.2 to be:
Var War = (1 (I=aanb(r) (r
tanh(r)
tanh (
(29)
One can then solve r. given the desired variance (V (VarlWhe he-n.del He 1
Glorot & Bengio
2010
B.1.3 MOMENTS AND INITIALIZATION MULTIPLICATION
Using second order multivariate Taylor approximation and some assumptions of uncorrelated stochas- tie variables, the expectation and variance of the NAC, layer can be estimated
He-1
(21,02)) = l + 2
Var (orr/Wnnnn Wo. hog(// e)²
E|
R (1,
Var Zhe
2 (4, (1,2)
E
= 0
(30)
Ozh
0L
QL D=ne
Var
Var
He (4,1) Var (W.
Ozn.
Var[zarn)
EE Bhy 11 +
te)
This is problematic because El 2 1, and the variance explodes for Elsh, , = 0. E Elehe ==0 is normally a desired property GGorot & Bengio, 2010). The variance explodes for E [(2 = 0. and thus not be initialized anything meaninguul.

---

Published conference paper at ICLR 2020
For our proposed NMU, the expectation and variance can be derived (see proof Appendix [B. using the same assumptions as before, although no Taylor approximation required:
H
Elahe]
2
dc
E
=0
02h1-1
(Var mel wul+ 4) He-1
Hi-1
Var(zhe]
(Var(zhe-il +11 M1--
(31)
0L
0L izne
Var
= Var
He
(VarlWan
mhl 1) He-1 (Var(ase .I He-1-1
These expectations are better behaved. It is unlikely that the expectation of a multiplication unit can become zero, since the identity for multiplication is I. However, for large Hz-1 it will be
The variance is also better behaved, but do not provide input-independen initialization strategy. We propose initializing with V Var assuming Va Var[=s, = 1 and large H3-1 (see proof in Appendix B.4.3). However, more exact solutions are possible if the input variance is known.
1. as this the solution to Var[=,,] = Var V'rrfa,_11
B.2 EXPECTATION AND VARIANCE FOR WEIGHT MATRIX CONSTRUCTION IN NAC LAYERS
The weight matrix construction in NAC, is defined in scalar notation as:
W Wheh Lanh(WWe..r (MRn.ee-))
(32)
Simplifying the notation of this, and re-expressing it using stochastic variables with uniform distribu- tions this can written as:
W tanh(WWa((M)
W U[-r,r]
(33)
M ~ U[-r,r]
Since tanh(W") is an odd-function and E[W] = 0, deriving the expectation E[W] is trivial.
E[W]= E[tanh((V)) = 0 E(o((N)) =
(34)
The variance is more complicated, however as W and AM are independent, it can be simplified to:
Var[W] = F "aaan""""""nnaAa"""""aaaaaa""""" E[tanh(")) F[(eant(FF)))]] (35)
These second moments can be analyzed independently. First for E[tanh(11)))]
C[tanh(WW)
tanh(x)² fu(-..)()))) dr
tanh(x)² dx
2r
(36)
2. (r tanh(r))
2r
tanh(r)

---

Published conference paper at ICLR 2020
Then for Ettanh( (M) )::
oxx) ful-),)(()) dr
-00
2r I" o(x)) de
(37)
(r- tanh
(,)
Which results in the variance:
( (r tanh r'
Var[W]=
(38)
2r
2
B.3 EXPECTATION AND VARIANCE OF NAC.
B.3.1 FORWARD PASS
Expectation Assuming that each 2he-1 are uncorrelated, the expectation can be simplified to:
Hr-1
E| = E
exp
Wheche log([=ne .I [1+
br-1=1
Hi-1
= E
exp(Wne.hhe.. log( 2he +c]
he-
He II
(39)
@lexp(Whe.hee log( 2 +e)]]
1=-==
Mlexp(Wnene Whene log(l=ne [c]]]]
Mr-1
E [1sheaal + Whe
E [[ Whe He-1
Here we define g as non-linear transformation function of two independent stochastic variables:
f Whephe = (( +e) Wa
(40)
We then apply second order Taylor approximation of f. around (E(sh-,) ElWhe be.hh-
El El Whe Ne )1 E
f(E (3he-.)
Of(e Waeihe Wake-s 0f(ane Wheihe OWhe
the-1 E 18the-1
Why 13 W hr
The-1 El E( Wheche-s E he
Elzhe W Whikke- W.W,,,
Zhe-1
(41)
8 ((3hy Wapdhee) 87
8² flan We Bome OWnehe 1 paa Wapdens W-Wneeee
8 82 Warden)) Ushe oWheeie
Zhz-i 11 Whecee E El he .]
Zhe- E[
Wheche -E EE WWehhee

---

Published conference paper at ICLR 2020
Because E 2
El O, BeWhhe. E.Whe be.keill and Coulzhe-1 Wheceell = O.
This simplifies to:
Mig(2hr-1> Whn Whinhe-il 11 of 13 M1 B.a,,,iWWWaa Bl[Wee,henll
gggg,n Woy
1 Var
The Whech
(42)
+ 2
Zhe-i 11/zhe-11 Wheche- Whebe--
Inserting the derivatives and computing the inner products yields:
(ehe Wheche zo (1h .I) + W.Wheeeel
Var (Zne 1lE .1l e) ElWh 2 (ElWhee 1)
(43)
+ 2 VarlWWeee M(E .ll + e) e) log(1E(za,
Dog(1 +
= 1 2 Logg
c)²
This gives the final expectation:
ElZhe] = E [g(22e 13 Wheihe ) Me-1
(1 1 VarlWhn.ke log(1 Eigh +c)²
He
(44)
We evaluate the error of the approximation, where Wheche- Ul-r [w]] and 244-1 These distributions are what used the arithmetic dataset. The error is plotted in figure
r.)
50
error
30
0.00
-0.02
-0.04
-0.06
10
0.0
0.1
0.2
0.3
0.4
0.5
Fw
Figure 5: Error between theoretical approximation and the numerical approximation estimated by random sampling of 100000 observations each combination of r. and
The variance can be derived using the same assumptions as used in "expectation", that all Zhy_, are uncorrelated.
Variance
Varl
E1 E[zne]
H (1 + e) Wheieess 7
H,
(|=m)aa + e) 2-WWhinee1
F
E
(45)
he-1=1
he-1=1
E [
2 W Whecher))
He-s
[Z Wheker)) ) 2.Hr

---

Published conference paper at ICLR 2020
We already have from the expectation result (43) that:
[f Whe.he .)] Rz 1 2 Var[Whenee llgg .Il + e)²
(46)
By substitution of variable we have that:
E [f 2. We- 1 2 Wh M[t c)²
(47)
1 =1+2. Var [Whe.h Log log( (2 - c)²
This gives the variance:
Var[zn.] E [g(Ene- 1.2 Whene 1] M E [f(e Whenee 1) 2-H
= (1 + VarlWae.hee log (( + c)?) He
(48)
- (1 1 Varl ho + c)² 2.He-1
B.3.2 BACKWARD
0shest Ushz
Expectation The expectation of the back- pocck-poppggtion term assuming that ERess and mutually uncorrelated:
are
H
0zhrt1
-=hr-1
Elön, E
Sheel
Heg1 Eldennn
(49)
0=he
Ozhe
1+41=1
Assuming that Zhets Wheee die and The are uncorrelated:
@zheet
aasss (2h,))
Tabs' (2ne)
E
E Eizeem [e(W
M[ 1. 1...
(50)
Ozne
|zn, +
lanel +
Variance
Deriving the variance is more complicated:
abss' (2ne)
Var
Var
Zheis Wherishe de
(51)
02he
lahe +
Assuming again that She4s
W Whete and The are uncorrelated, and likewise for their second moment:
aasss aabs((2he)
V ar
== E El
JE(WR
L.A,EE E
8hhc
1 +
Tabs' (shee
El | 1E 12
12hee +
abs's (2the
27
El WariWhentthl
(52)
|z +
abss' (Zne)
12.0.E
11h +
abs' (2nr)) 1zke +
El | Car(Wheesseee |
Using Taylor approximation around F[ we have:
abs' (ahee
)
27
~ (1E(=M,11+ e)² + 1
E
(1E) e)a Varlznd
|2] +
(53)
[1E[zn.]ll +) + (1E7=n))) + e)* Varland

---

Published conference paper at ICLR 2020
Finally, by reusing result E[zh,] from earlier the variance can expressed as:
Var
OC
Var
JC
He (1 2 Var[Whr. hr. log( E - ill + c)a) He-1
0=hr-1
(54)
1
3
Var[Whe.te .1
+
V Var(zhe
(1E(she.)]l+ c)
(1E(ehe + c)
B.4 EXPECTATION AND VARIANCE OF NMU
B.4.1 FORWARD PASS
Expectation Assuming that Ehy-1 are independent:
11 2h, = E
II
(2 + Whe
he-1=1
(55)
E Phe-s + Wh Whe-ichel He-1
= (E[W EE( +1 - ElW He-1
Assuming that (2 .1 = which a desired property and initializing E ElWhe and 1/2, the expectation is:
E [he
(E(Whe E. .J+1 E(Whe ..bee) He
He
R3
1+
2
(56)
H4-1
Variance Reusing the result for the expectation, assuming again that all 2h,-, uncorrelated, using the fact that W
W .b is initially independent from Zhz-1²
V Varlzn.l = E[=he]
E [
2-H--1
2e 1i
2
11)
2-H1-1
E
WW Phe-1 Who-n,te)
h1-:=1
(57)
Whe Line)) 39tte 3-H--1
E((Whe
be i The
- 2 he-1. li El + E(W
=
He
1
2-Hr-1
+ 2 Whe Elan,
2 whdel +
2
Assuming that Ejah,-1 = which is a desired property and initializing E W.Whe-she Lieel = 1/2, the variance becomes:
Var[an,] Re (E(Wh (El He-1 - - 2-Hr-1
~ ([Var[W E(Wne-s (Var(znonn + 1)) He
(1
2
(58)
2
= -WaarWWniihed + 1 Hh-1 2
2-He-
(Var[shrrs) He-s 1)
1
-
2

---

Published conference paper at ICLR 2020
B.4.2 BACKWARD PASS
Expectation For the backward pass the expectation can, assuming that related, be derived to:
SC Bang
and
B3ng
0sh.
uncor-
OC
ac
8che
F
Jah,
dzhe
0C
02ps
H2F
F0
(59)
ac
H
Whe-shee
0zh,
Whe-1 - +
W Who- the
HeE
OC
2he
E Whe-sseel
0zh,
Ry + W -he
Initializing
E(W he- ncel
1/2,
and inserting the result for the expectation
E
Shy
Wa 1h 22 2
Why 1ike
UC
Jc H
E
H&E
dche-11
azhe
2
2
(60)
OC
He-1
E
H₁
Dzne
2
Assuming that E
8C 8eh,
which a desired property Glorot Bengio 2010
ac
1
He-1
E
2 0-
H
θzh1-1
2
(61)
0
Variance
For the variance of the backpropagation term, assume that ac is uncorrelated with Jznc
8thr
Jzne-1
JC
OL. θzhr
Var
H,Var
8zhe-1
azns 02h1-1
ØC
2
0L
Oehe
He
Var
1zh.
Dzhe-
+ E
izna
Var
azhe-1
(62)
aL
azhi
+ V Var
Var
azh.
Ozhe
Assuming again that
ac Jehs
and reusing the result
(2) -1-1
8zn
QL
Oc
2-Hi-1
Ozhs
Var
Var
He
Var
(63)
J2he-1
Ozne
Ozhe-s
8znr Joue
Focusing now on Var
we have:
Var
Jene
Zhe
E(W2 M
8=n--1
W Why do Fhi-1 + W Whe-
(64)
Zhe
E
W Why-iih di The-s
+ W Whenio

---

Published conference paper at ICLR 2020
Inserting the result for the expectation E E/Wh, mhdel = 1/2.
and Initializing again
Wh Ahg-1 +1- W 11
Var
8=he -1
2 E
The W 1h,
W Who- ) +1
E mid
2:
(65)
E
2he
E E.Wh
he mhve
Wh +
Whe do
2-H2-1
1
2
Using the identity that E( ElWhe he .hhel 1/2.
he nnd
Var Wor[Whe-nkkel + ElW hiWhe-ndee' and again using
8zns
J] (Var[Whr invel 2)
2hr
Var
E
02h1-11
2he +
Wh Whe-idee
(66)
2.He
2
To derive E War Fhe-s +1 Whe the result for Var[zne] can be used, but for Hz_1
27
Hi-1 1, because there is less term. Inserting E (Wa Sn the
(Var(Wne ande + +)) 7 me-1- (Varlzne .I + 1) Me 1--1-1 we have:
dan,
VarlWa Lidel + 1 1) Hi-1-1 (Varles ,]+ 1
Var
2.H
WerlWWe-r.aell 1
(67)
②
Hh-)
1l (1 2
(Var WWrnWWe.ddell
2
(Var(zhenll
2
Inserting the result for Var
dane asn,
8C into the result for Var
0
0c Jzne
1 He
2.Hr
Var
= Var
Ozh.
He
( 2.H
1
(Varlan, 1
I
(68)
iC Dine
Var
Hr.
(( (VarlWinnnddd + 2)
He-
(Varlzn + H
20

---

Published conference paper at ICLR 2020
B.4.3 INITIALIZATION
The Whr-1.h should be initialized with 2 2. order not bias towards inclusion exclusion of Zhz-1. Using derived variance approximations (68). the variance should be according to the forward pass:
Var War[Whe-1.hh) = ((1 Var(z,,)) - He-1 [zr[znal + (Vrr(zn,)) H1-1) कi-
(69)
And according to the backward pass it should be:
(Varr(nee) + 1)2
8e-1
(70)
H2
Both criteria dependent the input variance. If the input variance is know then optimal initializa- tion is possible. However, as this is often not the case one can perhaps assume that V arr (ar(3 = 1. This is not an unreasonable assumption in many cases, as there may either be normalization layer somewhere the input normalized. If unit variance assumed, the variance for the forward pass becomes:
[Wee L.he] = (2- 8 1) m
((4tt + : -2)
(71)
4
And from the backward pass:
(2 (21-Be-1 Te-1
Var Whe
(72)
H₂
The variance requirement for both the forward and backward pass can be satisfied with Var War be mihel = for large Hy-1-
21

---

Published conference paper at ICLR 2020
C ARITHMETIC TASK
The aim of the "Arithmetic task" is directly arithmetic models ability to extrapolate beyond the training range. Additionally, our generalized version provides a high degree of flexibility in how the input is shaped, sampled, and the problem complexity.
Our "arithmetic task" identical to the "simple function task" in the NALU paper (Trask al. 2018 However, as they do not describe their setup details, we use the setup from Madsen Johansen (2019). which provide Algorithm 3] an evaluation-critemn to if and when the model has converged, the sparsity error, well as methods for computing confidence intervals for success-rate and the sparsity errr.
stibet
Overlap
subset
Figure 6: Shows how the dataset is parameterized.
C.1 DATASET GENERATION
The goal is two random subsets of vector x (a and b), and perform arithmetic operation on these (a o b).
Land
82 and
Fii 1 a
(73)
Algorithm|| I defines the procedure generate the data, where interpolation range will be used for training and validation and an extrapolation range will be used for testing. Default values are defined in table
Table Default dataset parameters for "Arithmetic task"
Parameter name
Default value
Parameter name
Default value
Input size
100
Interpolation range
U1, 2 U 6
Subset ratio
0.25
Extrapolation range
Overlap ratio
0.5
Algorithm Dataset generation algorithm for "Arithmetic task"
I: function DATASET(OP(-) :): Operation, i InputSSize, s SubsetRatio, o OverlapRatio,
R: Range)
2:
UNIFORM(J RRtwwer Rupper;
Sample i elements uniformly Sample offset
3:
k + UNIFORM(O, 1 28 0)
4:
a < SUM(X il: i(k + s))
> Create sum a from subset Create sum b from subset Perform operation on a and b
5:
b b+- SUM(x(i(k
0) i(k 2s -0))
6:
t OP(a,
7:
return æ,
C.2 MODEL DEFINTIONS AND SETUP
Models are defined in table 4 and are all optimized with Adam optimization (Kingma & Ba 2014) using default parameters, and trained over 5 10th iterations. Training takes about 8 hours on a single CPU core(8-Core Intel Xeon ES-26665 2.4GHz). We run 19150 experiments HPC cluster.
22

---

Published conference paper at ICLR 2020
The training dataset continuously sampled from the interpolation range where a different seed used for each experiment, all experiments use mini-batch size of 128 observations, a fixed validation dataset with 1 10s observations sampled from the interpolation range, and a fixed test dataset with 1 10 observations sampled from the extrapolation range.
Table 4: Model definitions
Model
Layer
Layer 2
Asparso
Astart
Aend
NMU
NAU
NMU
10
106
2. 10
NAU
NAU
NAU
0.01
5 10
5 10
NAC
NAC
NAC.
NAC
NAC
NAC.
NAC MM
NAC
NAC.
10
106
2 106
NAC+
NAC+
NAC+
NALU
NALU
NALU
Linear
Linear
Linear
ReLU
ReLU
ReLU
ReLU6
ReLU6
ReLU6
C.3
ABLATION STUDY
To validate our model, we perform an ablation on the multiplication problem. Some noteworthy observations:
None of the W constraints, such as Reparee and clamping to in [0, 1]. are necessary when the hidden size is just 2.
Removing Rsparse causes the NMU to immediately fail for larger hidden sizes.
3. Removing the clamping of W does
much difference. This is because Reparse also constrains W outside of [0, 1]. The regularizer used here is Ryparse min( which is identical to the one used in other experiments in [0, 1]. but also valid outside [o. 1]. Doing this gives only slightly slower convergence. Although, this can not be guaranteed in general, as the regularizer is omitted during the initial optimization.
WD).
4. Removing both constraints, gives somewhat satisfying solution, but with lower success- rate, slower convergence, and higher sparsity error.
In conclusion both constraints are valuable, as they provide faster convergence and sparser solution, but they critical the success-rate of the NMU.
Success rate
Solved at lteration step
Sparsity error
1.00
4e106
0.20
3e406
0.15
0.50
2e406
0.10
0.00
0e:00-
0.00
10
6
10
2
8
10
Hidden size
model
NMU, Repce
Regess W-clemp
W-clamp
Figure 7: Ablation study where Raprrse is removed and the clamping of W is removed. There are 50 experiments with different seeds, for each configuration.
23

---

Published conference paper at ICLR 2020
C.4 EFFECT DATASET PARAMETER
To stress test the models on the multiplication task, we vary the dataset parameters one at a time while keeping the others at their default value (default values in table 3). Each runs for 50 experiments with different seeds. The results, visualized in figure
changed, therefore the extrapolation-rang needs to be changed such it doesn't overlap. For each interpolation-rangee the following extrapolation- range is used: U -1] uses U -2]. U| 2, uses U[-6, -2] U22 6]. U|0, I] uses U[1,5]. U 0.2] uses 2, 2]. U11.1, 1.2] uses U|1.2, 6]. U[1. 2] uses U 6). U[10, 20] uses U[20, 40).
figure 3
interpolation-rang
Success rate
Solved at lteration step
5 Sparsity entor
1.00
1.2e107
8.0 8.0
0.50
0.1
4.0e+06
0.00
0.0e+00
0.0
100
200
300
100
200
300
100
200
300
S Size of the input vector
1.00
4e 406
0.75
0.015
2e+066
0.25
1e406
0.005
0.000
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1000
0.00
0.25
0.30
0.75
1.b0
The ratio of which subsets overlap with each other
1.00
4e+06
0.0020
0.75
3e406
0.0015
0.50
2e406
0.0010
0.25
1e+06
0.0005
0.00
De
0.0000
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
Relative size subsets compared to inputiize
model
NAC.ayu
NAC.o
NAC.
Gated NAUNNMU
NALU
NMU
Figure 8: Shows the effect of the dataset parameters.
C.5 GATING CONVERGENCE EXPERIMENT
In the interest of adding some understand of what goes wrong in the NALU gate, and the shared weight choice that NALU employs remedy this, we introduce the following experiment.
We train models second layer. The only difference is that one model shares the weight between NAC, and NAC, the NALU, and the other treat them as two separate units with separate weights. In both cases NALU should gate between N NAC, and NAC. and choose the appropriate operation. Note that this NALU
the arithmetic task. Both uses the NAC, in the first layer and NALU in the

---

Published conference paper at ICLR 2020
model different from one presented elsewhere in this paper, including the original NALU paper (Trask et al. 2018). The typical NALU model is just two NALU layers with shared weights.
Furthermore, we also introduce new gated unit that simply gates between our proposed NMU and NAU, using the same sigmoid seperate weights, as NMU and NAU use different weight constrains and can therefore not shared.
in the NALU. This combination is done with
The models are trained and evaluated over 100 different seeds on the multiplication and addition task. A histogram of the value for all seeds is presented in figure 9 and table contains a summary. Some noteworthy observations:
1. When the NALU weights separated far more trials converge select NAC+ for both the addition and multiplication task. Sharing the weights between NAC+ and NAC. makes the gating less likely to converge for addition.
The performance of the addition task dependent NALU selecting the right operation. In the multiplication task, when the right gate selected, NAC. do not converge consistently, unlike our NMU that converges more consistently.
Which operation the gate converges to appears to mostly random and independent of the task. These issues are caused by the sigmoid gating gating-mechanism mechanism and thus exists independent of the used sub-units.
These observations validates that the NALU gating-mechanism mechanism does not converge as intended. This becomes critical issues when more gates are present, as is normally the case. E.g. when stacking multiple NALU layers together.
Gated NAU/NMU
NALU (soparatt)
NALU (shared)
60
Multpiicalon
20
solved
count
FALSE
60
TRUE
20
0-
0.25
0.50
0.15
add
0.25
0.50
0.75
mul
0.25
0.50
0.75
add
Gate
Figure 9: Shows the gating-value in the NALU layer and variant that uses NAU/NMU instead of NAC,/NAC. Separate/shared refers the weights in NAC,/NAC, used in NALU.
Table 5: Comparison of the success-rate, when the model converged, and the sparsity error, with 95% confidence interval on the "arithmetic datasets" task. Each value summary of 100 different seeds.
Op
Model
Success
Solved at iteration step
Sparsity error
Rate
Median
Mean
Mean
Gated NAU/NMU
62%
+9%
1.5 108
1.5 8
+3.9-10"
10 +2.3-10 5
10%
-3.8-104
-1.8-10-b
NALU (separate)
22% +9%
2.8 106
3.3 106
+3.9-10"
5.8 10-2 +4.1-10
-..6-10a
-2.3-10-2
NALU (shared)
24%
+9%
2.9 106
3.3 80
+3.7-10"
1.0 10 3 +1.1-10
7%
-3.6-105
-4.5-10-4
Gated NAU/NMU
37%
+ 10%
1.9 104
10
+--3-100
44.6-10-2
9%%
-6.7-104
-4.0-10-3
NALU (separate)
51%
10%
1.4 10
1.9
+3.5-104
1.8.10-1
+1.4-00
10%
-4.3-104
-1.4-10-2
NALU (shared)
34% + 10%
1.8 10
105
14.3-10
1.8 10-1 +2.3-100
9%%
-5.4-10"
-2,1-10-2
25

---

Published conference paper at ICLR 2020
C.6 REGULARIZATION
The Astart and Aen are simply selected based on how much time it takes for the model to converge. The sparsity regularizer should not be used during early optimization as this part of the optimization exploratory and concerns finding the right solution by getting each weight the right side of
=0.5.
In figure 10 and 12 the scaling factor Asparso is optimized.
Astart 1);
Nsparse
Asparso max(min) (min(
(74)
Aend
Astart
Salved iteration step
Sparsity
1.00
1200000
0.25
0.20
800000
0.10
400000
0.25
0.05
0.00
0-
0.00
0.01
0.1
1
10
100
0.01 0.1
100
0
0.01
0.1
10
100
parse regualizer
Figure 10: Shows effect of Reparso in NAU on the arithmetic dataset for the + operation.
S Success rate
Solved at teration step
Sparsity error
1.00
0.75
4000
0.20
3000
0.15
0.50
2000
0.10
0.25
1000
0.05
0.00
0.01
0.1
10
100
0.01
0.1
10
100
0
0.01
0.1
10
100
Sparse regualizer
Figure 11: Shows effect of Aeparao in NAU on the arithmetic dataset for the operation.
Success rate
Solved teration step
Sparsity error
1.00
2000000
0.75
1500000
4e-04-
1000000
2e-04-
500000
0.00
0
0e400
0.01
0.1
10
160
0.01
0.1
10
100
0.b1
0.1
10
160
Sparse regualizer
Figure 12: Shows effect of Syparse in NMU on the arithmetic dataset for the x operation.
C.7 COMPARING ALL MODELS
Table compares all models on all operations used NALU (Trask et al. 2018). All variations of models and operations are trained for 100 different seeds to build confidence intervals. Some noteworthy observations are:
26

---

Published conference paper at ICLR 2020
Division does not work for any model, including the NAC. and NALU models. This may seem surprising but is actually in line with the results from the NALU paper (Trask et al. (2018), table I) where there is large error given the interpolation range. The extrapolation range has smaller eπrоr, but this an artifact of their evaluation method where they normalize with a random baseline. Since a random baseline will have a higher error for the extrapolation range, errors just appear to be smaller. A correct solution to division should have both small interpolation and extrapolation error.
2. NAC. and NALU are barely able to learn vz. with just 2% success-rate for NALU and 7% success-rate for NAC..
3. NMU is fully capable of learning =². It learn this by learning the same subset twice in the NAU layer, this is also how NAC. learn 2².
The Gated NAU/NMU (discussed section C.3 works very poorly, because the NMU initialization assumes that , = O. This is usually true, as discussed in section 2.6 but not in this case for the first layer. the recommended NMU model, the NMU layer appears after NAU, which causes that assumption to be satisfied.
Table 6: Comparison of the success-rate, when the model converged, and the sparsity error with 95% confidence interval on the "arithmetic datasets" task. Each value summary of 100 different seeds.
Op
Model
Success
Solved at iteration step
Sparsity error
Rate
Median
Mcan
Mean
93%
14%
1.8 108
2.0 10 +
9.5 10 -4.2-10-7
7%
-9.7-101
NAC
100%
+0%
2.5 106
2.6 106
48.8-10*
4.6 4.6.10-6 +5.0-10
-9%
-7:2-104
-3.6-10
NAC.
31%
+10%
2.8 106
3.0 106 42.9-10
5.8 10 44.8-10-
8%
-2.4-105
-2.6-10-
NAC+
0%
+4%
-0%
Gated
NAU
0% 14%
SMU
-0%
Linear
0% +-1%
X
NALU
0% +4%
NAU
0% +4%
0%
NMU
98% +1%
1.4 106
1.5 108
+5.0-10
4.2 4.2 10-7 +2.9-10 +2.9-111-
5%
-6.6 8.6
-2.9-10-8
ReLU
0% +1%
ReLU6
0% 14%
-0%
NAC. NAC.NMC NMU
0% +4%
-0%
NAC
0% +1%
00%
NAC.
0%
NAC
0%
+4%
-0%
Gated
NAU
0%
+4%
NMU
-0%
Linear
0%
14%
-0%
NALU
0% +4%
-0%
NAU
0% +4%
NMU
0% +4%
-0%
ReLU
0% 0%
14%
ReLU6
0% -0%
+1%
27

---

Published conference paper at ICLR 2020
Table 6: Comparison the success-rate, when the model converged, and the sparsity error, with 95% confidence interval the "arithmetic datasets" task. Each value is summary of 100 different seeds. (continued)
Op
Model
Success
Solved
Sparsity
Rate
Median
Mean
Mean
NAC. NMU
0% 14%
NAC.
0%
+1%
-0%
NAC.
0%
+4%
00%
NAC+
100% +0%
2.5 10
4.9 10a +5.2-10*
2.3 10-1 +6.5-10"1
-4.5-104
-6.5-10-3
Gated NAU
0%
+4%
NMU
-0%
Linear
100%
+0%
6.1 104
6.3 10 +2.5-10
2.5-10-1
+3.6.1
19%
-3.3-10a
-3.6-10-4
NALU
14%
1.5 1.5-10
1.6 10th
+3.8-10"
1.7-10-1
-3.3-10"
-2,5-10
NAU
100%
+0%
1.8 10
3.9 10 +4.5-10'
3.2 3.2 10
+13--0-5
4%
-3.7-104
-1.3-10
NMU
0% 447%
ReLU
62%
+9%
-
7.6 104 48.3-10
2.5 10 -2.4-10-3
10%
-7.0-10-
ReLU6
0% -0%
+47%
NAC. NMU NMU
0% +1%
00%
NAC..
0% +47
-0%
NAC.
0% +4%
-0%
NAC
100%
9.0 10
3.7 10S +3.8-10
2.3 10-1 45.4-10-3
-3.8-104
-3.4-10-3
Gated NMU
NAU
0%
+49
-0%
Linear
7%
+7%
3.3-109
1.4 10th +7.0-00
18810-11 +722-102
-4%
-6.1-103
-3.8-10-2
NALU
14%
1.9 1.9-10
1.9 1.9 10 +4.4-10"
2.1 10 -22210-
+22-10-2
6%
-4.5-10*
NAU
100%
+0%
5.0-
1.6 105 +1.7-10'
6.6 10- +225-10-2 -
44%
-1.6-104
-1.9-1022
NMU
56%
+9%
1.0 10
1.0 10 +5.8-10"
3.4 43322.0
+1% 10%
-08
-2.6-10-
ReLU
0%
00%
ReLU6
0%
+-1%
-0%
NAC.
3%
+5%
1.0- 106
1.0 106
1NaN-10-1nf
1.7 10
48.3-10-3
-2%
-NNN-10-fM
-5.1-10-a
NAC
0%
+-%
-0%
NAC.
7%
+7%
4.0 10
1.5 10
46.0-10
2.4-10 +1.7-10-2
-4%
-5.6-107
-17-10-3
NAC
0%
+-1%
-0%
Gated
NAU
0% 14%
SMU
-0%
Linear
0% +1%
0%
2
NALU
2% +5%
2.6- 105
3.3 1066 +18-10
5.0 10-1 +25-10-6
-1%
-2.2
-8.0-10
NAU
0% 44%
-0%
NMU
0% +4%
-0%
ReLU
0% +11
ReLU6
0% +1%
-0%
28

---

Published conference paper at ICLR 2020
Table 6: Comparison the success-rate, when the model converged, and the sparsity error, with 95% confidence interval the "arithmetic datasets" task. Each value is summary of 100 different seeds. (continued)
Op
Model
Success
Solved
Sparsity
Rate
Median
Mean
Mean
NAC. NMU NMU
100%
+0%
1.4 10
1.5 10 48.4-10*
2.9 10 +1.4-10-6
4%
-7.9-104
-1.4-10-
NAC.
100%
+0%
1.9 106
1.9 10 +5.3-10*
1.8 10 +4.3-10
-6.2-104
-4.3-10-4
NAC.
77%
+7%
3.3 106
3.2 10t +1.6-10"
1.8 10 +5.8-10
99
-2.0-10P
-3.7-10-4
NAC+
0%
+-4%
-0%
Gated
SAU
0% +-1%
NMU
-0%
Linear
0% +.4%
NALU
0% +1%
NAU
0%
+426
-0%
NMU
100%
+0%
1.2 106
1.3 106 43.1-104
3.7 45.4-10-
1%%
-3.6-10
-3.7-10-5
ReLU
0% +-4%
-0%
ReLU6
0% +1%
0%
29

---

Published conference paper at ICLR 2020
D SEQUENTIAL MNIST
D.I TASK AND EVALUATION CRITERIA
The simple function task is a purely synthetic task, that does not require a deep network. As such it does not test if an arithmetic layer inhibits the networks ability to be optimized using gradient decent.
The sequential MNIST task takes the numerical value of sequence of MNIST digits and applies a binary operation recursively. Such that 1, Op(4--1-25 2,), where 2, value. This is identical to the "MNIST Counting and Arithmetic Tasks" in Trask el al. (2018) section 4.2). We present the addition variant validate the NAU's ability to backpropagate, and we add additional multiplication variant to validate the NMU's ability backpropagate.
MNIST digit's 'ss numerical
The performance this task depends on the quality the image-to-scalat network and the arithmetic layer's ability to model the scalar. We use mean mean-square- square-error (MSE) to evaluate joint image-to umaaa-to-scalar and arithmetic layer model performance. To determine MSE threshold from the correct prediction we use an empirical baseline. This is done by letting the arithmetic layer be solved, such that only the image-to ceee-tt-sseeee to-scalar is learned. By learning this over multiple seeds an upper bound for an MSE threshold can be In our experiment a student-t distribution.
the 1% one-sided upper -nntrrvll assuming
Similar to the simple function task we use a success-criteria as reporting the MSE is not interpretable and models that do not converge will obscure the mean. Furthermore, because the operation is applied recursively, natural error from the dataset will accumulate over time, thus exponentially increasing the MSE. Using a baseline model and reporting the successfulness solves this interpretation challenge.
D.2 ADDITION OF SEQUENTIAL MNIST
Figure 1 shows results for sequential addition of MNIST digits. This experiment is identical the MNIST Digit Addition Test from Trask et al. (0118) section 4.2). The models are trained on a sequence of 10 digits and evaluated on sequences between and 1000 MNIST digits.
Note that the NAU model includes the R2 regularizer, similarly the "Multiplication of sequential MNIST" experiment section 4.2 However, because the weights are in [-1, 1), and not [0, 1]. and the idendity of addition is 0, and not 1, Rs is
He-1 (1- (Wh...
R₂ = H He
Σ he-1
(75)
byg
To provide fair comparison. variant of NAC4 that also this regularizer is included. this variant called NAC+ R. R.. Section D.3 provides an ablation study of the R, regularizer.
Success rate
Solved at iteration step
Sparsity error
1.00
0.12
75000
0.75
0.09
50000
0.50
25000
0.03
0.00
0
0.00
1
10
200
400
600
800
1000
10
200
400
600
sño 1000
10
200
460
600
800
1000
Extrapolation length
model
NAC.,.
NAC,
LSTM
NALU
NAU
Figure 13: Shows the ability of each model to learn the arithmetic operation of addition and back-
propagate through the arithmetic layer in order to learn an image-to-scalar value for MNIST digits. The model is tested by extrapolating to larger sequence lengths than what it has been trained on. The NAU and NAC+, R, models the R, regularizer from section 4.2
30

---

Published conference paper at ICLR 2020
D.3 SEQUENTIAL ADDTION WITHOUT R3 REGULARIZER
As an ablation study of the Rg regularizer, figure 14 shows the NAU model without the R. regularizer. Removing the regularizer causes reduction in the success-rate. The reduction is likely larger. compared sequential multiplication, because the sequence length used for training is longer. The loss function is most sensitive to the 10th output in the sequence, as this has the largest scale. This causes some of the model instances to just learn the mean, which becomes passable for very long sequences, which is why the success-rate increases for longer sequences. However, this not valid solution. A well- behavior model should be successful independent of the sequence length.
Success
Solved iteration stap
Sparsiy
0.12
75000
0.75
0.09
50000
0.06
0.25
25000
0.03
200
600
800
1000
10
200
600
800
1000
200
400
800
1000
Extrapolation length
model
NAC.2.
NAC,
LSTM
NALU
NAU
Figure 14: Same as figure 13 but where the NAU model do not use the R. regularizer.
D.4 SEQUENTIAL MULTIPLICATION WITHOUT
REGULARIZER
As an ablation study the R₂ regularizer figure T3 shows the NMU and NAC, NNMU models without the R, regularizer. The success-rate is somewhat similar to figure However, as seen in the "sparsity error" plot, the solution quite different.
5 Success rate
S Solved at iteration steo
parsity error
1.00
3e 105
0.3
2e+05
0.50
0.2
1e+05
0.1
0.00
0e+00
0.0
10
12
14
16
18
20
12
10
12
20
Extrapolation length
model
LSTM
NALU
NMU
Figure 15: Shows the ability of each model learn the arithmetic operation of addition and back- propagate through the arithmetic layer in order The model is tested by extrapolating to larger sequence lengths than what it has heen trained on. The NMU and NAC. NMU models do not the R, regularizer.
an image-to-scalar value for MNIST digits.
NMUU


Reviews:
[{"review_id": "S1gJpwkpqB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "DISCLAIMER: I reviewed a previous version of this paper at another venue.\n\nThis paper introduces Neural Addition Units (NAU) and Neural Multiplication Units (NMU), essentially redeveloped models of the Neural Arithmetic Logic Unit (NALU). The paper presents a strong case that the new models outperform NALUs in a few metrics: rate of convergence, learning speed, parameter number and model sparsity. The performance of NAU is better than NAC/NALU, as is the performance of NMU with a caveat that the presented NMU here cannot deal with division, though it can deal with negative numbers (as opposed to NALU).\n\nWhat this paper excels at is a thorough theoretical and practical analysis of NALU’s issues and how the authors design the two new models to overcome these issues. The presented issues of NALU are numerous, including unstable optimization space, expectations of gradients converging to zero, the inability of NALUs gating to work as well as intended and its issues with division, and finally, the intended values of -1, 0, 1 in NALU do not get as close to these values as intended. \n\nThe paper is easy to read, modulo a number of typos and admittedly some weirdly written sentences (see typos and minor issues later) and I would definitely recommend another iteration over the text to improve the issues with it as well as the style of writing. I am quite fond of the analysis and the informed design of the two new models, as well as the simplicity of the final models which are fairly close to the original models but have been shown both theoretically and practically that they work.\nIt is great to see that the paper improved since my last review and stands stronger on its results, but there are still a few issues with it that make me hesitant to fully accept the paper:\n- The conclusion of the paper is biased towards the introduced models, but it should clearly define the limitations of these models wrt NALU/NAC\n- The performance of NALu on multiplication is in stark contrast to the results in the original paper (Table 1). This should be commented in the paper why that is, as the original model presents no issues of NALU with multiplication, whereas this paper essentially says that they haven’t gotten a single model (out of 100 of them) to do multiplication.\n- Could you explicitly comment on the paper why is the parameter sparsity such a sought-after quality of these models?\n- You ‘assume an approximate discrete solution with parameters close to {1-, 0, 1} is important’. What do you have to back this assumption? Would it be possible to learn the arithmetic operations (and generalize) even with parameters different than those?\n- Why did you introduce the product of the sequential MNIST experiment but did not presents results on the original sum / counting of digits? The change makes it hard to compare with the results in the original paper, and you do not present the reason why. This also makes me ask why didn't you compare to NALU on more tasks presented in the paper?\n\nTo conclude, this paper presents a well-done experimental and theoretical analysis of the issues of NALU and ways to fix it. Though the models presented outperform NALU, they still come with their own issues, namely they do not support division, and (admittedly, well corroborated with analysis) are not joined in a single, NALU-like model, that can learn multiple arithmetic operations. The paper does a great analysis of the models’ issues, with an experimental setup that highlights these issues, however, it does that on only one task from the original paper, and a(n insufficiently justified) modification of another one (multiplication of MNIST digits)---it does not extensively test these models on the same experimental setup as the original paper does.\n\nTypos and smaller issues:\n- Throughout the text you say that NMU supports large hidden input sizes? Why hidden??\n- Figure 4 is identical to figure in D.2\n- Repetition that E[z] = 0 is a desired property in 2.2, 2.3, 2.4\n- In Related work, binary representation -> one-hot representation\n- Found empirically in () - remove parentheses and see\n- increasing the hidden size -> hidden vector size?\n- NAU and NMU converges/learns/doesobtains -> converge/learn/do/obtain\n- hard learn -> hard to learn ?\n- NAU and NMU ...and improves -> improve\n- Table 1 show -> shows\n- Caption Table 1: Shows the - quite unusual caption (treating Table 1 as part of the sentence), would suggest to rephrase (e.g. Comparison/results of … on the … task). Similarly for Table 2...and Figure 3\n- experiemnts -> experiments\n- To analyse the impact of each improvements….. - this sentence is missing a chunk of it, or To should be replaced by We\n- Allows NAC_+ to be -> remove be\n- can be express as -> expressed\n- The Neural Arithmetic Expression Calculator () propose learning - one might read this as the model proposes, not the authors / paper / citation propose…(also combine or combines in the next line)\n- That the NMU models works -> model works? models work?\n- We choice the -> we choose the \n- hindre -> hinder\n- C.5 seperate -> separate\n- There’s a number of typos in the appendix\n- convergence the first -> convergence of the first?\n- Where the purpose is to fit an unknown function -> I think a more appropriate statement would hint at an often overparameterization in practice done when fitting a(n unknown) function\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxOeWlCYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose the Neural Multiplication Unit (NMU), which can learn to solve a family of arithmetic operations using -, + and * atomic operations over real numbers from examples. They show that a combination of careful initialization, regularization and structural choices allows their model to learn more reliably and efficiently than the previously published Neural Arithmetic Logic Unit.\n\nThe NALU consists of two additive sub-units in the real and log-space respectively, which allows it to handle both additions/subtractions and multiplications/divisions, and combines them with a gating mechanism. The NMU on the other hand simply learns a product of affine transformations of the input. This choice prevents the model from learning divisions, which the authors argue made learning unstable for the NALU case, but allows for an a priori better initialization and dispenses with the gating which is empirically hard to learn. The departures from the NALU architecture are well justified and lead to significant improvements for the considered applications, especially as far as extrapolation to inputs outside of the training domain.\n\nThe paper is mostly well written (one notable exception: the form of the loss function is not given explicitly anywhere in the paper) and well executed, but the scope of the work is somewhat limited, and the authors fail to properly motivate the application or put it in a wider context.\n\nFirst, divisions being difficult to handle does not constitute a sufficient justification for choosing to exclude them: the authors should at the very least propose a plausible way forward for future work. More generally, the proposed unit needs to be exposed to at least 10K examples to learn a single expression with fewer than 10 inputs (and the success rate already drops to under 65% for 10 inputs). What would be the use case for such a unit? Even the NMU is only proposed as a step on the way to a more modular, general-purpose, or efficient architecture, its value is difficult to gauge without some idea of what that would look like.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgVknOCtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors extend the work of Trask et al 2018 by developing alternatives to the Neural Accumulator (NAC) and Neural Arithmetic Logic Unit (NALU) which they dub the Neural Addition Unit (NAU) and Neural Multiplication Unit (NMU), which are neural modules capable of performing addition/subtraction and multiplication, respectively. The authors show that their proposed modules are capable of performing arithmetic tasks with higher accuracy, faster convergence, and more theoretically well-grounded foundations.\n\nThe new modules modules are relatively novel, and significantly outperform their closest architectural relatives, both in accuracy and convergence time. The authors also go to significant lengths to demonstrate that the parameters in these modules can be initialized and learned in a more theoretically well-grounded manner than their NAC/NALU counterparts. For these reasons I believe this paper should be accepted.\n\nGeneral advice/feedback:\n- should provide an explanation of the row in Table 2 showing that a simple linear transformation is able to achieve accuracy and convergence times comparable to those of the NAU\n- should provide an explanation of the universal 0% success rate on the U[1.1,1.2] sampling interval in Figure 3\n- inconsistent captioning in Figure 2c, missing \"NAC• with\"\n- should clarify in Section 4.1 that the \"arithmetic dataset\" task involves summing only *contiguous* vector entries; this is implied by the summation notation, and made explicit in Appendix Section C, but not specified in Section 4.1\n- it is unclear what experiments you performed to obtain Figure 3, and the additional explanation in Appendix Section C.4 regarding interpolation/extrapolation intervals only adds to the confusion; please clarify the explanation of Figure 3, or else move it to the Appendix\n- the ordering of some of the sections/figures is confusing and nonstandard: Section 1.1 presents results before explaining what exactly is being measured, Figure 1 shows an illustration of an NMU 2 pages before it is defined, Section 3 could be merged with the Introduction\n\nGrammatical/Typesetting errors:\n- \"an theoretical\" : bottom of pg 2\n- \"also found empirically in (see Trask et al. (2018)\" : top of pg 4\n- \"seamlessly randomly\" : middle of pg 5\n- \"We choice\" : middle of pg 6\n- inconsistent typesetting of \"NAC\" : bottom of pg 6\n- \"hindre\" : middle of pg 8\n- \"to backpropergation\" : bottom of pg 8\n- \"=≈\" : top of pg 17\n- \"mathcalR\" : bottom of pg 23\n- \"interrest\" : bottom of pg 24\n- \"employees\" : bottom of pg 24\n- \"models, to\" : bottom of pg 24\n- \"difference, is\" : bottom of pg 24\n- \"consider them\" : bottom of pg 24\n- \"model, is\" : top of pg 25\n- \"task, is\" : top of pg 25\n- \"still struggle\" : top of pg 25\n- \"seam\" : top of pg 27\n- \"inline\" : top of pg 27\n- inconsistent typesetting of \"NAC\" : top of pg 27\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1l3vxgLcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to address several issues shown in the Neural Arithmetic Logic Unit, including the unstability in training, speed of convergence and interpretability. The paper proposes a simiplification of the paramter matrix  to produce a better gradient signal, a sparsity regularizer to create a better inductive bias, and a multiplication unit that can be optimally initialized and supports both negative and small numbers.\n\nAs a non-expert in this area, I find the paper interesting but a little bit incremental. The improvement for the NAC-addition is based on the analysis of the gradients in NALU. The modification is simple. The proposed neural addition unit uses a linear weight design and an additional sparsity regularizer. However, I will need more intuitions to see whether this is a good design or not. From the experimental perspective, it seems to work well.\nCompared to NAU-multiplication, the Neural Multiplication Unit can represent input of both negative and positive values, although it does not support multiplication by design. The experiments show some gain from the proposed NAU and NMU.\n\nI think the paper can be made more self-contained. I have to go through the NALU paper over and over again to understand some claims of this paper. Overall, I think the paper makes an useful improvement over the NALU, but the intuition and motivation behind is not very clear to me. I think the authors can strengthen the paper by giving more intuitive examples to validate the superiority of the NAU and NMU.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends work on NALUs, providing a pair of units which, in tandem, outperform NALUs. The reviewers were broadly in favour of the paper given the presentation and results. The one dissenting reviewer appears to not have had time to reconsider their score despite the main points of clarification being addressed in the revision. I am happy to err on the side of optimism here and assume they would be satisfied with the changes that came as an outcome of the discussion, and recommend acceptance.
Decision: Accept (Spotlight)