Learning Division with Neural Arithmetic Logic Modules
Authors: Bhumika Mistry
Katayoun Farrahi
Jonathon Hare
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Neural Arithmetic Logic Modules
Division
Extrapolation

Abstract: To achieve systematic generalisation, it first makes sense to master simple tasks such as arithmetic. 
Of the four fundamental arithmetic operations (+,-,$\times$,$\div$), division is considered the most difficult for both humans and computers. In this paper we show that robustly learning division in a systematic manner remains a challenge even at the simplest level of dividing two numbers. We propose two novel approaches for division which we call the Neural Reciprocal Unit (NRU) and the Neural Multiplicative Reciprocal Unit (NMRU), and present improvements for an existing division module, the Real Neural Power Unit (Real NPU). Experiments in learning division with input redundancy on 225 different training sets, find that our proposed modifications to the Real NPU obtains an average success of 85.3$\%$ improving over the original by 15.1$\%$. In light of the suggestion above, our NMRU approach can further improve the success to 91.6$\%$. 

Learning Division with Neural Arithmetic Logic Modules
Anonymous Author(s)
Affiliation Address email
Abstract
To achieve systematic generalisation, it first makes sense to master simple tasks such as arithmetic. Of the four fundamental arithmetic operations (+.,,x,+)) division is considered the most difficult for both humans and computers. In this paper we show that robustly learning division in a systematic manner remains a challenge even at the simplest level of dividing two numbers. We propose two novel approaches for division which we call the Neural Reciprocal Unit (NRU) and the Neural Multiplicative Reciprocal Unit (NMRU), and present improvements for an existing division module, the Real Neural Power Unit (Real NPU). Experiments in learning division with input redundancy on 225 different training sets, find that our proposed modifications to the Real NPU obtains an average success of 85.3 3%% improving over the original by 15.1%. In light of the suggestion above, NMRU approach can further improve the success to 91.6%.
2
6
10
12
Introduction
14
Imagine you must learn to divide numbers, but are only given 10 numbers and the target value. This task requires finding the 2 relevant operands, the order to divide the operands, and learning to divide. In machine learning, this is equivalent to a supervised regression task where the aim is to learn the underlying function between the inputs and output such that the solution is generalisable any input. The ability to select relevant features is a desirable property of neural networks, useful for improved intepretability, reduced pre-processing costs and greater generalisation [Chandrashekar and Sahin 2014]. The ability to model division, one of the four fundamental arithmetic operations, is necessary for expressing dynamical systems [Sahoo et al., 2018), and ahysics-based formulas [Udrescu and Tegmark 2020 However, even recent models still struggle to learn division when there is input redundancy [Schlör al.] 2020).
15
16
17
18
19
20
21
23
24
The main challenge of the above task comes from learning the selection and operation at the same time, which can lead conflicting priorities when learning network weights. Furthermore, the natural properties of division of values around zero leads to undesirable gradients. Models which deal with this naively (e.g. MLPs) are unable deal with the fluctuant gradients caused by the asymptotic nature and discontinuities in division Trask et al. 2018).
25
26
27
28
Can we build models which can learn division in the presence of its undesirable, yet valid, properties?
30 We aim to address this question in this paper. Specifically, we contribute the following!"
31
Improvements to the Real NPU [Heim et al. 2020 including: clipping, discretisation and
constrained initialisation to improve performance in learning division on different training ranges.
'Code (MIT license) available at: ros-review-oo 2E4C/
neur
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

33
Two novel division modules, the NRU and the NMRU. The NRU explores extending the NMU weight ranges from [0,1] to [-1.1] include division, where we find a weakness learning from negative ranges. Learning from the weaknesses of the NRU, the NMRU extends the NMU to learn division while keeping weights values between [0.1]. We further boost performance by using a
34
35
36
37
Real NPU inspired sign retrieval mechanism, enabling the NMRU to gain the best performance when using a mean squared error (MSE) loss.
38
39
New understanding into the hindrances in learning division including: training mixed-sign inputs, training on negative ranges, and division on extremely small values. We find these difficulties can be sufficiently identified using synthetic division tasks.
40
41
42
The broader impact of our work relates to interpretable Artificial Intelligence where our modules can be included in larger networks for applications such image classification analogy creation, whilst retaining the ability to produce transparent generalisable solutions. However, there are possible negative societal impacts. Such modules can be viewed as specialised feature selectors/aggregattors which do not require integrating domain knowledge. Therefore, a non-domain-expern tries inter- preting relations in the input data, they may incorrectly interpret causality, which be especially harmful if such a case occurs on medical or financial data. Mitigating against such downstream issues requires to first focus efforts producing robust modules to different distributions and understand their affect on learning other networks architectures (e.g. CNN). Understanding this will enable recognising situations where these modules can aid and where they should avoid being used.
43
44
46
47
48
50
51
52
2
Related Work
58
One approach to learn division would be symbolic regression networks (Sahoo et al. 2018). However, symbolic approach pre-defines the operations, which is not limitation of using Neural Arithmetic Logic Modules (NALMs).
55
56
NALMs are neural networks which learn arithmetic operations and input selection MMistry et al. 2021]. The weights of these networks are intepretable such that a discrete value represents a specific operation. For example, '-1' to represent division and '0' for selection. From this research field, we focus on the Real NPU and the NMU. Until now, the Real NPU only has learned division on training ranges of either 2/[0.1,2] or Sobol(0,0.5) [Heim et al. 2020]. It remains unclear if this module is robust to other training ranges even as a stand-alone unit. Robustness to training ranges is important as these module's applicational use comes from being part of larger end-to-end networks, where the input range into the module cannot be controlled. The NMU is a multiplication module which we extend also do division. The authors of the NMU believe such an extension incurs too many limitations for learning [adsen and Johansen 2020). We use this paper as an opportunity to explore this belief.
57
59
61
62
63
65
88
Trask et al. 2018 developed the Neural Arithmetic Logic Unit (NALU) which can model all four 67
arithmetic operations. However, studies show this module to be unstable in learning division (Schlör et al. 2020 Heim et al., 2020]. In particular, their gating method responsible for selecting an operation cannot learn consistently [Madsen and Johansen 2020]. [Shllö et al. (2020) developed iNALU additionally applying weight and gradient clipping, sign retrieval, regularisation, reinitialisation and
69
71
72
separating shared parameters to the NALU. Even with these modifications, they still find consistently learning division to high precision to remain unattainable. Furthermore, [Heim et al. 22200's 's results imply iNALU is outperformed by the Real NPU for division.
73
74
3 Architectures
75
76
This section introduces the architectures for the (Real) NPU, NRU, and the NMRU. The (Real) NPU is existing module, which improve in Section The NRU and NMRU are novel contributions. Appendix A summarises the important properties of these division modules.
78
79
3.1
Real Neural Power Unit
Heim et al. 2020 develop module to learn to multiply and divide, using the intuition from Trask et al. 2018 that multiplicative operations are additive operations log space. Their work extends this idea into complex space. The NPU can be used with its complex form (Equation I requiring both
81

---

83 a complex and real weight matrix (W W(+),, or only its real form the Real NPU (Equation 2
84 For improved gradients, relevance gate r (Equation 3 is used which converts inputs close (1.e.
85 irreevant features) to to avoid the resulting output evaluating to O. A gating vector g. learns to
86 select relevant input elements, where gate values are clipped between [0.1] during training.
NPU exp( (() log(r)- w(())) k) o cos( ((() (1) log(r) + w()))) k),
87
RealNPU : expp log(r)) o cos(W(r)k))
(2)
0
2 0 x4.
where r=g (|ær c) + (1 g) and k
(3)
Tgi
0
89
A weighted LI penalty is used when training. The weight value 3 grows between predefined values
90
Satart Bend and is increased every Setep- 10, 000 iterations by growth factor Pgrowth 10. We focus on the Real NPU over the NPU as the solution of the tasks in this paper can be captured using only real values meaning that the complex form not required.
91
92
93
3.2
Neural Reciprocal Unit
94
We propose the NRU, which can model multiplication and division. We extend the NMU, motivated
by division being multiplication of reciproculs. The range which weight values can be is extended
85
96
from [0,1] to [-1,1], where -1 represents applying the reciprocal on the corresponding input element.
97 A NRU output element 2 is defined as
1 I (sign(x;; xx/We.. 1=1
NRU: 20
Wi.,, -W.ool + |Waol)
(4)
where / is the number of inputs, Assuming weights are either I (multiply) or -1 (reciprocal), |x|| W.,- will apply the operation on an input element. The absolute value is used so that the module only operates in the space of real numbers, as x; W.. for negative input (z;,) when -1 < W < results in complex number. The of absolute means the sign of the input must be reapplied. For the no-selection case W = 0, we want the input element to convert to (the identity value). resulting in applying |W... +1++1--- oo The derivative of the absolute function at 0 is undefined meaning the
98
99
100
102
104
gradients of Equation + can contain points of discontinuity. To alleviate this issue, we approximate
105
the absolute function using a scaled lanh (inspired by Faber and Wattenhofer 2020 ). More formally,
tannh(100- W if training otherwise
W
106
The scale factor (1000) controls how close to the absolute function the approximation is, where larger values give more accurate approximation. For clipping and regularisation, the same scheme the Neural Addition Unit (NAU) (see Appendix B} is used.
108
109
3.3
Neural Multiplicative Reciprocal Unit
110
An alternate extension of the NMU, also motivated by division being multiplication of reciproculs the NMRU (Equation 5). We concatenate the reciprocal of the input (plus a small e) to the input resulting in module which only needs to learn selection. Hence, weights can be in the range [0.1].
111
112
2/
21
NMRU: 2. = II (W |x/] +1 W.,))
Σ
(cos(Wi,p) ka)) , where ky
0
x4
2
34 < 0
i=1
(5)
113
The iteration over 27 represents the going through all inputs and their reciprocals. We calculate the magnitude and sign separately, joining the result at the end. The magnitude is calculated passing absolute of the concatenated input through an NMU architecture and the sign by using cosine mechanism similar to the Real NPU. However, unlike the Real NPU only the weight matrix is required. The norm of the weight's gradients are clipped to 1 prior to being updated by the optimiser.
114
115
116
117
118 This is done to alleviate the issue of exploding gradients caused by including the reciprocal to the inputs. For clipping and regularisation, the same scheme the NMU (see Appendix B is used.
3

---

Table Interpolation (train/validation) and extrapolation (test) ranges used. Data (as floats) is drawn from a Uniform distribution with the range values as the lower and upper bounds.
Interpolation
[-20,-10)
[-2, -1)
[-1.2, -1.1)
[-0.2, -0.1)
[-2, 2)
Extrapolation
[-40, -20)
[-6, -2)
[-6.1,
(.22))
[[-6, -2), [2, 6)]
Interpolation
[0.1, 0.2)
(1.2)
[1.1,
[10, 20)
Extrapolation
[0.2, 2)
6)
[1.2, 6)
[20, 40)
Experiment Setup
120
121
We introduce the two main experiments used to evaluate modules, including: default parameters. train and test ranges, and evaluation metrics. The tasks evaluate the ability of single module to divide two numbers from an input vector in two settings: no redundancy and with redundancy.
122
123
Default parameters: All experiments use mean squared error (MSE) loss with an Adam optimiser Kingma and Ba 2015), with 10,000 samples for the validation and test sets. The best model for evaluation is taken using early stopping the validation set. All runs are over 25 different seeds. All inputs are required in the no redundancy setting, i.c., input size of 2. Training takes 50,000 iterations where each iteration consists of a different batch of size 128. The Real NPU uses a learning rate of 5e-3 with sparsity regularisation scaling during iterations 40,000 to 50,000. The NRU and NMRU sparsity regularisation scaling during iterations 20,000 to 35,000 and learning rate of 1 and 1e-2 respectively. In contrast, the redundancy setting uses an input size of 10, where 8 input values are not required for the final output. The total training iterations are extended to 100,000 with batch sizes of 128. The learning rates for the Real NPU, NRU and NMRU are 5e-3, 1e-3 and 1e-2 respectively.
125
127
129
131
132
139
134
Sparsity regularisation scaling occurs during iteration 50,000 75,000 for all modules. A summary of all relevant parameters is found in Appendix C
135
136
The interpolation (train/validation)) and extrapolation (test) ranges, are found in Table The chosen ranges are influenced by Madsen and Johansen 2022)]
Ranges:
137
138
Evaluation metrics: We use the Madsen and Johansen 2019]' 's evaluation scheme, consisting of
three evaluation metrics: the success the extrapolation dataset against near optimal solution (success rate), the first iteration which the task is considered solved (speed of convergence), and the extent of discretisation towards the weights' inductive biases (sparsity error). Sparsity error calculated by max max(min( l. 1 ll), measures the weight element which is the furthest away 1,0
140
142
143
from the acceptable discrete weights for the module. A success means the MSE of the trained model is lower than a threshold value (i.e. the MSE of a near optimal solution). We differ from Madsen and Johansen [2019] by using fixed threshold value le-5 rather than simulated MSE, there are no intermediate layers to accumulate numerical errors. We choose this precision as it can be guaranteed when working with 32-bit PyTorch Tensors. 95% confidence intervals (over the 25 seeds) are calculated from a specific family of distributions dependant on the metric. The success rate uses Binomial distribution because trials (i.e. run single seed) are either pass/ fail situations. The convergence metric uses a Gamma distribution and sparsity error uses a Beta distribution. Both Beta and Gamma can casily approximate the normal distribution and support its corresponding metric.
144
146
148
150
5
Improving the Real NPU's Robustness
152
153
We first improve the robustness of the Real NPU on different training ranges. We use the Single Module Task with no redundancy (see Section 4 to investigate the following questions:
154
155
1. Is LI regularisation required, and if so, do the regularisation parameters require tuning?
2. Does clipping the weight matrix aid learning?
3. Does enforcing discretisation on parameters improve convergence?
4. Can the weight matrix initialisation be improved?

---

rate
Success
1.00
0.75
0.50
0.5
0.25
0.00
000
U(-022-031 -0.3
Uf-1.3 22-111
U[-2,41
Ul-222)
Ut-20-10
Ujo U.0.1..22)
ufz 21
UB U1t0200
U1--2-1111
U--2-41
UK-2.3
UP-28- -0
ut.3
un11.
uno
ut-12
use
Interpo laton range
Interpolation range
(Ie-Ate-11
(e-d
- (te-),ee-))
L1
(te-5
LI regularisation
(b) Sweep over L1 (start,end) beta parameters
Figure 1: Exploring the effect and sensitivity of LI regularisation on the Real NPU
Success
Sucress
Success
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.25
0.25
0.25
0.00
Uf-02 -0.11
U110201
0.00
0.00
U1-022-0:11
(iss -11
U1-231
Ut-20- -101
ujp.1.0. 1.0.3)
Ul-02 1011 Ul Ii
(l0.1.02)
-101
Ut U1-1.2.-0
dh
Interpolation range
Interpolation range
Interpolation range
Xavter-Uniform
Constrained
(a) Clipping
(b) Discretisation regularisation
(c) W (re) initialisation schemes
Figure 2: Effect of clipping, discretisation, and the NAU initialisation scheme on the Real NPU,
To address each question in order, propose applying incremental modifications to the Real NPU. These modifications include: ablation study on the LI regularisation (including a sweep over the scaling range hyperparameters), clipping, enforcing discretisation, and scheme. We assume that we are optimising the Real NPU to perform multiplication or division. Therefore, we trade-off the flexibility of having non-discretised weights, which enables the success of modelling the SIR data in Heim et al. (2020) Section 4.1). in favour of sparse models with discrete weight values. All the modifications suggested also be generalised for the NPU architecture.
160
restrictive initialisation
162
163
184
186
LI regularisation required? (Yes) L1 encourages sparsity (i.e., zero weights) in solutions. Zero-valuse weights means not to select an input and return the identity value 1. For the task, the optimal weight values require selecting all inputs and therefore non-zero values, suggesting the application of LI could be damaging. Therefore, we compare against model which does not use LI regularisation, shown Figure Ta Removing LI proves to detrimental in five the nine cases shown and only shows minor improvements in two of the nine ranges (i.e., ((-1122--111 and 1(1.1,1.2)). Hence, we keep L1 regularisation. The L1 regularisation scaling (see Section 3.1]. requires setting the hyperparameters for the start (Bstart) and end (Bend) scaling values. We sweep over six different start and end values, denoted (<start>, <end>), displaying results in Figure lb We find the configuration (1e-9, le-7) is the most successful when considering performance all the ranges, and larger scaling values perform worse
168
170
172
174
175
176
Does clipping the learnable parameters help? (Yes) Division and multiplication operations are represented by weight values -1 and respectively. The current architecture does not constrain the weights which result in large weight values. The gate weights do get clipped and saved to another
178

---

Success rate
Solved at iteration step
5 Sparsity error
1.00
0.06
0.75
30000
0.04
0.50
20000
0.02
0.25
10000
0.00
0-
0.00
Ut-02 2.1
U1-221
UI -10-
Uilli (.113
ul-022 -0.11
Pe-- di Cra
1 't
U1-221
101
Ulo.1
Interpolation range
Real NPU (baseline)
Real (modified)
NMRU
Figure 3: Division without redundancy (input size 2).
180
variable during the forward pass, meaning after an update step the gate values can also be out of the range [-1,1]. Hence, we investigate the effect of applying clipping directly to the weight and gate values after every optimisation step. Results, shown in Figure 2a show clipping beneficial, with clipping on both weight and gate (or just on the weights) to improve over the baseline on all ranges (excluding U(1,2) where the baseline has already achieved full success).
182
183
184
185
Does enforcing discretisation help? (Yes) Modelling division in generalisable manner requires all learnable parameters to be discrete i.e., value from {-1, 0, 1). Using Madsen and Johansen 's200''s regularisation scaling scheme, we penalise weights for not being discrete. We modily the scaling factor to be Ā 1 and the regularisation go from 'off" to 'on' between iterations 40,000 to 50,000. Results, shown in Figure 2b show discretising the gate improves over the baseline but also discretising the weights is additionally beneficial (especially for range (/[-0.2,-0.1)) 2/[10,20) is the
186
189
190
191
only range where the baseline outperforms using discretisation, succeeding on two additional seeds.
Does using a more constrained initialisation help? (Yes) (+) uses Xavier-Unifome initial- isation [Glorot and Bengio 2010). This can result in weights initialised out of the range [-1,1]] Therefore, we use the initialisation for the Neural Addition Unit which is a constrained form of the Xavier-Unifome
192
194
that does not allow the fan values of the uniform distribution go beyond 0.5,
196
meaning that no weight value will be out of the range [-1,1] [Mddenn and Johansen 2020). Figure 2c shows using the constrained initialisation provides improvements over multiple ranges.
Results: Single Module Task
198
199
We analyse the results for the: Real NPU without using the modifications of Section with modifications, NRU, and NMRU.
Real NPU
200
201
6.1
No Redundancy
202
Figure 3 shows the baseline Real NPU without modifications struggles with all ranges except 111,2), struggling with sparsity on the larger ranges. Applying the modifications deals with the sparsity issue and improves the robustness such that only range 2/1 struggles (with success rate of 0.64). The
203
204
205
NRU and NMRU achieve full success over all ranges while solving the problem consistently fast and with low sparsity error. The success of the NRU is correlated with the learning rate (see Appendix E).
206
207
6.1.1
Mixed-signed Inputs
208
The remaining failure range the Real NPU is U(-2,2) where inputs can consist of arbitrary signed values (e.g. all positives, all negatives, or mixture of positive and negative values). We question if the failure due to the input samples in a batch having different signs from each other, or if the problem is due to the fact data samples can be close O (leading to singularity issues). To investigate
210

---

5 Success rate
S Solved at iteration step
5 parsity error
0.00075
0.4
5000
0.00050
0.2
2500
0.00025
0.0
Tro-s
0
0.00000
W1-221
uni.2) GUI-2.-11
TTop Cop-ere einn
Jn
Tt 1-012-1-99 (2'0o ain
(01---21 --in
Ul-2-003)
Conn
in
Interpolation range
Real NPU
Figure 4: Extrapolation results on training the Real NPU using mixed-sign datasets that control the sign of the input elements. The ranges in order of the datasets (i.c. dataset 1 to 5).
Input [a]; Output: 1/a
Input [a.b] Output: 1/
Input: [a.b]: Output: a/b
1e+17-
1e+17-
1e+17-
log(Test emor)
1e409
1e+01
1e+01
1e401
1e-07
1e-07
1e-07-
U10.1e-8)
(10.1e-7)
U10.1e-6)
Ul0.1e-5)
U1o. 16-4)
uro, Ur0,1e-3) U10.1e-2) (6-2))
U/0. U10.1e-1)
Uj0,1e+0)
Ulo Ul0.1e-8)
UIO. Ul0.le-T)
U10.1e
Ujo, Uj0.1e-5)
Ie-Si
uro. 1e-3)
U10.1e-2
1e-2)
U10. 1e-1)
1o0.1eed 1e+00
V10
UjD, Uj0.1e-77 16-7)
Ujo, 66-16--)
Ur0,1e-5)
U10, 1e)))
U10.1 U10.1e-3)
U10,1e-2
U10. 11011-11)
1ex0)
U1o
Data sample range
ReaiNPU
RealNPU (eps-0)
NRU
NMRU
Figure 5: Effect of the singularity issue on the Real NPU, NRU and NMRU over increasing input ranges. Left: Reciprocal for input size of I (no redundancy), Middle: Reciprocal for an input size of (with redundancy). Right: Division for an input size of (no redundancy).
212
this, we create additional mixed-sign datasets, controlling the range for each element in the input. The interpolation and extrapolation ranges for the different datasets can be found in Appendix C Datasets
213
214 1, 2, 4 and 5 sample positive value for one input element and negative value for the other element.
215
Dataset samples the signs randomly. Datasets 2 and avoid sampling close 0 values to mitigate
the singularity issue. As shown by Figure 4 the Real NPU struggles on all these ranges, implying that
217
core issue is not from different input samples having different signs or due the input samples being able to contain small values close to O. The underlying issue is therefore most likely correlated
219 to the each element in an input having different signs. When the denominator of the output is positive (dataset I or 2), the solution is found faster than when the denominator is negative value (dataset 4 or 5). When the signs for an input element are controlled, discretisatioon/paarity is no problem, in contrast when the signs are arbitrary the sparsity error slightly (though not significantly) higher.
221
223
6.2
Division by Small Numbers
224
Division by zero remains a challenge model due to the inability to provide an computational value for the output and gradient. Furthermore, the discontinuous nature at zero causes its neighbouring
225
226
values to have large gradients. To understand the extent this issue when learning, we explore
learning to divide by values close to zero using three tasks with increasing difficulty: 1) learning to
228
take the reciprocal of single input, 2) taking the reciprocal of the first input given two inputs, and 3)
diving the first input by the second given two inputs. F plots the test error for different modules
7

---

Success rate
Solved at iteration step
5 Sparsity error
1.00
0.15
0.75
30000
0.10
0.50
20000
0.05
0.25
10000
0.00
0.00
U1-22-
Uj0.1.033
-101
UIZ.
(,111
ul-02 -0.11
Z-ln
Ver In It titnn
011
U[-022.
U1
Ul
Interpolation range
Real NPU (baseline)
Real (modified)
NMRU
Figure 6: Division with redundancy (input size 10).
230
assuming the module weights are set to the 'gold' solution for the three tasks. As the range values become closer to zero, the test error thresholds become increasingly large. Therefore, even with the correct weights, relying on the test errors alone as an indicator become increasingly deceptive with
232
233
values close to zero. The Real NPU has larger test errors for all tasks and ranges, caused by adding e
to the input (see Equation 3). Setting = 0 reduces the test error at the cost of the ability to deal with zelo-valued inputs. Appendix (rrrendxxpproviiiee provides the corresponding experimental results for these tasks.
234
235
6.3
With Redundancy
237
Introducing redundancy (Figure 6 causes failure modes to arise. Failures on range l([-2,2) become more prevalent. The baseline Real NPU produces high sparsity errors relative to the other modules suggesting struggle with discretisation. Using the modified Real NPU improves over all ranges of the baseline (which were not already at full success) in terms of success, speed and sparsity that complex weights do not fix the issue, we test the NPU module with all the modifications used on the real weight matrix (see Appendix G). Complex weights hinders success and convergence speeds of negative ranges. Assuming the global solution only uses the real weights, we enforce the complex weights be clipped between [-1,1] and to go to 0 during the regularisation stage using LI penalty. This did not result in any significant improvements against the Real NPU results. Input redundancy effects the NRU the most, resulting in full failures on all the negative ranges. The NMRU is the
239
[ To ensure
241
243
244
245
247
only module with success for the range U(-2,2), which is result of using the sign mechanism (see Appendix H). performs well over all ranges though can be outperformed by the modified Real NPU for negative ranges. Multiple ranges for the NMRU are solved around 50,000 iterations correlating to the sparsity regularisation being turned on.
248
249
250
251
6.3.1
Gradient Difficulties with the NRU
252
The partial derivative for the NRU weights, Equation 6
can give insight to the struggles of the NRU.
0y =
log(==))++
Ow; 2000 sech(1000w::))
(6)
2000 sech(1000w:)
NRUR
253 NRU (x:) (x) applies the NRU to all inputs excluding x; influencing the gradient values between
subsequent update steps. Factoring out this term, the following observations are made. If z, 2 0 and wi x 0 then gradients become increasingly large. 24 s and -1 S wy < then as w, + -1 all gradients for I; where |x,]| >> 1 become increasingly small. The gradients for 2; -1 and x; 1 are O regardless the value of W;. If w; = then the gradient 0 for all w; a result of using the tanh approximation. Even if the sign and magnitude are calculated separately and then combined (see Appendix Th to try to control the gradient better, the problem remains. Therefore, we conclude that extending the NMU to divide using weight of -1 is
255
257
259
poor choice when there redundant inputs.
Except for the sparsity error for range 2({10.20).
8

---

(a) Real NPU
(b) NRU
(c) NMRU
Figure 7: Root Mean Squared loss curvature for the NAU stacked with either a ReaINPU, NRU, wi
or NMRU. "The weight matrices are constrained to W1 Me uy 121
W₂ a0 m2 The
problem is (I) =22) (I1 + x2 + I3 + x4) for x (1, 1.2, 1." [Madsen and Johansen 2020
The ideal solution is W1 = = 1, though other valid solutions do exist e.g., W1 -1, W2
(The NMRU's weight matrix would be W2 = w2 wa J.. and the Real NPU's g [11].)
281
6.3.2
The Real NPU's and NMRU's Exploitation of Multiplicative Rules
262
The NMRU solutions exploit the inverse rule of division in that a; - = Since the input also contains the reciprocals, numerous extrapolative solutions exist. However this comes at the cost of finding a 'simple' solution which contains ones only for relevant inputs. The Real NPU exploits the
263
264
rules a; () = O and 1 = enabling non-zero weight values if the corresponding gate value is O. However, we can avoid this by allowing to also not penalised during sparsity regularisation stage (see Appendix G). We find this alleviates the exploitation issue with no cost to performance.
266
267
288
7
Discussion
269
In this paper, we demonstrate the limitations of intepretable neural networks in learning to divide. Using the no redundancy setting (size 2). we find that the Real NPU challenged when training data consists of mixed-signed inputs even with our applied improvements. Increasing the difficulty to have an input redundancy (with 8 redundant and 2 relevant input values) magnifies this issue, but also introduces failure modes for the NRU and NMRU for negative ranges. The NRU is unable to not wise to use with MSE. Alternate losses can improve certain failure cases though sometimes at the cost of performance on other ranges. For further details see Appendix which displays results on a correlation and scale-invariant based loss.
270
272
273
274
handle any negative ranges, which we conclude
275
276
277
Our NMRU is the only module with reasonable success over all tested ranges, requiring only 27 x O learnable parameters. However, this comes at the cost of the simplicity of the solution due to its exploitation of the identity rule; an issue the Real NPU does not have.
279
Once robust modules are attainable in single layer setting, the next step would be to question performance when learning stacked modules, e.g. learning a stacked additive and multiplicative module. Previously, Madsen and Johansen 2020, Figure 2] illustrates the troubles for multiplicative models with the capacity for division. They show how a stacked summative- summative-muliiliccaie multiplicative module can
281
283
lead to an exploding loss when the output of the summative module is close to 0 and the multiplicative model tries to divide. In Figure 7 we recreate their setup to produce the loss surfaces for the NAU- Real F NAU-NRU and NAU-NMRU respectively We find similar issue with the Real-NPU
285
286
287
and NRU, as both these units use a weight range of [- In contrast, the NMRU, whose weight's range is limited [0, does not have exploding losses.
288
289
In conclusion, division remains a challenge to learn using intepretable neural networks, even for the simplest tasks. Nevertheless, by identifying the specific areas causing difficulty (e.g., training ranges), and useful architecture properties (e.g., using a sign retrieval mechanism), we hope the community has better intuition for dealing with division and develop more robust modules to learn division.
291
'hhe NAU is a summative module Madsen and Johansen, 2020 Appendix K displays larger versions of these plots.

---

References
293
294
Girish Chandrashekar and Ferat Sahin. A survey on feature selection methods. Computers &
295
Electrical Engineering. 40(1):16-28, 2014. ISSN 0045-7906. doi:
eleceng. 20110011111000 024. https://w: httpss///wwwsccinncerrettt... sciencedirect...
296
297
045790613003066 40th-year commemorative issue.
298
Lukas Faber and Roger Wattenhofer. Neural status registers. CoRR, abs/2004.07085, 2020. URL
wttp://wwww..rrr////////.000 07085
300
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249 256. JMLR Workshop and Conference Proceedings, 2010. URL http:
301
302
303
pppoceedings.nin mlr pttpp/////g..oooooaag//ooo//////
Niklas Heim, Tomák Pevný, and Václav Smídl. Neural power units. Advances in Neural Information Processing Systems, 33, 2020. httphtts::/
304
306
ao/d Paper. pdf
307
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2015. httthttss:///rrtttt http:///axxxv..../ptooo wttp://..rr//p///11...0021. pdf
308
309
Andreas Madsen and Alexander Rosenberg Johansen. Measuring arithmetic extrapolation perfor- mance. In Science meets Engineering of Deep Learning at 33rd Conference on Neural Information Processing Systems (NeurIPS 2019). volume abs/1910.01888. Vancouver, Canada, October 2019.
310
311
URL buttts://arxiie org/ pttpp/ff/999/ /9900 010.00011....0 pdf
313
Andreas Madsen and Alexander Rosenberg Johansen. Neural arithmetic units. In International http:pppwwweeeeeeeeeeeeeereeeeee
Conference on Learning Representations, 2020. URL
315
HigNOeHKPS]
316
Bhumika Mistry, Katayoun Farrahi, and Jonathon Hare. A primer for neural arithmetic logic modules, 2021. httpt:tt///aattt org/ /df///df///// /2101. 09530. pdf
317
318
Subham Sahoo, Christoph Lampert, and Georg Martius. Learning equations for extrapolation and
control. In International Conference on Machine Learning, pages 4442-4450. PMLR, 2018.
320
Daniel Schlör, Markus Ring, and Andreas Hotho. inalu: Improved neural arithmetic logic unit. 2020.00071. Frontiers in Artificial Intelligence, 3:71, 2020. ISSN 2624-8212. doi: 10.00007f...2220000077. URL httpp:://wwtt fwontiersinnnnnm wwrg/articl///ii wtte///ee...330////1/ /frai. 2020. 000711
322
323
Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. In Advances in Neural Information Processing Systems, pages 8035-8044, 2018. URL https://openrevier: mwt/pdfffiieeee id=i1 gNoedd gNOeHKPS
325
326
Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for symbolic
327
regression. Science Advances, 6(16), 2020. doi: 10. 10.112
URL. https
328
Checklist
329
330
1. For all authors...
331
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's s
332
contributions and scope? [Yes] See contributions in the introduction. For Real NPU
improvements see Section 5 For two novel modules see Section 3.2 and
3.3 and results
in Section 6) For hindrances in learning division, see Section 6
334
395
(b) Did you describe the limitations of your work? [Yes] See Section 7
336
(c) Did you discuss any potential negative societal impacts of your work? [Yes] This paper focuses on the ML techniques and the foundational research required to learn division a systematic manner. Once robust modules for the arithmetic operations (i.e. NALMs) are achievable the community will possess trainable modules with significant advantages regarding model transparency and generalisability. That being said, we discuss how this Ieads to negative societal impact in the end of Section [
337
338
340
10

---

342
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the guidelines and our work does use human-derivee data.
343
345
2. If you are including theoretical results...
348
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
347
(b) Did you include complete proofs of all theoretical results? [N/A]
348
3. If you ran experiments...
349
(a) Did you include the code. data, and instructions needed reproduce the main ex- perimental results (either in the suppiemental material or as URL)? [Yes]] See link provided to the code base. Data is generated in real time and the code to generate is available in the linked repository.
350
351
352
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section I and Appendix [C]
354
355
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See plots and experiment details in Section for further details.
358
357
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix [D]
359
380
4. If you are using existing assets code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See footnote link
362
provided to the code base and Section
363
(b) Did you mention the license of the assets? [Yes] We state the MIT licence when giving the link to the codebase.
384
365
(c) Did you include any new assets either the supplemental material or as URL? [Yes] All code for model, data and experiments are available through the link to the codebase.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
368
369
(e) Did you discuss whether the data you are using/curating contains personally identifiable
370
information or offensive content? [No] All data synthetic, containing no such information.
372
5. If you used crowdsourcing or conducted research with human subjects...
373
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
375
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
378
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
378


Reviews:
[{"review_id": "tIM0nWNAmDb", "reviewer": "Reviewer_mB6g", "summary": "In this paper, the authors explore neural arithmetic architectures that could learn to perform division tasks, both in the case of input redundancy as well as with only 2 operators. Besides improving an existing architecture (NPU), other two related architectures are introduced (NRU and NMRU), both based on the idea that division can be framed as a multiplication of reciprocals. Extrapolation capabilities are probed by considering testing intervals that do not overlap with training intervals. The newly introduced architectures achieve better performance, though in the case of input redundancy all models are still error prone.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This research work appears mostly incremental in nature. It extends and improves previous model architectures, which are empirically tested over previously established tasks.\n\nQuality: The research topic is very actual, given the recent interest in testing mathematical learning skills of deep nets, though the scope of the work appears somewhat limited given the specificity of the research approach. The applicability of these models to broader contexts / domains is not particularly evident. The paper builds on the methodological setup established in previous work, which appears sound and well-designed. As a possible improvement, given that the proposed NMRU achieves reasonable accuracy over all tested ranges it would be interesting to further test its extrapolation capabilities by including even more challenging numerical ranges.\nOverall, my feeling is that this is a well-conducted research work, but probably NeurIPS might not be the most appropriate venue to present these findings.\n\nClarity: The paper is well written and properly organized. I found the opening sentence a bit convoluted: I think there could be more effective ways to introduce to the reader the challenging aspects of the division problem. Moreover, the authors do not  clearly explain why we should initially focus on the problem of learning division when there is input redundancy.\n\nSignificance: The issue of learning mathematical knowledge with deep nets is timely and challenging, and the deep learning community is spending increasing efforts in this direction. However, the significance of this paper for the broader NeurIPS community might be limited, because the current setup is very specific. The “broader impact” proposed by the authors (e.g., ability to produce transparent generalizable solutions) appears quite speculative and should be more clearly demonstrated.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors properly discuss the limitations of their research approach. The potential negative societal impact discussed by the authors seems quite speculative.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "efxuROmEwOk", "reviewer": "Reviewer_ZiPP", "summary": "The paper is an addition to the line of study concerning neural arithmetic. Several new architectures are proposed to tackle the problem of learning the division operation. Out of the four arithmetic operations division is arguably the more challenging one and previous works have struggled with it.\n\nThrough several experimental results, the new architectures are shown the outperform previous designs, both in training accuracy and in other desirable properties", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Firstly, I should mention that I am a non-expert in this area. After reviewing the relevant literature I find the topic of the paper interesting and relevant. Overall, the paper is well written and presents the motivations in a compelling way.\n\nHowever, there are two issues which I would like to raise:\n\n1. The ideas underlying the new architectures seem to be a bit incremental with respect to previous works. While it is definitely a serious paper that merits publication,  I am just not 100% convinced that NeurIPS is the correct venue.\n\n2. While the authors support their choice of architecture with a plethora of experiments, I feel that the paper lacks some theoretical foundations, which do appear to some extent in previous works. This makes it hard to interpret some of the experimental results. For example (but definitely not the only one), in Figure 3 why should I expect to have different behaviors between different interpolation ranges. How can one explain the fact that the real NPU is competitive with the new architectures on some ranges, and why those specific ranges? \nMoreover, as the authors note, the most challenging ranges are those that include 0. However, only one such range is tested ([-2,2]). I would expect to have more such ranges, both in different scales and with different levels of symmetry/asymmetry.\n\nMinor comments:\n- Display (1): Perhaps it is better to mention that the multiplication is the Hadamard (or Schur) product.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": " The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "W0LSTQC9OoN", "reviewer": "Reviewer_H6xb", "summary": "This manuscript proposed new types of neural arithmetic modules called Neural Reciprocal Units (NRU) and Neural Multiplicative Reciprocal Units (NMRU). NRU and NMRU extend Neural Multiplication Units by applying power terms and reciprocal inputs, respectively, and thus they enable divide operation. They also proposed improvement method of Neural Power Units (NPU).", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This manuscript introduces novel arithmetic layers, which enables divide operation. Proposed NRU and NMRU achieved higher success rates and fast learning ability compared to Real NPU in various experiments.\nThe manuscript seems to cover too broad subject. Proposing an improvement method of NPU does not seemed to be highly related to NRU and NMRU. It would be better to add comments explain relationship among them, or focusing more on NRU and NMRU. Additionally, there is a concern about novelty on the proposed improvement method of NPU, since they are briefly commented on ‘Neural Power Units’(NIPS 2020). \nIn Equation (5), they used summation of cosine function to calculate sign value of output, but I think it should be product of cosine like Equation (5) in supplementary material.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "PxhGC2pnjf1", "reviewer": "Reviewer_mhZR", "summary": "This paper study possible adaptations of Neural Arithmetic Linear Units (Trask 2018) to better handle division. The authors first propose improvements for training Real Neural Power Unit (Heim 2020), a neural arithmetic unit which performs division as substraction in log-space, using cosines to handle signs, and a gating mechanism to select operands, while constraining all weights to remain close to 0, 1 or -1. They show that introducing L1 regularisation, gradient clipping, and tuning weight constraint and initialization greatly improve its accuracy when dividing numbers.\n\nThey introduce two new designs, the Neural Reciprocal Unit, which functions as an addition in log-space with weights constrained to -1, 0 and 1, and can therefore perform multiplication or division of numbers (and \"unselect\" them when weights are zero, which dispenses of the gating mechanism), and the Neural Reciprocal Modified Unit, where the reciprocals are provided as additional inputs, constraining weights to 0 and 1 (as a selection mechanism), and adds a cosine function to handle signs. \n\nTesting over a variety of generated samples, they show that whereas the new designs improve on the original Real NPU, most of the still struggle on input of mixed signs, and have difficulty working with numbers close to zero, and learning both selection (choosing the operands) and division.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality** \nThe paper builds on existing designs and experiments. The techniques introduced in section 5 to improve performance of Real NPU are standard. The new designs proposed are incremental or crossovers between existing NALU modules. \n\n**Clarity** \nThe paper is very technical, but well written, and provides a lot of details about proposed architectures and experiments. \n\nIn section 3.1, it would be useful to define  the symbol used for element-wise multiplication. Also, the notation W^r W^i for the real and imaginary weight matrice conflict with the r symbol used for the gate. The ordering of the interpolation datasets in figure 3 and 6 is counter-intuitive: decreasing order of negative numbers, then increasing of positive)\n\n**Quality** \nVery detailed and interesting experiments. The results are extremely convincing, and the experiments are the strong point of this paper. \n\nA few questions and comments :\n\n1- In section 3.2: is the use of tanh to deal with the local discontinuity of the derivative of absolute value really necessary? RELU and absolute value activation functions are routinely used in feedforward networks, despite having the same gradient discontinuity. \n\n2- All the data in the training and test set are uniformly sampled from relatively small intervals. This has an impact on the distribution of results. In the case of multiplication and division, it might be interesting to test over a very large interval, with a logarithmic (Benford) distribution.\n\n3- In section 5, experiments with L1 regularization suggest that it is needed but in very small amounts. Could L2 regularization be a better choice? (as it would have the same stabilizing effect, but less enforce sparseness).  \n\n4- Being able to extrapolate out of the training distribution support is one of the main justification of NALU. In the paper, most experimental results are calculated from interpolation test set. Adding more extrapolation results would reinforce the paper.\n\n**Significance** \nThe main significant result is the improvement of the Real Neural Power Unit training. NRU fail to convince because of their problem on mixed sign operands. The experiments do a good job at documenting the difficulty of training NALU on division.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They have", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper study possible adaptations of Neural Arithmetic Linear Units (Trask 2018) to better handle division. 
While the contribution is worthwhile, the reviewers concern that it is too marginal and of limited scope
Decision: Reject