Exploring the Learning Mechanisms of Neural Division Modules
Authors: Bhumika Mistry
Katayoun Farrahi
Jonathon Hare
Venue: Accepted by TMLR ()

Abstract: Of the four fundamental arithmetic operations (+, -, $\times$, $\div$), division is considered the most difficult for both humans and computers. 
In this paper, we show that robustly learning division in a systematic manner remains a challenge even at the simplest level of dividing two numbers. We propose two novel approaches for division which we call the Neural Reciprocal Unit (NRU) and the Neural Multiplicative Reciprocal Unit (NMRU), and present improvements for an existing division module, the Real Neural Power Unit (Real NPU). In total we measure robustness over 475 different training sets for setups with and without input redundancy. We discover robustness is greatly affected by the input sign for the Real NPU and NRU, input magnitude for the NMRU and input distribution for every module. Despite this issue, we show that the modules can learn as part of larger end-to-end networks. 

Published in Transactions on Machine Learning Research (09/2022)
Exploring the Learning Mechanisms of Neural Division Modules
Bhumika Mistry
bm4g15@soton.cce.
Department of Vision Learning, and Control Electronics and Computer Science Universily of Southamplon
Katayoun Farrahi
jffrrmhi@sstoon.cc ac.uk
Deportment of Vision Learning, and Control Electronics and Computer Science University of Southampton
Jonathon Hare
jsh2@ecs.sooon.com uk
Department of Vision Learning, and Control Electronics and Computer Science University of Southampton
Reviewed on OpenReview: https: // opwnreviee.........mmmo net/ forun? id- Hjelo
Abstract
Of the lour fundamental arithmetic operations (+, -, x, +), division is considered the most difficult for both humans and computers. In this paper, we show that robustly learning division in a systematic manner remains a challenge even at the simplest level of dividing two numbers. We propose two novel approaches for division which we call the Neural Reciprocal Unit (NRU) and the Neural Multiplicative Reciprocal Unit (NMRU), and present improvements for existing division module, the Real Neural Power Unit (Real NPU). In total we measure robustness over 475 different training sets for setups with and without input redundancy. We discover robustness is greatly affected by the input sign for the Real NPU and NRU, input magnitude for the NMRU and input distribution for every module. Despite this issue, we show that the modules can learn as part of larger end-to-end networks.
1 Introduction
Division is one of the four fundamental arithmetic operations and is necessary for expressing real-world
dynamical systems (Sahoo et al. 2018a)) or physics-based formulas (Udrescu & Tegmark 2020). However,
the properties of division values around zero leads to undesirable gradients for training Neural Networks
(NN) through backpropagation, making it the hardest operation to learn (Mistry et. al.
2022). Networks
which try to learn division naively such as multilayer perceptrons (MLPs) are unable deal with the fluctuant gradients caused by the asymplotic nature and discontinuities in division Trask al. 2018). Furthermore, if the network lacks appropriate bias for learning division, it can also lead to poor generalisation out-of-distribution (OOD) data. For example, a network could learn to achieve reasonable loss on a soatiing//alidaaiion training/ validation range between [1, 2] but be unable to maintain a reasonable loss when tested outside of
[, 2] such as range [3, 10).
In particular, imagine you must learn to divide 2 numbers from a list of numbers, but are only given the 10 numbers and the expected result. This task requires finding the 2 relevant operands, the order to divide the operands, and learning to divide. In machine learning, this is equivalent to supervised regression task

---

Published in Transactions on Machine Learning Research (09/2022)
where the aim is to learn the underlying function between the imputs and output such that the solution is generalisable to any input. For NNs, the main challenge of this task comes from learning the selection and operation at the same time, which can lead to conflicting priorities when learning weights.
Selecting relevant inputs aaputs/featurs features is a desirable property of neural networks useful for improved interpretabil- ity, reduced pre-processing costs and greater generalisation (Chandrashekar Sahin| 2014). In particular, there exists a class of NNs called Neural Arithmetic Logic Modules (NALMs) which learn to select features and learn arithmetic operations simultancously (Trask et al. 2018 Mistry et al.) 2022 NALMs are designed to provide systematic generalisation for arithmetic operations such that if the appropriate parameters are learnt once training ends the NALM can extrapolate to unseen OOD data. Furthermore, as differentiable spe- cialist modules (such as those for arithmetic operations) can be integrated with overparametrized networks as an intermediate module, being able to successfully select only the relevant inputs is important (Madsen & Johansen 2020p. However, even recent models still struggle to learn division when there is input redundancy (Schlör et al. 2020). Can we build models which can learn division in the presence of its undesirable, yet valid, properties We aim to address this question in this paper. Specifically, we contribute the following
We show how additional biases improves learning of an existing division module, the Real NPU Heim et al., 2020), by including: clipping, discretisation and constrained initialisation.
We develop two novel division modules, the NRU and the NMRU, by extending an existing multiplication unit. Through rigorous experiments we find both modules outperform the Real NPU for the no input redundancy tasks, with the NMRU also outperforming the Real NPU for the imput redundancy tasks.
We identify the types of data which hinders learning division for each module, including training on: mixed-sign inputs, negative ranges, extremely small values and different distributions. These difficulties can be sufficiently identified using synthetic division tasks.
We show how NALMs can learn in larger end-to-end network using an arithmetic MNIST task.
2 Related Work
Learning to robustly divide provides stepping stone for NNs in achieving symbolic regression. Sym- bolic regression searches the space of expressions to predict mathematical expression from a given set of input-oulput observations. Compared to black-box NN functions, expressing a mathemalical function is significantly easier to interpret. Symbolic regression can be implemented via Genetic Programming (GP) using Evolutionary Algorithms (EA) which learn mathematical expressions (Koza) 1994 Schmidt & Lipson 2009). EAs maintain a population of expressions where individuals of the population get selected based-off a fitness function and modified via techniques such as crossover and mutation. This procedure is repeated until& a stopping criterion is met. Hybrid methods which combine GP with local search can also be used to further boost generalisation results (Kommenda et al.) 2020), however both the pure EA and hybrid methods do not scale well due to the combinatorial nature of the method.
Alternatively, a fully differentiable NN approach can be taken by incorporating biases to improve the in- terpretability of the network. [Sahoo et al. (2018b) sets activation functions in a layer to different symbolic operations rather than using a traditional non non-linear activation like ReLU. They also encourage only using relevant weights through sparsity regularisation scheme which varies strength depending on how much training has occurred. However, to gain the best performance requires using selection strategies over many trained modules which is costly and can be unreliable (Sahoo et al., 2018a).. In contrast. Udrescu & Tegmark (2020) exploits patterns in the data by designing physies related biases such transnational symmetry multiplicative separability into their architecture. Due to the strong prior which assumes the dataset con- tains an underlying physics representations ions, the model performs poorly when trained on datasets without such representations (Cava et al.) 2021
AA desiderata for building a division module provided in Appendix A 2 Code (MIT license) available at: hnttps://gáthi: httm//mistrrr//aammmiiiii divissio
2

---

Published in Transactions on Machine Learning Research (09/2022)
Another type of differentiable NNs are NALMs which have specially designed architectures biased towards learning arithmetic operations (Mistry et al. 2022) This work focuses modelling division using NALMs. The weights of NALMs are interpretable such that a discrete value represents a specific operation. For example, '-1' would represent division and "0' for no selection. Trrak et al. (2018) developed the Neural Arithmetic Logie Unit (NALU), the first NALM, which can model all four arithmetic operations. The NALU consists of two sub-units; one to model addition and subtraction and another to model multiplication and division. For each input corresponding weight value is learnt to represent the exact operation and a gate value is learnt to select between sub-units. However, multiple studies show the NALU is unstable in learning division (Schlör et al.) 2020 Heim et al. 2020). In particular, their gating method responsible for selecting an operation cannot learn consistently (Madsen & Johansen 2020). To improve the NALU, SShlör et al. (2020) developed the iNALU which applies weight and gradient clipping, sign relrieval, regularisation, reinitialisation and separating shared parameters. Even with these modifications, they find consistently learning division a high precision to remain unattainable. MMdsen & Johansen (2020) create the Neural Multiplication Unit (NMU) which only models multiplication but has significant performance gains compared to the NALU. Focusing multiplication and division, Heim et al. (2020) developed module which learns in the real and complex parameter space. Their results showed their Real NPU to outperform the iNALU for division. Until now, the Real NPU only has learned division on training ranges of either [(001,2 or (0,05 (Heim et al. 2020). Tt remains unclear if this module is robust to other training ranges even as a stand-alone unit. Robustness to training ranges is important as these module's applicational use comes from being part of larger end-to-end networks, where the input range into the module cannot be controlled.
3 Architectures
This section introduces the architectures for the (Real) NPU, NRU, and the NMRU. The (Real) NPU is existing module, which we improve in Section 5 The NRU and NMRU are novel contributions which extend the existing NMU (see Appendix B] to do division. Since these architectures are NALMs they can be viewed as regression modules trained via supervised learning. The imputs have some underlying mathematical relation to the outputs which is modelled using division. An input is represented as a vector of features, where only certain input features are relevant to the output.
3.1 Real Neural Power Unit
Heim et al. 20200) develop module to multiply and divide using the intuition from Trask ct al. (0018) that mullippicatiive operalions are uddilive operulions in log space. For example, x b exp(In(a) In(b) and a exp(In(a) - In(b)). Heim et al. (2020) extends this idea into complex space. The NPU can be used with its complex form (Equation I requiring both a complex and real weight matrix (WIM, RRE of shape Inputs|] x |Outputs)), or only its real form the Real NPU (Equation 2 We focus on using the Real NPU over the NPU as the solution of the tasks in this paper can be captured using only real values meaning that the complex form is not required. For improved gradients, a relevance gate r (Equation 3) converts inputs close to 0 (i.e. irrelevant features) to to avoid the resulting output evaluating to 0. A gating vector g. learns to select relevant input elements, where gate values clipped between [0,1] during training (Equation 5
NPU yo = cxp
Σ (WRE. In(r:))
Σ (WIM kss)
(1)
cos Σ
(W.M In(r:)) + Σ (WRE ki) 10
1=1
RcaINPU: : yo = =xx
Σ (WRR in(r:)))
cos Σ
(W.L
RE ki)))
(2)
1=1
3

---

Published in Transactions on Machine Learning Research (09/2022)
where
rj 9: (|2,) c) + (1-9.),
(3)
4 0
I2 2
(4)
I2 <0-
and
9: min(max(g; 0).
(5)
Following Heim al. (2020 L1 penalty scaled by factor 3 is used when training. B grows between predefined values Astart Bend and increases every Bstep 10, 000 ilerations by growth factor Bgrouth = 10.
3.2 Neural Reciprocal Unit
We propose the NRU, which can model multiplication and division, by extending the NMU motivated by the fact that division is the mulliplication. of reciprocals. The range which weight values can be is extended from [0,1] to [-1,]], where-1 represents applying the reciprocal on the corresponding input element. A NRU output element 2o is defined as
NRU to II (sign(x;) (xs |W + W2
(6)
1=1
where I is the number inputs. Assuming weights are either 1 (multiply) or -1 (reciprocal), |x, W will apply the operation on an input element. The absolute value is usod so that the module only operates in the space of real numbers, as z; Wio for a negative input (z:) when -1 W2 < 1 results in a complex mumber. The use of absolute means the sign of the input must be reapplied. For the no-selection case W = O, we want the input clement convert to (the identity value), resulting applying o + |W,.o- The derivative of the absolute function at is undefined meaning the gradients of Equation can contain points of discontinuity. To alleviate this issue, we approximate the absolute function using scaled tanh (inspired by Faber & Wattenhofer (2020)). ). More formally,
ta00(1000 W if training otherwise
W
The scale factor (1000) controls how close the absolute function the approximation is, where larger values give more accurate approximation For clipping and regularisation, the same scheme the Neural Addition Unit (NAU) (see Appendix B is used which forces weight elements to converge to -1, 0 or 1.
3.3 Neural Multiplicative Reciprocal Unit
An alternate extension of the NMU, also motivated by division being mullippication of reciprocals is the NMRU (Equation 7). We concatenate the reciprocal of the input (plus small e) to the input resulting in a module which only needs to learn selection. Hence, weights can be in the range [0,1].
21 II (W... |x| + W.,) cos (W... ka)) where k;
34
NMRU: =
(7)
<<<0 0
The iteration over 27 represents the going through all inputs and their reciprocals. We calculate the magni- tude and sign separately, joining the result the end. The magnitude is calculated by passing the absolute of the concatenated input through an NMU architecture and the sign is caleulated by using cosine mech- anism similar to the Real NPU. However, unlike the Real NPU only the weight matrix is required. The morm of the weight's gradients are clipped to prior to being updated by the optimiser. This is done to alleviate the issue of exploding gradients caused by including the reciprocal to the inputs. For clipping and regularisation, the same scheme as the NMU (see Appendix is used.
SSee Appendix tanh scaling experiments.
4

---

Published in Transactions on Machine Learning Research (09/2022)
Table 1: Nine interpolation Data (as floats) is drawn from a Uniform distribution with the range values as the lower and upper bounds.
ranges with their corresponding extrapolation (test) range.
Interpolation -10)
12
-1122,
-0.1)
22 [[6, 2), 6)
0.11, 0.2)
[1, 2)
[11, 1.2)
[10, 20)
Extrapolation
-40, -20)
[6, 22)
[.11, -122
[-2, -0.2)
[0.2, 2)
2. 6)
[1.2, 6)
[20. 40)
4 Single Layer Arithmetic Experiment Setup
We introduce the two main experiments used to evaluate modules, including: default parameters, train and test ranges, and evaluation metrics. The tasks evaluate the ability of a single module to divide two numbers from an input vector in two settings: no redundancy (2 inpuls) and with redundancy (10 inputs). During training models will have discretisation regularisation applied to weights which enforces exact selection and precise application of arithmetic operations.
Default parameters: A summary of all relevant parameters is found in Appendix All experiments use mean squared error (MSE) loss with an Adam optimiser (Kingma & Ba 2015) and 10,000 samples for the validation and test sets. Training uses batch sizes of 128 and the best model for evaluation is taken using early stopping on the validation set. All runs are over 25 different seeds. All inputs are required in the no redundancy setting, i.c., imput size 2. Training takes 50,000 iterations where each iteration consists of different batch. The Real NPU uses a learning rate of 5e-3 with sparsity regularisation scaling during iterations 40,000 to 50,000. The NRU and NMRU use sparsity regularisation scaling during iterations 20,000 to 35,000 and a learning rate of and 1c-2 respectively. In contrast, the redurdancy setting uses an input size of 10, where 8 inpul values are not required for the final output. The total training iteralions are extended to 100,000. The learning rates for the Real NPU, NRU and NMRU are 5c-3, le-3 and 1e-2 respectively. Sparsity regularisation scaling occurs during iteration 50,000 to 75,000 for all modules.
Ranges:
The ranges used for training, validation and testing is task dependant. For cach task the train and validation sets are sampled from interpolation range, whilst the test set is sampled from extrapolation range. The interpolation ranges will not overlap with the extrapolation ranges meaning no data in the test set will ever have been seen during training.
evaluation scheme, consisting of three evalu- alion metrics: the success on the extrapolation dataset against near optimal solution (success rale). the first iteration which the task is considered solved (speed of convergence), and the extent of discretisation towards the weights' inductive biases (sparsity error). Sparsity error is calculated by max(min(( - 1.0
Evaluation metrics: We use the Madsen & Johansen (0019
D)) measuring the weight element which is the furthest away from the acceptable discrete weights for the mod- ule. A success means the MSE of the trained model is lower than threshold value (i.e., the MSE of a near optimal solution). We differ from Madsen & Johansen (2019) by using a fixed threshold value 1c-5 rather than a simulated MSE. We choose this precision as can be guaranteed when working with 32-bit PyTorch Tensors. 95% confidence intervals (over the 25 seeds) are calculated from a specific family of distributions dependant on the metric. The success rate uses a Binomial distribution because trials (for single seed) are either pass/fail situations. The convergence metric uses Gamma distribution and sparsity error uses Beta distribution. Both Beta and Gamma can easily approximate the normal distribution and support its corresponding metric.
5 Improving the Real NPU's Robustness
We first improve the robustness of the Real NPU against different training ranges. We use the Single Module Task with no redundancy (see Section 4 to investigate the following: (1) Is L1 regularisation required, and if so, do the regularisation parameters require tuning? (2) Does clipping the learnable parameters aid learning? (3) Does enforcing discretisation on parameters improve convergence? (4) Can the weight matrix initialisation be improved?
5

---

Published in Transactions on Machine Learning Research (09/2022)
Ixtrapolation range success rate
Estrppllaion
1.00
1.50
0.75
0.50
0.50
0.25
0.00
U[-0.2.-0.1 -0.10
Uf-1.2-1211
-1.1
U[-2-11
U1-2.2)
u(-20, 20--10)
(f0.1.0.2)
Ut1
U[1.1.1.3) 2.2.
Uf10
0.00
01-02-841
uf-222
uf-en- -10
ue U8-14.1
un.3
Un82200
uP-12
Interpolation range
Interpolasion range
(1-11.te-5)
(te-9Ie-1)
(1e-8.te4: ne-72e-5)
L1 off
L1 on
(e--51o-31
12-2.1e-1]
(2-1118)
(a) L1 regularisation
(b) Sweep over L1 (start,end) beta parameters
Figure 1: Exploring the effect and sensitivity of Ll regularisation on the Real NPU
To address each question, we apply incremental modifications to the Real NPU. Modifications include an ablation study on the L1 regularisation (including a sweep over the scaling range hyperparameters), clipping, enforcing discretisation, and more restrictive initialisation scheme. We assume that we are optimising the Real NPU to perform multiplication or division. Therefore, we trade-off the flexibility of having non- discretised weights, which enables the success of modelling the SIR data in Heim et al. 2020 Section 4.1), in favour of sparse models with discrete weight values. All the modifications can also be generalised for the NPU architecture. The ranges (Table 1 are influenced by the ranges from Madsen & Johansen (2020) as they provide good coverage.
Is L1 regularisation required? (Yes.) L1 encourages sparsity (i.e., zero weights) in solutions. Zero- valued weights means to select an input and return the identity value For the task, the optimal weight values require selecting all inputs and therefore non-zero values, suggesting the application of L1 could be damaging. Therefore, in Figure Ta Removing L1 proves to be detrimental in five of the nine cases shown and only shows minor improvements in two of the nine ranges (i.e., U|-1.2,-1.1) and U(1.111.2)). Hence, we keep L1 regularisation The L1 regularisation scaling (see Section 3.1. requires setting the hyperparameters for the start (8start) and end (Bend) scaling values. We run a sweep over six different start and end values, denoted (<start>, <end>), displaying results in Figure Tb. We find the configuration (1c-9, 1o-7) the most successful when considering performance on all the ranges, and larger scaling values perform worse
compare against model which does not L1 regularisation, shown
Does clipping the learnable paramcters help? (Yes.) Division and multiplication represented by weight. values of -1 and I respectively. The current architecture does not constrain the weights which can result in large weight values. The gate weights do get clipped and saved to another variable during the forward pass, moaning after an update step the gate values can also be out of the range [-1,1]. Hence, we investigate applying clipping directly to the weight and gate values after every optimisation step. Figure 2a shows clipping is beneficial, with clipping on both weight and gate (or just. on the weights) improve over the baseline on all ranges (excluding U(1,2) where the baseline has already achieved full success).
Does enforcing discretisation help? (Yes.) Modelling division in generalisable manner requires all learnable parameters to be discrete i.e., value from (-1, 0, 1). Using Madsen & Johansen (2020) 's regularisation scaling scheme (see Appendix B). we penalise weights for not being discrete. We modify the scaling factor to be == 1 and the regularisation to go from 'off' to 'on' between iterations 40,000 to 50,000. Figure 2b shows discretising the gate improves over the bascline but also discretising the weights
WW also experimented with using 1.2 regularisation but found to perform better (see Appendix

---

Published in Transactions on Machine Learning Research (09/2022)
Extrapolation range rate
Extrapolation range success
Extrapolation range success rate
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.25
0.25
0.25
0.00
U[-22]
U(-20. -101
Ujo. 1.0.1.0.3
0.00
0.00
U122
Utl
U[-0.2-0.1)
U1-1.2-1111
-1.11
U(-2.-11
U(-2.2)
UI-20 -10)
up Upp.1.021)
Uf1
Ull Ut10.
Uf-02, M1.02.0.11
UI-122-111 -211)
.0.11
UI-2,
-11
Ul-2.2)
Ul-20,
-10)
U00.1.0.2 2.)
Uf1.2]
Utz 1.1.2)
110.000
U1-00
U
Interpolation range
Interpolation range
Interpolation range
Xavier-Unffrrm
None
GW
None
GW
Xavier-Unilorm Constrained
(a) Clipping
(b) Discretisation regularisation
(c) W RE initialisation schemes
Figure 2: Elfect of clipping, discretisation and the NAU initialisation scheme on the Real NPU.
is additionally beneficial (especially for range u(-0.2,-0.1). 4(10,20) is the only range where the baseline outperforms using discretisation, succeeding on two additional seeds.
Does using a more constrained initialisation help? (Yes.) W uses a Xavier-Uniform initialisa- tion (Glorot Bengio 2010), meaning weights can be initialised out the range [-1,1]. Therefore, we use the initialisation for the Neural Addition Unit which constrained form of the Xavier-Uniform that does not allow the fan values of the uniform distribution to go beyond 0.5, meaning that weight value will be out of the range [-1,1] (Madsen & Johansen 2020). Figure 2c shows using the constrained initialisation provides improvements. For this Real NPU configuration learning rate of 50-3 works best (see Appendix E
6 Uniform Range Datasets
We now compare learning Table [ Uniform ranges on all modules including the NRU and NMRU for the no redundancy and redundancy setups. On the no redundancy setup (Figure 3 the NRU and NMRU achieve full success while solving the problem consistently fast and with low sparsity error, while the baseline Real NPU without modifications struggles with success on all ranges and with sparsity on the larger ranges. Applying the Real NPU modifications described in Section 5 deals with the sparsity issue and improves the robustness such that only range 2(22,2) struggles (with success rate of 0.64).
Introducing redundancy (Figure causes failure modes to arisc on all modules. The baseline Real NPU produces high sparsity errors relative to the other modules suggesting struggle with discretisation. The modified Real NPU improves over all ranges of the baseline (which were not already at full success) in terms of success, speed and sparsity (except for the sparsity in 4[[0020)). To ensure that complex weights do not fix the issue, we test the NPU module with all the modifications used on the real weight matrix but find significant improvements (see Appendix F). The redundancy affects the NRU the most, resulting in full failures all the negative ranges. The NMRU is the only module with success range 2(-2,2) due to its
sign mechanism (see Appendix Gb. It performs well over all ranges but can be outperformed by the modified Real NPU for negative ranges.
7 Mixed-Sign Input Datasets
The Uniform ranges results showed that the Real NPU (modified) and NRU have difficulty in learning when inputs can consist of arbitrary signed values (c.g. all positives, all negatives, or mixture of positive and negative values) such as 2(-2,2). We question if the failure is due to the input samples in batch having
different signs from each other, or if the problem is due to the fact data samples can be close to O (leading
7

---

Published in Transactions on Machine Learning Research (09/2022)
Extrapolation range success rate
Solved iteration step
Sparsity
1.00
40000
0.06
0.75
30000
0.04
0.50
20000
0.25
10000
0.00
0
0.00
U[-0.2.-0.11
U/-1.2,
2,-1.1
UI-2.
U(-2.-1
U1-2.2)
U/-20. -10)
Uf0.1002)
U(1.2)
U(1.1.1.21
U110,201
U[-0.2.-0.11
UI-1.2
UI-2.
U(-2.2)
UI-20.-11 -10)
Uj0.1.0.21
U(1.2)
ui (1..1.21
1000000
UI-02 21-022-000
UI-12
2.111
-
U1-2.21
UI-20-
1.0.1.0.
-101
Interpolation range
Real NPU (baseline)
NPu (modified)
NRU
NMRU
Figure 3: Division without redundancy (input size 2) on Uniform ranges.
Extrapolation range success rate
Solved at iteration step
Sparsity error
1.00
50000
0.15
0.75
40000
30000
0.10
0.50
20000
0.05
0.25
10000
0.00
0.00
U1-0.2,-0.1 -0.1)
UF-122--1 11-1.2.-111 111111
U1-2. -1]
U(-222
Ul-20, -10)
010.1,0.2)
Uj0.1.00
211.2)
U01.1 U02.1.1.2)
2/10200)
U1-0.22-0.11
Uf-1.2.-1...
U1-2-11
UI-2,2)
U[-20, -10)
Uj0.1.0.2)
Ul1.3]
t0.11 1.2)
2010.201
Ul-0.2,-40 40.1)
U[- U1-.22-111
2-111)
U[-2
2-1
U1-2.2)
U[-20- -10)
U10.1.0.2
U0.11133
10001 201
Interpolation range
Real NPU (baseline)
Real NPU (modifled)
NRU
NMRU
Figure 4: Division with redundancy (input size 10) on Uniform ranges.
to singularity issues). Five mixed-sign datasets which can control the range for each element in the input are used. The interpolation and extrapolation ranges can be found in Appendix C Datasets 1, 2, 4 and 5 sample a positive value for one input element and a negative value for the other element. Dataset 3 samples the signs randomly. Datasets and avoid sampling close to 0 values to mitigate the singularity issue.
Figure shows the Real NPU struggles all these ranges while the NMU and NMRU do not. This implies that the core issue for the Real NPU is not from different input samples having different signs or due to input values being close to 0. The underlying issue is most likely correlated to each element in input having different signs. When the denominator of the output is positive (datasct 1 or 2), the solution is found faster than when the denominator is a negative value (dalaset or 5). When the signs for an input element are controlled, discretisation is no problem, in contrast when the signs are arbitrary the sparsity enror are slightly (though not significantly) higher. Learning with input redundancy (Figure G), causes the Real NPU and NRU swap in performance. The Real NPU performs significantly better than the no redundancy task all ranges except 4((2222,, while the NRU longer works any range.
8

---

Published in Transactions on Machine Learning Research (09/2022)
Extrapolation range rate
Solved iteration step
Sparsity
1.00
7500
0.00075
0.50
5000
0.00050
0.25
2500
0,00025
0.00
0,00000
(EO- fro----n-- (2'rnn rain
aannanaa Un
w-22-01) S
(ro-z
te' (t-2-J 2-
Cee-in in
1ro (20-E-In (2t0on roi
(et & Pee2An
U--2-0.1)
ut
ur
Interpolation range
NPU (madified)
NRU
NMRU
Figure 5: Division without redundancy on the mixed-sign datasets that control the sign of the input elements. The ranges are in order of the datasets (i.e. dataset to 5).
Extrapolation range success rate
Solved at iteration step
Sparsity error
1.00
50000
0.75
40000
0.06
0.50
30000
0.04
20000
10000
0.02
0.00
0
0.00
JETOIn
1222 CG-1
110----n Beroonn
ce-in
Settaa (Et)) Un
6
2------- 3 ('rooln
1000--1
tn
1-1
Interpolation range
NPU (modified)
NRU
NMRU
Figure 6: Division with redundancy on the mixed-sign datasets that control the sign the input elements. The ranges are in order of the datasets (i.e. dataset to 5).
8 More Challenging Distributions: Larger Magnitudes and Mixed-Signs
To further stress test the modules, we explore the effect of larger Uniform ranges and different distributions (i.e., Benford and Truncated Normal). The ranges are found in Table E The Uniform ranges tests how mixed-sign dataset is influenced by larger ranges (with magnitudes of 50 and 100). The Benford distribution also tests learning on large magnitude values. Tt follows a more natural distribution compared to the Uniform, known to underlie real world data such as accounting data (Hill 1995). The Truncated Normal (TN) distributions also investigate mixed-sign datasets. Normal distribution allows to set biases via the mean value (and is to either -1, 1), while the truncation allows the extrapolation range not overlap with the interpolation range. Results for the 2-input and 10-input setting are shown in Figures 7 and
Larger ranges are found to be challenging when redundant inputs exist. On the 2- 2-imput setup, both NRU and NMRU have full success, while the Real NPU (modified) has failure cases for both Uniform distributions (with success rates of 0.72 on 2f(-100,100) and 0.76 on u(-50,50)). On the 10-input size setup, all modules fail for runs for both ranges.
Uniform distributions:
Additional discussion and experiments for learning on small numbers found in Appendix

---

Published in Transactions on Machine Learning Research (09/2022)
Table 2: Interpolation (train/valiin (rrain/valiaation) and extrapolation (test) ranges for different distributions. Data is drawn with the lower and upper bound ranges. TN Truncated Normal in the form TN(mean, sd)[lower bound, upper bound). Benford. U Uniform.
Interpolation
TN(-1, 3)(-5, 10)
TN(O, ()-5, 5)
TN(1, 3)(-10, 5) TN(10, 3) 15)
Extrapolation
TN((10, 3)(-15, -5)
[5,,,, 15)
Interpolation
B[10, 100)
Ul-100, 100)
U|-50, 50)
Extrapolation
B[100, 1000)
U[-000, -100) U [100, 200)]
U[[-100, -50) U [50, 100)]
Extrapolation success rate
Solved at iteration step
S parsity error
1.00
15000
0.20
0.75
0.15
10000
0.50
0.10
0.25
5000
0.05
0.00
8:110.100
THIO TN(-1 131---1-110
1h1--
TN(1 TNI1 3H(110.5)
U-j U-J-100,100 100)
U U:f-50.500
8:100.100
335--5100
111-5.5)
TN 31 -10.5)
u Uf-100. 1001
(05-50,50)
8:100.100)
TN(-1 TN(-1.3)1-55100 1.135--51100
TN10,11d-5.5
TNO,3) (1(3)-1-1.5)
U1-1000.10 100)
U11-50.50)
TN(-1 TN(-1.3)
3.33
TNO,
Interpolation range
Real NPU (modified)
NRU
NMRU
Figure 7: Division without redundancy on the Benford, Truncated Normal and Uniform distribution.
Extrapolation range
Solved iteration step
Sparsity
1.00
50000
0.20
0.75
40000
0.15
30000
0.50
20000
0.10
0.25
10000
0.05
0.00
0
0.00
8110.100)
TNL1 331---0555)
U:1-100,10 100)
U:[-50,50 50)
8:10100)
TNI-133M-5100 1.3M-5100
TNd 304-10,5
Us-1000 100)
Uil UHf-50,50)
8110.100)
TNI-1 (WI-1.3)--51100
TN(0111-555
U.i U:I-100.10 TNG.3) 3-15000055)
100)
U:f-50.50)
TM(O,1)
TN(O.
Interpolation range
Real NPU Imodified)
NRU
NMRU
Figure 8: Division with redundancy on the Benford, Truncated Normal and Uniform distribution.
Benford distribution: On the 10-input setting, the NRU and modified Real NPU have full success implying the Uniform distributions failures are due to using mixed-signed inputs rather than the large ranges. The NMRU shows majority failures (failure rate 0.84) suggesting that large ranges are also area of struggle.
Truncated Normal distributions: On the 2-input setup, both NRU and NMRU have full success but the Real NPU (modified) has failure cases for all three distributions (with success rales 0.48, 0.6, 0.64 respectively). When trained using the 10-input setup, both the NRU and Real NPU (modified) have no succoss. The NMRU's success rate greatly varies depending on the range (being 0.48, 0.04 and 0.92 for TN(-1, 3)|-5, 10), (0,11)) 5) and TN(1, 3)0- 5) respectively). This suggests the NMRU works better when a majority of the inputs are likely to have the same sign and struggles with values around zero
10

---

Published in Transactions on Machine Learning Research (09/2022)
B:(10,100)
TN(-13:1-5_10)
TN(0,1M-5,55
1.00
0.75
0.50
Sparsity failure
0.00
U:1-100,100)
U:[-50,50)
1.00
0.50
0.25
0.00
1e-5
1e-1
25e-2
1e-5
1e-1 Epsilon
25e-2
1e-5
1e-1
25e-2
Real NPU (modified)
NRU
NMRU
Figure 9: The sparsity failure rates (with 95% confidence intervals) for the division with redundancy task on the Benford, Truncated Normal and Uniform distributions.
8.1 Sparsity Failure Rates
For more insight as to how the failures are occurring, Figure shows the rate of non-discrete parameters which lie more than a given epsilon away from the nearest discrete value (-1, or 1) for the redundancy setup. The rates are calculated for model's best validation epoch (because of carly stopping) and are averaged over seeds and not just the successful seeds like in Figure S The high sparsity failure rales the most forgiving epsilon threshold (0.25) suggests that there was high uncertainty in the learnt parameters resulting them being unable to discretisc. This is evident for the NMRU the Benford and largest Uniform distributions. Distributions with high extrapolation success rates in Figure X is reflected by the low failure rate all the epsilons. In contrast, high reduction in failure over the epsilons occurs, then it suggests the discretisation is working but the values which parameters were converging to we not correct.
9 Traits of Modules when Learning on the Redundancy Setting
Gradient difficulties with the NRU. For insight to why the NRU performs poorly with input redundancy, we look al the gradients with respect to the weights. The partial derivative for the weights is,
Og tanh ((a000000000
Dw:
(8)
2000 sech((0000:) 2000 sech(1000w:;; (X)
The NRUxexi (x) (RUUxxX(xx)) subsequent update steps. Factoring out this term, the following observations are made: If x, as 0 and R 0 then gradients become increasingly large; 2, ss 0 and -1 I w; < 0 then as w; - -1 all gradients for x; where [xy] >> become increasingly small; the gradients for 24 = and x; = 1 are regardless of the value of wi; w; then the gradient is for all Hi result of using the tanh approximation; and, even if the sign and magnitude are calculated separately and then combined (see Appendix [J] to try control the gradient better, the problem remains. Therefore, conclude that extending the NMU to divide using weight of -1 is poor choice when there are redundant inputs.
term applies the NRU to all inputs excluding inftuencing the gradient values between
The Real NPU's and NMRU's exploitation of multiplicative rules. In the redundancy setting, modules with extrapolative solutions learn to exploit rules for multiplication. The NMRU exploits the inverse rule of division i.c., a; 1 1. Since the module's input contains the reciprocals numerous extrapolative solutions exist, however this comes the cost of finding a 'simple' solution containing non-zero weights only enabling non-zero weights if the
for relevant inputs. The Real NPU exploits the rules a; 0 0 and 1a:
11

---

Published in Transactions on Machine Learning Research (09/2022)
Table 3: Test accuracies of the output label for the MNIST task. The predictions and targets are rounded to d.p. before the accuracy is calculated. The mean accuracy over 10-folds is given with the standard error.
DIV
MLP
Real NPU (mod.)
NRU
NMRU
Test Acc. (5 d.p.)
97.497=0.183
0.004=0.004
97.147=0.242
94.215=3.627
14.69#13.841
NNRU
Roal NPU (nad)
accurcty (%)
Figure 10: Test accuracies on the d.p. output values.
corresponding gate value is 0. However, this can be avoided by allowing 0 to not be penalised during sparsity regularisation stage (see Appendix F): this alleviates the exploitation issue with no cost to performance.
10
MNIST Arithmetic
To determine if NALMs can learn in larger end-to-end networks we investigate learning to divide the labels of an image composed of two MNIST digits. Summaries of parameters can be found in Appendix D
10.1 Setup and Network Architecture
Following Bloice et al. 22021) the dataset contains permutation pairs of MNIST digits side-by-side with the target label being the product of the digits, e.g. input 4 with output 4(= 4 1). Importantly, although there is no overlap between the permutation pairs in the train and test set, all individual digits (between 1-9) are seen during training. E.g., the pair '54' would exist in the test set and not the train set but the digits '5' and '4' would exist in other pairs of the train set such as '15' or '47'. All instances zero are removed from the datasets to avoid division by zero case from occurring. The network learns a map from the input image to the labels of the two digits (digit classifier), followed by map from the two labels to their divided value (division layer). The digit classifier is convolutional network We separate the two digits to single digits, classify per digit and the recombine the two labels. There are three possibilities for the division layer: (1) a solved division baseline model (DIV), (2) MLP made of 2 hidden layers with 256 hidden units and ReLU activations and 1.2 regularisation, and (3) a NALM being either the Real NPU (modified), NRU, or NMRU. As the DIV baseline only requires learning to classify the images to their respective labels, considered a strong bascline. NALM should perform similar to the DIV bascline; if NALM outperforms the bascline it implies the NALM can also aid learning of downstream layers aswell as learn division.
10.2 Metrics and Results
The output accuracy is given based on the predicted and target values rounded to decimal places (d.p.) avoid issues caused by floating point rounding. Results are taken over 10-fold cross validation setting
@Taken from the PyTorc MNIST example http://:ww..cccccccccacaaaaaaaaa
12

---

Published in Transactions on Machine Learning Research (09/2022)
Table 4: Summary of the types division tasks the models can/cannot solve. Using redundancy means there are irrelevant inputs (10-input setup). The values are the mean success rate (out of 1) for the specific input task, bold values are the best model for the respective row.
Redun- Input type dancy?
Distribution
Real NPU
NRU
NMRU
Figure
(modified)
Mixed-signs
Uniform
0.56
1
1
Mixed-signs
Truncated Normal
0.57
Negative
Uniform
1
Positive
Uniform
1
No
Large magnitude Large magnitude Close to 0
Uniform
0.74
Benford
1
Truncated Normal Uniform
0.6
see [N(0,)) 5)
Close to 0
0
0.17
0.4
24
Mixed-signs
Uniform
0.77
0.99
Mixed-signs
Truncated Normal
0
0.48
Negative
Uniform
0.92
0.82
Yes
Positive
Uniform
1
Large magnitude
Uniform
0
0
Large magnitude
Benford
1
0.16
Close to 0
Truncated Normal
0
0.04
see (N(0))) 5)
and the NALM's initialisation is the same for each fold. Table 3 and Figure 10 displays the results. The MLP is not used in the violin plot so the distributions of the other modules can be better seen. The DIV baseline performs the best as expected since only the classification network requires to be learnt. The Real NPU (modified) has consistent accuracy on par with the DIV results. The NRU is less robust than the Real NPU (modified) but the better fold can outperform even the DIV. The NMRU performs the worst out of all NALMs struggling with robustness and the MLP is the worst division layer showing nearly no success across all seeds. More extensive experiments subject future work.
11 Discussion
Single layer division robustness. We summarise the key challenges for learning independent modules in Table 4 and give the ranges used to generate the values Table 10 In the no redundancy setting (2-inputs). the Real NPU challenged when the training data consists of mixed-signed inputs even with our applied improvements. Increasing the difficulty to have an input redundancy (with 8 redundant and relevant input values) improves performance when the Uniform distribution is used but magnifies the issue when ranges are samples from Truncated Normal distribution. The NRU and NMRU have strong performance across the no redundancy tasks but show failures when redundancy is included. In particular, the NRU loses its ability to learn successfully on most of the input settings. Negative ranges also becomes an issue for the NRU, in which we conclude is not wise to use with MSE. Alternate losses can improve certain failure cases though sometimes at. the cost of performance on other ranges. See Appendix M which displays results on a correlation and scale-invariant based loss. The NMRU drops most in performance on large magnitude datasets regardless the distribution. In the redundancy setup, the NMRU's robustness comes al the cost of the simplicity of the solution due to its exploitation of the identity rule; issue the Real NPU does not have, The Truncated Normal distribution causes the greatest learning difficulties to all the modules. Learning to divide values around zero remains challenging for all modules, even on the no redundancy setup, implying an alternate method for dealing with zero denominators should be open for exploration.
NALM can be used as part of larger networks. The MNIST experiment shows NALMs can act as downstream layers in non-trivial regression experiment which requires an intermediary classification network without direct classification loss. This is promising it implies uses of NALMs in more complex
13

---

Published in Transactions on Machine Learning Research (09/2022)
tasks, however two points of caution should be considered. Firstly, the results show that there is not direct correlation between the performance of NALM in the single layer tasks to their performance if embedded in larger networks. For example, the NRU and NMRU which outperform the Real NPU on the single layer tasks perform worse in the MNIST task. Secondly, if such units are to be utilised in larger embedded networks, we encourage performing lests in the larget domain before employing NALMs in the wild. Therefore, a future direction for this work and NALMs general includes developing more challenging experimental tasks with rigorous evaluations.
Number of learnable parameters. The NRU requires I x O parameters, the Real NPU requires I(O + 1) parameters and the NMRU requires 27 x O parameters. Although the NRU has the lowest parameter count, it performs the worst when redundancy is involved. The doubling of the input dimensionality in the NMRU results in more parameters, especially the output dimension is high. Additionally, as half the inputs of the modules require being inverted (which includes the irrelevant elements), scaling difficulties can arisc.
Two-layer learning. Once robust modules are attainable in single layer selting, the next step would be to question performance when learning stacked modules, e.g. learning a stacked additive and multiplicative module. Madsen & Johansen (2020) Figure 2) illustrates the troubles for multiplicative models with the capacity for division. They show how a stacked sumnative-uuliillcatiiv module can lead to an exploding loss when the output of the summative module is close to and the multiplicative model tries to divide. We recreate their setup (in Appendix N to produce loss surfaces for the NAU-Real NP NAU-NRU and NAU-NMRU respectively. A similar issue exists with the Real NPU and NRU which use a weight range of [+1,1], whereas the NMRU whose weight's range is limited to [0.1] does not have exploding losses.
Exploring alternate discretisation methods. We focus on using existing discrctisation techniques for NALMs to enforce weights loward values such as -1, 0 and 1. In future work, it would also be interesting to explore the effects of other types of regularisation influenced by the works from the network compression literature such as binary/ternary networks (Hubara et al. 2016 Zhu et al. 2017).
In conclusion, division remains a challenge to learn using interpretable neural networks, even for the simplest tasks. Nevertheless, by identifying the specific areas causing difficulty (e.g., training ranges), and useful architecture properties (e.g., using sign retrieval mochanism), we hope the community has better intuition for dealing with division and developing more robust specialist modules.
Acknowledgments
We would like to thank the anonymous reviewers who have help improve the manuscript. B.M. sup- ported by the EPSRC Doctoral Training Partnership (EP (EP/R513325/11 J.H. received funding from the EPSRC Centre for Spatial Computational Learning (EP/ /SS0006699)) The authors acknowledge the use of the IRIDIS High-Peeforrance Computing Facility, the ECS Alpha Cluster, and associated support services at the University of Southampton in the completion of this work.
References
Marcus Bloice, Peter M. Roth, and Andreas Holzinger. Performing arithmetic using neural network trained on images of digit permutation pairs. Journal of Intelligent Information Systems, 08 2021. doi: 10.1007/ s10844-001-006662-9 URL
William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabricio Olivetti de Franca, Marco Virgolin, Ying Jin, Michael Kommenda, and Jason H. Moore. Contemporary symbolic regression methods and their relative performance. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1),
Girish Chandrashekar and Ferat Sahin. A survey feature selection methods. Computers & Electrical En- gineering, 40(1):16 28, 2014. ISSN 0045-7906. doi: httprr//thaioggg///ooUooooooooop 1..... https: memorative issue.
/ http://wwwccococcoooo/oooa/aaaaa
40th-year com-
7The NAU is summative module Madsen & Johansen 2020
14

---

Published in Transactions on Machine Learning Research (09/2022)
Lukas Faber and Roger Wattenhofer. Neural status registers. CoRR, abs/2004.07085, 2020. URL https:
7/arxiv
.wriii.orrg/aoo org/ httgg/bb/2000..0000
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249 256. JMLR Workshop and Conference Proceedings, 2010. URL http://proceedingg mlr www.mml..r......... press/v9/
Poorot00a/glooo11000
Niklas Heim, Tomáš Pevný, and Václav Smídl. val Information Processing Systems, 33, 2020. URL https://papers nips cc/paper/2020/f11. pdf.
Neural power units. Advances in Neu-
Theodore Hill. A statistical derivation of the iggnficcattdigh law. Statistical science, pp. 354-363, 1995.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio, Bina- rized neural networks. In NIPS, pp. 4107 4107-4115, 2016.
URL
caaurallnetworke.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/ 141..6998., 2015. URL https://aaxix: orgoooggttt..ooo httg//df/14410 1412. 11441.6680.... 69f pdf
Michael Kommenda, Bogdan Burlacu, Gabriel Kronberger, and Michael Affenzeller. Parameter identification for symbolic regression using nonlinear least squares. Genetic Programming and Evolvable Machines, 21 (3):471 501, 2020. URL attps://ddo httg:/00..000/000000000007773330
John R Koza. Genetic programming mcans for programming computers by natural selection. Statistics and compaling, 4(2), 1994.
Andreas Madsen and Alexander Rosenberg Johansen. Measuring arithmetic extrapolation performance. In Science meets Engineering of Deep Learning at 33rd Conference on Neural Information Processing Systems (NewrIPS 2019), volume abs/1910.01855 Vancouver, Canada, October 2019. URL
pdf/1910.000 01w.8.p8888.88.. pdf
Andreas Madsen and Alexander Rosenberg Johansen. Neural arithmetic units. In International Conference on Learning Representations, 2020. URL httts::/ httt://oorr/.dddI1ggl0llaii.iee
Bhumika Mistry, Katayoun Farrahi, and Jonathon Hare. A primer for neural arithmetic logic modules. Jour- nal of Machine Learning Research, 23(185):1-58, 2022. URL
html
Subham Sahoo, Christoph Lampert, and Georg Martius. Learning equations for extrapolation and control. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4442 4450. PMLR, 10 15 Jul 2018a. URL https::/proceeedingg .ml.ggsmmrr... wwt..pres//////ahoollo html
Subham Sahoo, Christoph Lampert, and Georg Martius. Learning equations for extrapolation and control. In International Conference on Machine Learning, pp. 4442 4450. PMLR, 2018b.
Daniel Schlör, Markus Ring, and Andreas Hotho. inalu: Improved neural arithmetic logic unit. Frontiers in Artificial Intelligence, 3:71, 2020. ISSN 2624-8212. doi: 10.3389/fral http:/f.a.2220000../00 URL httpt::/t/wee
frontiersin org/ artice/101 wttp://ee.......fraaa.ea 2020. 00071
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. science, 324 (5923):81-85, 2009.
Andrew Trask, Felix Hill, Scott E Rood, Jack Rac, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. In Advances in Neural Information Processing Systems, pp. 8035-8044, 2018. URL https:
15

---

Published in Transactions on Machine Learning Research (09/2022)
Silviu-Marian Udreseu and Max Tegmark. Ai feynman: A physics-inspires method for symbolic regression. Science Aduances, 6(16), 2020. doi: (0112//sciavv.aay//. URL. https::/advancess.sett http://ddvanccssssseemmmmeee org/
content/6/16/eaay2631
Chenzhuo Zhu, Song Han, Huizi Mao, and William J. Dally. Trained ternary quantization. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.ne,, 2017. URL http://ppaarrrreeeeeeeeeorrmnaaa
16

---

Published in Transactions on Machine Learning Research (09/2022)
Appendix
Table of Contents
A Properties of Division Module
18
B Neural Addition and Neural Multiplication Units' (NAU NMU)
20
Experiment Parameters
21
Parameter Initialisation
22
D Hardware and Time to Run Experiments
23
Real NPU; Single Module Task (without Redundancy): Additional Experiments
24
Real NPU; Single Module Task (with Redundancy): Additional Experiments
25
G NMRU; Single Module Task with Redundancy: Additional Experiments
26
H NRU; the Single Module Task (no Redundancy): Effect of Learning Rate
NRU; Single Module Task (no Redundancy); Tanh Scale Factor
29
NRU; Single Module Task (with Redundancy): Calculating the Sign Separately
30
K
Division by Small Values:
K.1 Impact of Singularity Issue on Gold Solutions
K.2 Experimental Results
31
Ranges Used for the Single Layer Task Summary Table
M Effect of Different Losses on the Single Module Task (with Redundancy)
34
RMSE Loss Landscapes
17

---

Published in Transactions on Machine Learning Research (09/2022)
A Properties of a Division Module
When building division module, the following properties should be included:
Ability to multiply: Without multiplication the module limited to expressing reciprocals
Interpretable weights: A good division module should produce generalisable solutions to out-of-bounds data. Using interpretable weights to represent exact operations is way of doing so, c-g., -1 to divide, 1 to multiply, 0 to not select. For the scope of this paper, focus discrete weights, however fractional weights can also be considered interpretable. For example, the Real NPU can express / using a weight value of -0.5.
Calculating the output: This can be decomposed into three tasks: magnitude calculation, sign caleulation and input selection.
Refers to calculating the output value for a calculation. This is achieved using discrete weight parameters. For example, the Real NPU and NRU use a weight value -1 for calculating reciprocals of selected input and 1 for multiplication, while the NMRU uses for sclecting an input element resulting in either mulliplication reciprocal depending on the weight's position index.
Magnitude calculation:
Sign of the output: Calculating the sign value (1/-1) of the output can occur at element level in which the sign is calculated for each intermediary value as each input element is being processed, or at the higher input level in which the sign is calculated separately from the magnitude and then applied once the final output magnitude is calculated. The NRU uses the prior method while the Real NPU and NMRU use the latter method. If an input is 0 or considered irrelevant then the output sign will be (Ablation studies on the NMRU, Figure
15 suggest the latter option which separately calculates the sign to be more beneficial).
The Real NPU and NMRU use the cosine function to calculate the final sign of the module's output neuron. Below shows the state diagram of how the sign value (i.e., the state) of the output would change depending the imputs and relevant parameters being processed. We only consider the discrete parameters for simplicity. Both the Real NPU and NMRU use the same state diagram but have different conditions for a state transition to occur.
b(s) 1 a(s)
b(s)
The conditions for the Real NPU transition functions a(s) - and b(s) 8, where is the state value -1, 1, are defined as follows:
a(>) w.: < 0i,o, E 11 b(s) NT; 2 OV W1oo = 0 V = -
Transitioning from one sign to another only occurs if the input element (xi) is negative and is considered relovant i.c. the gate (g:) and weight value (wi.o) is non-O. In contrast, to remain at a state requires either the input element to be 2 or not be considered relevant.
The conditions for the NMRU transition functions a(s) = -s and b(s) = s, where is the state value -1, or 1, are defined as follows:
(s)) :a; 0 W1o0 11, b(s) n.; 0 Wioo 0.
Transitioning from one sign to another only occurs if the input element (x;) is negative and is considered relevant İee. the weight value (wi.,) is 1. To remain at a state requires either the input element to be 2 0 or the weight value to not select the input.
Selection: Not all inputs are relevant for the output value. To process any irrelevant input elements can be ddiiiiiiiioooiiiiiiiniioooo
interpreted converting to the identity value of multiplication/ division (=1). The identity property means
18

---

Published in Transactions on Machine Learning Research (09/2022)
that any value multiplied/diviiee by the identity value remains at the original number. Hence, irrelevant inputs are converted into 1 (rather than being masked out to 0). For the multiplication case, this stops the output becoming O, and for division it avoids the divide by 0 case, For all the explored modules, a weight value will deal with the irrelevant imput case. However, the Real NPU goes a step further by also having an additional gale vector with the purpose learning to select relevant inpuls. Such gating has been proven be helpful for an NPU based module (Heim et al. 2020), but may not be necessary when dealing with weights between [0,1] like in the NRMU (see Appendix G
19

---

Published in Transactions on Machine Learning Research (09/2022)
B Neural Addition and Neural Multiplication Units' (NAU & NMU)
Madsen & Johansen 2020) develop two modules: one for dealing with addition and subtraction (the NAU) and the other for multiplication (the NMU). NAU output element is defined as
NAU do Σ (W xx)
(9)
where I is the number of inputs. The NMU output element mo is defined as
NMU = II (Wi. x; + 1- -,WW),
(10)
Before passing input through module, the weight malrix is clamped to [-1,1] for the NAU or [0,1] for the NMU. Weights are ideally discrete values, where the NAU is 0, 1, or -1, representing no selection,
addition and subtraction, and the NMU is 0 or 1, representing no selection and multiplication. To enforce discretisation of weights both units have a regularisation penalty for a given period training. The penalty
is
ΣΣ min (IWi,),11 1 (W
(11)
T
where O is the number of outputs and À is defined
min (ileration; - Astart .))
x= x max
(12)
Xend Astart
Regularisation strength is scaled by a predefined Å. The regularisation will grow from to X between iterations Astart and Aend, after which it plateans and remains at X.
The iterations for switching on and scaling the discretisation regularisation are chosen to allow weights have the opportunity to explore and begin move far enough the ideal discrete value (within 0.5), therefore the carliest regularisation is only switched results, we have shown this works empirically. As the warmup scaling used for the regularisation occurs over many epochs, there is some leniency in the epochs to start and end the scaling without significantly impact success. In some cases, it could be possible to tune the start and end epoch for faster convergence but this was out of the scope for this study.
after 50% of the epochs have occurred. From
20

---

Published in Transactions on Machine Learning Research (09/2022)
C Experiment Parameters
Tables and
0 for the breakdown of parameters used in the Single Module Tasks. The paramcters of Table 0
are laken from
Heim et al. 2020 Section 4.1) which we confirm work empirically in Figure 1b
Table gives the interpolation and extrapolation ranges used in the mixed-sign datasets tasks.
Table shows the breakdown of parameters used the MNIST experiments. All experiments for the MNIST Tasks were trained using single GeForce GTX 1080 GPU.
Table 5: Parameters which are applied all modules. Parameters have been split based on the experiment. "Validation and test datasets generate one batch of samples at the start which gets used for evaluation for all iterations. the Real NPU modules value of 1.
Parameter
Without redundancy
With redundancy
Layers
1
1
Input size
2
10
Total iterations
50,000
100,000
Train samples
128 per batch
128 per batch
Validation
10000
10000
samples'
Test samples'
10000
10000
Seeds
25
25
Optimiser
Adam (with default parame- Adam (with default parame- ters)
ters)
Xt
10
10
Table 6: Parameters specific to the Real NPU modules for the Single Module Tasks.
Parameter
Value
(BstartBend))
(1e-9,1e-7)
ggrouth
10
Satop
10000
Table 7: Mixed-Sign Datasets:
The interpolation and extrapolation ranges to sample the two input elements for single data sample. The target expression to learn is: input 1 + input 2.
INTERPOLATION
EXTRAPOLATION
Dataset
Input
Input
Input
Input
U[-2, -0.1)
U(0.1, 2)
Ul-6, -2)
U[2, 6)
2
-l-
U[1, 2)
U|-6, -2)
U[2,
3
U[-2, 2)
U[-2, 2)
Ul-6, -2)
U[2, 6)
4
U(0.1,
U[-2,
U[2, 6)
U[-6, -2)
5
U(1
U(-2,-1)
U[2 6)
U[-6, -2)
21

---

Published in Transactions on Machine Learning Research (09/2022)
Table 8: MNIST experiment parameters.
Parameter
Two digit MNIST
Epochs
1000
Samples per permuta-
1000
tion
Train:Test
90:10
Batch Size
128
Train samples
72,000 (1 fold)/73,000 /70,00 (9 folds)
Test samples
9,000 (1 fold)/8,000 (9 folds)
Folds/Seeds
10
Optimiser
Adam (with default parameters)
Criterion
MSE
Learning rate
1e-3
Astart Xend epochs
30-40
2
grad norm clip
MLP None; Real NPU None; NRU 1; NMRU None
C.1 Parameter Initialisation
We give the initialisations used on the different module parameters:
Real NPU: The real weight matrix uses the Pytorch's Xavier Uniform initialisation. The gate vector initialises all values to 0.5. (This is the same initialisation used in Heim et al. (0200.)
NPU: The imaginary weight matrix initialised to 0. The rest of the parameters are initialised same as the Real NPU. (This the same initialisation used in Heim et al. (2020).)
NRU: The weight matrix uses Xavier Uniform initialisation which have maximum range between -0.5 to 0.5 (depending the network sizes). (This is the same initialisation the Neural Addition Unit uses
[Madsen & Johansen 2020
NMRU: The weight matrix uses a Uniform initialisation which can have a maximum range between 0.25 to 0.75 (depending on the network sizes). (This is the same initialisation the Neural Multiplication unit uses
Madsen & Johansen 2020
22

---

Published in Transactions on Machine Learning Research (09/2022)
D Hardware and Time to Run Experiments
All experiments were trained on the CPU, as training on GPUs takes considcrably longer. All Real NPU experiments were run on Iridis (the University of Southampton 's supercomputer), where a compute node has 40 CPUs with 192 GB of DDR4 memory which uses dual 2.0 GHz Intel Skylake processors. All NRU and NMRU experiments were run on a 16 core CPU server with 125 GB memory 1.2 GHz processors.
Table displays time taken for each experiment run single seed for single range. Timings are based on single rather than the runtime of a script execution because the queuing time from jobs when executing scripts is not relevant to the experiment timings. For single model, single experiment would have 225 runs (for 9 training ranges and 25 seeds).
Table 9: Timings of experiments.
Experiment
Model
Approximate time for completing 1 seed (mm:ss)
Real NPU
03:20
No redundancy (size 2)
NRU
02:00
NMRU
03:00
Real NPU
05:30
With redundancy (size 10)
NRU
05:00
NMRU
05:15
23

---

Published in Transactions on Machine Learning Research (09/2022)
E Real NPU; Single Module Task (without Redundancy): Additional Experiments
Figure 11 shows results of using the NPU for the 2-input task. Of the 9 tested ranges, L2 has a lower success rate than L1 for 5 ranges and has the same success rate for the remaining 4 ranges. If L2 regularisation is used instead of no regularisation, it performs worse in 3 (of the 9) ranges, better on 3 ranges and the same on the remaining ranges.
Extrapolation range success rate
1.00
0.75
0.50
0.25
0.00
U1-122
U[-2-11
U[-2.2)
U[-20.-10)
Ul0.1,0.2)
U(12)
Uf1.1.1.2)
U(10,20)
Interpolation range
No reg
L1
L2
Figure 11: Applying regularisation, L1 regularisation and L2 regularisation to enforce sparsity in weights.
Figure 12 displays the effect of different learning rates for the modified Real NPU. A learning rate of 5e-3
has the best performance over all ranges.
Extrapolation range success rate
Solved at iteration step
Sparsity error
1.00
30000
0.00075
0.75
20000
0.00050
0.50
0.25
10000
0.00025
0.00
10
0.00000
1.11
10)
11111 tin
11
2I-2.
11-2.2
Ur-20-11
-0-0--00)
U.0.1.0.22
(tti
U(-0.2. -0.1)
U1-122-11.
--11
-1
U10.1.0.3
U(1.))
U12. (1.11
0[10.20)
uI-022
Ujo
U1-20
Interpolation range
5e-1
5e-2
5e-3
5e-4
Figure 12: Different learning rates the Real NPU (mod) for the Single Module Task (no redundancy).
24

---

Published in Transactions on Machine Learning Research (09/2022)
F Real NPU; Single Module Task (with Redundancy): Additional Experiments
We test the NPU module with all the modifications used on the real weight matrix. Also, assuming the global solution only uses the real weights, we enforce the complex weights to be clipped between [-,]]] and to go to during the regularisation stage using a L1 penalty. Figure 13 shows the complex weights without any constraints hinders success and convergence speeds negative ranges. Applying clipping and regularisation constraints does not result any significant improvements against the Real NPU results.
Extrapolation range
Solved iteration step
Sparsity
1.00
60000
0.75
0.15
40000
0.50
0.10
0.25
20000
0.05
0.00
0.00
UI-22.111 2211111
0.11
U(-2. U[-2.11
U1-2,2)
Uf-20- -10)
1.022
U11.2)
U(1.1.1.2) 21
U/10,20)
U1-2--1
U(-2,2 2)
U/-20- -10)
Uja.1 1.0.1
U[1.]
iin 1.1.2)
U1-02.-0000
UI-12.111
-0.31
(ll --11
-100
0j0.1.0.21
U11.2]
U[1.1 Uf1.1.1.21
U(30,20)
Vio.
UI-20.
Interpolation range
Real NPU (modified)
NPU (no constraints)
NPU [clip reg)
Figure 13: Adapting the Real NPU use complex weights (NPU) on the Single Module Task with re- dundancy. Compares the NPU architecture with the Real NPU modifications (i.c. NPU (no constraints)) and the same model but with the imaginary weights clipped to [-1,]] and LI sparsity regularisation the complex weights (i.e. NPU (clip & reg)).
Figure 14 shows how modifying the Real NPU's weight discretisation to not penalise weights at 0 does not affect success.
Extrapolation range rate
Solved iteration
Sparsity
1.00
0.100
40000
0.75
30000
0.50
20000
0.050
0.25
10000
0.025
0.00
0.000
U(-0.2,-0.1) 0.11)
U1-1.2.111
-111
U(-2. -1
(1-2,2)
UI-20. -10)
Uj0.1.0.2) 1,0.2)
U(1.))
UfL.1.1.2)
UI-022-0.11
UH-122
-.11)
-1-2-111
U(-222
UI-20
-101
010.1.0.2)
Ufa.21
ul1.1. 1.1.22
203
U1-0.2, 0.11
U(-1.2.-1
-1.11)
U(-2,2
Uf-20-1101
-101
U10.1.0.22
Uf1.2)
1f1.1.1.2)
U[10,20)
Interpolation range
F11)
(-1.0.1)
Figure 14: Comparing weight discretisation the NPU weights which penalises not having weight of (-1,1} 1, 1).
25

---

Published in Transactions on Machine Learning Research (09/2022)
Extrapolation range success
Solved iteration
Sparsity
1.00
60000
7.5e-08
0.75
40000
5.0e-08
0.50
0.25
20000
2.5e-
0.00
0.0e+00
01-0.2.
Ui-12
0..1
UI 5 1
11-20
U10.1...
-101
u(-0.21 111
Ult
Ul-22)
Ui-20. -10)
Ujo. 1.0.21
U(L.2]
Mill- (.1.1.
U(-0.2, 0.11
U1-1.2.-1
1.11
Uf-2.
U(-2.11
U(-2.2
U/-20. -101
U10.1.0.21
((1.2)
UR..1.1.2)
U(10,20]
Interpolation range
vanilla
gnc
sign
gnc+
gnctsign-ggting
Figure 15: Ablation study for the NMRU,
G NMRU; Single Module Task with Redundancy: Additional Experiments
This section further explores the NMRU architecture.
Figure 15 shows an ablation study on different components of the NMRU architecture. Removing both the sign retrieval and grad norm clipping performs poorly over majority of ranges (including positive ranges). Gradient norm clipping alone is unable to solve the issue in learning negative ranges, however fully succeeds on the U(-2,2) range. Using the sign retrieval without the gradient clipping gains successes for the negative ranges, though performance on 2(2 affected. However, including both gradient clipping and sign retrieval results in separating the calculation of the magnitude of the output and its sign while having reasonable gradients, gaining the most improvement
the vanilla NMRU. Further including learnable gate vector (like the Real NPU), which is applied to the input vector, hinders performance. The largest solved at iteration step seems to be bounded at approximately 50,000 iterations which correlates to the point at which the sparsity regularisation begins, which highlights the importance of discretisation. Even with the different ablations, the sparsity crrors of the successful scods remain extremely low (which is not always the case for the Real NPU (see Figure
Figure 16 shows the effect of using different learning rates on the NMRU (with grad norm clipping and sign retrieval) using Adam optimiser. Too low learning rate struggles the mixed-sign range 41-2,2). Too high learning leads to no success on multiple ranges.
Extrapolation range success rate
Solved at iteration step
Sparsity error
1.00
60000
0.75
40000
4e-04
0.50
0.25
20000
2e-04
0.00
0e+00
110.
0011
ul-20-
(0-10000
1.0.1.0.1
Tr tin
1111 in
111
In
02-10
11.15 1Jn
1L-22-
un
Interpolation range
le-3
1e-2
le-1
Figure 16: Effect of different learning rates on the NMRU
26

---

Published in Transactions on Machine Learning Research (09/2022)
Figure Lr compares training the NMRU with either Adam and SGD optimiser. As expected, Adam outperforms SGD in all ranges (except two, where both perform equally). This difference in performance can be accounted for by Adam's ability to scale the step size of each weight, which can complement the clipped gradient norm of the NMRU, contrast to the SGD's global step size.
Extrapolation range success rate
Solved iteration step
Sparsit error
1.00
8e-05
60000
0.75
60-05-
40000
0.50
4e-05
20000
2e-05
0.00
0e+00
01-022-0.11
U1-12.-1
-1.1
01-3
(11-2.31
UI-20.
01-20--101
U10.1.0.21
-101
Uf1.1.1.3)
U1-0.2.-01 -011
U1-12,-11
-1111
U(-2.11
U1-221
U/-20-110 -10
vi0.1.0.3
11.12.1111
-1111
U[-2-1
U1-2.31
UI-20.-10;
UI0.1.0.23
U11.2]
Uf1.1.1.3
Interpolation range
sgd
adam
Figure 17: Effect of optimiser on the NMRU, SGD Stochastic Gradient Descent.
27

---

Published in Transactions on Machine Learning Research (09/2022)
H NRU; the Single Module Task (no Redundancy): Effect of Learning Rate
displays the effect of different learning rates for the NRU. A learning rate of gets full success on all ranges with performance deteriorating as the learning rate reduces.
Figure 18
E Extrapolation range success rate
Solved Iteration step
Sparsity emoo
1.00
2.0e-07-
0.75
30000
1.5e-07
0.50
20000
1.0e-07
0.25
10000
5.0e-08
0.00
0.0e+00
11-1112-111
--1
U1-2-3
Un 01-20:11
110.1.0.
Uf-0.2.-0.
U(-1.2.-1.
-1.1)
Ul-2--1
Ul-2.2)
U(-20.-
Ul-20-110)
ujo.1 U10.1.0.2) 1.0.2)
UI1.21
uf1.1.122
u(10220)
U(-0.2, Uf-0.2.-0.11 0.11
U(-1.2.-1
U(-2--11
U[-2.2)
UI-20.-10
U10.1.0.21
-101
U(1.2)
Uf2.1.1.2)
Uj10,20)
Interpolation range
1e-3
1e-2
1e-1
Figure 18: Different learning rates on the NRU for the Single Module Task (no redundancy)
28

---

Published in Transactions on Machine Learning Research (09/2022)
NRU; Single Module Task (no Redundancy); Tanh Scale Factor
Figure 19 shows the impact of changing the tanh scale factor. We find lager scale factors work better with
a factor of 1000 being the best. This correlates to the findings in Faber & Wattenhofer 2020 Figure 5).
Extrapolation range success rate
Solved at iteration step
Sparsity error
1.00
25000
20000
0.75
6e-08-
0.50
10000
4e-08-
0.25
5000
2e-08
0.00
0e+00
102
Uf-0.2.001
U(-122.-11
-1.1)
U1-2.-1 -1
U1-2.-)
01-20-1100
U[0.1.0.3)
U[1.2]
Uf1.1.1.3)
Ur-022-000
2-111)
U1-2-11
UI-2a1
((-20--10)
v(0.1.02)
01-02.-0.11 -0.1
-.11
-11
10010000
ui
0I-12
Interpolation range
tanh scale
10
100
1000
Figure 19: Effect of the tanh scale factor for the NRU on the 2-input setting.
29

---

Published in Transactions on Machine Learning Research (09/2022)
J NRU; Single Module Task (with Redundancy): Calculating the Sign Separately
The 'separate NRU' modulc calculates the magnitude and sign separately and then combinos them using multiplication together once all input elements are accounted for. The following definition is used to calculate a NRU with separate magnitude and sign calculation,
2 II (Ix² Wi, |Weol + - -W II sign
) round(W..))
(13)
Figure 20 shows results, where the separate sign method shows no difference success to the original NRC architecture.
Extrapolation range success rate
Solved at iteration step
Sparsity error
1.00
8e-05
15000
0.75
6e-05-
10000
0.50
4e-05
0.25
5000
2e-05
0.00
10
0
0e+00
U[-0.2,-00
Uf-122-111
-0.1)
-1.11
U1-21
-1-11
11-2--1
UI-2a
ul-20. -10)
U10.1.0.2
UtL.1
1.02--011
U1-2221
Uf-20. -101
U10.1.0.33
([1.2]
U[2 (11.1.1.2)
U(10,20)
11-20
1.0.2.2..
Interpolation range
together
separate
Figure 20: NRU on the redundancy experiment comparing a module which calculates the magnitude and sign together vs calculating the magnitude and sign separately and then combining them.
30

---

Published in Transactions on Machine Learning Research (09/2022)
K Division by Small Values:
The discontinnous nature of division at zero results in the inability to provide computational value for the sulput/gradient and causes neighbouring values to have large gradients. To understand the extent of this issue when learning, we explore learning to divide by values close to zero using three tasks with increasing difficulty: 1) learning to take the reciprocal of a single imput, 2) taking the reciprocal of the first input given two imputs, and 3) dividing the first input by the second given two inputs.
K.1 Impact of the Singularity Issue on Gold Solutions
Figure 21 plots the test error assuming the module weights are set to the 'gold' solution for the three tasks. As the range values become closer zero, the test error thresholds become increasingly large. Therefore, even with the correct weights, relying on the test errors alone an indicator become increasingly deceptive with values close to zero. The Real NPU has larger test errors for all tasks and ranges, caused by adding to the imput (see Equation 3/. Setting
0 reduces the test error at the cost of the ability to deal with zaro-valued inputs. Appendix K.2 provides the corresponding experimental results finding that only modelling reciprocals can be learnt with extremely small values.
Input: (a] Output: 1/a
Input: [a.b] Output: 1/a
Input: (a.b]: Output: a/b
1e+17
1e417-
1e+17
log(Test emor
1e+09
1e+09
1e+09
1e+01
1e+01-
1e401
1e-07
1e-07-
1e-07
U1O, U10.1e-8)
U10, 18-7)
U10.1e-0 ()
UID. 1e-5)
UIO. 16-4)
U10. U10, -21
UtO, le-1)
U10, 40-
UJ0 16-8)
U10.1e-7)
UIO, 16-6)
Ula. Ulo.le-5)
ulo Uro, 1e-3)
Uto. V10. 1e-1)
1e-2)
U10. tev0)
V10 V10.1e-0)
UID, 1e-7)
V10.1e-5 .e-5)
U10.1e-4t (e)1))))
UIO 10-1)-
U10. (0.1)-2)
UID. 1e-1)"
U10 U10.1e+0)
Uto.
Data sample range
ReaiNPU
PealN (eps=0)
NRU
NMRU
Figure 21: Effect of the singularity issue on the Real NPU, NRU and NMRU over increasing input ranges. Left: Reciprocal for an input size of (no redundancy). Middle: Reciprocal for an input size of (with redundancy). Right: Division for an input size of 2 (no redundancy).
K.2 Experimental Results
This section shows the results on trying to learn the reciprocal/division of values close to zero using the Real NPU, NRU and NMRU. We train and test on the ranges where the lowest bound is 0 and the upper bounds are: 1e-4, 1e-3, 1e-2, 1e-1 and 1. Unless stated otherwise, the hyperparameters what is used for the Single Layer Task without redundancy. The first task runs for 5,000 iterations with no regularisation for any module. The second and third tasks both run for 50,000 iterations.
model are set to
Due to precision errors, solution with the ideal parameters will not evaluate to MSE of 0. Therefore, we calculate thresholds which the test MSE should be within. A threshold value for task is calculated from evaluating the MSE of each range's test dataset for each module, using the 'golden' weight values and adding an epsilon term to the resulting error which takes into account precision errors. All experiments are run using 32-bit precision.
@The term the Pytorch default eps value, borch.finfo()epp
31

---

Published in Transactions on Machine Learning Research (09/2022)
In general, successful runs take longer to solve as the input ranges become smaller. The simplest task, of taking the reciprocal when the input size is 1 (Figure 22 is achieved with ease for all modules, though for 1e-44,,4), we find the NRU begins to start struggling.
S Success rate
S olved at iteration step
Sparsity emoo
1.00
1.05e-16-
0.75
900
7.75e-17-
0.50
600
5.00e-17
0.25
300
2.25e-17-
0.00
0
-5.00e-18-
U10,0.0001]
U(0,0.001)
010,0.01)
Uj0.0.1)
U(8.1)
U10.0.0001)
010,0.001)
v10.0.11)
U10.0.1)
U10,1)
u10.0.0001)
U(0.0.001)
U10,0.01)
U(0.0.1)
U10.1)
Ul0.0.
Interpolation range
model
Real NPU
NRU
NMRU
Figure 22: Input: [a], output
Learns reciprocal when there is no input redundancy,
Introducing a redundant input (Figure 23 greatly impacts performance with only the NMRU able to achieve reasonable success for the larger ranges, The successes shown for the Real NPU at range 0d[0, 1e-4) are false positives caused by the in the architecture used for stability. Test false positives can also be indicated by the high sparsity error of the weights.
Solved iteration step
parss ity error
1.00
25000
20000
0.09
0.75
15000
0.06
0.50
10000
0.25
5000
0.03
0.00
0
0.00
0001)
U10.0.001)
U(0.0.01)
U10,0.1)
U10 1)
ujo 000.0.0001)
010,0.011)
010.0.01)
U(0,0.1)
U10.1)
Ujo 010.0.0001)
U10,0.001)
U10.0.01)
U10.0.1)
U10.1)
Interpolation range
model
Real NPU
NRU
NMRU
Figure 23: Input: [a,b], output a- Learns reciprocal of the first input when there is redundancy.
Modifying the task to division (Figure 24 mcaning the redundant input now relevant, shows improvement for the NMRU and NRU for the larger ranges.
32

---

Published in Transactions on Machine Learning Research (09/2022)
Success rate
Solved iteration step
Sparsity
1.00
10000
5e-08-
0.75
7500
4c-08
3e-08-
0.50
2e-08-
0.25
2500
1e-08-
0.00
0e+00
U10.0.0001)
Uj0.0.001)
010.0.011
U10,0.11
U10.1]
0j0.0.00011
010.0.001)
010,0.111
u100.3)
ujo.1)
uj0.0.0001)
0j0,0.001)
U10,0.011
U10,0.1)
U10.1)
Interpolation range
model
Real NPU
NRU
NMRU
Figure 24: Input: [a,b], output g. Learns division of the first and second value when there redundancy.
L Ranges Used for the Single Layer Task Summary Table
Table 10 shows the ranges used lo generate the summary statistics. Note that even though the interpolation ranges are given (to make it easier to compare against the relevant Figures), it is the success rate on the extrapolation range which is used the table summary.
33

---

Published in Transactions on Machine Learning Research (09/2022)
Table 10: The relevant ranges used to calculate the summary statistics Table
Redun- dancy?
Input type
Distribution
Figure Interpolation Ranges
sixed-signs
Uniform
All 5 ranges: U(-2,-0.1) & U(0.1,2), U1-2,- 1) & U(12),, U(-2,2), U(0.1,2 & 0(-2,0.1) and U(1,2) & U(-2,-1)
-ixed-signs
Truncated Normal
All 3 TN ranges: TN [-5,10),
(N(0,1):(-5.5) and (N(1,3)
Negative
Uniform
3
Only pure negative ranges: U[-0.2,0.1), Uf-1.2-1.11, Ul-2,-1) and Ul-20,-10)
No
Positive
Uniform
3
Only pure positive ranges: U(0.1,0.2), U(1,2), U(1.1,1.2 and U(10,20)
Large magnitude
Uniform
Ul-100,100) and Ul-50,50)
Large magnitude
Benford
B[10,100)
Close to 0
Uniform
24
All 5 ranges: U(0,0.0001), Uj0,0.001), U(0,0.1),, U(0,0.1) and U(0,1)
Close to
Truncated Normal
TN(0.1) 5)
Mixed-signs
Uniform
6
All ranges: Ul-2,-0.1) & U(0.1,2), UI-,,-
1) & U(1,2), Ul-2,2), U(0.1,2) & U(-2,-0.1) and U(1,2) U(-2,-1)
Mixed-signs
Truncated Normal
All 3 TN ranges: TN(-1,3): (-5,10),
IN((01):{-555) and (--1,3)
Yes
Negative
Uniform
Only pure negative ranges: U(-0.2,-0.1), U(-1.2,1111), U(-2,-1) and Uf-20,-10)
Positive
Uniform
Only pure positive ranges: U(0.1,0.2),
U(1,2), U(1.1.1.2) and U[10,20)
Large magnitude
Uniform
U(-100,100) and Ul-50,50)
Large magnitude
Benford
B[10,100)
Close to
Truncated Normal
TN(0,1)(-5,) 5)
M Effect of Different Losses on the Single Module Task (with Redundancy)
Different losses induce different loss landscapes impacting the areas of success for module. We explore the effects of three different losses including the MSE, Pearson's Correlation Coefficient (Equation 15), and the Mean Absolute Procision Error (Equation 16). We use the division task with inputs. The propertics of each loss summarised in Table II All experiment parameters match the original MSE runs in the main experiments. The only difference is the loss used.
34

---

Published in Transactions on Machine Learning Research (09/2022)
Table 11: The properties of different loss functions.
MSE
PCC
MAPE
Batch mean
Standardisation
Difference of prediction from target Projection
Mean centering
Extrapolation range success rate
Solved at iteration step
Sparsity error
1.00
60000
0.100
0.75
0.075
0.50
0.050
20000
0.25
0.025
0.00
0.000
U[-1.2,11.1
-0.11
-1-11
UI-2.-1
Ul-2.2)
Uf-20,
U10.1,0.2)
U(L2)
U(1.1.1.2)
(]30,20)
Uf-0.2. -011
Uf-122, -1.1)
Uj-2,-11
U(-2.21
Ul-20
-00
0.0.1.0.2)
Uf2.21
U11.1.
1.1.3
U(-1.2.111
0.11
-1.11
01-2.-1
-1
U(-2221
UI-20.
-10
Ul0.1.0.2)
U(1.2]
U11.1 (11.1.1.2)
U[10,20]
U1-0.2,
Uf-022
Interpolation range
MSE
Figure 25: Single Module Task with redundancy the Real NPU, comparing different loss functions.
N
Uz.i (0i -
-0, 8x
clamp(
2
c)
N
Uy
(y/ 8). Sy
clamp(
22
M.89 E)
(14)
Un.i
Uy
8. + 6
By +
pec loss :=1
(15)
where N is the batch size, and the means (6 and taken over the batch. is used to provide better nunerical stability. The clamping refers to setting the minimum of the values to
1 N
1//
mape loss
(16)
y/i
Both the Real NPU and MAPE are able lo get success on the 4(22,2) range, which the MSE completely fails on, implying that having loss with standardisation useful. However, gain successes in the mixod-sign range, the other negative ranges have reduced in success for both PCC and MAPE. Both speed and sparsity retain similar performance MSE in majority of cases, with PCC solving especially fast for all tested ranges.
Real NPU (Figure 25
35

---

Published in Transactions on Machine Learning Research (09/2022)
Extrapolation range
Solved iteration
Sparsity
1.00
0.00015
0.75
75000
0.00010
0.50
50000
0.25
25000
0.00005
0.00
0.00000
01-0.2. 0..1 00.1
U1-1.2-11
2
U1-2.-
-11
UI-2
U[-20- -10
U10.1.0.22 2.2.1
U[1.2]
011.1.1.3
Ui-111 0.11
Ui-20 -10)
1.0.1
Ut1.2)
Ul1 (..1..1.)
U(30220)
U(-0.2.-0.11
U1-1.2.-11
UI-2 U1-2.11
U[-222
U/-20 -10)
U10.1.0.21
U(1.2
U/L.1.1.2)
U[10,20)
uja.
Interpolation range
MSE
PCC
MAPE
Figure 26: Single Module Task with redundancy on the NRU, comparing different loss functions.
Different losses have little effect the NRU. All three losses perform well the positive ranges. Compared to the Real NPU, the PCC loss on the NRU takes longer to converge to success for negative ranges.
NRU (Figure 26
NMRU (Figure 27 All three loses perform reasonably well, with the PCC struggling the most. Unlike the other units, -(- causes the most trouble, whereas 2((2,22 gains near to full success on two of the three losses.
Extrapolation range success rate
Solved at iteration step
Sparsity error
1.00
60000
2.0e-07-
40000
1.5e-07
0.50
1.0e-07
20000
0.25
5.0e-08
0.00
0.0e+00
1n 100o
Ul-2.2)
U(- Ur-20-10 -10)
ujo. 1.0.2)
(010200
U(-0.2.-40 0011
2-11
U[-2.2]
U[-20.-101
-100
U10.1.0.21
U/1.2)
U(2.1.1.2)
U110,20j
Ul 1.22-000
01-12--11
U1-0.2.-0.1
11-12.2
2
U1-122-144
U(-2.
Interpolation range
MSE
PCC
MAPE
Figure 27: Single Module Task with redundancy on the NMRU, comparing different loss functions.
36

---

Published in Transactions on Machine Learning Research (09/2022)
N RMSE Loss Landscapes
The following shows the Root Mean Squared loss curvature for the NAU stacked with either a RealNPU, NRU, or NMRU. "The weight matrices are constrained to W1 =
wi wi 0 0 wy w
W = [ wy]. The problem
wi wi
is ( + 22) (I1 + 22 + 03 + x4) for I = (1, 1.8, 2) (Madsen & Johansen
2020 The ideal solution is
W1 = W2 = 1, though other valid solutions do exist e.g., W1 = -1, u2 1.
The NMRU's weight matrix
would be W2 = wy wy 00, and the Real NPU's g [11].
98
78
68
68
loss
48
3B
00.
28
w2
w1
18
0.5
0.5
(a) NAU-Real NPU (where e le-
37

---

Published in Transactions on Machine Learning Research (09/2022)
9x10
8x10
7x10a
0.8x109
6x10a
0.6x10
0.4x109
0.2x10m
3x10
0.5
2x10
w2
w1
1x10a
0.5
0.5
(b) NAU-NRU
15
0.2
005
0.6
W2
0.9
(c) NAU-NMRU
Figure 28: Enlarged loss landscapes different stacked summmative-mulliplcaaaiii units.
38


Reviews:
[{"review_id": "D6yMEYVYSa", "reviewer": "Reviewer_UuYH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper's main contribution is also its main strength: a detailed experimental study that compares the studied modules (Modified Real NPU, NRU and NMRU) on a diverse range of synthetic test problems. The problems differ on two main axes: one is the range of inputs used for training/validation and test sets, and the other is the \"no redundancy\" setting with 2 inputs (where the task is learn to divide one input by the other) vs. \"with redundancy\" setting with 10 inputs (where the network should also learn to ignore the 8 irrelevant inputs). With this coverage, despite the synthetic nature of the tasks, I find that the evaluation is sufficient to understand the strengths and failure modes of the various models.\n\nThe NRU/NMRU are very intuitive extensions of the NMU, and \"why can't the NMU be extended to division?\" seems to be a very natural question to ask. This experimental study fills a clear gap in the literature by exploring this possibility.\n\nAnother strength of the paper is that it thoroughly evaluates the importance of choices related to training, regularization and initialization in addition to the model design. Based on the results, it can be observed that these choices may in many cases be more important than the model design itself. It also closely examines a series of specific issues related to learning division in detail such as gradient difficulties with the NRU, exploitation of multiplicative rules, and extrapolation to large ranges. Through these experiments, I believe that this paper sets a good standard for future work.\n\nI did not find any major weaknesses in this study. A small concern is whether the hyperparameters for Modified Real NPU (mainly the learning rate of 5e-3) were tuned, similar to NRU/NMRU (based on results in appendix), or not. Sec. 5 does indicate that the L1 regularization hyperparameters were tuned.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is concerned with the problem of training neural networks to learn the division operation using gradient descent. The motivation is that using the ability to represent precise mathematical functions will be useful for training neural networks in situations where there is aprecise mathematical relationships between variables in the data, such as certain physical systems. Another potential use of such modules is in learning interpretable functions of the inputs (such as controllers).\n\nThe paper builds on two prior ideas: Real Neural Power Unit (Real NPU), an existing module that can learn multiplication and addition,  and Neural Multiplication Unit (NMU), a module for learning multiplication. For the Real NPU, it discusses and experimentally demonstrates its failure modes when learning division, and demonstrates that techniques based on the NMU paper can be used to substantially improve its performance and robustness. The paper takes inspiration from NMU to propose two new modules for learning division: Neural Reciprocal Unit (NRU) and Neural Multiplicative Reciprocal Unit (NMRU). The key insight is that if we have a module that is effective at learning multiplication, then it can be extended to learn division easily if a) the ability to take reciprocals is added to the module (NRU), or b) the reciprocal of each input is additionally provided as an input to the module (NMRU). \n\nThe main contribution is a detailed experimental study evaluating the Real NPU, Real NPU with proposed modifications, NRU, and NMRU under a large variety of training/test set ranges. The experiments are designed to evaluate robustness to different training ranges, ability to extrapolate to different testing ranges, and ability to learn to divide relevant inputs while ignoring irrelevant inputs. In addition, there are further experiments focused on specific challenges and failure cases for learning division.\n\nOverall, the Modified Real NPU and NMRU are identified as promising modules for future work. Failure cases for both of these modules are discussed, which are mainly related to mixed-signed inputs, large input ranges and redundant inputs.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "None of the following changes are critical.\n\n- Indicate how the learning rate of Real NPU was tuned and whether the measured performance is indicative of the best that can be achieved with reasonable tuning.\n- Fix typo in pg. 1, second to last para: \"differentable\"\n- Fix extra space after \"Section 4.1)\" in Sec. 5\n- Add a table of contents to the appendices since there are several sections\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VLjBhqaf2h", "reviewer": "Reviewer_3o96", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) The authors do a thorough investigation across ranges and ablations.\n2) The new units proposed do clearly work better than the older Real NPU variant on the tasks tested here.\n\nWeaknesses:\n\n1) There is a big disconnect between the potentially desired uses of these division units and the tests performed in this paper. While reading this paper, I was left asking myself, why would anyone want to train these modules in these ways? To unpack that statement: the motivation for having a neural unit that can learn division is surely so that it can be included in a larger neural network being trained on more complex, interesting tasks. If we were modelling some system (be it some physics problem, or whatever), and our network included these units, and training to match the system being studied led these units to learn division of certain variables, then we would potentially have some interpretable description of the system. Great! But, I can think of no situation in which one would actually want to train these units in isolation on literal division tasks as the authors do here. Now, the intuition may be that if we cannot show that the units learn division when explicitly trained to do so in isolation then we certainly can't be confident that they would work when trained as part of a larger network. That may be, but it's hard to say - networks behave very differently than individual units and as the authors' own data shows that if a different loss is used it can also effect training. Moreover, we also cannot say that if these units work when trained explicitly to do division in isolation that they will then learn division when trained as part of a larger network. As such, the work here essentially demonstrates that these units work better on a series of tasks that no one would ever actually want to use them for, and doesn't provide any evidence that they work well on the sorts of tasks that one might actually want to use them for. This means that the technical prowess of this system for the actual desired uses is not demonstrated at all.\n\n2) The clarity of the exposition is weak in a few places. It took me a while to fully understand the various components, a number of parameter choices were opaque, and some of the tables and figures are hard to interpret. I will provide details of this below. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores simple neural modules for learning division operations. The goal is to develop interpretable units that could be included in neural networks trained with gradient descent that could learn to perform division in a manner that generalizes outside of the training range, even when there are multiple inputs, including irrelevant ones. The authors design two novel units (the NRU and NMRU), and they explore the performance of these units (as well as an older one, the Real NPU) across a variety of domains and task set-ups for explicit division, and with various ablations and manipulations of the parameters and components of the units. Their results provide clear evidence that the NRU and NMRU work better than the Real NPU on the sort of tasks explored.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1) This work is lacking a major piece of data, namely, any kind of demonstration that these units learn division when they are part of a larger network learning some other task. Without any data of this nature the work is not convincing as to the technical capabilities of these units in the sort of applications one would actually want to use them on. Really, to make this work relevant to other researchers, the authors would show some non-trivial learning of division as a result of network modelling of a known ground truth system that includes division as a key operation.\n\n2) There are a number of areas where the clarity of the paper could be improved:\n\n- In equation 3, is g a function of the input (as would be the case in, e.g. an LSTM), or literally just an extra set of fixed parameters?\n- What exactly is meant by a \"weighted L1 loss\"? There are a couple of possible interpretations of this, but it appears that here there is a single weight, so do the authors simply mean that the add an L1 loss whose impact is weighted by beta? Also, why is a schedule for this weight necessary?\n- How exactly are the various weights and parameters clipped to remain in range? Do you use simply set them to the max/min whenever those values are passed?\n- In equation 4, is o simply an index for the output unit? Why include that in the absence of any tests with multiple outputs?\n- How did you choose the scale factor of 1000 for the tanh version of absolute value?\n- Why bother using cos to do sign calculations? Is that more efficient than simply multiplying the signs of the input elements together?\n- What do the different columns in Table 1 correspond to?\n- How were the default parameters selected?\n- Why use the ranges of Madsen & Johanson (2020)\n- Typo page 4: \"Sparsity error (is) calculated by:\"\n- Why use parametric distributions to calculate confidence intervals? With only 25 seeds one could easily just show the data. Alternatively, one could use a non-parametric bootstrap method to calculate the CIs.\n- On page 6 the scaling factor is mentioned before being introduced.\n- Bottom of page 6: what modifications to the NPU are used here, exactly?\n- Typo page 8: \"10 input task(s)\"\n- Typo page 9: \"regardless (of) the value\"\n- Typo page 11: \"the use*d* of mixed signed inputs\"\n- Figure 9: the text is too small too read. I know the figures are presented in larger form in the Appendix, but still, it's not good practice to include any figures where one cannot read the text.\n- Table 5: similar to above, how were these parameters chosen?\n- Equation 12: what is the \"clamp\" operation here, exactly?\n", "broader_impact_concerns": "I do not have any concerns about the broader impact of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HMVCkLsFur", "reviewer": "Reviewer_fzJX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro\n * The paper introduces four sensible improvements to the baseline method and evaluates these through thorough ablation studies.\n * The paper introduces two novel neural modules for division, which are simple but theoretically sound, and evaluates their performance through a battery of simple experiments.\n * Results are consistently computed from 25 repetitions over random seeds, with appropriate statistics, which leads to highly reliable comparisons.\n\nCon:\n * To make the paper accessible to a wider audience, it lacks a more general introduction and explanation of the goals and challenges with Neural ALUs. This is fine for an expert audience, but I think there is some room for improvement.\n * It is unclear how experimental settings were designed. One experiment consists of a certain distribution over inputs (for training and validation set), and another non-overlapping distribution over inputs for the test set (typically implying larger-magnitude values than the training set). There are two concerns with this:\n   * Can it be ruled out that experiments were not picked to favor the novel method over the modified baseline, in particular NMRU (indirect cherry picking to tick as many lines as possible in Table 3)? This is a minor concern since some experiments show shortcomings of the proposed method (which is good for any follow-up work). See Requested Changes for more discussion.\n   * The main story of the paper could perhaps be told in a more compact fashion by selecting a subset of all experiments in the paper and using them as the primary example. For instance, why not have one set of experiments that include mainly mixed-sign ranges with two-sided generalization (extrapolation on both sides of the interval)? It seems to me like the latter is the main experiment, the single-sign ranges with one-sided generalization can sometimes offer additional info, but overall Table 1 seems to have too many redundant settings, and not enough interesting ones (which are presented later). See Requested Changes for more discussion.\n * At the end of the paper it is still unclear whether the way forward is to come up with more, or better, modifications of the baseline (or NRU/NMRU), or whether a new Neural Division Model is needed to address the shortcomings shown in the paper. This is, of course, not an easy task - but it makes the current findings somewhat inconclusive (which is still nice for future work building on these results, but the paper would certainly be stronger with a stronger answer to this question).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper follows a line of research on Neural ALUs (Arithmetic Logical Units). Given the crucial role that ALUs play in modern processors, the Neural ALU is supposed to be a learned equivalent that is comprised of (roughly) neural-network building blocks. Importantly, the Neural ALU needs to be able to learn arithmetic/logic computation that generalizes to arbitrary inputs (of the same numerical type). Building and learning Neural ALUs has been a challenge in the field for a few years now - the most elusive arithmetic computation being division. This is the problem that this paper aims to make progress on. It is phrased mainly as a problem of information-selection and routing (which roughly requires binary or ternary weights).\n\nIn particular, the paper makes the following main contributions:\n * Empirical investigation of 4 improvements to a previously proposed neural module to solve the problem (main idea: division via log-space subtraction). The improvements are: L1 regularization, parameter clipping, enforcing binary/ternary parameter values, non-standard initialization. These improvements are fairly obvious given the problem at hand, but nonetheless they lead to a significant increase in performance of the baseline method, as is shown by a series of well-conducted ablations.\n * Proposal of two alternative neural modules, where the main idea is to implement division via multiplication with the reciprocal. The modules require differentiability (to optimize parameters via SGD), and, like the baseline method, binary or ternary parameter values (ideally). The proposals are theoretically sound, and seem fairly straightforward. Both proposals are empirically compared against the (modified) baseline method and often perform favorably.\n * The main finding is that the unmodified baseline method can be improved, and that the new proposals perform even better in some cases. The paper also shows (sometimes systematic) limitations of all three methods, which highlights the need for further research to find a general Neural Division Module.\n\nThe main evaluation of the paper is on two types of tasks:\n * No redundancy: the module is fed with two one-dimensional inputs and needs to produce an output value corresponding to one input divided by the other.\n * Redundancy: same as above, but in addition the module is fed eight non-informative inputs that the module needs to learn to ignore.\n\nOverall I think the paper shows a large range of modifications of the baseline method and ablations for all three modules under a large set of experimental conditions. In its current form the manuscript and set of experiments feels a bit unorganized and rushed. I think the experiments could be reorganized and trimmed down to have an easier-to-read main text without compromising the narrative and main points. There is a lot of work and results in the paper, but the focus could be sharpened (I know it can be tempting to present everything that was tried and investigated, but at least in the main paper less can be more sometimes). Having said that, I think the main ingredients for an acceptance are there but I want to encourage the authors to try and streamline and publish the best version of this paper (rather than just getting it out the way and moving on).\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Major/important changes\n1. Perhaps the main experiments can be distilled down into a smaller, more informative, set with the remaining ones moved to the appendix. In particular, why does Table 1 only have a single mixed-sign experiment and only a single two-sided extrapolation experiment? I would have liked to see more of the datasets that went into Figure 4 and Table 2 as the main experiment (and only one or two representative single-sign, one-sided extrapolation experiments). Though this is listed as a major change, I am OK if this is not addressed - but personally think that it would be a major improvement (and would at least like to hear the authors’ opinion on why the current presentation is advantageous).\n2. Cherry-picking: were some additional experiments selected to make NMRU look like a “clear winner” in Table 3? Throughout the discussion of the results and looking at the plots I had a sense that there are some small, but clear improvements from NMRU, but an overall still quite mixed story; whereas Table 3 gives the impression that NMRU is a massive improvement over the other two. For instance: in Fig 7, RealNPU passes the 75% threshold for one large-magnitude uniform task but not the other, such that it does not pass overall (though clearly it does perform worse than NRU/NMRU). It would be helpful if there was more systematicity to how the ranges are selected. Or perhaps it would also help a lot to report numbers in Table 3 and simply make the entries that currently have a check-mark bold. Since this is not a method paper, but mainly an empirical analysis, I think it is more important to give a differentiated overview over strengths and weaknesses, rather than producing a clear winner method.\n3. It would be nice to see NRU/NMRU also evaluated on the tasks in Fig. 4. Any reason to exclude them?\n4. The paper could be better organized. It would help to describe the tasks (somewhat) before the models, and even within the models I feel like 3.1 is much harder to follow than 3.2 and 3.3 (perhaps also move 3.1 after 3.3?).\n5. The bibliography seems very short (unless I missed something I am counting 9 directly related references and 13 references in total; the last one is a duplicate) and the related work discussion could go into a bit more details about the differences and similarities. While there is no need for dozens of references, and I am not expert enough to point out which ones are missing, it still seems to me that a more comprehensive literature review might be beneficial.\n6. More detailed and differentiated discussion. My takeaway from the results is that RealNPU performs fairly broadly but most often a bit worse than NRU (when it works) and NMRU. NRU has a systematic shortcoming as shown in Fig 6, and NMRU is also not completely free of issues. In particular, it is still a bit unclear that NMRU is a practical alternative to RealNPU, since NMRU requires double the input dimensionality and, crucially, inversion of the whole input (even the irrelevant parts). This makes NMRU scale less well. It would be nice to discuss all of this with a bit more detail.  \n7. Throughout the paper I would have been keen to see the percentage of non-sparse or non-discrete parameters (what percentage lies more than some epsilon away from the nearest discrete value, i.e. -1, 0, or 1?). This would be an interesting additional measure, perhaps even more insightful than the sparsity error.  \n\nMinor (not crucial for acceptance)\n\nThere is a fairly large body of literature on learning neural networks with binary or ternary weights, and pruning weights (sparsity) in the field of neural network compression. It might be worth (for the future) to have a look at the state-of-the-art for further improvements a la section 5.\n\nBeginning of Sec. 4: explain sparsity regularisation scaling and why it is needed (it can be inferred from context, and will be clear to an expert reader, but adding a sentence would not hurt)?\n\nHow were the different iterations for switching on/off the sparsity regularisation determined? \n\nTable 1 is missing some labeling or description: what are the different rows, what is the difference between rows 1, 2 and 3, 4?\n\nResults in Sec. 5: There is a succession of ablations; does each subsequent setting use the best outcome from the previous ablation? E.g. all later experiments use L1 regularization with the best range, all experiments after Fig. 2a use GW clipping, etc?\n\nFig. 3, panel 3: The sparsity error is dominated by two outliers, which makes it really hard to see what happens in the rest of the plot (the scaling of the y-axis is too large). Consider providing a plot without RealNPU and/or a log-scale y-axis in the main paper.\n\nA few typos/misspellings throughout (e.g. diving instead of dividing on P7, effects instead of affects on P9, …). Make sure to do another pass for errors/typos.\n\nWall-clock timings in Table 7 cannot be meaningfully compared when experiments are run on very different machines.\n\nHave you experimented with a “more held-out” validation set (that already requires some extrapolation), rather than an i.i.d. validation set?\n", "broader_impact_concerns": "I have no concerns regarding the broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Paper Summary**

This paper focuses on different types of neural units for learning division with backpropagation and gradient descent.  The paper empirically evaluates the Real NPU, Real NPU with proposed modifications, NRU, and NMRU across a large variety of datasets, tasks and domains. The main contribution of the paper is the thorough experiments, and ablations of different neural modules for learning division operation. 

**Decision**

Overall, the authors did a very good job in the rebuttal phase and successfully addressed the concerns raised by the reviewers. The paper is well-written, clear and well-motivated. The experiments are very thorough and evaluation protocol is carefully thought. The authors seem to provide sufficient details about their experiments to be able to reproduce their results. The reviewers agreed that the findings of this paper can be beneficial for the TMLR and broader machine learning community and the authors incorporated the changes requested by the reviewers into the paper. In a nutshell, this paper is ready to be published at TMLR in this form.
Decision: Accept (poster)