Learning Division with Neural Arithmetic Logic Modules

To achieve systematic generalisation , it first makes sense to master simple tasks1 such as arithmetic . Of the four fundamental arithmetic operations ( + , - , × , ÷ ) ,2 division is considered the most difficult for both humans and computers . In this3 paper we show that robustly learning division in a systematic manner remains a4 challenge even at the simplest level of dividing two numbers . We propose two5 novel approaches for division which we call the Neural Reciprocal Unit ( NRU ) and6 the Neural Multiplicative Reciprocal Unit ( NMRU ) , and present improvements for7 an existing division module , the Real Neural Power Unit ( Real NPU ) . Experiments8 in learning division with input redundancy on 225 different training sets , find that9 our proposed modifications to the Real NPU obtains an average success of 85.3 % 10 improving over the original by 15.1 % . In light of the suggestion above , our NMRU11 approach can further improve the success to 91.6 % .12 1 Introduction13 Imagine you must learn to divide 2 numbers , but are only given 10 numbers and the target value . This14 task requires finding the 2 relevant operands , the order to divide the operands , and learning to divide.15 In machine learning , this is equivalent to a supervised regression task where the aim is to learn the16 underlying function between the inputs and output such that the solution is generalisable to any input.17 The ability to select relevant features is a desirable property of neural networks , useful for improved18 intepretability , reduced pre-processing costs and greater generalisation [ Chandrashekar and Sahin,19 2014 ] . The ability to model division , one of the four fundamental arithmetic operations , is necessary20 for expressing dynamical systems [ Sahoo et al. , 2018 ] , and physics-based formulas [ Udrescu and21 Tegmark , 2020 ] . However , even recent models still struggle to learn division when there is input22 redundancy [ Schlör et al. , 2020 ] .23 The main challenge of the above task comes from learning the selection and operation at the same24 time , which can lead to conflicting priorities when learning network weights . Furthermore , the natural25 properties of division of values around zero leads to undesirable gradients . Models which deal with26 this naively ( e.g . MLPs ) are unable to deal with the fluctuant gradients caused by the asymptotic27 nature and discontinuities in division [ Trask et al. , 2018 ] .28 Can we build models which can learn division in the presence of its undesirable , yet valid , properties ? 29 We aim to address this question in this paper . Specifically , we contribute the following:130 • Improvements to the Real NPU [ Heim et al. , 2020 ] including : clipping , discretisation and31 constrained initialisation to improve performance in learning division on different training ranges.32 1Code ( MIT license ) available at : https : //anonymous.4open.science/r/nalu-stable-exp-neur ips-review-2E4C/ . Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . • Two novel division modules , the NRU and the NMRU . The NRU explores extending the NMU33 weight ranges from [ 0,1 ] to [ -1,1 ] to include division , where we find a weakness in learning from34 negative ranges . Learning from the weaknesses of the NRU , the NMRU extends the NMU to learn35 division while keeping weights values between [ 0,1 ] . We further boost performance by using a36 Real NPU inspired sign retrieval mechanism , enabling the NMRU to gain the best performance37 when using a mean squared error ( MSE ) loss.38 • New understanding into the hindrances in learning division including : training on mixed-sign39 inputs , training on negative ranges , and division on extremely small values . We find these difficulties40 can be sufficiently identified using synthetic division tasks.41 The broader impact of our work relates to interpretable Artificial Intelligence where our modules42 can be included in larger networks for applications such as image classification or analogy creation,43 whilst retaining the ability to produce transparent generalisable solutions . However , there are possible44 negative societal impacts . Such modules can be viewed as specialised feature selectors/aggregators45 which do not require integrating domain knowledge . Therefore , if a non-domain-expert tries inter-46 preting relations in the input data , they may incorrectly interpret causality , which can be especially47 harmful if such a case occurs on medical or financial data . Mitigating against such downstream issues48 requires to first focus efforts on producing robust modules to different distributions and understand49 their affect on learning other networks architectures ( e.g . CNN ) . Understanding this will enable50 recognising situations where these modules can aid and where they should avoid being used.51 2 Related Work52 One approach to learn division would be symbolic regression networks [ Sahoo et al. , 2018 ] . However,53 a symbolic approach pre-defines the operations , which is not a limitation of using Neural Arithmetic54 Logic Modules ( NALMs ) .55 NALMs are neural networks which learn arithmetic operations and input selection [ Mistry et al.,56 2021 ] . The weights of these networks are intepretable such that a discrete value represents a specific57 operation . For example , ‘ -1 ’ to represent division and ‘ 0 ’ for no selection . From this research field,58 we focus on the Real NPU and the NMU . Until now , the Real NPU only has learned division on59 training ranges of either U [ 0.1,2 ] or Sobol ( 0,0.5 ) [ Heim et al. , 2020 ] . It remains unclear if this60 module is robust to other training ranges even as a stand-alone unit . Robustness to training ranges is61 important as these module ’ s applicational use comes from being part of larger end-to-end networks,62 where the input range into the module can not be controlled . The NMU is a multiplication module63 which we extend to also do division . The authors of the NMU believe such an extension incurs too64 many limitations for learning [ Madsen and Johansen , 2020 ] . We use this paper as an opportunity to65 explore this belief.66 Trask et al . [ 2018 ] developed the Neural Arithmetic Logic Unit ( NALU ) which can model all four67 arithmetic operations . However , studies show this module to be unstable in learning division [ Schlör68 et al. , 2020 , Heim et al. , 2020 ] . In particular , their gating method responsible for selecting an operation69 can not learn consistently [ Madsen and Johansen , 2020 ] . Schlör et al . [ 2020 ] developed iNALU70 additionally applying weight and gradient clipping , sign retrieval , regularisation , reinitialisation and71 separating shared parameters to the NALU . Even with these modifications , they still find consistently72 learning division to a high precision to remain unattainable . Furthermore , Heim et al . [ 2020 ] ’ s results73 imply iNALU is outperformed by the Real NPU for division.74 3 Architectures75 This section introduces the architectures for the ( Real ) NPU , NRU , and the NMRU . The ( Real ) NPU76 is an existing module , which we improve in Section 5 . The NRU and NMRU are novel contributions.77 Appendix A summarises the important properties of these division modules.78 3.1 Real Neural Power Unit79 Heim et al . [ 2020 ] develop a module to learn to multiply and divide , using the intuition from Trask80 et al . [ 2018 ] that multiplicative operations are additive operations in log space . Their work extends81 this idea into complex space . The NPU can be used with its complex form ( Equation 1 ) requiring both82 a complex and real weight matrix ( W ( i ) , W ( r ) ) , or only its real form the Real NPU ( Equation 2 ) .83 For improved gradients , a relevance gate r ( Equation 3 ) is used which converts inputs close to 0 ( i.e.84 irrelevant features ) to 1 to avoid the resulting output evaluating to 0 . A gating vector g , learns to85 select relevant input elements , where gate values are clipped between [ 0,1 ] during training.86 NPU : = exp ( W ( r ) log ( r ) −W ( i ) k ) cos ( W ( i ) log ( r ) +W ( r ) k ) , ( 1 ) 87 RealNPU : = exp ( W ( r ) log ( r ) ) cos ( W ( r ) k ) ( 2 ) 88 where r = g ( |x|+ ) + ( 1− g ) and ki = { 0 xi ≥ 0 πgi xi < 0 . ( 3 ) A weighted L1 penalty is used when training . The weight value β grows between predefined values89 βstart to βend and is increased every βstep = 10 , 000 iterations by a growth factor βgrowth = 10.90 We focus on the Real NPU over the NPU as the solution of the tasks in this paper can be captured91 using only real values meaning that the complex form is not required.92 3.2 Neural Reciprocal Unit93 We propose the NRU , which can model multiplication and division . We extend the NMU , motivated94 by division being multiplication of reciprocals . The range which weight values can be is extended95 from [ 0,1 ] to [ -1,1 ] , where -1 represents applying the reciprocal on the corresponding input element.96 A NRU output element zo is defined as97 NRU : zo = I∏ i=1 ( sign ( xi ) · |xi|Wi , o · |Wi , o|+ 1− |Wi , o| ) , ( 4 ) where I is the number of inputs . Assuming weights are either 1 ( multiply ) or -1 ( reciprocal ) , |xi|Wi , o98 will apply the operation on an input element . The absolute value is used so that the module only99 operates in the space of real numbers , as xWi , oi for a negative input ( xi ) when −1 < Wi , o < 1 results100 in a complex number . The use of absolute means the sign of the input must be reapplied . For the101 no-selection case Wi , o = 0 , we want the input element to convert to 1 ( the identity value ) , resulting102 in applying ·|Wi , o|+1−|Wi , o| . The derivative of the absolute function at 0 is undefined meaning the103 gradients of Equation 4 can contain points of discontinuity . To alleviate this issue , we approximate104 the absolute function using a scaled tanh ( inspired by Faber and Wattenhofer [ 2020 ] ) . More formally,105 |Wi , o| = { tanh ( 1000 ·Wi , o ) 2 if training |Wi , o| otherwise . The scale factor ( 1000 ) controls how close to the absolute function the approximation is , where larger106 values give a more accurate approximation . For clipping and regularisation , the same scheme as the107 Neural Addition Unit ( NAU ) ( see Appendix B ) is used.108 3.3 Neural Multiplicative Reciprocal Unit109 An alternate extension of the NMU , also motivated by division being multiplication of reciprocals110 is the NMRU ( Equation 5 ) . We concatenate the reciprocal of the input ( plus a small ) to the input111 resulting in a module which only needs to learn selection . Hence , weights can be in the range [ 0,1 ] .112 NMRU : zo = 2I∏ i=1 ( Wi , o · |xi|+ 1−Wi , o ) · 2I∑ i=1 ( cos ( Wi , o · ki ) ) , where ki = { 0 xi ≥ 0 π xi < 0 . ( 5 ) The iteration over 2I represents the going through all inputs and their reciprocals . We calculate the113 magnitude and sign separately , joining the result at the end . The magnitude is calculated passing114 absolute of the concatenated input through an NMU architecture and the sign by using a cosine115 mechanism similar to the Real NPU . However , unlike the Real NPU only the weight matrix is116 required . The norm of the weight ’ s gradients are clipped to 1 prior to being updated by the optimiser.117 This is done to alleviate the issue of exploding gradients caused by including the reciprocal to the118 inputs . For clipping and regularisation , the same scheme as the NMU ( see Appendix B ) is used.119 We introduce the two main experiments used to evaluate modules , including : default parameters,121 train and test ranges , and evaluation metrics . The tasks evaluate the ability of a single module to122 divide two numbers from an input vector in two settings : no redundancy and with redundancy.123 Default parameters : All experiments use a mean squared error ( MSE ) loss with an Adam optimiser124 [ Kingma and Ba , 2015 ] , with 10,000 samples for the validation and test sets . The best model for125 evaluation is taken using early stopping on the validation set . All runs are over 25 different seeds . All126 inputs are required in the no redundancy setting , i.e. , input size of 2 . Training takes 50,000 iterations127 where each iteration consists of a different batch of size 128 . The Real NPU uses a learning rate of128 5e-3 with sparsity regularisation scaling during iterations 40,000 to 50,000 . The NRU and NMRU129 use sparsity regularisation scaling during iterations 20,000 to 35,000 and a learning rate of 1 and 1e-2130 respectively . In contrast , the redundancy setting uses an input size of 10 , where 8 input values are not131 required for the final output . The total training iterations are extended to 100,000 with batch sizes132 of 128 . The learning rates for the Real NPU , NRU and NMRU are 5e-3 , 1e-3 and 1e-2 respectively.133 Sparsity regularisation scaling occurs during iteration 50,000 to 75,000 for all modules . A summary134 of all relevant parameters is found in Appendix C.135 Ranges : The interpolation ( train/validation ) and extrapolation ( test ) ranges , are found in Table 1.136 The chosen ranges are influenced by Madsen and Johansen [ 2020 ] .137 Evaluation metrics : We use the Madsen and Johansen [ 2019 ] ’ s evaluation scheme , consisting of138 three evaluation metrics : the success on the extrapolation dataset against a near optimal solution139 ( success rate ) , the first iteration which the task is considered solved ( speed of convergence ) , and140 the extent of discretisation towards the weights ’ inductive biases ( sparsity error ) . Sparsity error141 calculated by max i , o ( min ( |Wi , o| , 1− |Wi , o| ) ) , measures the weight element which is the furthest away142 from the acceptable discrete weights for the module . A success means the MSE of the trained model143 is lower than a threshold value ( i.e . the MSE of a near optimal solution ) . We differ from Madsen144 and Johansen [ 2019 ] by using a fixed threshold value 1e-5 rather than a simulated MSE , as there145 are no intermediate layers to accumulate numerical errors . We choose this precision as it can be146 guaranteed when working with 32-bit PyTorch Tensors . 95 % confidence intervals ( over the 25 seeds ) 147 are calculated from a specific family of distributions dependant on the metric . The success rate uses148 Binomial distribution because trials ( i.e . run on a single seed ) are either pass/ fail situations . The149 convergence metric uses a Gamma distribution and sparsity error uses a Beta distribution . Both Beta150 and Gamma can easily approximate the normal distribution and support its corresponding metric.151 5 Improving the Real NPU ’ s Robustness152 We first improve the robustness of the Real NPU on different training ranges . We use the Single153 Module Task with no redundancy ( see Section 4 ) to investigate the following questions:154 1 . Is L1 regularisation required , and if so , do the regularisation parameters require tuning ? 155 2 . Does clipping the weight matrix aid learning ? 156 3 . Does enforcing discretisation on parameters improve convergence ? 157 4 . Can the weight matrix initialisation be improved ? 158 To address each question in order , we propose applying incremental modifications to the Real NPU.159 These modifications include : ablation study on the L1 regularisation ( including a sweep over the160 scaling range hyperparameters ) , clipping , enforcing discretisation , and a more restrictive initialisation161 scheme . We assume that we are optimising the Real NPU to perform multiplication or division.162 Therefore , we trade-off the flexibility of having non-discretised weights , which enables the success of163 modelling the SIR data in Heim et al . [ 2020 , Section 4.1 ] , in favour of sparse models with discrete164 weight values . All the modifications suggested can also be generalised for the NPU architecture.165 Is L1 regularisation required ? ( Yes ) L1 encourages sparsity ( i.e. , zero weights ) in solutions.166 Zero-valued weights means not to select an input and return the identity value 1 . For the task , the167 optimal weight values require selecting all inputs and therefore non-zero values , suggesting the168 application of L1 could be damaging . Therefore , we compare against a model which does not use169 L1 regularisation , shown in Figure 1a . Removing L1 proves to be detrimental in five of the nine170 cases shown and only shows minor improvements in two of the nine ranges ( i.e. , U [ -1.2 , -1.1 ) and171 U [ 1.1,1.2 ) ) . Hence , we keep L1 regularisation . The L1 regularisation scaling ( see Section 3.1 ) ,172 requires setting the hyperparameters for the start ( βstart ) and end ( βend ) scaling values . We run a173 sweep over six different start and end values , denoted ( < start > , < end > ) , displaying results in Figure 1b.174 We find the configuration ( 1e-9 , 1e-7 ) is the most successful when considering performance on all175 the ranges , and larger scaling values perform worse.176 Does clipping the learnable parameters help ? ( Yes ) Division and multiplication operations are177 represented by weight values of -1 and 1 respectively . The current architecture does not constrain the178 weights which can result in large weight values . The gate weights do get clipped and saved to another179 variable during the forward pass , meaning after an update step the gate values can also be out of the180 range [ -1,1 ] . Hence , we investigate the effect of applying clipping directly to the weight and gate181 values after every optimisation step . Results , shown in Figure 2a , show clipping is beneficial , with182 clipping on both weight and gate ( or just on the weights ) to improve over the baseline on all ranges183 ( excluding U [ 1,2 ) where the baseline has already achieved full success ) .184 Does enforcing discretisation help ? ( Yes ) Modelling division in a generalisable manner requires185 all learnable parameters to be discrete i.e. , a value from { -1 , 0 , 1 } . Using Madsen and Johansen186 [ 2020 ] ’ s regularisation scaling scheme , we penalise weights for not being discrete . We modify the187 scaling factor to be λ̂ = 1 and the regularisation to go from ‘ off ’ to ‘ on ’ between iterations 40,000 to188 50,000 . Results , shown in Figure 2b , show discretising the gate improves over the baseline but also189 discretising the weights is additionally beneficial ( especially for range U [ -0.2 , -0.1 ) ) . U [ 10,20 ) is the190 only range where the baseline outperforms using discretisation , succeeding on two additional seeds.191 Does using a more constrained initialisation help ? ( Yes ) W ( r ) uses a Xavier-Uniform initial-192 isation [ Glorot and Bengio , 2010 ] . This can result in weights initialised out of the range [ -1,1 ] .193 Therefore , we use the initialisation for the Neural Addition Unit which is a constrained form of194 the Xavier-Uniform that does not allow the fan values of the uniform distribution to go beyond 0.5,195 meaning that no weight value will be out of the range [ -1,1 ] [ Madsen and Johansen , 2020 ] . Figure 2c196 shows using the constrained initialisation provides improvements over multiple ranges.197 6 Results : Single Module Task198 We analyse the results for the : Real NPU without using the modifications of Section 5 , Real NPU199 with modifications , NRU , and NMRU.200 6.1 No Redundancy201 Figure 3 shows the baseline Real NPU without modifications struggles with all ranges except U [ 1,2 ) ,202 struggling with sparsity on the larger ranges . Applying the modifications deals with the sparsity issue203 and improves the robustness such that only range U [ -2,2 ) struggles ( with a success rate of 0.64 ) . The204 NRU and NMRU achieve full success over all ranges while solving the problem consistently fast and205 with low sparsity error . The success of the NRU is correlated with the learning rate ( see Appendix E ) .206 6.1.1 Mixed-signed Inputs207 The remaining failure range of the Real NPU is U [ -2,2 ) where inputs can consist of arbitrary signed208 values ( e.g . all positives , all negatives , or a mixture of positive and negative values ) . We question209 if the failure is due to the input samples in a batch having different signs from each other , or if the210 problem is due to the fact data samples can be close to 0 ( leading to singularity issues ) . To investigate211 this , we create additional mixed-sign datasets , controlling the range for each element in the input . The212 interpolation and extrapolation ranges for the different datasets can be found in Appendix C. Datasets213 1 , 2 , 4 and 5 sample a positive value for one input element and a negative value for the other element.214 Dataset 3 samples the signs randomly . Datasets 2 and 5 avoid sampling close to 0 values to mitigate215 the singularity issue . As shown by Figure 4 , the Real NPU struggles on all these ranges , implying that216 the core issue is not from different input samples having different signs or due to the input samples217 being able to contain small values close to 0 . The underlying issue is therefore most likely correlated218 to the each element in an input having different signs . When the denominator of the output is positive219 ( dataset 1 or 2 ) , the solution is found faster than when the denominator is a negative value ( dataset 4220 or 5 ) . When the signs for an input element are controlled , discretisation/sparsity is no problem , in221 contrast when the signs are arbitrary the sparsity error are slightly ( though not significantly ) higher.222 6.2 Division by Small Numbers223 Division by zero remains a challenge to model due to the inability to provide an computational value224 for the output and gradient . Furthermore , the discontinuous nature at zero causes its neighbouring225 values to have large gradients . To understand the extent of this issue when learning , we explore226 learning to divide by values close to zero using three tasks with increasing difficulty : 1 ) learning to227 take the reciprocal of a single input , 2 ) taking the reciprocal of the first input given two inputs , and 3 ) 228 diving the first input by the second given two inputs . Figure 5 plots the test error for different modules229 assuming the module weights are set to the ‘ gold ’ solution for the three tasks . As the range values230 become closer to zero , the test error thresholds become increasingly large . Therefore , even with the231 correct weights , relying on the test errors alone as an indicator become increasingly deceptive with232 values close to zero . The Real NPU has larger test errors for all tasks and ranges , caused by adding 233 to the input ( see Equation 3 ) . Setting = 0 reduces the test error at the cost of the ability to deal with234 zero-valued inputs . Appendix F provides the corresponding experimental results for these tasks.235 6.3 With Redundancy236 Introducing redundancy ( Figure 6 ) causes failure modes to arise . Failures on range U [ -2,2 ) become237 more prevalent . The baseline Real NPU produces high sparsity errors relative to the other modules238 suggesting struggle with discretisation . Using the modified Real NPU improves over all ranges of the239 baseline ( which were not already at full success ) in terms of success , speed and sparsity.2 To ensure240 that complex weights do not fix the issue , we test the NPU module with all the modifications used on241 the real weight matrix ( see Appendix G ) . Complex weights hinders success and convergence speeds242 of negative ranges . Assuming the global solution only uses the real weights , we enforce the complex243 weights to be clipped between [ -1,1 ] and to go to 0 during the regularisation stage using a L1 penalty.244 This did not result in any significant improvements against the Real NPU results . Input redundancy245 effects the NRU the most , resulting in full failures on all the negative ranges . The NMRU is the246 only module with success for the range U [ -2,2 ) , which is a result of using the sign mechanism ( see247 Appendix H ) . It performs well over all ranges though can be outperformed by the modified Real NPU248 for negative ranges . Multiple ranges for the NMRU are solved around 50,000 iterations correlating to249 the sparsity regularisation being turned on.250 6.3.1 Gradient Difficulties with the NRU251 The partial derivative for the NRU weights , Equation 6 , can give insight to the struggles of the NRU.252 ∂ŷ ∂wi = tanh ( 1000wi ) ( sign ( xi ) |xi| ( tanh ( 1000wi ) log ( |x| ) + 2000 sech ( 1000wi ) 2 ) − 2000 sech ( 1000wi ) 2 ) × NRUx̃∈x\ { xi } ( x̃ ) . ( 6 ) NRUx̃∈x\ { xi } ( x̃ ) applies the NRU to all inputs excluding xi influencing the gradient values between253 subsequent update steps . Factoring out this term , the following observations are made . If xi ≈ 0 and254 wi ≈ 0 then gradients become increasingly large . If xi ≈ 0 and −1 ≤ wi < 0 then as wi → −1 all255 gradients for xi where |xi| > > 1 become increasingly small . The gradients for xi = −1 and xi = 1256 are 0 regardless the value of wi . If wi = 0 then the gradient is 0 for all xi , a result of using the tanh257 approximation . Even if the sign and magnitude are calculated separately and then combined ( see258 Appendix I ) to try to control the gradient better , the problem remains . Therefore , we conclude that259 extending the NMU to divide using a weight of -1 is a poor choice when there are redundant inputs.260 2Except for the sparsity error for range U [ 10,20 ) . ( a ) Real NPU ( b ) NRU ( c ) NMRU Figure 7 : Root Mean Squared loss curvature for the NAU stacked with either a RealNPU , NRU , or NMRU . `` The weight matrices are constrained to W1 = [ w1 w1 0 0 w1 w1 w1 w1 ] , W2 = [ w2 w2 ] . The problem is ( x1 + x2 ) · ( x1 + x2 + x3 + x4 ) for x = ( 1 , 1.2 , 1.8 , 2 ) '' [ Madsen and Johansen , 2020 ] . The ideal solution is w1 = w2 = 1 , though other valid solutions do exist e.g. , w1 = −1 , w2 = 1 . ( The NMRU ’ s weight matrix would be W2 = [ w2 w2 0 0 ] , and the Real NPU ’ s g = [ 1 1 ] . ) 6.3.2 The Real NPU ’ s and NMRU ’ s Exploitation of Multiplicative Rules261 The NMRU solutions exploit the inverse rule of division in that ai · 1ai = 1 . Since the input also262 contains the reciprocals , numerous extrapolative solutions exist . However this comes at the cost of263 finding a ‘ simple ’ solution which contains ones only for relevant inputs . The Real NPU exploits the264 rules ai · 0 = 0 and 1ai = 1 enabling non-zero weight values if the corresponding gate value is 0.265 However , we can avoid this by allowing 0 to also not be penalised during sparsity regularisation stage266 ( see Appendix G ) . We find this alleviates the exploitation issue with no cost to performance.267 7 Discussion268 In this paper , we demonstrate the limitations of intepretable neural networks in learning to divide.269 Using the no redundancy setting ( size 2 ) , we find that the Real NPU is challenged when training data270 consists of mixed-signed inputs even with our applied improvements . Increasing the difficulty to271 have an input redundancy ( with 8 redundant and 2 relevant input values ) magnifies this issue , but272 also introduces failure modes for the NRU and NMRU for negative ranges . The NRU is unable to273 handle any negative ranges , in which we conclude it is not wise to use with MSE . Alternate losses274 can improve certain failure cases though sometimes at the cost of performance on other ranges . For275 further details see Appendix J which displays results on a correlation and scale-invariant based loss.276 Our NMRU is the only module with reasonable success over all tested ranges , requiring only 2I ×O277 learnable parameters . However , this comes at the cost of the simplicity of the solution due to its278 exploitation of the identity rule ; an issue the Real NPU does not have.279 Once robust modules are attainable in a single layer setting , the next step would be to question280 performance when learning stacked modules , e.g . learning a stacked additive and multiplicative281 module . Previously , Madsen and Johansen [ 2020 , Figure 2 ] illustrates the troubles for multiplicative282 models with the capacity for division . They show how a stacked summative-multiplicative module can283 lead to an exploding loss when the output of the summative module is close to 0 and the multiplicative284 model tries to divide . In Figure 7 , we recreate their setup to produce the loss surfaces for the NAU-285 Real NPU3 , NAU-NRU and NAU-NMRU respectively.4 We find a similar issue with the Real-NPU286 and NRU , as both these units use a weight range of [ -1,1 ] . In contrast , the NMRU , whose weight ’ s287 range is limited to [ 0,1 ] does not have exploding losses.288 In conclusion , division remains a challenge to learn using intepretable neural networks , even for the289 simplest tasks . Nevertheless , by identifying the specific areas causing difficulty ( e.g. , training ranges ) ,290 and useful architecture properties ( e.g. , using a sign retrieval mechanism ) , we hope the community291 has better intuition for dealing with division and develop more robust modules to learn division.292 3The NAU is a summative module [ Madsen and Johansen , 2020 ] . 4Appendix K displays larger versions of these plots . References293 Girish Chandrashekar and Ferat Sahin . A survey on feature selection methods . Computers & 294 Electrical Engineering , 40 ( 1 ) :16–28 , 2014 . ISSN 0045-7906. doi : https : //doi.org/10.1016/j.comp295 eleceng.2013.11.024 . URL https : //www.sciencedirect.com/science/article/pii/S0296 045790613003066 . 40th-year commemorative issue.297 Lukas Faber and Roger Wattenhofer . Neural status registers . CoRR , abs/2004.07085 , 2020 . URL298 https : //arxiv.org/abs/2004.07085.299 Xavier Glorot and Yoshua Bengio . Understanding the difficulty of training deep feedforward neural300 networks . In Proceedings of the thirteenth international conference on artificial intelligence and301 statistics , pages 249–256 . JMLR Workshop and Conference Proceedings , 2010 . URL http:302 //proceedings.mlr.press/v9/glorot10a/glorot10a.pdf.303 Niklas Heim , Tomáš Pevnỳ , and Václav Šmídl . Neural power units . Advances in Neural Information304 Processing Systems , 33 , 2020 . URL https : //papers.nips.cc/paper/2020/file/48e5900305 0d7dfcf6c1d96ce4a603ed738-Paper.pdf.306 Diederik P. Kingma and Jimmy Ba . Adam : A method for stochastic optimization . CoRR,307 abs/1412.6980 , 2015 . URL https : //arxiv.org/pdf/1412.6980.pdf.308 Andreas Madsen and Alexander Rosenberg Johansen . Measuring arithmetic extrapolation perfor-309 mance . In Science meets Engineering of Deep Learning at 33rd Conference on Neural Information310 Processing Systems ( NeurIPS 2019 ) , volume abs/1910.01888 , Vancouver , Canada , October 2019.311 URL https : //arxiv.org/pdf/1910.01888.pdf.312 Andreas Madsen and Alexander Rosenberg Johansen . Neural arithmetic units . In International313 Conference on Learning Representations , 2020 . URL https : //openreview.net/forum ? id=314 H1gNOeHKPS.315 Bhumika Mistry , Katayoun Farrahi , and Jonathon Hare . A primer for neural arithmetic logic modules,316 2021 . URL https : //arxiv.org/pdf/2101.09530.pdf.317 Subham Sahoo , Christoph Lampert , and Georg Martius . Learning equations for extrapolation and318 control . In International Conference on Machine Learning , pages 4442–4450 . PMLR , 2018.319 Daniel Schlör , Markus Ring , and Andreas Hotho . inalu : Improved neural arithmetic logic unit.320 Frontiers in Artificial Intelligence , 3:71 , 2020 . ISSN 2624-8212. doi : 10.3389/frai.2020.00071.321 URL https : //www.frontiersin.org/article/10.3389/frai.2020.00071.322 Andrew Trask , Felix Hill , Scott E Reed , Jack Rae , Chris Dyer , and Phil Blunsom . Neural arithmetic323 logic units . In Advances in Neural Information Processing Systems , pages 8035–8044 , 2018 . URL324 https : //openreview.net/pdf ? id=H1gNOeHKPS.325 Silviu-Marian Udrescu and Max Tegmark . Ai feynman : A physics-inspired method for symbolic326 regression . Science Advances , 6 ( 16 ) , 2020. doi : 10.1126/sciadv.aay2631 . URL https:327 //advances.sciencemag.org/content/6/16/eaay2631.328 Checklist329 1 . For all authors ... 330 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s331 contributions and scope ? [ Yes ] See contributions in the introduction . For Real NPU332 improvements see Section 5 . For two novel modules see Section 3.2 and 3.3 and results333 in Section 6 . For hindrances in learning division , see Section 6.334 ( b ) Did you describe the limitations of your work ? [ Yes ] See Section 7.335 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] This336 paper focuses on the ML techniques and the foundational research required to learn337 division in a systematic manner . Once robust modules for the arithmetic operations ( i.e.338 NALMs ) are achievable the community will possess trainable modules with significant339 advantages regarding model transparency and generalisability . That being said , we340 discuss how this leads to a negative societal impact in the end of Section 1.341 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to342 them ? [ Yes ] We have read the guidelines and our work does not use human-derived343 data.344 2 . If you are including theoretical results ... 345 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 346 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 347 3 . If you ran experiments ... 348 ( a ) Did you include the code , data , and instructions needed to reproduce the main ex-349 perimental results ( either in the supplemental material or as a URL ) ? [ Yes ] See link350 provided to the code base . Data is generated in real time and the code to generate it is351 available in the linked repository.352 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they353 were chosen ) ? [ Yes ] See Section 4 and Appendix C.354 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-355 ments multiple times ) ? [ Yes ] See plots and experiment details in Section 4 for further356 details.357 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type358 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] See Appendix D.359 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 360 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] See footnote link361 provided to the code base and Section 4.362 ( b ) Did you mention the license of the assets ? [ Yes ] We state the MIT licence when giving363 the link to the codebase.364 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 365 All code for model , data and experiments are available through the link to the codebase.366 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re367 using/curating ? [ N/A ] 368 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable369 information or offensive content ? [ No ] All data is synthetic , containing no such370 information.371 5 . If you used crowdsourcing or conducted research with human subjects ... 372 ( a ) Did you include the full text of instructions given to participants and screenshots , if373 applicable ? [ N/A ] 374 ( b ) Did you describe any potential participant risks , with links to Institutional Review375 Board ( IRB ) approvals , if applicable ? [ N/A ] 376 ( c ) Did you include the estimated hourly wage paid to participants and the total amount377 spent on participant compensation ? [ N/A ] 378

Summary: This paper study possible adaptations of Neural Arithmetic Linear Units (Trask 2018) to better handle division. The authors first propose improvements for training Real Neural Power Unit (Heim 2020), a neural arithmetic unit which performs division as substraction in log-space, using cosines to handle signs, and a gating mechanism to select operands, while constraining all weights to remain close to 0, 1 or -1. They show that introducing L1 regularisation, gradient clipping, and tuning weight constraint and initialization greatly improve its accuracy when dividing numbers.  They introduce two new designs, the Neural Reciprocal Unit, which functions as an addition in log-space with weights constrained to -1, 0 and 1, and can therefore perform multiplication or division of numbers (and "unselect" them when weights are zero, which dispenses of the gating mechanism), and the Neural Reciprocal Modified Unit, where the reciprocals are provided as additional inputs, constraining weights to 0 and 1 (as a selection mechanism), and adds a cosine function to handle signs.   Testing over a variety of generated samples, they show that whereas the new designs improve on the original Real NPU, most of the still struggle on input of mixed signs, and have difficulty working with numbers close to zero, and learning both selection (choosing the operands) and division.