Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: DISCLAIMER: I reviewed a previous version of this paper at another venue.

This paper introduces Neural Addition Units (NAU) and Neural Multiplication Units (NMU), essentially redeveloped models of the Neural Arithmetic Logic Unit (NALU). The paper presents a strong case that the new models outperform NALUs in a few metrics: rate of convergence, learning speed, parameter number and model sparsity. The performance of NAU is better than NAC/NALU, as is the performance of NMU with a caveat that the presented NMU here cannot deal with division, though it can deal with negative numbers (as opposed to NALU).

What this paper excels at is a thorough theoretical and practical analysis of NALU’s issues and how the authors design the two new models to overcome these issues. The presented issues of NALU are numerous, including unstable optimization space, expectations of gradients converging to zero, the inability of NALUs gating to work as well as intended and its issues with division, and finally, the intended values of -1, 0, 1 in NALU do not get as close to these values as intended. 

The paper is easy to read, modulo a number of typos and admittedly some weirdly written sentences (see typos and minor issues later) and I would definitely recommend another iteration over the text to improve the issues with it as well as the style of writing. I am quite fond of the analysis and the informed design of the two new models, as well as the simplicity of the final models which are fairly close to the original models but have been shown both theoretically and practically that they work.
It is great to see that the paper improved since my last review and stands stronger on its results, but there are still a few issues with it that make me hesitant to fully accept the paper:
- The conclusion of the paper is biased towards the introduced models, but it should clearly define the limitations of these models wrt NALU/NAC
- The performance of NALu on multiplication is in stark contrast to the results in the original paper (Table 1). This should be commented in the paper why that is, as the original model presents no issues of NALU with multiplication, whereas this paper essentially says that they haven’t gotten a single model (out of 100 of them) to do multiplication.
- Could you explicitly comment on the paper why is the parameter sparsity such a sought-after quality of these models?
- You ‘assume an approximate discrete solution with parameters close to {1-, 0, 1} is important’. What do you have to back this assumption? Would it be possible to learn the arithmetic operations (and generalize) even with parameters different than those?
- Why did you introduce the product of the sequential MNIST experiment but did not presents results on the original sum / counting of digits? The change makes it hard to compare with the results in the original paper, and you do not present the reason why. This also makes me ask why didn't you compare to NALU on more tasks presented in the paper?

To conclude, this paper presents a well-done experimental and theoretical analysis of the issues of NALU and ways to fix it. Though the models presented outperform NALU, they still come with their own issues, namely they do not support division, and (admittedly, well corroborated with analysis) are not joined in a single, NALU-like model, that can learn multiple arithmetic operations. The paper does a great analysis of the models’ issues, with an experimental setup that highlights these issues, however, it does that on only one task from the original paper, and a(n insufficiently justified) modification of another one (multiplication of MNIST digits)---it does not extensively test these models on the same experimental setup as the original paper does.

Typos and smaller issues:
- Throughout the text you say that NMU supports large hidden input sizes? Why hidden??
- Figure 4 is identical to figure in D.2
- Repetition that E[z] = 0 is a desired property in 2.2, 2.3, 2.4
- In Related work, binary representation -> one-hot representation
- Found empirically in () - remove parentheses and see
- increasing the hidden size -> hidden vector size?
- NAU and NMU converges/learns/doesobtains -> converge/learn/do/obtain
- hard learn -> hard to learn ?
- NAU and NMU ...and improves -> improve
- Table 1 show -> shows
- Caption Table 1: Shows the - quite unusual caption (treating Table 1 as part of the sentence), would suggest to rephrase (e.g. Comparison/results of … on the … task). Similarly for Table 2...and Figure 3
- experiemnts -> experiments
- To analyse the impact of each improvements….. - this sentence is missing a chunk of it, or To should be replaced by We
- Allows NAC_+ to be -> remove be
- can be express as -> expressed
- The Neural Arithmetic Expression Calculator () propose learning - one might read this as the model proposes, not the authors / paper / citation propose…(also combine or combines in the next line)
- That the NMU models works -> model works? models work?
- We choice the -> we choose the 
- hindre -> hinder
- C.5 seperate -> separate
- There’s a number of typos in the appendix
- convergence the first -> convergence of the first?
- Where the purpose is to fit an unknown function -> I think a more appropriate statement would hint at an often overparameterization in practice done when fitting a(n unknown) function

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper aims to address several issues shown in the Neural Arithmetic Logic Unit, including the unstability in training, speed of convergence and interpretability. The paper proposes a simiplification of the paramter matrix  to produce a better gradient signal, a sparsity regularizer to create a better inductive bias, and a multiplication unit that can be optimally initialized and supports both negative and small numbers.

As a non-expert in this area, I find the paper interesting but a little bit incremental. The improvement for the NAC-addition is based on the analysis of the gradients in NALU. The modification is simple. The proposed neural addition unit uses a linear weight design and an additional sparsity regularizer. However, I will need more intuitions to see whether this is a good design or not. From the experimental perspective, it seems to work well.
Compared to NAU-multiplication, the Neural Multiplication Unit can represent input of both negative and positive values, although it does not support multiplication by design. The experiments show some gain from the proposed NAU and NMU.

I think the paper can be made more self-contained. I have to go through the NALU paper over and over again to understand some claims of this paper. Overall, I think the paper makes an useful improvement over the NALU, but the intuition and motivation behind is not very clear to me. I think the authors can strengthen the paper by giving more intuitive examples to validate the superiority of the NAU and NMU.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors extend the work of Trask et al 2018 by developing alternatives to the Neural Accumulator (NAC) and Neural Arithmetic Logic Unit (NALU) which they dub the Neural Addition Unit (NAU) and Neural Multiplication Unit (NMU), which are neural modules capable of performing addition/subtraction and multiplication, respectively. The authors show that their proposed modules are capable of performing arithmetic tasks with higher accuracy, faster convergence, and more theoretically well-grounded foundations.

The new modules modules are relatively novel, and significantly outperform their closest architectural relatives, both in accuracy and convergence time. The authors also go to significant lengths to demonstrate that the parameters in these modules can be initialized and learned in a more theoretically well-grounded manner than their NAC/NALU counterparts. For these reasons I believe this paper should be accepted.

General advice/feedback:
- should provide an explanation of the row in Table 2 showing that a simple linear transformation is able to achieve accuracy and convergence times comparable to those of the NAU
- should provide an explanation of the universal 0% success rate on the U[1.1,1.2] sampling interval in Figure 3
- inconsistent captioning in Figure 2c, missing "NAC• with"
- should clarify in Section 4.1 that the "arithmetic dataset" task involves summing only *contiguous* vector entries; this is implied by the summation notation, and made explicit in Appendix Section C, but not specified in Section 4.1
- it is unclear what experiments you performed to obtain Figure 3, and the additional explanation in Appendix Section C.4 regarding interpolation/extrapolation intervals only adds to the confusion; please clarify the explanation of Figure 3, or else move it to the Appendix
- the ordering of some of the sections/figures is confusing and nonstandard: Section 1.1 presents results before explaining what exactly is being measured, Figure 1 shows an illustration of an NMU 2 pages before it is defined, Section 3 could be merged with the Introduction

Grammatical/Typesetting errors:
- "an theoretical" : bottom of pg 2
- "also found empirically in (see Trask et al. (2018)" : top of pg 4
- "seamlessly randomly" : middle of pg 5
- "We choice" : middle of pg 6
- inconsistent typesetting of "NAC" : bottom of pg 6
- "hindre" : middle of pg 8
- "to backpropergation" : bottom of pg 8
- "=≈" : top of pg 17
- "mathcalR" : bottom of pg 23
- "interrest" : bottom of pg 24
- "employees" : bottom of pg 24
- "models, to" : bottom of pg 24
- "difference, is" : bottom of pg 24
- "consider them" : bottom of pg 24
- "model, is" : top of pg 25
- "task, is" : top of pg 25
- "still struggle" : top of pg 25
- "seam" : top of pg 27
- "inline" : top of pg 27
- inconsistent typesetting of "NAC" : top of pg 27

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors propose the Neural Multiplication Unit (NMU), which can learn to solve a family of arithmetic operations using -, + and * atomic operations over real numbers from examples. They show that a combination of careful initialization, regularization and structural choices allows their model to learn more reliably and efficiently than the previously published Neural Arithmetic Logic Unit.

The NALU consists of two additive sub-units in the real and log-space respectively, which allows it to handle both additions/subtractions and multiplications/divisions, and combines them with a gating mechanism. The NMU on the other hand simply learns a product of affine transformations of the input. This choice prevents the model from learning divisions, which the authors argue made learning unstable for the NALU case, but allows for an a priori better initialization and dispenses with the gating which is empirically hard to learn. The departures from the NALU architecture are well justified and lead to significant improvements for the considered applications, especially as far as extrapolation to inputs outside of the training domain.

The paper is mostly well written (one notable exception: the form of the loss function is not given explicitly anywhere in the paper) and well executed, but the scope of the work is somewhat limited, and the authors fail to properly motivate the application or put it in a wider context.

First, divisions being difficult to handle does not constitute a sufficient justification for choosing to exclude them: the authors should at the very least propose a plausible way forward for future work. More generally, the proposed unit needs to be exposed to at least 10K examples to learn a single expression with fewer than 10 inputs (and the success rate already drops to under 65% for 10 inputs). What would be the use case for such a unit? Even the NMU is only proposed as a step on the way to a more modular, general-purpose, or efficient architecture, its value is difficult to gauge without some idea of what that would look like.


review_title: Official Blind Review #2

Meta-review: This paper extends work on NALUs, providing a pair of units which, in tandem, outperform NALUs. The reviewers were broadly in favour of the paper given the presentation and results. The one dissenting reviewer appears to not have had time to reconsider their score despite the main points of clarification being addressed in the revision. I am happy to err on the side of optimism here and assume they would be satisfied with the changes that came as an outcome of the discussion, and recommend acceptance.
Decision: Accept (Spotlight)