Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper study possible adaptations of Neural Arithmetic Linear Units (Trask 2018) to better handle division. The authors first propose improvements for training Real Neural Power Unit (Heim 2020), a neural arithmetic unit which performs division as substraction in log-space, using cosines to handle signs, and a gating mechanism to select operands, while constraining all weights to remain close to 0, 1 or -1. They show that introducing L1 regularisation, gradient clipping, and tuning weight constraint and initialization greatly improve its accuracy when dividing numbers.

They introduce two new designs, the Neural Reciprocal Unit, which functions as an addition in log-space with weights constrained to -1, 0 and 1, and can therefore perform multiplication or division of numbers (and "unselect" them when weights are zero, which dispenses of the gating mechanism), and the Neural Reciprocal Modified Unit, where the reciprocals are provided as additional inputs, constraining weights to 0 and 1 (as a selection mechanism), and adds a cosine function to handle signs. 

Testing over a variety of generated samples, they show that whereas the new designs improve on the original Real NPU, most of the still struggle on input of mixed signs, and have difficulty working with numbers close to zero, and learning both selection (choosing the operands) and division.
main_review: **Originality** 
The paper builds on existing designs and experiments. The techniques introduced in section 5 to improve performance of Real NPU are standard. The new designs proposed are incremental or crossovers between existing NALU modules. 

**Clarity** 
The paper is very technical, but well written, and provides a lot of details about proposed architectures and experiments. 

In section 3.1, it would be useful to define  the symbol used for element-wise multiplication. Also, the notation W^r W^i for the real and imaginary weight matrice conflict with the r symbol used for the gate. The ordering of the interpolation datasets in figure 3 and 6 is counter-intuitive: decreasing order of negative numbers, then increasing of positive)

**Quality** 
Very detailed and interesting experiments. The results are extremely convincing, and the experiments are the strong point of this paper. 

A few questions and comments :

1- In section 3.2: is the use of tanh to deal with the local discontinuity of the derivative of absolute value really necessary? RELU and absolute value activation functions are routinely used in feedforward networks, despite having the same gradient discontinuity. 

2- All the data in the training and test set are uniformly sampled from relatively small intervals. This has an impact on the distribution of results. In the case of multiplication and division, it might be interesting to test over a very large interval, with a logarithmic (Benford) distribution.

3- In section 5, experiments with L1 regularization suggest that it is needed but in very small amounts. Could L2 regularization be a better choice? (as it would have the same stabilizing effect, but less enforce sparseness).  

4- Being able to extrapolate out of the training distribution support is one of the main justification of NALU. In the paper, most experimental results are calculated from interpolation test set. Adding more extrapolation results would reinforce the paper.

**Significance** 
The main significant result is the improvement of the Real Neural Power Unit training. NRU fail to convince because of their problem on mixed sign operands. The experiments do a good job at documenting the difficulty of training NALU on division.

limitations_and_societal_impact: They have
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: This manuscript proposed new types of neural arithmetic modules called Neural Reciprocal Units (NRU) and Neural Multiplicative Reciprocal Units (NMRU). NRU and NMRU extend Neural Multiplication Units by applying power terms and reciprocal inputs, respectively, and thus they enable divide operation. They also proposed improvement method of Neural Power Units (NPU).
main_review: This manuscript introduces novel arithmetic layers, which enables divide operation. Proposed NRU and NMRU achieved higher success rates and fast learning ability compared to Real NPU in various experiments.
The manuscript seems to cover too broad subject. Proposing an improvement method of NPU does not seemed to be highly related to NRU and NMRU. It would be better to add comments explain relationship among them, or focusing more on NRU and NMRU. Additionally, there is a concern about novelty on the proposed improvement method of NPU, since they are briefly commented on ‘Neural Power Units’(NIPS 2020). 
In Equation (5), they used summation of cosine function to calculate sign value of output, but I think it should be product of cosine like Equation (5) in supplementary material.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors explore neural arithmetic architectures that could learn to perform division tasks, both in the case of input redundancy as well as with only 2 operators. Besides improving an existing architecture (NPU), other two related architectures are introduced (NRU and NMRU), both based on the idea that division can be framed as a multiplication of reciprocals. Extrapolation capabilities are probed by considering testing intervals that do not overlap with training intervals. The newly introduced architectures achieve better performance, though in the case of input redundancy all models are still error prone.
main_review: Originality: This research work appears mostly incremental in nature. It extends and improves previous model architectures, which are empirically tested over previously established tasks.

Quality: The research topic is very actual, given the recent interest in testing mathematical learning skills of deep nets, though the scope of the work appears somewhat limited given the specificity of the research approach. The applicability of these models to broader contexts / domains is not particularly evident. The paper builds on the methodological setup established in previous work, which appears sound and well-designed. As a possible improvement, given that the proposed NMRU achieves reasonable accuracy over all tested ranges it would be interesting to further test its extrapolation capabilities by including even more challenging numerical ranges.
Overall, my feeling is that this is a well-conducted research work, but probably NeurIPS might not be the most appropriate venue to present these findings.

Clarity: The paper is well written and properly organized. I found the opening sentence a bit convoluted: I think there could be more effective ways to introduce to the reader the challenging aspects of the division problem. Moreover, the authors do not  clearly explain why we should initially focus on the problem of learning division when there is input redundancy.

Significance: The issue of learning mathematical knowledge with deep nets is timely and challenging, and the deep learning community is spending increasing efforts in this direction. However, the significance of this paper for the broader NeurIPS community might be limited, because the current setup is very specific. The “broader impact” proposed by the authors (e.g., ability to produce transparent generalizable solutions) appears quite speculative and should be more clearly demonstrated.

limitations_and_societal_impact: The authors properly discuss the limitations of their research approach. The potential negative societal impact discussed by the authors seems quite speculative.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper is an addition to the line of study concerning neural arithmetic. Several new architectures are proposed to tackle the problem of learning the division operation. Out of the four arithmetic operations division is arguably the more challenging one and previous works have struggled with it.

Through several experimental results, the new architectures are shown the outperform previous designs, both in training accuracy and in other desirable properties
main_review: Firstly, I should mention that I am a non-expert in this area. After reviewing the relevant literature I find the topic of the paper interesting and relevant. Overall, the paper is well written and presents the motivations in a compelling way.

However, there are two issues which I would like to raise:

1. The ideas underlying the new architectures seem to be a bit incremental with respect to previous works. While it is definitely a serious paper that merits publication,  I am just not 100% convinced that NeurIPS is the correct venue.

2. While the authors support their choice of architecture with a plethora of experiments, I feel that the paper lacks some theoretical foundations, which do appear to some extent in previous works. This makes it hard to interpret some of the experimental results. For example (but definitely not the only one), in Figure 3 why should I expect to have different behaviors between different interpolation ranges. How can one explain the fact that the real NPU is competitive with the new architectures on some ranges, and why those specific ranges? 
Moreover, as the authors note, the most challenging ranges are those that include 0. However, only one such range is tested ([-2,2]). I would expect to have more such ranges, both in different scales and with different levels of symmetry/asymmetry.

Minor comments:
- Display (1): Perhaps it is better to mention that the multiplication is the Hadamard (or Schur) product.
limitations_and_societal_impact:  The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: This paper study possible adaptations of Neural Arithmetic Linear Units (Trask 2018) to better handle division. 
While the contribution is worthwhile, the reviewers concern that it is too marginal and of limited scope
Decision: Reject