Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new method for the hardware implementation of deep neural networks (DNNs) which are quantized but use floating-point (FP) activations. Prior art has attempted to solve this problem by devising new data formats e.g., bfloat16 and Block Floating Point number for weights and activations, but they have limitations (high overhead to overcome rounding errors and lack of configurability for given hardware platform respectively). 

The paper presents an alternative approach which optimizes the matrix multiplication (MatMul) operations using a special design block, iFPU. It involves decomposing quantized weights into bit-planes (taking out common factors and reducing weights into a series of +1/-1 values). Activation values (FP/bfloat16 format) are similarly reduced by normalizing them to the largest common exponent and converting the mantissa bits to integers. Thereafter, in this reduced space the multiply ops are converted to a series of integer adds (subs) which helps reduce the Matmul complexity.

 Further, the authors further simplify the add/sub ops by truncating the least significant bits of the integer values added upto the precision of the output plus a few bits. This truncation helps reduce the complexity of integer add/sub operations significantly. Once the add operations are done, the resulting values are converted back to FP format followed by scaling with values in each bit-plane (removed from weights) and a floating point add.

The authors demonstrate through detailed exploration of FP add operations that their proposed truncation technique doesn’t cause any error higher than the rounding error of existing FP add operations. Additionally, they also demonstrate that this continues to hold when testing on real networks by showing a strong match between the activation values calculated (in network layers) and comparing the output accuracy generated by FP and iFPU HW implementation. 

Lastly, authors compare their proposed iFPU implementation with two other HW implementations, FP (default) and FP-ADD, which does similar decompose and add/sub in bit-plane space steps as iFPU without any truncation. The results show that iFPU achieves higher energy efficiency and area efficiency than the other implementations.

strength_and_weaknesses: The paper presents a good overview of the target application by highlighting the limitations of the floating-point format and prior approaches. The diagrams presented are clear and help explain the proposed ideas. The authors derive the conclusion in Remark1 with detail which helps establish the important point that iFPU maintains the same error at the output despite lowering complexity of the operations performed compared to FP. The presented experiments are useful and highlight the capabilities of the proposed hardware implementation, iFPU as opposed to prior art. 

While the authors present their ideas on truncation for FP multiplication, the idea of truncation in binary multipliers is not new, e.g., N. Petra, et. al., "Truncated Binary Multipliers with Variable Correction and Minimum Mean Square Error," in IEEE Transactions on Circuits and Systems I: Regular Papers, 2010, presents similar ideas which discuss truncating the bits from the partial products derived after binary multiplication and their impact on the output accuracy. In fact, the authors in Petra et. al., paper define an additional correction function estimated based on the truncated bits which further helps reduce the error measured. I request the authors to cite this paper and elaborate how their contributions are unique from those in that paper. 

An important assumption in the proposed technique is the ability to simplify the quantized weight values to respective bit-planes, which allow simplifying the multiply ops to add/subs. For the paper the authors assume the use of 4-bit quantized weights which might favor the simplification of weights into +1/-1 easily. But often it is harder to achieve the same level of accuracy for a given ML task with a DNN with lower quantization bit sizes. As the quantization bit size increases to 8/16bit integer values, how do the benefits of the proposed iFPU scale? Since for larger bit size the number of bit-planes will increase which will increase the cost of the Scale and Accumulator block shown in Figure 5. 

FP-Add is a useful baseline added by the authors which highlights the individual contribution of the bit-plan decomposition and truncation ideas used. I am curious why authors select Int8 Matmul engine considering that the iFPU assumes 4-bit quantized weights, it would be helpful to add an Int4 Matmul engine data as well cause that would be the baseline of the energy efficiency achievable.

clarity,_quality,_novelty_and_reproducibility: Stating the size of the adder and multiplier blocks used in the PE and Scale and Accumulator blocks will be helpful to project the scalability to different quantization widths. Kindly also add figures showing the internal circuitry of these blocks to help improve the understanding of their design, similar to the pre-alignment unit. 

Questions regarding the novelty and reproducibility (to larger quantized bit-size networks) are listed in Strengths and weakness section. 


summary_of_the_review: The paper presents strong results showing the accuracy and energy/area efficiency of the proposed HW implementation of FP activation and 4-bit integer quantized weights Matmul operations. But its not obvious that the benefits observed will scale for other quantization bit-widths. Further, as shown in Figure 9 a significant part of the energy/area efficiency benefits are derived due to truncation of FP add operations which relies on similar experiments as another paper referenced above. If that is removed from the results, the improvement in energy/area efficiency due to the other ideas presented in the paper are smaller.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use integer arithmetic on truncated floating-point operands. The idea is that FP arithmetic anyway suffers from numerical errors associated with rounding. Therefore, the author propose that FP arithmetic can be replaced altogether by pre-aligned integer arithmetic. Essentially, the representation is kept in FP, which preserves accuracy, but compute is done in integer, which is more efficient.
strength_and_weaknesses: The proposed idea is very interesting. However, I have several questions to the authors:

1) The proposed work focuses much on the compute efficiency. I wonder if this is really critical, as the memory access cost is known to usually be a bottleneck for DNN accelerator architecture. Can the authors comment on how significant reducing the cost of compute is when memory access is taken into account?

2) The authors claim that FP is needed to maintain accuracy. This is usually true when quantization methods do not have a good handle on the range vs resolution trade-off in the data. However, for a properly clipped quantizer, integer quantization can be shown to be as accurate as FP [1]. In such a case, integer arithmetic can naturally be used. Can the authors compare their work to such an fully-integer baseline?
[1] Sakr, Charbel, et al. "Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training." International Conference on Machine Learning. PMLR, 2022.

3) I am curious as to how the DNN accuracy number were obtained in the experimental section. Specifically, how is the FP to INT MatMul emulation performed? The implementation of CUDA kernels, which are invoked by deep learning frameworks, are fast because of the reduction operations that can eliminate the need to store intermediate results in memory. To emulate the proposed method, the authors need to perform an element-wise multiplication, followed by truncation, and then successive additions and truncations. This is very memory and time consuming. Do the authors have a trick to bypass these challenges in the emulation? Or are the experiments simply very slow?
clarity,_quality,_novelty_and_reproducibility: The proposed concept is presented clearly and is novel. Regarding reproducibility, please answer question 3 above.
summary_of_the_review: The paper introduces a new approximate way of computing FP MatMuls using integer arithmetic. The idea is novel and interesting. I have a few questions I hope the authors can answer.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed an efficient matrix multiplication hardware. Concretely, the paper performs MatMuls with quantized weights and floating-point activations operants and outputs a numerical approximation of the floating-point result.
strength_and_weaknesses: # Strength 
- The paper introduces an interesting approach for inferencing neural networks faster.
- The method only requires the weights to be quantized. This puts much less stress on the stability of the numerics and training compared to a fully quantized network where both the weights and the activations are quantized.
- Extensive experimental evaluation on NLP and CV tasks with comparison to fp32 and bf16 computations.

# Weaknesses
The paper misses an evaluation of important computer vision architectures. Particularly, the paper evaluates VGG9, which is undoubtedly outdated and not relevant anymore (only 92% on CIFAR, where 96% can be easily obtained with a ResNet and CUTOUT augmentation).
Moreover, the ResNet variant for ImageNet is rather small (ResNet18), whereas the "gold standard" for CV evaluations is the ResNet50. I believe larger models or new models (EfficientNet, ConvNext) would make the evaluation of the CV part more relevant for ICLR 2023.

The method seems to rely heavily on weight quantization, i.e., requiring the assumption that the network still performs well if the weights are expressed by low-bit integers.
It would be, therefore, relevant for the paper to provide results on how much the performance of a network suffers from the weight quantization.
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clear to understand.

However, Figure 1 would be more helpful if the example calculation was done on multiply and addition (as it appears in MatMul) to explain how the pre-alignment process works. 

Questions:
The results on BERT-base seem inconsistent with the numbers presented in the original paper and on the huggingface model hub. Please elaborate.
summary_of_the_review: Overall interesting approach with high potential real-world usage due to only weights required to be quantized.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: For neural network inference, the paper proposes to use integer arithmetic on (truncated) floating-point operands, specifically for already quantized neural network weights.
While reviewers liked the efficiency improvements, questions remained on the level of novelty, and it only remains narrowly on the accept side even after the author feedback and discussion.

We hope the authors will incorporate the several points mentioned by the reviewers in the final version. 
We'd also encourage authors to cite some more of the related work, including e.g.

- Deep Neural Network Training without Multiplications,
  Tsuguo Mogami, 2020

which the authors have already commented on in the discussion on openreview, thanks for that
justification_for_why_not_higher_score: Concerns remained on the level of novelty
justification_for_why_not_lower_score: This can be bumped down, given the high bar for ICLR. Reviewers please ackn the feedback and get back to us
Decision: Accept: poster