PLUM: Improving Inference Efficiency By Leveraging Repetition-Sparsity Trade-Off
Authors: Sachit Kuhar
Yash Jain
Alexey Tumanov
Venue: Accepted by TMLR ()

Abstract: Efficient inference of Deep Neural Networks (DNNs) on resource-constrained edge devices is essential. Quantization and sparsity are key techniques that translate to repetition and sparsity within tensors at the hardware-software interface. This paper introduces the concept of repetition-sparsity trade-off that helps explain computational efficiency during inference. We propose PLUM, a unified co-design framework that integrates DNN inference systems and quantization (forward and backward pass) to leverage the repetition-sparsity trade-off to improve inference efficiency. Our results demonstrate that PLUM’s quantization method is more accurate than binary quantization with the same number of non-zero weights. Detailed analysis indicates that signed binarization generates a smaller distribution of effectual (non-zero) parameters nested within a larger distribution of total parameters of latent full-precision weights for a DNN block. Finally, the proposed PLUM framework achieves a 26% speedup on real hardware, doubles energy efficiency, and reduces density by 2.8× compared to binary methods while retaining top-1 accuracy when compared to prior-art methods for ResNets on ImageNet (by achieving 66.2% top-1 accuracy), presenting an alternative solution for deploying efficient models in resource-limited environments

Published in Transactions on Machine Learning Research (10/2024)
PLUM: Improving Inference Efficiency By Leveraging Repetition-Sparsity Trade-Off
Sachit Kuhar; Yash Jain, Alexey Tumanov Georgia Institute of Technology
Reviewed on OpenReview: https: // openreview. net/ htt//frmm? id- IEKt/MSSC 1m
Abstract
Efficient inference of Deep Neural Networks (DNNs) on resource-constrained edge devices essential. Quantization and sparsity are key techniques that translate to repetition and sparsity respectively within tensors at the hard interface. This paper introduces the concept of repetition-sparsity trade-off that helps explain computational efficiency during inference. We propose PLUM, unified co-design framework that inte- grales DNN inference systems and quantization (forward and backward pass) to leverage repetition-sparsity trade-off to improve inference efficiency. Our results demonstrate that PLUM's quantization method is more accurate than binary quantization with the same number of non-zero weights. Detailed analysis indicates that signed binarization generates a smaller distribution of effectual (non-zero) parameters nested within a larger distribution of total parameters of latent full-precision weights for a DNN block. Finally, the proposed PLUM framework achieves a 26% speedup on real hardware, doubles energy efficiency, and reduces density by 2.8x compared to binary methods while retaining top-1 accuracy when compared to prior-art methods for ResNets on ImageNet (by achieving 66.2% top-1 accuracy), presenting an alternative solution for deploying efficient models in resource-limited environments. Code available at https:// httpl/uu...ooocacciaaaaaaaaaaaaa
1 Introduction
Despite significant strides in accuracy, the burgeoning complexity and resource demands of deep learning models pose challenges for their widespread adoption across a wide range of domains (He et al.) 2016 Brown et al. 2020 Graves et al.) 2013 Jain et al.) 2022 Mandlekar et al.) 2021 Jain et al., 2024). This requires the development of innovative techniques to enhance DNN elficiency during inference on edge devices. Two such techniques have been studied extensively: binarization and sparsity. Binarization, a form of quantization, results in weight repetition as only two values appear repeatedly in the weight tensor (Courbariaux et al. 2015). This approach significantly trims the memory footprint of the weight tensor, thereby decreasing memory I/O during inference (Hegde et al., (018)). In contrast, sparsity leads to zero weight values. Since anything multiplied by zero is zero, weight sparsity leads to ineffectual multiplications (Wu et al.) 2022]) This approach reduces memory T/O during inference by nol reading activations that would be multiplied by zero weights (Gong et al. 2020). Thus, both these techniques are geared towards reducing memory 1/O during inference.
DNN inference efficiency is usually achieved by leveraging either binarization or sparsity. Introducing spar- sity by adding a zero-valued weight in conjunction with binary weights is termed ternary quantization. Ternary was conceived with the reasonable assumption that transitioning from binary to ternary models would only minimally impact inference latency due to the effect of zero weights (Li et al. 2016). However, advances in the contemporary hardware-software systems have revealed substantial increase in latency during such transitions (Prabhakar et al. 2021 Fu et al. 2022 This work aims to perform faster inference on contemporary hardware-software systems while retaining model accuracy.
Principal Investigator; Correspondence: Shharssachtt&@mmil.comm

---

Published in Transactions on Machine Learning Research (10/2024)
Classic Partitioned Design
PLUM's Unified Framework
Quant Design
Quant Co-Design
Backward Propagation
PLUM Quantization:
Signed-Binary
DNN Architecture (ResNet)
Forward Propagation
Deploy DNN
DNN Architecture (ResNet)
Deploy Co-designed System
System Co-Design
System
System Design
PLUM Inference
Weight Repetition
Keree
Weight Sparsity
Legend: Updating Latent FP Weights
Quantization
Inference System
The conventional, isolated approach where DNN inference systems and quantization methods are designed separately, resulting in being ignorant of repetition-spariily trade-off, leading lo inefficient inference. On the right: PLUM, a unified design framework, performs quantization-systen co-design to exploit the repetition-sparsity trade-off, thereby enhancing computational efficiency.
Figure 1:
On the left:
The key insight of this work the concept of the repelilion-sparsiiy Irade-off. This trade-off explains the inference inefficiency of binary and ternary weight quantization. A traditional binary network chooses max- imization of weight repetition while being ignorant of weight sparsity, whereas ternary network introduces weight sparsity at the expense of weight repetition. For instance, transitioning from binary to ternary net- works increases the number of possible unique 3x3 2D filters from 512 (29) to 19683 (3"). This makes it exponentially harder to extract efficiency using filter repetition.
The conventional approach to deploying efficient DNNs has largely been a two-stage process: train- ing the DNN with binarization or ternarization (Bai at. 2018). followed by formulating system design that reduces memory 1/O by leveraging both weight repetition and/or sparsity (Hegde et al. 2018). This partitioned design approach has revealed a pitfall: increased memory T/O during inference due to the repetition-sparsity trade-off. However, recent ad- vancements in DNN inference systems can now ex- ploit repetition and sparsity to improve efficiency shown in (Prabhakar et al. 2021 Fu et al. 2022). To this end, this paper proposes quantization-syytee
Conventional System w/ Binary
Vs
PLUM
Inference Speed
Inference Energy
+26% speedup
2x reduction
2. 2.8x reduction
Density
+2. 2..5% accuracy
co-design framework called PLUM (PLUs-Minus)
Accuracy
Framework as seen in Figure 0
vs Total Parameters
Accuracy Effectual Parameters
This framework aims to demonstrate and lever- age the existence of repetition-sparsity trade-off. PLUM re-imagines different levels of stack working together in synergy to enhance computational efficiency during inference while retaining the model's accuracy. As seen in Figure compared to a classic partitioned design for binary, the PLUM framework improves inference efficiency (with respect to energy, latency, and throughput) and reduces model densily to allow the of bigger models, resulting in more accuracy. Based our analysis and empirical evidence, we envision that the PLUM framework has the potential to positively shape the evolution of more efficient and high-performing deep learning models.
Figure 2: PLUM vs. Prior-Art: For RosNet Im- ageNet, the pronounced spread indicates that PLUM holistically outperforms prior-art method of parti- tioned design using binary quantization. It retains competitive accuracy and pushes the Pareto front, exhibiting a +2.5% improvement when both methods employ comparable number of effectual parameters. Moreover, our method enhances inference efficiency, achieving 26% speedup, doubling energy efficiency, and reducing density by 2.8x for the same backbone.
2

---

Published in Transactions on Machine Learning Research (10/2024)
We make the following contributions in our work:
We present the concept of repelition-sparsity trade-off along with the PLUM framework. While the former explains the inference inefficiency of binary and ternary weight quantization, the latter leverages (and thus proves) this insight to improve the efficiency of DNN inference.
Compared to prior-art quantization schemes, we demonstrate that PLUM's signed-binary quantiza- tion scheme retains model accuracy (of 90.7% and 66.2% top-1 accuracy on CIFAR1O and ImageNet) while requiring significantly fewer effectual parameters. We offer detailed insights on co-dependent learned features in the DNN when using PLUUM by visualizing the distribution of quantized and latent full-precision weights.
We perform DNN inference on Intel CPUs to prove the repetition-sparsity trade-off and demonstrate that the PLUM framework leads to the fastest inference. We further demonstrate the benefits of sparsity during inference when using PLUM via energy reduction experiment.
2 Background
Quantization Quantization in deep learning involves mapping of real-valued numbers to a select set of discrete values to streamline and optimize computations. The Binary Quantization technique assigns any real-valued number to either +1 or (Courbariaux et al. 2015). The primary objective behind this is to simplify operations in DL hardware, turning multiply-ande accumulates (MACs) into mere accumulate operations (Rastegari al. 2016). Conversely; Ternary Quantization designates values +1, -1, or 0 (Li et al. 2016). This determination is based on threshold, A. Values exceeding A are assigned +1, those below receive -1, while others become zero. For both quantization methods, the output values can undergo scaling using a factor, a (Bulat & Tzimiropoulos 2019 Bulat et al.) 2019 There is rich literature on improving the accuracy of these methods (Qin et al. 2023 Zhang et al. 2018 Gong et al. 2019 Qin et al. 2020b; Hu et al.] 2018: Pouransari et al. 2020 Liu et al. 2018). These methods are not aware of the repetition-spant trade-off. In PLUM framework, we combine the benefits of both binary and ternary create efficient. inference.
multiply-and-accummlate
Weight Sparsity Weight Sparsity in DNN implies that there is repetition of weight with value equal to zero. The basic idea is that since 0x = 0 for any real-valued scalar x, if the weight is zero, the multiplication is ineffectual and should be skipped (Gong et al., (2020). Sparsity in weights is static during inference (Dai let al., 2020). Therefore, if the value of weight is zero, we can choose not to load activations corresponding to that weight (Qin et al.) [2020a]. This can lead to a reduction in data movement, memory accesses, and MACs thereby reducing computations and hence resulting in efficient DNN inference. This approach has been effective on ASICs and general-purpose devices (Hegde al. 2019 Wang et al., 2021 Dai et al., 2020;
Gong al. 2020
Nayak et al. 2023 PL.UM exploits weight sparsity during inference to improve efficiency.
Weight Repetition Quantization of weights leads to the same value being repeated again and again in the weight tensor. This phenomenon is known as weight repetition (Hegde et al., 2018 Sze et al. 2020). Since the weights are fixed during DNN inference (Dai et al.) (020), this leads to opportunities for improving efficiency during inference with respect to time and energy by exploiting the repetition of weights and reducing memory accesses (Sze al. 2020). This concept was first introduced BNN (Courbariaux et al. 2016). They demonstrated that for a CNN, only 42% of filters are unique per layer on average which can lead to reducing the number of operations by 3x.
UCNN (Hegde et al. 2018) demonstrated how to leverage widespread and abundant weight repetition present across range of networks (like ResNet, GoogleNet) that are trained on a variety of datascts. It reorders the weights and thus reorders activations, leading to reduced memory access and arithmetic operations, resulting in 4x more efficient inference. For example, if the filter weights are {a, b, a, a] and activations are [w, x. y, 2], UCNN would reorder it as a X (w + y + 2) + b x (x) for efficient inference (Sze et al., (020) but does not exploit weight sparsity. SumMerge (Prabhakar et al., 2021) uses both weight repetition and weight sparsity for efficient DNN inference. For example, if b = 0 in the previous example, SumMerge would
3

---

Published in Transactions on Machine Learning Research (10/2024)
Full Precision
Ternary
Filters
Charnels
Repetition
Spersity
Efficiency
Rews
Binary
Good
X
Average
(+1.-1)
Columns
Binary
PLUM's Quant: Signed-BBinn
Ternary
Poor
Poor
(+1.
Signed
Binary
Good
Good
(1.0),-1.0y
Ful Precision
Poative Weight
Negative Weigt(-1)
Zero Weight(()
Figure 3: Concept Diagram the left: The diagram shows the comparison of Binary, Ternary, and Signed Binary Quantization in terms of visual representation of their quantized weights. Qualitative Evaluation on the right: The table qualitatively evaluates them in terms of weight sparsity, weight repetition, and inference efficiency.
compute X (w + during inference. While it reduces arithmetic operations up to 20x, it uses a greedy strategy to reduce arithmetic operations. Q-Gym (Fu et al. (002)) treats weight repetition as a combinatorial optimization problem to perform inference on CPUs and GPUs. But, all these methods are designed for standard pre-built quantization and assume repetition and sparsity quantized DNNs be rigid. All these systems do not perform the dot product of activation and weight tensors in a single step. Instead, they split the dot products of input and weights into smaller blocks via tiling to improve data locality during inference. PLUM exploits this for faster inference through quantization and DNN-inference co-design.
a range of DNNs trained diverse datasets (Hegde et al. 2018). The impact of weight sparsity weight repetition be most easily visualized for x convolution filters shown in Figure 3 Binary models have 2 unique filters while ternary models employ 3 unique filters. Recent works (Hegde et al. 2018 Prabhakar et al. 2021 Fu et al. 2022) show a significant slowdown in inference when using ternary when compared with binary. This extends to bit-utilization; binary systems can potentially be represented by a single bit, whereas ternary systems require two bits due the inclusion of zero-valued weights. Moreover, recent findings indicate that runtime doubles when transitioning from one to two bits during the deployment of ResNet18 on ARM CPUs, showcasing the influence of bit-width distinction inference time (Cowan al. (020).LUU aims to balance out exploitation of repetition and sparsity for faster inference.
Repetition, Sparsity & Inference Latency Weight repetition prevalent
3 PLUM
The objective is to create a framework that leverages trade-off between repetition and sparsity. aspires to retain competitive accuracy while combining the merits both repetition and sparsity for faster inference.
3.1
Leveraging Repetition-Sparsie Trade-off
Conventional binary networks offer 2 unique weight choices per element, creating dense setup. Conversely, ternary networks take additional bit to represent 3 unique weight choices per element to create sparse setup. Continuing the 3 x 3 convolution filter illustration from the previous section, binary models can have 29 unique filters. In contrast, ternary models employ 3 unique filters, inadvertently causing an exponential decrease in repetition by a factor of 38.4x. These design decisions can be interpreted as follows: Binary quantization prioritizing the maximization of weight repetition, overlooking the potential benefits of weight sparsity. In contrast, ternary quantization induces weight sparsity but compromises weight repetition. This disparity places the two schemes at opposite ends of spectrum, creating repelilion-sparsily trade-off (as
4

---

Published in Transactions on Machine Learning Research (10/2024)
PLUM
Weight1:
Weight2
PS = (a C)
X
O
X
y
Sopeliton-Sparshie Trade-off
Aware Design
Output (PS +++)
Input:
D
C
O
O
O
O
O
Output 2 (PS_0
e
x
x
x
x
X
PS 0 (a c) Output (PPSS++)
PS (d
Oulput_1 Input" Weight1 2 Input Weight2
Prior Art System
Output (PS_0
PPS_0+0 + x)* (PS_
Weight1
Weight2
Conventional Inference w/ Binary Quant
Figure 4: PLUM framework leads to efficient inference as acknowledges repetition-sparsity trade-off off through co-design: Visualizing inference when using recent systems (Prabhakar al., 2021; Fu et al. 2022 Weight repetition enables binary models to skip work by re-using partial sums within and across filters. PLUM takes this even further by reducing the number of effectual operations by leveraging sparsity while retaining repetition. (details in Supp B
in Figure In given DNN block, we have the flexibility to assign each latent full-precision weight one of four possible quantization function value sets: (1,-1, {1,0), (0,-1), and {1,0,-1 This variety of assignments to individual weights allows for the creation of unique configurations within the block, each representing a distinct point along the repetition-sparsite spectrum. PLUM framework aims to identify a more efficient point on the repetition-sparsity spectrum. PLUM's quantization, i.c., signed-binary makes the design decision to use two quantization functions with value sets {1,0} and {0,-1} as they are sparse and could potentially be represented using one bit per latent. full-precision weight.
Let the convolutional layer have R x S kernel size with C input channels and K filters. The quantization function takes latent full-precision weights W as imput and outputs the quantized weight Wruant The quantized weight. Wyuand would be the product of the sign-factor B and the bitmap U.
Q: W Wonant
Wauant = BU
(1)
V B -1; UE {0.1}
(2)
3.2 Co-design Exploration
Developing an efficient framework requires redesigu due to its distinct nature from existing methods, latent FP weight can be assigned at any of the two quantization functions. This leads to challenges: (1) retaining the ability to learn meaningful representations and (2) providing faster inference when using state- of-the-art inference systems. Our method addresses these concerns by revisiting design decisions on different layers of the stack with the goal of improving efficiency while retaining accuracy. For in-depth insights into our experimental setup and implementation details, please refer the supplementary section C For additional ablations, please see supplementary E and
3.2.1 DNN inference in PLUM
The aim is to achieve faster inference by using two quantization functions for a DNN block. As explained in Section 2 (weight repetition paragraph), modern DNN inference systems enhance data locality during inference by splitting the dot products of inputs and weights into smaller blocks through tiling. For instance, Prabhakar et al. (2021) divides every filter into smaller blocks by introducing sub-dimension denoted C", representing the tile size along the C' dimension. We leverage this insight and make the design decision of local binarizalion. This allows a single processing step during PLUM inference lo see one signed binary quantization function, leading to efficiency as shown in Figure [ We define the region to be sign-binarized as R Sx C, where C, = max(C,, kC") and & E Zt For given 3, Wouant
JU where W E R RxSCC and U E 1) Hence, this results in global ternorization for a DNN block. Please refer section 5.1 for ablation on DNN inference for PLUM Intel CPU.
5

---

Published in Transactions on Machine Learning Research (10/2024)
Arch
FP
T
SB
%60, 1} filters
%(0,- filters
Acc
ResNet20
92.10
90.86
90.20
90.05
1
88,84
ResNet32
92.90
92.03
91.51
91.55
0.25
0.75
89.32
ResNet44
93.30
92.40
91.93
91.98
0.5
0.5
90.05
ResNet56
93.63
92.90
92.42
92.52
0.75
0.25
89.30
ResNet110
93.83
93.33
92.64
92.68
89.07
Table 1: PLUM's Sogned-Binary does not lead to accuracy degradation wrt Binary while leading to efficient inference.
Table 2: Equal percentage of usage of %60, 1} and %6[0, -13
regions leads to best model accuracy in PLUM.
EDET
Ace
Region
Acc
A
Acc
Disabled
88.4
C.
0.01 max [W]
90.01
Enabled
88.7
C. C/2
87.9
max [W]
90.05
Table 3: Enabling Adapted EDEt during backpropagation improves model accuracy PLUM.
Table 4: Region size of C2 C/ (wrt.
Table 5: Quantization functions used in PL.UM not sensitive to the choice of threshold A.
R S x Ct) leads competitive accuracy PLUM.
Ablations for PLUM: ResNet models trained on CIFAR-10 dataset for the method. Tables 2-5 use ResNet20. Default configurations for method are marked in grey "Adapted Error Decay Estimator
(EDE) Qin et al. (2020b]
details in Section 3.2.3
3.2.2 Quantization in PLUM: Signed Binary
The challenges arise from the need to design signed-binary quantization functions, that enable the model learn meaningful representations. Local binarization techniques must accommodate a global ternarization strategy, which presents additional challenges such as determining suitable regions for local binarization and assigning these regions to their designated signed binary quantization functions. In response to these challenges, we have designed our method, supported by systematic ablation studies conducted on ResNets trained with the CIFAR-10 datasct (sce supplementary | for additional ablations).
Signed Binary Quant Functions: Our method involves the strategic use two distinct quantization functions (as shown in Figure 3 In this section, we intend to meticulously design these functions, charac- terized by the value sets {0,1} where the sign-factor 81 equal 1, and {0.-1} where the sign-factor $_1 is -1. The scaling factor a; mirrors Bi where +1. Following ŞZhu et al., 2016), define the threshold value A 0.05 x maxx|W/) To assess the efficacy and sensitivity of the chosen thresholds, experiment with different As using signed-binary quantization (sce Section 4.2.2) to find stable performance across various configurations, as depicted in Table
These are defined as:
if W 2 A and B =
Wytaant
a-1
if W < -A and - Otherwise
(3)
0L oW
01
aT.
if W A and B 8=1 W - and B =1
-011
(4)
1
Otherwise
Intra-Filter Signed-Binary Quant In this section, we delve deeper into signed binary quantization by introducing variations in C, with respect. lo the constant value C, aiming to identify the optimal setting for C.. We design this approach to emphasize the changes in performance across different thresholds. Table [ assesses the impact on representation quality by adjusting the C, values during the training. The result shows that intra-filter signed binary co-design preserves competitive level of representation quality even with reduced C, and setting of C. C. works best.

---

Published in Transactions on Machine Learning Research (10/2024)
Inter-Filter Signed-Binary Quant Building on intra-filter, where C = C, this is the simplest signed binary quantization, and we term it as Inter-Filter Signed binary quantization. In essence, this method involves assigning each filter to one of two distinct signed binary quantization functions, emabling an efficient representation as C. = =C We contrast it with binary and ternary quantization schemes in an apples- lo-apples comparison. Table [ illustrates ResNets of varying depths trained using different. quantization schemes. The signed-binary quantization maintains comparable accuracy against traditional quantization approaches across different architectures. The largest chunk of convolution filter that can processed during PLUM inference at given instance is the entire convolution filter itself L.., C C. Inter-filter signed- binary quantization automatically results in the PLUM inference processing region corresponding to one quantization function at any given time.
Value Assignment of Signed- -inarry Quant Functions To fine-tune our network's performance, Ex- amining how different value assignments within filters affect accuracy is crucial. This experiment focuses on studying the effects of varying the proportions of filters with {0.1} and {0,-1} value assignments. As illus- trated in Table 2 we explore the delicate balance between positive and negative binary values to find the best way to optimize network performance. The data indicates that having equal mix of these values is more beneficial, clearly outperforming the methods that use only a single type of sparse quantized function. This observation underscores the importance of using both positive and negative binary regions to achieve effective learning of representations.
3.2.3 Backpropagation in PLUM
In this section, we delve into tuning latent full-precision weights during training to enhance model accuracy. Traditional binary networks experience fluctuations in latent weight assignments at zero due to the use of the one quantization function, i.c., sign
maiiiion-mmiistiingg as a peak latent full-precision weights at zero (Bai et al., 2018: Qin el al. 2020b). This issue exacerbated in PLUM. Employing two quantization functions such that individual signed binary regions co-dependently learn diverse features, results in two distinct peaks at non-zero values of =A as illustrated in Figure 6b This dual peak structure necessitates process to ensure stable and accurate weight updates. In response, we have adapted traditional binary's EDE (Qin et al. 2020b) that approximates the derivative of the sign function during backward propagation to improve gradient accuracy and update capability. PLUM stabilizes fluctuations in latent full-precision weights around A = =0.05 max(W) with EDE, thereby fostering improved representations as delincated in Thax Table This is done by first determining & and & using the equations / = Tiin 10 xlog Tiin k = max (2, 1) where is the current epoch and N the total number of epochs, Tmin 10-1 and Tmax = 10 Subsequently, we adapt g'(-) for PLUM: g((x) kt tanh A))), to compute the gradients with
OL OL respect to w as dw (wj I (w)
4 PLUM Pushes the Pareto-Frontier
In this section, we evaluate the efficacy of PLUM in preserving the quality of learned representations when transitioning from the conventional binarization technique. Initially, we train ResNets on the CIFAR10 and ImageNet datasets using PLUM, and benchmark the results against existing state-of-the-art methods in subsection Additionally, subsection 4.3 examines the latent full-precision and quantized weights in detail to provide a holistic understanding of the retained representational capacity of the trained model.
4.1
4.1 Comparison with other works
To ascertain the true potential of PLUM codesign, compare PLUM's signed-binary quantization with SOTA binary-quantization methods.. We benchmark on CIFAR10 and ImageNet datasets by training ResNet-(20,32) and ResNee-(11,344) models. Conversely, model parameters are of two types, effectual (or non-zero valued) parameters and ineffectual (or zero valued) parameters (Wu et al. 2021 Nayak et al. 2023) Since only effectual parameters lead to computation during inference, we compare different binary methods on two axes: model accuracy on the Y-axis and effectual binary model parameters on the X-axis (refer supplementary D & C for baselines and setup). As shown in Figure PL.UM exhibits Pareto optimality
7

---

Published in Transactions on Machine Learning Research (10/2024)
PLUM's Signed-Binany pushes Pareto frontier
92.0
DIR-Net SI-BNN
1.5x(4) effectual
IR-Net
DIR-Net
70
Bi Real
params
acc
IRevett
1K Accuracy
ABC-Net
CIFAR-10 Accuracy
91.5
PO BC
DIR DiR-Net
IReN
66
91.0
Convantooel Nethids
ImageNet-IK 1K
2.8x(1)
BWHN
Conventional Binary Menotd
DSQ
2.5x(1)
ABC-Ngt
ProxQuart
62
90.5
HWGG!
BWN
60
DSQ
LQ-Net
90.0
DoReFa
58
SQ_BVIN
Effectual Parameters (M)
Effectual Parameters (M)
* PLUM's Signed-Binary vs Binary
Figure 5: Comparison of PLUM and conventional binary methods on CIFAR10 and ImageNet datasets. PLUM pushes the Pareto frontier, providing superior accuracy with respect to effectual parameters and exhibiting a significant reduction effectual parameters of equivalent models.
against state-of-the-ar methods. achieves a -0.7% and 22.5 increase in accuracy on the CIFAR10 and ImageNet datasots, respectively, coupled with 1.4x and 1.5x reduction in effectual parameters respectively. Concurrently, it realizes approximately 2.5x and 2.8x decrease in effectual parameters for equivalent models the respective datasets. We find significant training overhead for PLUM when compared to conventional binary quantization schemes. Supplementary E trains signed-binate with binary till saturation under an identical setting with an equal training time of 320 epochs (4 hour training NVIDIA T4 GPU) on the CIFAR10 dataset. to observe comparable accuracy with equal total parameters and higher accuracy
with comparable effectual parameters. All hyperparameters are listed in supplementary D These results emphasize the potential of PLUM co-design in creating efficient and accurate models.
4.2 Comparing Against Full Precision Models on Additional Datasets
We train VGG, AlexNet, and ResNet on CIFARIO, SVHN (svh 2011
and Tiny ImageNe (Le & Yang)
datasets, respectively. AlexNet* aale 2016) and VGG** (Cai et al. 2017
are derivative architectures based
on AlexNet (Krizhovsky et al. 2012) and VGG
Simonyan & Zisserman,
2015) for smaller datasets as used
in prior works Prabhakar et al. 2021 Fu et al. 2022).
Model
Dataset
Accuracy Signed Binary
Accuracy Full Precision
A
SVHN
97.2
97.7
VGG**
CIFAR10
92.9
93.8
ResNet18
CIFARI00
75.83
77.8
ResNet18
TinyImageNet
56.9
59.72
Table 6: Accuracy Comparison between Signed Binary and Full Precision Models
4.3 Visualizing Latent FP and Quantized Weights
Latent FP weights. Binarization quantization leads to zero-mean Laplacian distribution of latent full- precision weights Xu et al. 2021) For 1esNet18 trained on Imagenet in Figure 0 observe that despite the
8

---

Published in Transactions on Machine Learning Research (10/2024)
Negatives
Latent Full-Precision Weights
Signed Binarrr Weight Parcentagg
Conv Block
Filters
Neg Filters
Frequency
stods Loane Bloot tiie blo peredt
piort cpter
tloret crter
alos coree
glont gpeed
cpnnt
cooed
ploat come
Blusd mome
Convolutional Layers
134
Value
(a) Quantized Weights
(b) Latent Full Precision Weights
On the left: Quantized Weights, are depicted, illustrating distribution reminiscent of ternary networks but with weights distinctively segregated across filters, enhancing weiglit-repetition and inference efficiency. On the right: Latent Full Precision Weights in signed-binary conv block maintain blue distribution akin to binary's Laplacian. In sign-binary, total parameters can be divided between positive and negative signed-binary filters. These parameters can be subdivided into non-zero valued effectual and zero-valued ineffectual ones. Notably, while the total parameter's looks like binary's zero-mean Laplace dis- tribution, individual filters do not. The effectual and ineffectual parameters' green-red and red- grregreen distri- butions resemble Laplace and Gaussian distributions, respectively. Sign Binary reduces computations during inference when compared lo binary, il reduces effectual parameters from blue to green-red distribution.
Figure 6:
pronounced sparsity introduced by signed binary quantization, the distribution of latent FP weights across entire convolutional block resembles zero-mean Laplacian distribution. Signed-binary filters are neither zero mean, nor exhibit a Laplacian distribution (supplementary H This shows that even if individual filters don't align with a zero-mean or Laplacian distribution, collectively, the convolution block resembles these characteristics. Morcover, the presence of four distinctive peaks overlaying Laplacian resembling distribution for full- full-precision latent weights are due to: (a) Two peaks the extremes appear because of clamped weights at +1 and -1, and (b) two intermediate peaks arising from the employment of threshold functions defined + analogous to valu-valued peak observed in binary network due to the use of sign quant function.
Quantized weights. We investigate the distribution in quantized weights of trained signed-binas model by plotting the percentage distribution of quantized signed-binans convolutional blocks in ResNet18 trained Imagenet. Figure 0 reveals roughly consistent, equal proportion of both positive and negative weights. This observation of a roughly equal proportion of positive and negative weights is also observed in ternary quantization (Zhu et al.] 2016). However, a crucial distinction arises in the distribution of positive and negative weights within a convolutional layer. Like ternary, both positive and negative valued weights are present within layer. However, signed binary co-design bifureates them across different filters. This design decision improves inference efficiency.
5 PLUM improves Inference Efficiency
This section shows that transitioning from conventional partitioned design for binary to PLUM co-design framework enhances inference efficiency for ResNet18 trained on ImageNet. We highlight PLUM's superiority over conventional partitioned design using binary and ternary by considering the inpctition-sparsiii trade- off. Additionally, we explore how weight sparsity in DNNs, specifically in signed-binary ResNet18 trained ImageNet, benefits DNN inference.
5.1 Exploiting Repetition Sparsity with PLUM
We posit that the performance of binary and ternary weight quantization methods inference may be hindered due their neglect of the crucial repetition-sparsi trade-off. Weight sparsity allows us to utilize upcoming sparse tensor operation technology to accelerate inference by skipping zeros during inference (Wu

---

Published in Transactions on Machine Learning Research (10/2024)
1.5
Signed-Binary w/ Sparsity Support
Speedup
Signed-Binary w/o Sparsity Support
1.0
Binary
Ternary w/ Sparsity Support
Ternary w/o Sparsity Support
0.
block1 ccnvv
blook1 Leov2
blook2 convt
block2 conv2
blook convt
blook conv2
blocka shorteut
blockd, convt
blook blookd convz
blookS conv2
convt
blockS. shortecu
blockd, convt
blocké, conv2
block? .convl
blook conv2
blook? Lahorteat
block8 convt
blook8. convz
Total
Layer
Figure 7: Efficiency Analysis wrt Binary ResNet18 on Intel CPU: Our study shows signed-binary ex- celling in every convolutional layer, depicted by bars for signed-binary: PLUM (w/ sparsity support) and
w/o sparsity support: ternary: w/ sparsity support and w/o sparsity support; ; and one for binary indicating its 100% density, Please refer to Figure G The inference computations for signed-binary w/o sparsity support is aligned with blue distribution, matching binary performance. On the other hand,
PLUM (w/ sparsity support) reduces computation to green-red distribution. Because negative and positive valued quantized weights exist in separate regions of the network by design (sce Figure 3 PLUM retains repetition, resulting in speedup.
et al. 2021 Concurrently, exploiting weight repetition leads to diminishing arithmetic operations and data movement (Hegde al.] 2018). However, traditional binary networks prioritize the maximization of weight repetition, overlooking the potential benefits of weight sparsity, while ternary networks induce weight sparsity but compromise on weight repetition. In contrast, we hypothesize that PL.M framework, being attuned to this tradc-off, promises enhanced efficiency in actual device inference. To validate our hypothesis, we deploy quantized ResNet-11 models on Intel CPUs, rigorously measuring inference times under varying conditions.
Experimental Setup and Methodology We use SumMerge (Prabhakar et al. 2021 for performing inference of quantized and sparse DNNs on Intel CPUs (details in supplementary Ap. All experiments are conducted under identical test environments and methodologies. Within our experiments utilizing Sum- Merge, we explore two distinct configurations: (1) with sparsity support deactivated, the software does not distinguish between zero and non-zero weights, relying solely on weight repetition; (2) with sparsity support activated, the software additionally omits compulations involving zero weights. We provide detailed analysis into both per-layer speedups and the aggregate speedup across different quantization strategies relative to binary quantization.
Result and Analysis Please refer to Figures and 7 We observe in Figure 7 that PLUM, i.e., signed- binary with kernel exploiting repetition and sparsity, is most efficient for every quantized layer and the model overall by 1.26x and 1.75x faster respectively when exploiting both repetition and sparsity.
This result can be explained as follows: (A) When sparsity support is turned off: The software is only relying on repeating values within the weight tensor for speedup. Because binary and signed-binary have two unique values per convolutional filter, they take similar time for DNN inference. Ternary is much slower as it has three unique values per convolution filter which makes extracting efficiency by using weight repetition exponentially harder. (B) When sparsity support is turned on: The software not only cares about the repeating values in the weight tensor but also skips computations on zero weights to improve the runtime. Here we observe that ternary is slower than binary, because the reduction in work due to sparsity is not able to compensate for the exponential decrease in weight repetition. On the other hand, our method, PLUM, does not suffer from this problem and can exploit weight repetition and weight sparsity to the fullest and is most efficient. Thus, PLUM framework that relies on pppctiiion-spasiiyy aware-inference performs better than using prior-art inference with binary and ternary methods.
10

---

Published in Transactions on Machine Learning Research (10/2024)
Arithmetic Operations We report the reduction in arithmetic operations Prabhakar et al. (0221 Fu
et al. (2022) required for single inference relative to binary. We find that ResNet 18 trained using signed- binary takes 20% fewer operations during w/ sparsity support enabled inference wrt binary. On the other hand, when trained using ternary, inference requires 35% more operations during w/ sparsity support
enabled inference wrt binary. Please refer to supplementary G for detailed ablations.
5.2 Understanding benefits of Sparsity during inference
In this section, aim to understand the impact of weight sparsity given fixed weight repetition in PLUM framework. We change the percentage of weight sparsity when using one bit quantization. To do this, we count the number of quantized weights with zero values and divide it by the total number of quantized weights to calculate the percentage of sparsity. We find that signed-binary ResNet-118 trained on ImageNet has 65% sparsity. Since density is (1 sparsity) (Wu et al. 2021], ResNet-118 has 35% density (see Figure 66a) if we switch from conventional binary to PLUM's signed-bing binary, we decrease the density from 100% to 35%. We would like to leverage the low density to reduce the amount of computation activity during inference:
Throughput In model with low density, represented by 1/z, there is one effectual multiplication for every total multiplications. By eliminating the ineffectual computations and the time associated with them, there is the potential to improve throughput by a factor of x (Emer et al. 2021). Given that the PLUM's higeed-iiary has a 35% aggregate density meaning only 35% of the multiplications are effectual- exploiting sparsity during inference can lead to a potential 2.86x increase throughput compared to dense binary (1,-1}. In practice, the spoodup we observe on real hardware is the range 1.26x-1.705 the support for unstructured spansity on general-purper
devices is an active area of research (Wu et al. 2021 This speedup is comparable to the speedup due to unstructured sparsity general-purpose devices in recent
papers (Gong et al. 2020 Hegde et al. 2019h.
Energy Reduction To estimate energy reduction due to the unstructured sparsity during inference, we use the cycle-level miero-architectural simulator (Muiioz-Matrinez et al.] 2021] of a sparsity supporting ASIC (Qin et al. 2020a). We take their publicly relcased code and use it under default configuration (details in supplementary AJ. We observe that decreasing density from 100% to 35% for one-bit ResNet leads to ~xx reduction energy during inference. Thus, switching from binary to PLUM's signed- bineerybbirry would lead to significant improvements in power consumption on ASICs.
6 Discussion and Future Work
The paper introduces the concept of repetition-sparsity trade-off and proposes PLUM (PLUs-Minus) quant-system co-design framework that leads to improvement of inference efficiency while retaining accuracy.. PLUM exhibit Pareto optimality compared to prior conventional methods, improving accuracy per effectual parameter and enhancing computational efficiency during inference with respect to speed, energy, and density. However, PLUM requires training from scratch, which can be time-consuming and computationally expensive. PL.UM requires the use of two quantization functions, i.c., with value sets of {1,0) and {-1,0)- Only using a single quantization function (just (1,03) could lead to improving inference efficiency. Following the terminology from Section [.1 while binary can be represented using at least R S x C x K bits, signed-binary requires one additional bit per filter to signify the corresponding signed-binary quantization function, resulting in R x S x C' x K + K bits. Further, the tile size of the modern inference system should be set such that a single processing step in PLUM should see only one signed binary quantization function. Additionally, the understudied repercussions of quantization and, consequently, signed binary quantization on model bias warrant further exploration. Despite these limitations, PI.UM presents a promising approach for training efficient models that are both accurate and computationally efficient, particularly well-suited for applications where hardware resources are limited, such mobile devices and embedded systems.
11

---

Published in Transactions on Machine Learning Research (10/2024)
7 Reproducibility Statement
We provide all the hyperparameters for the key experiments including instructions on how to train the models in the supplementary C Further, since our work is about quantization system co-design, we provide all the hyperparameters and configurations to reproduce our inference experiments in supplementary implementation details and baselines for our method can be found in supplementary C
A Finally,
Adhnowledgements. We lhank Tushar Krishna, Christopher Fletcher, Twinkle Kuhar and Paramvir Singh for their insightful discussions, as well as the anonymous reviewers for their very helpful feedback. We'd also like to thank Francisco Munoz-Martinez and Raveesh Garg for helping with Stonne for our experiments.
References
The Street View House Numbers (SVHN) Datasct.
2011.
AlexNet Derivative 2.
http:///h......c.c.aa/a/aaa/aaaa pyj 2016.
Y. Bai, Y. X. Wang, and E. Liberty. Proxquant: Quantized neural networks via proximal operators. Tn ICLR'19, arXiv (2018), 2018.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vors Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005.14166 2020.
Adrian Bulat and Georgios Tzimiropoulos. Xnor-net+++: Improved binary neural networks. arxiv preprint
MXXi:1909.13866 2019.
Adrian Bulat, Georgios Tzimiropoulos, Jean Kossaifi, and Maja Pantic. Improved training of binary networks for human pose estimation and image recognition. arXiv preprint XrXi::1904 05868, 2019.
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. 2017 IEEE Conference Compuler Vision and Pallern Recognition (CVPR), Jul 2017. doi: 10.1109/cvpr.20177.
URL http::/dxxdoo dow...og/o.... org/10. 1wt.1.......R....../ 20t.0////V............ .74
M. Courbariaux, Y. Bengio, and J. P. David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NIPS'15, 2015.
M. Courbariaux, I. Hubara, D. Soudry, R. El- Yaniv, and Y. Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1. In NIPS'16, 2016.
M. Cowan, T. Moreau, T. Chen, J. Bornholt, and L. Ceze. Automatic generation of high-performance quantized machine learning kernels. In CGO'20, 2020.
Pengcheng Dai, Jianlei Yang, Xucheng Ye, Xingzhou Cheng, Junyu Luo, Linghao Song, Yiran Chen, and Weisheng Zhao. Sparsctrain: Exploiting dataflow sparsity for efficient convolutional neural networks training, In 2020 57th ACM/IEEE Design Automation Conference (DAC), pp. 1 6. IEEE, 2020.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR '09, 2009.
Li Dong, Shaohan Huang, Furu Wei, Mirella Lapata, Ming Zhou, and Ke Xu. Learning to generate product reviews from attributes. In Proceedings of the 15th Conference of the European Chapler of the Association for Computational Linguistics: Volume 1, Long Papers, volume 1, pp.
2017.
Joel Emer, Angshuman Parashar, Vivienne Sze, Po-An Tsai, and Yannan Nellie Wu. Sparse tensor acceler- ators: Abstraction and modeling. In ISCA Tutorial 2021, 2021.
12

---

Published in Transactions on Machine Learning Research (10/2024)
Cheng Fu, Hanxian Huang, Bram Wasti, Chris Cummins, Riyadh Baghdadi, Kim Hazelwood, Yuandong Tian, Jishen Zhao, and Hugh Leather. Q-gym: An equality saturation framework for dnn inference exploiting weight repetition. In Proceedings of the International Conference on Parallel Architectures and Compilation Techniques, pp. 291-303, 2022.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In Proceedings of the IEEE/CVF international conference computer vision, pp. 4852 4861, 2019.
Zhangxiaowen Gong, Houxiang Ji, Christopher W Fletcher, Christopher Hughes, Sara Baghsorkhi, and Josep Torrellas. Save: Sparsity-aware vector engine for accelerating dnn training and inference on cpus. In 2020 53rd Annual IEEE/ACM International Symposium on Mierourchälecture (MICRO), pp. 796 810. IEEE, 2020.
Alex Graves, Abdel rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In IEEE international conference on acoustics, speech and signal processing, 2013.
He, X. Zhang, Ren, and J. Sun. Deep residual learning for image recognition. In CVPR 16, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance imagenet classification. In Proceedings of the IEEE international conference computer vision, pp. 1026-1034 2015.
K. Hegde, J. Yu, R. Agrawal, M. Yan, M. Pellauer, and C. Fletcher. Uenn: Exploiting computational reuse in deep neural networks via weight repctition. In ISCA '18, 2018
Kartik Hegde, Hadi Asghari-Moghaddaam, Michael Pellauer, Neal Crago, Aamer Jaleel, Edgar Solomonik, Joel Emer, and Christopher W Fletcher. Extensor: An accelerator for sparse tensor algebra. In Proceedings of the 52nd Annual IEEE/ACK AAM International Symposium on Microarchitecture, pp. 319 139--33, 2019.
Q Hu, Wang, and J. Cheng. From hashing to cnns: Training binaryweight networks via hashing. Tn AAAT118, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 03167, 2015.
Yash Jain, Chi Ian Tang, Chulhong Min, Fahim Kawsar, and Akhil Mathur. Collossl: Collaborative self- supervised learning for human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wear- able and Ubiquitous Technologies, 6(1):1 28, 2022.
Yash Jain, David Chan, Pranav Dheram, Aparna Khare, Olabanji Shonibare, Venkatesh Ravichandran, and Shalini Ghosh. Multi-stage multi-modal pro-training for automatic speech recognition. arXiv preprint arXiv:2403. 19822, 2024.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXio preprint arXiv:1412.6900 2014.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
A. Krizhevsky, Sutskever, and G. E. Hinton. Imagenct classification with deep convolutional neural networks. In N2 2012.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. ffev. hhttps://githnie http://ttttuu..cmottotttteeeeeee wwt.cooommm..eee wwmm/iiffc///fer/// 2022.
Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. Xixiv preprint or 1rx:::6605.000 2016.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. Advances in neural information processing systems, 30, 2017.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng, Bi-real net: Enhancing the performance of 1-bit cmns with improved representational capability and advanced training algorithm. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 722-737, 2018.
Andrew I. Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. In JCML'IS 13, ICML'13, 2013.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference Robot Learning (CoRL), 2021.
Francisco MatrieMa José L. Abellán, Manuel E. Acacio, and Tushar Krishna. Stonne: Enabling cycle- level microarchitectural simulation for dnn inference accelerators. In 2021 IEEE International Symposium on Workload Characterization (IISWC), 2021.
Nandeeka Nayak, Toluwanimi O Odemuyiwa, Shubham Ugare, Christopher Fletcher, Michael Pellauer, and Joel Emer. Teaal: A declarative framework for modeling sparse tensor accelerators. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecturr, pp. 1255 1270, 2023.
H. Pouransari, Z. Tu, and O, Tuzel. Least squares binary quantization of neural networks. In CVPRW'20, 2020.
Rohan Prabhakar, Sachit Kuhar, Rohit Agrawal, Christopher J Hughes, and Christopher Fletcher. Sun- merge: an efficient algorithm and implementation for weight repetition-aware dun inference. In Proceedings of the ACM International Conference on Supercompuling, pp. 279 279-290, 2021.
Eric Qin, Ananda Samajdar, Hyoukjun Kwon, Vineet Nadella, Sudarshan Srinivasan, Dipankar Das, Bharat Kaul, and Tushar Krishna. Sigma: A sparse and irregular gemm accelerator with flexible interconnects for dnn training. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 58-70. IEEE, 2020a.
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song, Forward and backward information retention for accurate binary neural networks. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp.
2020b.
Haotong Qin, Xiangguo Zhang, Ruihao Gong, Yifu Ding, Yi Xu, and Xianglong Liu. Distribattion-senstivee information retention for accurate binary neural network. International Journal of Computer Vision, 131 (1):26 47, 2023.
M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV'16, 2016.
K Simonyan and A Zisserman. Very deep convolutional networks for large-scale image recognition. In 3nd International Conference on Learning Representations (ICLR 2015). Computational and Biological Learning Society, 2015.
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel Emer. Efficient processing of deep neural networks. Synthesis Lectures on Computer Architecture, 15(2):1-344 341, 2020.
Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cas- cade token and head pruning. In 2021 IEEF International Symposium on High-PPrfformant Computer Architecture (HPCA), pp. 97 110. IEEE, 2021.
Yannan Nellie Wu, Po-An Tsai, Angshuman Parashar, Vivienne Sze, and Joel Emer. Sparseloop: An analytical, enorgy-focusod design space exploration methodology for sparse tensor accelcrators. In 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 232 234. TEEE, 2021.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. In Proceedings of the International Conference on Computer Vision (ICCV), pp. 5198 5208, 2021.
Dongqing Zhang, Jiaolong Yang, Donggiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In European Conference Computer Viston (ECCV). 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.00 66660666666,,. 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained lernary quantization. ar preprinl arXiv:1612.01000 01064, 2016.
15

---

Published in Transactions on Machine Learning Research (10/2024)
A Experiment Setup Efficiency
Deploying on CPUs We use SumMerge Prabhakar et al. (2021) for this task. We run all experiments on Intel Xeon Gold 6226 CPU. In order to make our test environment as close as possible to the test environment of the authors of Prabhakar et al. (2021), we disable simultaneous multi-threading, and enable 2MB huge pages and disable dynamic frequency scaling as well. The test methodology is exactly the same as used by the authors Prabhakar et al. [2021] J. i.c., each experiment is run 50 times when the machine is unloaded and the values for run with the lowest execution time are reported. All arithmetic operations in floating boating-point. All DNN inference experiments subject to identical test environments and methodology.
We use STONNE Inference Accelerator SIGMA Qin et al. (2020a) for this experiment. We use the docker image released by the authors of Matrine et al. (2021). We use the standard configuration of SIGMA with 256 multiplier switches, 256 read ports in SDMemory and 256 write ports in SDMemory. The reduction networks is set to ASNETWORK and the memory controller is set SIGMA SPARSE GEMM. We use SimulatedConv2d function in the PyTorch frontend version of STONNE. For given convolutional layer, we STONNE twice, once with 0% sparsity and once with 65% sparsity. We calculate the reduction in energy consumplion by dividing the energy of the dense convolutional layer by the energy of the sparse convolutional layer. Since the weights' precision (or bit-width) is a parameter of SIGMA, the reduction in energy due to sparsity when compared to the dense model is not a function of the precision of the weights of the DNN.
ASIC
et al. [2021), a cycle-level microarchitectural simulator for DNN
B Visualizing Exploitation of Repetition and Sparsity
Output1: x*(a A
C
2
Input:
Output2: x* +
PS0: ca (d
x
x
x
x
Outputt: + e) + AMSPS1
A=
x
A=
A=
A=
x b) + (PS1 +e)
Filter1
Filter2
Output1:
x'a + A*b x And x Aff
4
PS0: (a c);
Recall: ==
Output2: x"a And Are
Output1: e)
Output2: x*(PSO + b)
Figure 8: Visualization of repetition and sparsity during inference in modern systems: (1) Input and two filters that need to be multified (2) naive multiplication of weights and activations to create output 1 and output 2. (3) Re-ordering of weights and activations to simplify work within each filter (and exploit intra filter repetition phenomenon (Hgdde et al. 2018 (3) Using partial sums to reduce the work across filters (and exploit intra filter repetition phenomenon (Hegde et al. 2018)) (4) exploiting sparsity (by acknowledging repetition of zero weights as a special case of weight repetition (Sze et al. 2020 Hegde et al. 2018 Prabhakar
et al. 2021
C
Experimental Setup Accuracy
CIFAR10 The data loader pipeline consists of simple augmentations - padding by 4 pixels on cach size, random crop 32 32, Random Horizontal Flip with probability 0.5, and normalization. We train from scratch for 350 epochs and the Adam Optimizer Kingma & Ba 2014). We start with initial learning
16

---

Published in Transactions on Machine Learning Research (10/2024)
rate of 0.01 and reduce it by a factor of 10 at epochs 150, 200, and 320. For apples-to-apples comparison with binary and ternary, we do sweep over batch sizes (16, 32, 64, 128, 256) and activation functions (ReLU, PReLU, Tanff) and report the best top-1 validation accuracy. For ablations on (1) value assignment percentage and (2) comparison with binary networks with comparable effectual operations, we select the batch size the to be 32 and activation function to be PReLU. We compare the binary and signed-binary methods on their non-zero valued parameters of quantized layers as other aspects would be similar. When comparing against prior art works, we run these methods on our setup and report numbers. Further ablations on batch size and activation function provided in the appendix.
ImageNet We train ResNet-18 He et al. 2016)) using SBWN on ImageNet Deng et al.
2009 We use
standard practices used to train binary networks like (1) normalize the input using batch-norm loffe &
Szegody (0015) before convolution instead of after convolution Rastegari et al. (016) (2) the first and the
last layers are not quantized Zhu et al. (2016); Pouransari et al. (2020): Li et al. (2016)
: Bai et al.
2018
We use first order polynomial learning-rate anncaling schodule with Adam optimizer Kingma & Ba
(2014)
along with PReLU He et al. (2015); Maas et al.] (2013). We use FFCV dataloader Leclere et al. (2022) with
simple augmentations Random Resize Crop to 224 x 224, Random Horizontal Flipping and Color Jitter with (brightness, contrast, saturation, huc) set as (0.4, 0.4, 0.4, 0). We decrease the learning rate from 2.0 x e-t to 2.0 x while training for 320 epochs and do not use weight decay and batch size of 256 for training. We compare the binary and signed-binary methods on their non-zero valued parameters of quantized layers as other aspects would be similar. When comparing against prior art works, we report numbers as is from the literature due to high compute cost of running these experiments. Furthermore, for ResNet 34, we simply increase the model size.
Implementation Ligned-Binary is a local binarization scheme, ie., the quantization function takes full- precision latent weights of region of a convolutional layer input and maps it to either {0, 1) 1)Rx55xC or {0, {0,-1]RxxXC;, The values of the quantization function for these predetermined regions of convolutional layer are determined randomly before training commences and remain unchanged. Different regions can have distinct quantization functions for a convolutional layer. This framework categorizes these regions into two buckets, optimizing efficiency by grouping them based on their quantization function values for more streamlined training. For example, if C, = C, signed binarization becomes a per filter quantization scheme and cach filter will have a different quantization function. We quantize the full-precision latent weights of convolutional layer from R (RXXXXCX to (x11)xxxxXXXx and {0, -1 - fX xxXxXXXXXXxX x(1-P) where P is the percentage of filters whose quantization functions have the values {0,1} such that K x P is an integer.
D Baselines
Baselines for Comparison with Prior work: Figure
5 shows comparison against prior-art methods.
The numbers are reported from the literature: DIR-Net
Qin et al. (2023) LQ-Net Zhang et al. (2018).
DSQ Gong et al. (219)) BC Courbariaux
al. (2015),
ProxQuant Bai et al. (2018). IR-Net Qin et al.
(2020b), DoReF'A Zhou et al. (2016), SQ-BWN Dong et al. et al. 2018), ABC-Net Lin et al. (2017), BWHN Hu et al.
2017
BWN RWstRgarrrri et al.
2016
BWNH Hu
(2018). LS Pouransari et al.
2020 Bi-Real Liu
et al. 2018
E
<Signed-Be Signed-Binary vs Binary wrt Effectual parameters
We would like to compare binary with signed-binary when the DNN has the same number of non-zero parameters. Signed-Binary ResNet trained on CIFAR1O has slightly greater than 50% sparsity. If we reduce the total number of parameters of the binary ResNet by half, the resulting model would have comparable number of non-zero weights to signed-binary ResNet. This is done by reducing depth (sce Table 7ab and by reducing width (see Table 7bp. We train these models under identical conditions (setup details in the appendix). To clarify, row 1 & row 2 of Table 7 have the same number of total parameters while row & row 3 have a comparable number of non-zero parameters. Thus, signed-binary leads to a higher accuracy than binary when both methods have comparable number of effectual operations.
17

---

Published in Transactions on Machine Learning Research (10/2024)
Quant
# Parameters
Depth
Acc
Quant
# Parameters
Width
Acc
SB
0.46M
32
91.55%
SB
0.27M
1x
90.05%
B
0.46M
32
91.22%
B
0.27M
1x
90.20%
B
0.27M
20
90. 16%
B
0.14M
[0.7x]
88.5%
(a) Reducing number of parameters by reduc- ing depth: We observe that accuracy of binary is 1.3% lower than signed-binary with comparable non- zero weights.
(b) Reducing number of parameters by reduc- ing width: We observe that accuracy of binary is 1.7% lower than segned-binary with comparable non- weights.
Table 7: Binary vs Signed-Binary with comparable non-zero weights: We observe that Signed- Binary achieves higher accuracy when compared to binary with comparable effectual operations.
F Additional Ablations on CIFAR-10
Batch Size
Accuracy (Top-1)
Non-Linearity
Accuracy (Top-1)
16
89.44
ReLU
88.64
32
90.05
PReLU
90.05
64
89.62
TanH
88.75
128
89.59
LReL.U
89.22
(a) A Ablation on Batch Size: The setup is identi- cal across batch sizes and the non-linearity used is PReLU, We observe decrease in accuracy when a high batch size 256 is used.
(b) Ablation on Non-Linearity: The setup is identical across non-linearity and the batch size used is 32. observe that PReLU works best our method.
Table 8: Additional Ablations on CIFAR10: Abations on batch size and non-lincarity for SB.
We perform ablation (1) batch sizes, (2) non-linearity on CIFAR10 KKizhevsky & Hinton 2009 dataset and report the numbers in Table Sa and gb respectively. The setup is the same as mentioned above. We observe that for our method, there is drop in accuracy with higher batch size, PReLU (Maas et al. 2013
works best and it is not sensitive to the choice of delta.
G Arithmetic Reduction Ablation
The fewer arithmetic operations required during inference for given layer, the more efficient the quantization
scheme is for both CPUs and GPUs (Hegde et al. 2018 Prabhakar et al.) 2021 Fu et al.
2022 This can be
measured using arithmetic reduction (Hegde et al. 2018 Prabhakar et al. 2021 Fu et al. 2022 which is defined as the ratio of arithmetic operations required using naive dense computation (unaware of repetition and sparsity) to the arithmetic operations taken during repetition-sparsity aware inference for a given DNN block. This metric indicates inference efficiency at the algorithmic level (Hegde et al. 2018 Prabhakar al. 2021 Fu al. p022p. Figure compares the arithmetic reduction across different quantization schemes. The experiment follows the original lest settings and methodologies described by the authors of (Prabhakar al., 2021), using synthetic, uniformly distributed weights in the DNN block. The results show that quantization is the most efficient, providing the highest arithmetic reduction across all conv layers.
We extend this experiment by varying the sparsity percentage from 0% to 100%, with equal percentages of positive and negative weights across all three quantization schemes. Figure [10 compares the arithmetic roduction (Y-axis) and the percentage sparsity (X-axis) for convolutional block of shape [3,3,512,512] Since binary quantization uses +1, -1 and does not leverage sparsity, its performance is represented by a horizontal line. Ternary quantization initially behaves like binary when sparsity is negligible and performs
18

---

Published in Transactions on Machine Learning Research (10/2024)
worse with moderate sparsity before improving under high sparsity conditions. This be explained by the repetition-sparrity trade-off. Signed-binary quantization consistently outperforms ternary due to higher weight repetition with equal sparsity and outperforms binary due to the presence of sparsity. The highly efficient behavior of signed-binary under both very low and very high sparsity can be explained: with approaching 0% sparsity, signed binary gets converted into monolithic filters containing one unique weight per filter, and with high sparsity, most operations become ineffectual, resulting significant savings.
Arithmetic Reduction vs. Percentage of Zero Weights
Arithmetic Reduction vs. Layer
Arithmetic Reduction (log scale)
Binary
40x
Ternary
Signed Binary
Ternary
Binary
Signed Binary
Arlthmetic
6x
10k
3,3,16,16
3,3,64,64
3,3,128,128 3,3,256,256 3,3,512,512 Layer
0.0
0.2
0.4
0.6
0.3
Percentage of Zero Weights
Figure 9: Arithmetic Reduction (higher is botter) for Binary, Ternary, and Signed-Binarg across different. DNN blocks.
Figure 10: Arithmotic Reduction (higher is botter) for Binary, Ternary, and Signed-Binary across dif- ferent degrees of sparsity.
H Latent Full-Precision Weights & Standardization
Binary quantization has shown to improve accuracy from standardization of latent full- precision weights. However, this trend is not observed in signod-binary networks.
Standardization Stratogy
Accuracy (%)
Local
Regions
59.1
Global Signed-Binary Block
61.2
No Standardization
61.4
Table Comparison of Standardization Strategies on Accuracy
(a) Negative SB Filters
(b) Positive SB Filters.
(c) SB Conv Block.
Distribution of Latent Full-Precision Weights: While the local signed-binary weights are neither zero mean, nor Laplacian, the entire conv block is zero mean Laplacian distribution. The peaks in the image are because of clipping at +1 along with thresholding at +A.
Figure 11:
19

---

Published in Transactions on Machine Learning Research (10/2024)
Additional ImageNet Ablations
We provide additional ImageNet ablations by training ResNet-18 architecture.
{0,1) filters %{0,-1 filters
Acc
EDE
Acc
Delta A
Acc
0
55.23
Disabled 62.73
0.01 x max [W]
64.1
0.25
0.75
61.94
Enabled 63.17
0.05 x max W
64.3
0.5
0.5
62.29
Table 11: EDEt Accuracy
Table 12: Delta Accuracy
Table 10: Accuracy with Filters
J
Datasets
Dataset
License
Source
ImageNet
Non- Non-Commercial
ILSVRC2012
CIFAR10
N/A
CIFAR
Table 13: Dataset with Licenses: License and source of the datasets used.
Licenses of ImageNet (Dng et al. 2009 and CIFAR10 (Krizheskyy & Hinton 2009 datasets used in this paper are listed Table 1. Every accuracy reported in this paper on validation set of the dataset. ImageNet and CIFAR10 are standard publicly used datasets. Since they do own their images, therefore they do not have release license. Actual images may have their copyrights but ImageNet provides stipulations for using their dalaset (for non-commercial use). We do not recommend using the resulting signed-binary models trained on these datasets for any commercial use.
20


Reviews:
[{"review_id": "wDZ6SO3n6k", "reviewer": "Reviewer_KJZp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is well written and easy to follow.\n2. The proposed new binary representation is interesting and seems very promising.\n\nWeaknesses:\n1. Most of the experiments are conducted on CIFAR-10 which greatly undermined the credibility of the method.\n2. Speed up experiments seems only conducted on CPU, No GPU results.\n3. How to combine with activation binarization. Can PLUM be adopted in FPGA as vanilla BNN?\n4. The training time and cost is not clear too.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces PLUM, a framework that integrates DNN inference systems with quantization to leverage a repetition-sparsity trade-off for improved efficiency. PLUM’s quantization method is more accurate than binary quantization, and signed binarization effectively reduces non-zero parameters within DNNs. The framework achieves a 26% speedup, doubles energy efficiency, and reduces density by 2.8× compared to binary methods, while maintaining a 66.2% top-1 accuracy on ImageNet for ResNets. PLUM offers a viable solution for deploying efficient models in resource-limited environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses above.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QpDiARGg5P", "reviewer": "Reviewer_bHEV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The approach presents a simple-yet-effective framework, instead of a overly complicated design.\n2. I think the experiments are complete and comprehensive.\n\nWeakness:\n1. The models used in experiments seem to be small scale. I wonder would it be possible to evaluate on larger scale of the models?\n2. Figure presentation: Figure 3 seems to be a bit blurry; Figure 5 has overlapped legends.\n3. Need to check for Typos. Appendix B is empty to me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces PLUM, a unified co-design framework that integrates DNN inference systems and quantization to leverage the repetition-sparsity trade-off to improve inference efficiency. Experiment results demonstrate more accuracy quantization over binary quantization with the same number of non-zero weights and 26% speedup on real hardware and doubled energy efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness", "broader_impact_concerns": "N.A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YMia817ivl", "reviewer": "Reviewer_JW4J", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The concept of repetition-sparsity trade-off provides a novel and insightful perspective for analyzing existing quantization methods, helping to understand these methods' limitations better.\n2. The PLUM framework demonstrates the importance of co-designing quantization methods with inference systems, offering a new approach for developing efficient inference solutions.\n3. Through comprehensive experiments, the authors thoroughly demonstrate the effectiveness of the PLUM framework, including accuracy comparisons on standard benchmark datasets, visualization analysis of weight distributions, and inference performance evaluations on real hardware platforms.\n4. The visualization analysis in the paper provides interesting insights into the learned features of the PLUM quantization scheme, helping readers better understand how the method works.\n5. The inference experiments on Intel CPUs not only prove the effectiveness of the PLUM framework but also provide empirical evidence for the repetition-sparsity trade-off, strengthening the core argument of the paper.\n\nWeaknesses:\n1. The ablation studies for PLUM are conducted on a limited scale, which weakens the persuasiveness of the experiments. More comprehensive ablation studies on a larger scale would strengthen the empirical evidence supporting the effectiveness of the proposed method.\n2. While the authors demonstrate the effectiveness of PLUM on image classification tasks using ResNet architectures, it would be beneficial to see the framework's performance on other tasks (e.g., object detection, segmentation) and with different network architectures to assess its generalizability.\n3. The paper could benefit from a more detailed discussion of the limitations of the PLUM framework, such as any potential trade-offs between model size, accuracy, and inference efficiency.\n4. The authors could provide more insights into the selection of hyperparameters for the PLUM framework, such as the choice of quantization function value sets and the impact of different choices on performance.\n5. The paper would benefit from a discussion on the potential challenges in deploying PLUM on other hardware platforms (e.g., GPUs, edge devices) and any specific optimizations that might be required.\n6. While the authors mention that PLUM requires training from scratch, which can be time-consuming and computationally expensive, they do not provide a detailed analysis of the training overhead compared to other quantization methods or the potential impact on the practical adoption of the framework.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The authors introduce the concept of repetition-sparsity trade-off to explain the inefficiencies in inference performance of binary and ternary quantization methods.\n2. The authors propose PLUM, a quantization-system co-design framework that leverages the repetition-sparsity trade-off to improve inference efficiency while maintaining model accuracy.\n3. Through extensive experiments, the authors demonstrate the effectiveness of the PLUM framework:\na) PLUM's signed-binary quantization scheme achieves accuracy comparable to existing quantization methods on CIFAR10 and ImageNet datasets while using fewer effectual parameters.\nb) Visualization analysis reveals the characteristics of the quantized weight distribution in PLUM and its relationship with the latent full-precision weights.\nc) Inference experiments on Intel CPUs, measuring latency and energy consumption, validate the existence of the repetition-sparsity trade-off and showcase PLUM's advantages in accelerating inference speed and reducing energy consumption.\nThese experimental results indicate that the PLUM framework can significantly improve model inference performance while maintaining accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical changes:\n\n1. Conduct more comprehensive ablation studies on a larger scale to strengthen the empirical evidence supporting the effectiveness of the PLUM framework. This will help address the concern about the limited scale of the current ablation experiments and increase the persuasiveness of the results.\n2. Provide a more detailed discussion on the limitations of the PLUM framework, including any potential trade-offs between model size, accuracy, and inference efficiency. This will help readers better understand the scope and applicability of the proposed method.\n3. Include a detailed analysis of the training overhead of PLUM compared to other quantization methods, and discuss the potential impact on the practical adoption of the framework. This information is crucial for practitioners considering the use of PLUM in real-world scenarios.\n\nSuggested improvements:\n\n4. Evaluate the performance of the PLUM framework on other tasks (e.g., object detection, segmentation) and with different network architectures to demonstrate its generalizability. This will strengthen the paper's contribution and broaden its impact.\n5. Provide more insights into the selection of hyperparameters for the PLUM framework, such as the choice of quantization function value sets and their impact on performance. This will help readers better understand how to apply the proposed method effectively.\n6. Discuss the potential challenges in deploying PLUM on other hardware platforms (e.g., GPUs, edge devices) and suggest possible optimizations. This will provide valuable information for practitioners looking to implement the framework on different hardware setups.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9OuqD7DyAb", "reviewer": "Reviewer_WV4c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Minor: The right part of Figure 3 contains two sparsity items.\n2. As indicated in Eq(2), each weight requires a sign-factor and a bitmap. Therefore, is each weight represented by 2 bits? If so, the advantage of PLUM partially comes from more efficient utilization of bits compared to ternary quantization. However, the model size for PLUM is also double that of binary quantization.\n3. GPUs are optimized for parallel computation. A potential drawback of this scheme is that unstructured sparsity may hinder inference speedup. in GPU", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper highlights that binary quantization is an extreme form of repetition, often overlooking weight sparsity. In contrast, ternary quantization introduces weight sparsity but results in reduced efficiency. To address this issue, the authors propose a trade-off quantization scheme. Additionally, they co-designed the quantization algorithm and the inference kernel. Two strengths are as follows:\n1. The idea is intriguing and logical; such a trade-off scheme can push the boundaries of quantization.\n2. The proposed quantization successfully introduces sparsity into binary quantization while maintaining accuracy and speeding up inference.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see the weaknesses for details", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7PPMrjtrBd", "reviewer": "Reviewer_3Yto", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed PLUM framework is a relatively simple yet effective approach that considers repetition-sparsity trade-off in the DNN inference process. Experiments validate its accuracy retaining with less effectual parameters.\n2. The paper provides an analysis of the experiments of PLUM, providing detailed charts and figures.\n\nWeaknesses:\n1. The contribution of the paper is not fully analyzed. There is a gap between the concept of repetition-sparsity trade-off and the proposed scheme.\n2. Experiments are not convincing enough, where baseline methods and models for evaluation are limited.\n3. The paper is not clearly presented and organized. \n4. Figures and tables are not clearly explained.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors introduce the concept of repetition-sparsity trade-off, based on which the proposed method PLUM is designed, which integrates DNN inference systems and quantization to improve inference efficiency. Experiments show that PLUM retains model accuracy with fewer effectual parameters due to its signed-binary quantization scheme. Demonstrations on Intel CPUs show that PLUM improves inference speed and energy efficiency.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The requested changes in the following comments are critical for acceptance.\n1. In the introduction, the novelty and contributions are not clearly stated. The concept of repetition-sparsity trade-off seems straightforward, but how is PLUM designed based on this concept? The contributions and novelty should be made more clear. It is now lacking in the introduction.\n2. Following the above comment, there is a gap between the repetition-sparsity trade-off concept and the proposed scheme. In what sense is PLUM an optimal solution to solve this trade-off? A mathematically rigorous analysis is required, otherwise, the method design looks ad-hoc.\n3. A concept of repetition-sparsity trade-off is essential in this paper. Is there any similar concept mentioned in previous works, or is it the first time this concept is introduced? A comprehensive literature review is required to show how this concept of repetition-sparsity trade-off differs from the previous works.\n4. The references in Section 2 are mostly published before 2020, more recent references (published in 2023 and 2024) should be included, especially when the reference is called \"recent work\". \n5. the ablation for PLUM comes out of a sudden in Section 3.2.2. it should be included in the experimental part to avoid distraction. Moreover, the explanation for this ablation is missing.\n6. In Figure 5, what are the methods that PLUM is compared to? Please carefully cite and explain the methods used for comparison.\n6. Appendix B is empty? At least some explanation or description is needed. Fig. 8 is not even referred to.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the repetition-sparsity trade-off concept and PLUM framework for efficient neural network inference. The authors demonstrate PLUM's effectiveness through comprehensive experiments on standard datasets and real hardware.

The authors have addressed most reviewer concerns by adding ablation studies, discussing limitations and practical considerations, and clarifying training details. While some limitations remain, PLUM significantly improves inference efficiency while maintaining accuracy, valuable for resource-constrained environments.

Regarding the minor revision, the AE would like to recommend the authors to
- Add training time cost in the main part of the paper, not in the last discussion session. Provide some numerical results on the time cost;
- Move some contents in appendix I and J to the main part of the paper, as the audience will be interested in seeing larger models;
- Consider adding models larger than ResNet18 for more validation.
Decision: Accept (poster)