Multi-Precision Policy Enforced Training (MuPPET) : A precision-switching strategy for quantised fixed-point training of CNNs
Authors: Aditya Rajagopal
Diederik A. Vink
Stylianos I. Venieris
Christos-Savvas Bouganis
Venue:  (2020)

Abstract: Large-scale convolutional neural networks (CNNs) suffer from very long training times,  spanning from hours to weeks, limiting the productivity and experimentation of deep learning practitioners.  As networks grow in size and complexity one approach  of  reducing  training  time  is  the  use  of  low-precision  data  representation and computations during the training stage.  However, in doing so the final accuracy suffers due to the problem of vanishing gradients.  Existing state-of-the-art methods combat this issue by means of a mixed-precision approach employing two different precision levels, FP32 (32-bit floating-point precision) and FP16/FP8 (16-/8-bit floating-point precision), leveraging the hardware support of recent GPU architectures for FP16 operations to obtaining performance gains. This work pushes the boundary of quantised training by employing a multilevel optimisation approach that utilises multiple precisions including low-precision fixed-point representations.  The training strategy, named MuPPET, combines the use of  multiple  number  representation regimes  together  with  a  precision-switching mechanism that decides at run time the transition between different precisions. Overall, the proposed strategy tailors the training process to the hardware-level capabilities of the utilised hardware architecture and yields improvements in training time and energy efficiency compared to state-of-the-art approaches. Applying MuPPET on the training of AlexNet, ResNet18 and GoogLeNet on ImageNet (ILSVRC12) and targeting an NVIDIA Turing GPU, the proposed method achieves the same accuracy as the standard full-precision training with an average training-time speedup of 1.28× across the networks.

Under review as a conference paper at ICLR 2020
MULTI-PRECISION POLICY ENFORCED TRAINING (MUPPET) A PRECISION-SWITCHIE STRATEGY FOR QUANTISED FIXED-POINT TRAINING OF CNNS
Anonymous authors
Paper under double-blind review
ABSTRACT
Large-scale convolutional neural networks (CNNs) suffer from very long training times, spanning from hours to weeks, limiting the productivity and experimen- tation of deep learning practitioners. As networks grow in size and complexity, training time reduced through low-precision data representations and computa- tions. However, in doing the final accuracy suffers due to the problem of van- ishing gradients. Existing state-of-the-ar methods combat this issue by means of mixed-precision approach utilising two different precision levels, FP32 (32-bit floating-poine point precision) and FP16/FP8 (16-/8-bit floating -ointt precision), lever- aging the hardware support of recent GPU architectures for FP16 operations obtain performance gains. This work pushes the boundary of quantised training by employing multilevel optimisation approach that utilises multiple precisions including low-precision fixed-point representations. The novel training strategy, MuPPET, combines the use of multiple number representation regimes together with precision-switchee between precision regimes. Overall, the proposed strategy tailors the training pro- cess to the hardware-level capabilities of the utilised hardware architecture and yields improvements in training time and energy efficiency compared to state-of- the-art approaches. Applying MuPPET on the training of AlexNet, ResNet 8 and GoogLeNet on ImageNet (ILSVRCI2) and targeting an NVIDIA Turing GPU, MuPPET achieves the same accuracy as standard full-precision training with an average training- -ime speedup of 1.28 x across the networks.
mechanism that decides at run time the transition point
INTRODUCTION
Convolutional neural networks (CNNs) have demonstrated unprecedented accuracy in various ma- chine learning tasks, from video understanding (Gan et al. 2015 He et al. 2018) to drone navigation (Loquercio al.) 20188] Kouris & Bouganis 2018). To achieve such high levels of accuracy in inher- ently complex applications, current methodologies employ the design of large and complex CNN models (Szegedy et al., [017 Huang et al.) 2017) trained over large datasets (Deng et al.) 2009
Lin et al. 2014). Nevertheless, the combination of large models and massive datasets results in long training times. This in turn leads long turn-around times which limits the productivity of deep learning practitioners and prohibits wider experimentation. For instance, automatic tuning and search of neural architectures (Cai et al. 2018 Zhong et al.) (2018) is rapidly advancing area where accelerated training enables improving the produced networks.
To counteract these long turn turn-around times, substantial research effort has been invested in hyper- parameter tuning for the acceleration of training, with a particular focus on batch size and content. One line work maximises memory and hardware utilisation by changing the batch size, order to perform CNN training specific prefetching, scheduling and dependency improvement (Chen et al.) 2019 Rhu et al. 2016). Other works focus on altering batch size or reconstructing minibatches to improve the convergence rate while sustaining high hardware utilisation (Devarakonda et al. 2017
Johnson & Guestrin 2018 Peng et al. 2019), demonstrating up to 6.25x training speedup.
A number of studies have focused the of precision arithmetic involves the utilisation of data formats that have smaller wordlengths than the conventional 32-bit floating-point (FP32) representation and is approach for co-optimising pro-
training schemes. Reduced-

---

Under review as a conference paper at ICLR 2020
cessing speed, memory footprint and communication overhead. Existing literature can be cate- gorised into works that use reduced precision to accelerate only the training stage while targeting an FP32 model, and those that produce networks with quantised weights. Regarding the former, [Cour- bariaux et al. (2015) and Gupta et al. (2015) utilise dynamic quantisation and stochastic rounding respectively as means combat the accuracy loss due quantisation.
Nevertheless, the effectiveness of the proposed schemes have only been demonstrated on small- scale datasets such as CIFAR-10 and MNIST, and on a limited set of networks. Furthermore, the range of quantisation levels that has been explored varies greatly, with a number of works attacking the problem by focusing on mild quantisation levels such as half-- salff-rreiiion floating-poinh point (FP16)
(Micikevicius et al. 2018). while others focus on lower precisions such as 8-bit floating-poin (FP8) Wang et al. 2018). Finally, quantisation has also been used as a means of reducing the memory
and communication overhead in distributed training (De Sa et al., 2015 2017 Alistarh et al. 2017
At the same time, the characteristics of modern CNN workloads and the trend towards quantised models have led to emergence of specialised hardware processors, with support for low-precision arithmetic at the hardware level. From custom designs such as Google's TPUs (Jouppi et al. al.] 2017) and Microsoft's FPGA-based Brainwave system (Fowers et al., 2018) to commodity devices such as NVIDIA's Turing GPUs, existing platforms offer native support for reduced- pprccisioo data types including 16-bit floating-poin (FP16), and 8- (INT8) and 4-bit (INT4) fixed fixed-point, providing in- creased parallelism for lower bitwidths. Although these platforms have been mainly designed for the inference stage, the low-precision hardware offers significant opportunities for accelerating the time-consuming training stage. In this respect, there is an emerging need to provide training algo- rithms that can leverage these existing hardware optimisations and provide higher training speed.
This work tackles the field of reduced-preciision training at an algorithmic level. Independently of the number of quantisation levels chosen, or how extreme the quantisation is, this work proposes a met- ric that estimates the amount of information each new training step obtains for a given quantisation level, by capturing the diversity of the computed gradients across epochs. This enables the design of a policy that, given a set of quantisation levels, decides at run time appropriate points to increase the precision of the training process at that current instant without impacting the achieved test accuracy compared to training in FP32. Due to its agnostic nature, it remains orthogonal and complementary to existing low-precision training schemes. Furthermore, by pushing the precision below the 16-bit bitwidth of existing state-of-the -rrt---r-ttt-tttte techniques, the proposed method is able to leverage the low- precision capabilities of modern processing systems to yield training speedups without penalising the resulting accuracy, significantly improving the to-curacyy trade-off.
2 BACKGROUND AND RELATED WORK
The state-of-the-art method in training in reduced precision is mixed-precision training (Micikevi- cius et al. 2018). The authors propose to employ low-precision FP16 computations in the training stage of high-precision CNNs that perform inference in FP32. Along the training phase, the algo- rithm maintains a high-precision FP32 copy of the weights of the network, known master copy. At each minibatch, the inputs and weights are quantised to FP16 with all computations of the for- ward and backward pass performed in FP16, yielding memory footprint and runtime savings. Under this scheme, each stochastic gradient descent (SGD) update step entails accumulating FP16 gradi- ents into the FP32 master copy of the weights, with this process performed iteratively throughout the training of the network. Micikevicius et al. (2018) evaluate their scheme over a set of state-of- art models on ImageNet, and show that mixed-precision training with FP16 computations achieves comparable accuracy to standard FP32 training.
Wang et al. (2018) also presented a method to train an FP32 model using 8-bit floating point (FP8). The authors propose hand-crafted FP8 data type, together with a chunk-based computation tech- nique, and employ strategies such as stochastic rounding to alleviate the accuracy loss due to train- ing at reduced precision. For AlexNet, ResNet18 and ResNet50 on ImageNet, Wang al. (2018) demonstrates comparable accuracy to FP32 training while performing computations in FP8.
Additionally the works presented in (Zhou et al. 2016 Chen et al. 2017) approach the problem of seduced-precisiin training employing fixed-poin point computations. FxpNet (Chen et al. 2017) was only evaluated CIFAR-10, failing to demonstrate performance ImageNet. DoReFa-net (Zhou et al. (2166) was tested on ImageNet but only ran on AlexNet missing out on wate--oftthe-an networks such as GoogLeNet and ResNet.
more complex datasets such as
2

---

Under review as a conference paper at ICLR 2020
All related works focus on accelerating the training of FP32 model through reduced- precision computations. At the hardware level, 8-bit fixed-point multiplication uses 18.5x less energy and x less area with up to 4x lower multiplication times than FP32 (Sze et al. 2017). Consequently, this work attempts to push the boundaries of reduced-precd precccsnne training by moving to reduced- precision fxxedpoont computations while updating FP32 model.
Preliminary tests (Sec. 4.4 for details) demonstrated that training solely in 8-bit fixed-point results in significant degradation of validation accuracy compared to full FP32 training. This work aims to counteract this degradation by progressively increasing the precision of computations throughout training an online manner determined by the proposed metric inspired by gradient diversity (Yin) et al. 2018). Additionally by operating in an online fashion, MuPPET tailors the training process to best suit the particular eetwork-datase pair at each stage of the training process.
Gradient diversity was introduced by Yin et al. (2018) as a metric of measuring the dissimilarity between sets of gradients that correspond different minibatches. The gradient diversity of a set of gradients is defined
As(w)= = Ex
E VV((w]
(1)
[V((W) + (iyy((iii Vfs(w))
where (w) represents the gradient of weights for minibatch i.
The key point to note in Eq. I is that the denominator contains the inner product between two gra- dients from different minibatches. Thus, orthogonal gradients would result in high gradient diversity, while similar gradients would result in low gradient diversity. The proposed framework, MuPPET, enhances this concept by considering gradients between minibatches across epochs and proposes the developed metric as a proxy for the amount of new information gained in each training step. Section further expands on how gradient diversity is incorporated into the MuPPET algorithm.
METHODOLOGY
3.1 MULTILEVEL. OPTIMISATION FOR TRAINING CNNSS
Conventionally, the training process of a CNN can be expressed as in Eq. (2). Given CNN model parameterised by a set of weights w E RP,, where D is the number of weights of f. training involves search for weight values that minimise the task-specific empirical loss, Loss, the target dataset. Typically, a fixed arithmetic precision is employed across the training algorithm with FP32 currently being the de facto representation used by the deep learning community.
min (1933 ER O
Loss Loss(s(w (((
The proposed method follows a different approach by introducing a multilevel optimisation scheme (Migdalas et al. 2013) that leverages the performance gains of reduced- precision arithmetic. The single optimisation problem of Eq. (2) is transformed into a series of optimisation problems with each one employing different precision for computations, but maintaining weights storage FP32 precision. Under this scheme, an N-level formulation comprises N sequential optimisation prob- lems be solved, with each level corresponding to "finer" model.
peduced-precision
Overall, this formulation adds a hierarchical structure to the training stage, with an increasing arith- metic precision across the hierarchy of optimisation problems. Starting from the N-th problem, the inputs, weights, and activations of the CNN model f are quantised with precision q. which is the lowest precision in the system and represents the coarsest version of the model. Each of the N lev- els progressively employs higher precision until the first level is reached, which corresponds to the original problem of Eq. (2). Formally, at the i-th level, the optimisation problem formulated as
min (f s.t. V {w(e') LB, LB]] w
(3)
where LB and UB are the lower and upper bound in the representational range of precision g². The target CNN model ʃ uses a set of weights quantised with precision q and hence the solution of this optimisation problem can be interpreted as an approximation to the original problem of Eq. (2). To transition from one level to the next, the result of each level of optimisation is employed as starting point for the next level, up the final outermost optimisation that reduces to Eq. (2)
3

---

Under review as a conference paper at ICLR 2020
3.2 THE MUPPET ALGORITHM
Hacc Capy
Fig. [ presents the process of train- ing CNN using the proposed al- gorithm. All figures in this pa- per are shown in the Appendix B at a larger scale for enhanced read- ability. Within each epoch, MuP- PET performs mixed-preed -xeeddprecision train- ing where the weights are stored in an FP32 master copy and are quan- tised to the desired fixed-point preci- sion un-the-fly fly. At epoch j, the com- putations for the forward and back- ward passes (F and B blocks respec- tively) are performed at the current quantised precision (q') and the ac- tivations as well as the gradients obtained from each layer are quantised by the quantiser module before being passed on to the next layer, or stored, After each minibatch, the full-precision mas- ter copy of the weights is updated using quantised gradient matrix. As discussed in Section 3.1 quantisation level is gradually increased over the period of the training. In MuPPET, switching between these optimisation levels at the correct times is crucial in order not compromise the fi- validation accuracy. this respect, MuPPET introduces a precision switching policy based on an inter-epoch gradient diversity (Yin et al., 2018) metric that dictates when to switch to the next precision. Details of the switching policy are presented Section 3.3
Lppdore
Loss
Tpodh Profioe
Sieerity
Guanber Credents
Figure 1: Precision-switching training scheme of MuPPET.
3.2.1 QUANTISATION STRATEGY
In order to implement quantised training. a quantisation strategy needs to be defined. The proposed dynamic quantisation strategy utilises block floating-poii arithmetic (also known as dynamic fixed- point), where each fixed-point number is represented as a pair of an WL -bi signed integer x and scale factor s, such that the value is represented as x x 2
During the forward and backward passes of the training process, the weights and feature maps are both quantised, and the multiplication operations performed the same low precision. The quantisation method employs stochastic rounding methodology (Gupta et al. 2015). The accu- mulation stage of the matrix-multiply operation accumulated into a 32-bit fixed-point value to prevent overflow on the targeted networks The result of this matrix multiplication is subsequently quantised to the target wordlength before being passed input to the next layer. Following the block floating point scheme, quantisation is performed such that each weight and feature map ma- trix in the network has a single scale factor shared by values within the matrix. The quantisation configuration for the i-th level of optimisation and the I-th layer, of. and the full set of configurations, q², are given by left- and right-hand side of Eq. respectively.
of = (WWL"
weights
VI E [1] and q' = 149./VZ E (1.1C1)
where [4] the number of layers of the target network, WI. is the fixed wordlength across the network, i1 weights and sh are the scaling factors for the weights and activations respectively, of the th layer for the i-th level of optimisation. As result, for N levels, there are N distinet quantisation schemes; N of these schemes are with varying fixed-point precisions, and the finest level of quantisation, g² single-precision floating-point (FP32). The scaling factor for matrix X is first calculated as shown in Eq. (5) and individual elements are quantised as in Eq.
s (weights, act)
= Ilog2 min
UB + 0.5 (weights ant
LB 0.5
(5)
X X
[weights, act) min
[weights, act) Te quuant
x
weights, act)
ga (night
+ Unif (-0.5, 0.5)
(6)
'The accumulator wordlength is large enough accommodate the current CNN models, without overflow.
4

---

Under review as a conference paper at ICLR 2020
where X (weights, act) {max, min} of the current layer, LB and UB are the lower and upper bound of the current wordlength WL and bni represents sampling from the uniform distribution the range [a.b]. Eq. (5) adds 0.5 and -0.5 to UB and LB respectively to ensure maximum utilisation of WL net
is either the maximum or minimum value in the weights feature maps matrix
3.2.2 INFORMATION TRANSFER BETWEEN LEVELS
Employing multilevel training for CNNs requires an appropriate mechanism for transferring infor- mation between levels. To achieve this, the proposed optimiser maintains a master copy of the weights in full precision (FP32) throughout the optimisation levels. Similar to mixed-precision training (Micikevicius et al. (2018), at each level the SGD update step is performed by accumulating fixed-point gradient value into the FP32 master copy of the weights. Starting from the coarsest quantisation level i = N, to transfer the solution from level to level - 1, the master copy is quantised using the quantisation scheme q²=1, With this approach, the weights are maintained in FP32 and are quantised on- mnthe--fy during run time in order to be utilised in each training step.
3.3 PRECISION SWITCHING POLICY
The metric to decide when to switch between levels of quantisation is inspired by Yin et al. (2018) and based on the concept of gradient diversity (Eq. ①.. MuPPET computes As (w) between gradients obtained across epochs as proxy measure the information that is obtained during the training process; the lower the diversity between the gradients, the less information this level of quantisation provides towards the training of the model. Therefore, the proposed method comprises a novel normalised inter-epoch version of the gradient diversity along with a run-time policy to determine the epochs to switch precision.
The following policy is employed to determine when precision switch is to be performed. For a network with layers L and a quantisation scheme q' that was switched into epoch et
For each epoch j and each layer E C, the last minibatch's 's gradient, VJ (w), is stored.
2. After (resolution) number of epochs, the inter-epoch gradient diversity at epoch j is
E/= EE VSf (w)
//
As(w)² Evie
(
3. At an epoch j. given a set of gradient diversities 8(j) As(w) Vest Vesi<sj. the ratio max Slx) is calculated.
4. An empirically determined decaying threshold T' = a Be (8) is placed on the ratio p.
5. If the violates T more than 7 times, a precision switch is triggered and S(j) 0.
As long as the gradients across epochs remain diverse, As(w)' (Eq. sustains high value and the value of p remains low. However, when the gradients across epochs become similar, As(w): decreases and the value of becomes larger. Generalisability across epochs is obtained as p accounts for the change in information relative the maximum information available since the last precision change. Hence, the metric acknowledges the presence of temporal variations in information provided by the gradients. Generalisability across networks and datasets is maintained measures ratio. Consequently, the absolute values of gradients which could vary between networks and datasets, matter less. Overall, MuPPET employs the metric p as a mechanism to trigger precision switch whenever violates threshold T more than times.
at the denominator of
The likelihood of observing gradients across epochs that have low gradient diversity, pecially at early stages of training is low. The intuition applied here is that when this does happen given precision, it may be indication that information is being lost due to quantisation and thus corresponds to a high p value, which argues to move to a higher bitwidth.
3.3.1 HYPERPARAMETERS
The hyperparameters for the proposed MuPPET algorithm are the following: 1) values of a, B, and À that define the decaying threshold from Eq. (8). 2) the number of threshold violations allowed
5

---

Under review as a conference paper at ICLR 2020
Trannyy Exochs for Mlollet
Parc Thonkhal Tranding Eoret Peofect
Trrad addy
P932
Fparfhc
(a) AlexNet CIFAR-10
(b) ResNet20 CIFAR-10
(c) GoogLeNet-
CIFAR-10
munic Thosshol br tranking Shaerss
Pagic Thoesteld Traning SpaAd
Momiic Lient Ty over
Hag
206
(d) AlexNet ImageNet (e) ReReee111- ImageNet
(1) GoogLeNer ImageNet
Figure 2: Demonstration of the generalisability of p over networks, datasets and epochs.
before the precision change is triggered (y), 3) the resolution 4) the set of precisions at which training is performed, and 5) the epochs at which the learning rate is changed. The values of a, B, X, r, and were set at 1, 1.5, 0.1, and 2 respectively after empirical vols-vaali,tion. These were tuned by running training on AlexNet and ResNet20 on the CIFAR-10 dataset. All MuPPET hyper- parameters remain the same regardless of network or dataset. Regarding training hyperparameters. batch size was increased from 128 to 256 going from CIFAR-10 to ImageNet. All other training hyperparameters, including learning rate remained constant. Analysis of generalisability and the training hyperparameters used are presented in Section 4.1 The -aaheenlly-choeen quantised pre- cisions at which training was performed were 8-, 12-, 14- and 16-bit fixed- -oont...t. Precisions below this did not result in any progress towards convergence for any network.
Overall, MuPPET introduces a policy that allows to decide at run time an appropriate point to switch between quantisation levels. After training at 16-bit fixed-point, the rest of the training is performed at FP32 until the desired validation accuracy is reached. Decaying the learning rate causes a finer ex- ploration of the optimisation space as does increasing the quantisation level. Therefore, the learning rate was kept constant during quantised training and was decayed only after switching to FP32.
EVALUATION OF MUPPET
4.1 GENERALISABILITY
The MuPPET framework was evaluated on its applicability across epochs, networks and datasets. Fig. 2 shows the value of the metric p over the epochs in blue, and the decaying threshold described in Eq. (8) in orange. The number of epochs for which training in each precision was performed is shown by the various overlay colours. The first violation is denoted by a red dot and the second violation is not seen as it occurs exactly at the point of switching. The graphs show that across various networks and datasets, the values of p stay relatively similar, backing the choice of a uni- versal decaying factor. Furthermore, empirical results for CIFAR-10 indicated that changing from one fixed- point precision to another too early in the training process had a negative impact on the final validation accuracy. Using decaying threshold ensures that the value of p needs to be much higher in the initial epochs to trigger precision change due to the volatility of in early epochs of training.
4.2 PERFORMANCE EVALUATION
The accuracy results presented in this section utilised the proposed stochastic quantisation strategy. The methodology was developed using PyTorch. As the framework does not natively support low- precision implementations, all quantisation and computations corresponding to 8-, 12-, 14-, and 16- bit precisions were performed through emulation on floating-point hardware. All hyperparameters

---

Under review as a conference paper at ICLR 2020
not specified below were left as PyTorch defaults. For all networks, an SGD optimiser was used with batch sizes 128 on CIFAR-10 or 256 on ImageNet, momentum of 0.9 and weight decay of le te-4.
As baseline, FP32 model with identical hyperparameters (except for batch size) was trained. The baseline FP32 training was performed by training for 150 epochs and reducing the learning rate by factor of 10 at epochs 50 and 100. In order to achieve comparable final validation accuracy to the FP32 baseline, once MuPPET triggered a precision change out of 16-bit fixed-point, 45 training epochs FP32 precision were performed. The learning rate was reduced by a factor of 10 every 15 FP32 training epochs. For AlexNet, ResNet18, ResNet20, and GoogLeNet, the initial learning rate was set to 0.01, 0.1, 0.1, and 0.001 respectively. The detailed breakdown of the ImageNet training runs with training and validation loss curves can be found in the Appendix A.
CIFAR-10 MuPPET
ImageNet MuPPET
FP32
DifT (pp)
FP32
Diff (pp)
AlexNet
75.45
74.49
-0.96
56.21
55.33
-0.88
ResNet
90.08
90.86
0.78
69.48
69.09
-0.39
GoogLeNet
89.23
89.47
0.24
59.15
63.70
4.55
Table Top-1 validation accuracy (%) on CIFAR- 10 and ImageNet for FP32 baseline and MuPPET
Table presents the achieved Top-1 validation accuracy of MuPPET and the FP32 baseline, together with the accuracy difference in percentage points (pp). As shown on the table, MuPPET is able to provide comparable Top-1 1 validation accuracy to standard FP32 training across both networks and datasets. Due sub-optimal training setup of GoogLeNet on ImageNet, the baseline and MuPPET training severely underperformed compared to the reported state-of-the-art works. Nevertheless, the results demonstrate the quality of training with MuPPET using identical hyperparameters. As a result, MuPPHT's performance demonstrates the effectiveness of the precision switching strategy in achieving significant acceleration of training time (Section 4.3) at negligible cost in accuracy by running many epochs at lower precision, particularly on very large datasets.
4.3 WALL-CLOCK TIME IMPROVEMENTS
This section explores the gains in estimated wall-clock time of the current implementation of MuP- PET (Current Impl.) with respect baseline FP32 training, Mixed Precision by Micikevieius et al.) 2018) and MuPPET's ideal implementation (Table 2) For all performance results, the target plat- form was an NVIDIA RTX 2080 Ti GPU, At the moment, deep learning frameworks, such Py- Torch, do not provide native support for reduced-pres reduced-precision hardware. Consequently, the wall-clock times in Table were estimated using performance model developed with NVIDIA's CUTLASS library (Kerr et al., 2018) for reduced-precisee
general matrix- matrix-muliplicatioon (GEMM) employing the latest Turing architecture GPUs. The GEMMs that were accelerated were in the convolutional and fully-connected layers of each network. INT8 hardware was used to profile the 8-bit fixed-point computations, while FP16 hardware was used to profile 12-, 14-, and 16-bit fixed-point computa- tions as well Mixed Precision (Micikevicius et al., 2018) wall-clock time. CUTLASS (Kerr al.) 2018) natively implements bit bit-packing to capitalise on improved memory-bandwiddhh utilisation. The model for the current implementation is limited by the fact that frameworks force quantisation to happen to and from FP32. For the MuPPET (Ideal) scenario, the model assumes native hardware utilisation which would reduce the overhead by removing this restriction.
As shown on Table 2, MuPPET consistently achieves 1 1.25-1.32x speedup over the FP32 baseline across the networks when targeting ImageNet on the given GPU. With respect to Mixed Precision, the proposed method outperforms it on AlexNet by 1.23x and delivers comparable performance for ResNet18 and GoogLeNet. Currently, the absence of native quantisation support. and hence the necessity to emulate quantisation and the associated overheads. is the limiting factor for MuPPET to achieve higher processing speed. In this respect, MuPPET run on native hardware would yield 1.05 x and 1.48x x speedup for ResNet18 and GoogLeNet respectively compared to Mixed Precision. As result, MuPPET demonstrates consistently faster to Mixed Precision across the benchmarks. Additionally, while Mixed Precision has already reached its limit by using FP16 on FP16-native GPUs, the 8-, 12-, 14- and 16-bit fixed -oint computations enabled by MuPPET leave space for further potential speedup when targeting next- and current- generation (Fowers et al. 2018) precision- precisio--otimmieed fixed-point platforms. Similar to the analysis in Section 4.2 Micikevicius al. (2018) and Wang et al. (2018) compare their schemes to baseline
(Coleman et al., 2019) compared
7

---

Under review as a conference paper at ICLR 2020
FP32 training performed by them. The reported results demonstrate that their methods achieve similar accuracy results to our method by lying close to the respective FP32 training aceuracy. As Wang et al. (2018) do not provide any results in terms of gains in wall-clock times and since they use custom HP8 hardware, their work could not be directly compared to our method.
FP32 (Baseline)
Mixed Prec
MuPPET
MuPPET (Ideal)
Micikevicius et al. 2018
(Current Impl.)
AlexNet
30:13 (1x)
29.20 (1.03x)
23:52 (1.27x)
20:25 (1.48x)
ResNet18
132:46 (1x)
97:25 (1.36x)
100:19 (1.32x)
92:43 (1.43x)
GoogLeNet
152:28 (1x)
122:51 (1.24x)
122:13 (1.25x)
82:38 (1.84x)
Table 2: Wall-clock time (GPU hours:mins) & relative acceleration for networks targeting ImageNet
4.4 PRECISION SWITCHING
Accuracy- Trade-off for Resnet20 on CIFAR100
To evaluate the ability of MuPPET to effectively choose an epoch to switch precision at, AlexNet and ResNet20 were first trained using MuPPET on the CIFAR-100 dataset. The hyper- parameters for MuPPET were kept the same across all runs. From the
muppet
random
66.0
static
Accuracy (N)
65.5
65.0
results it was noted that training at
64.5
reduced precision and not switch- ing at all causes a drop in vali- dation accuracy of 1.4% and 1.3% for Aeex and ResNer20 respec- tively, hence demonstrating the need to switch precisions when training at bit-widths as low 8-bit fixed-point.
63.5
1500
2000
3000
4000
4500
5000
Training Timats)
Figure 3: Accuracy vs time trade-off for ResNet20
MuPPET runs on CIFAR-100.
To demonstrate the benefits of a pre- cision switching methodology, two further sets of experiments were conducted ResNet20 using CIFAR100 as depicted in Fig. 3 First, 34 training runs were performed (34 red dots in Fig. 3 where for each training four epochs along the standard training duration were randomly selected and used as the switching points. Second, the switching strategy MuPPET generated for AlexNet and GoogLeeNet was applied to ResNet20 (2 blue dots in Fig. 33. Fig. shows the best test accuracy achieved by each of the runs and the training time as estimated by our performance model described in Sec. 4.3 It shows that for given bine-budget, MuPPET runs (6 green dots) outperform on average all other experiment sets, demonstrating the need for a precision switching policy that is real-time and agnostic network and dataset in order to achieve good accuracy- couuracy-to-trinnee to-training time trade-off.
CONCLUSION
This paper proposes MuPPET, a novel low- low-precision CNN training scheme that combines the use of fxeetppoint and floating-point representations produce network trained for FP32 inference. By introducing a precision-swichiing mechanism that decides at run time an appropriate transi- tion point between different precision regimes, the proposed framework achieves Top-1 validation accuracies comparable to that achieved by state-of-the-art FP32 training regimes while delivering significant speedup in terms of training time. Quantitative evaluation demonstrates that MuPPET's training strategy generalises across CNN architectures and datasets by adapting the training process to the target CNN-dataset pair during run time. Overall, MuPPET enables the utilisation of the low- precision hardware units available on modern specialised processors, such as next-generee eext-eeneration GPUs, FPGAs and TPUs, to yield improvements in training time and energy efficiency without impacting the resulting accuracy. Future work will focus on applying the proposed framework to the training of LSTMs, where the training process is more sensitive to gradient quantisation, as well as on the extension of MuPPET to include batch size and learning rate as part of its hyperparameters. Fur- thermore, we will explore improved quantisation techniques that could enable training convergence for bitwidths even lower than 8-bit fixed-point.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovie. QSGD: Communication-Effiiiee SGD via Gradient Quantization and Encoding. In Advances Neu- ral Information Processing Systems 30, pp. 1709-1720. 2017.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient Architecture Search by Network Transformation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Xi Chen, Xiaolin Hu, Ningyi Xu, Hucheng Zhou, Hucheng Zhou, and Ningyi Xu. ExpNet: Training deep convolutional neural network in fixed-point representation. In International Joint Confer- ence on Neural Networks (IJCNN 2017), May 2017.
Xiaoming Chen, Danny Ziyi Chen, Yinhe Han, and Xiaobo Sharon Hu. moDNN: Memory Optimal Deep Neural Network Training on Graphics Processing Units. IEEE Transactions on Parallel and Distributed Systems, 30(3):646-60 161446666,226 2019.
Cody Coleman, Daniel Kang, Deepak Narayanan, Luigi Nardi, Tian Zhao, Jian Zhang, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia. Analysis of DAWNBench,
time-to-Accuracy Machine Learning Performance Benchmark. SIGOPS Oper. Syst. Rev. 53(1):14 25, July 2019. ISSN 0163-5980. doi: 10. 10.1144533522203332222 URLhttp::/ /to::///oii.o acm. org/ 10../11.114//
3352020. 3352024
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. In Workshop in International Conference in Learning Representa- fions, 2015.
Christopher De Sa, Matthew Feldman, Christopher Ré, and Kunle Olukotun. Understanding and Optimizing Asynchronous Low-Precision Stochastic Gradient Descent. In 44th Annual Interna- tional Symposium on Computer Architecture (ISCA), pp. 561-574, 2017.
Christopher M De Sa, Ce Zhang, Kunle Olukotun, Christopher Ré, and Christopher Ré. Taming the Wild: A Unified Analysis of Hogwild-Sty Algorithms. In Advances in Neural Information Processing Systems 28, pp. 2674-2682. 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In 2009 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp.
2009.
Aditya Devarakonda, Maxim Naumov, and Michael Garland. AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks. 17 12.0229 [cs, stat], 2017.
Fowers, K. Ovtcharov, M. Papamichael, T. Massengill, M. Liu, Lo, Alkalay, M. Haselman, L. Adams, M. Ghandi, S. Heil, P. Patel, A. Sapek, G. Weisz, L. Woods, S. Lanka, S. K. Reinhardt, A. M. Caulfield, S. Chung, and D. Burger. Configurable Cloud-Scale DNN Processor for Real-Time AI. In 2018 ACM/IEEE 45th Annual International Symposium Computer Architec- ture (ISCA), pp. 1-14, 2018
C. Gan, Naiyan Wang, Y. Yang, Din Yeung, and A. G. Hauptmann. DevNet: A Deep Event Network for Multimedia Event Detection and Evidence Recounting. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2568-2577, 2015.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep Learning with Limited Numerical Precision. In 32nd International Conference Machine Learning (ICML), pp. 1737-1746, 2015.
K. He, G. Gkioxari, P. Dollar, and R. Girshick. Mask R-CNN. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2018.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely Connected Convolutional Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2261-2269, 2017.

---

Under review as a conference paper at ICLR 2020
Tyler B Johnson and Carlos Guestrin. Training Deep Models Faster with Robust, Approximate Importance Sampling. Advances in Neural Information Processing Systems 31, pp. 7265- 7275. 2018.
Norman P. Jouppi et al. In-Datacenter Performance Analysis of a Tensor Processing Unit. In 44th Annual International Symposium on Computer Architecture (ISCA), pp. 1-12, 2017.
Andrew Kerr, Duane Merril, Julien Demouth, and John Tran. CUTLASS: Fast Lin-
ear Algebra in CUDA C, Sep 2018. URL hatppsr//arvolooss..viii..... com/
cut lass- linear algebra-cudas
A. Kouris and C. Bouganis. Learning to Fly by MySelf: A Self-Supervised CNN-Based Approach for Autonomous Navigation. In 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1-9, 2018.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitiick. Microsoft COCO: Common Objects in Context. European Conference on Computer Vision (ECCV), pp. 740-755, 2014.
Antonio Loquercio, Ana I. Maqueda, Carlos R. del Blanco, and Davide Scaramuzza. DroNet: Learn- ing to Fly by Driving. IEEE Robotics and Automation Letters, 3(2): 2018.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksïi Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. In International Conference on Learning Representations (ICLR), 2018.
Athanasios Migdalas, Panos M Pardalos, and Peter Värbrand. Multilevel Optimization: Algorithms and Applications, volume 20. Springer Science & Business Media, 2013.
Xinyu Peng, Li Li, and Yue Wang. Accelerating Minibatch Stochastic Gradient Descent using Typicality Sampling. arXiv:1903.04192 [cs, math, stat], 2019.
Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W. Keckler. vDNN: Virtualized Deep Neural Networks for Scalable, nemory-efficieee Neural Network Design. In The 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 18:1- 18:13, 2016
V. Sze, Y. Chen, T. Yang, and J. S. Emer. Efficient Processing of Deep Neural Networks: A Tutorial and Survey. Proceedings of the IEEE, 105((2):2295-2333 2017.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, Inception- ResNet and the Impact of Residual Connections on Learning. In AAAI Conference on Artificial Intelligence, 2017.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan. Train- ing Deep Neural Networks with 8-bit Floating Point Numbers. In Advances Neural Information Processing Systems 31, pp. 7675-7684. 2018.
Dong Yin, Ashwin Pananjady, Max Lam, Dimitris Papailiopoulos, Kannan Ramchandran, and Peter Bartlett. Gradient Diversity: a Key Ingredient for Scalable Distributed Learning. In 21st Interna- tional Conference on Artificial Intelligence and Statistics (AISTATS), pp. 1998-2007, 2018.
Z. Zhong, J. Yan, W. Wu, J. Shao, and C. Liu. Practical Block-Wise Neural Network Architecture Generation. In 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2432,, 2018.
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. CoRR, 2016.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX A
The graphs Fig. dedemoottttttttt both the training and validation loss AlexNet, ResNet18 and GoogLeNet for MuPPET and FP32 runs the ImageNet dataset. For each graph, the light gray lines indicate the point at which precision was switched in the MuPPET run. The green lines are used show MuPPET behaviour and blue to show FP32 behaviour. Solid lines show validation loss and dashed lines show training loss.
AlexNet FP32 VS MUPPET loss curves
PP32-Vaidation
MuPPET- MuPPET-Valldation
100
120
Epoch
Figure 4: AlexNet training and validation loss values for FP32 and MuPPET on ImageNet.
ResNet18 FP32 vs MuPPET loss curves
FP32-Train
5.0
FP32-Valddation
MuPPET-Train
4.5
MuPPET-Valldation
4.0
35
3.0
2.5
2.0
15
10
20
40
60
100
120
140
Figure 5: ResNet18 18 training and validation loss values for FP32 and MuPPET on ImageNet.
11

---

Under review as a conference paper at ICLR 2020
GoogLeNet FP32 vs MuPPET loss curves
FP32-Train
FP32-Validttion
MuPPET-Train
8-
2-
40
60
100
120
140
Figure 6: GoogLeNet training and validation loss values for FP32 and MuPPET on ImageNet.
B
APPENDIX B
This section contains the larger versions of all the figures in the paper for enhanced clarity.
12

---

Under review as a conference paper at ICLR 2020
FP32 Master Copy
Quantiser
F
F
F
Weight Update
Weights
Activations
w
of i+1
o' F2
Loss
Gradient wunt weights
Gradient wr. inputs
B
B
B
Epoch Precision
qu+1
Quantiser
(w)
Gradient Diversity
v/--(ww
Metric (p)
V)((m)
Quantised Gradients
Figure Precision-switching switching training scheme of MuPPET.
13

---

Under review as a conference paper at ICLR 2020
Accuracy-Time Trade-off for Resnet20 on CIFAR100
muppet
random
66.0
static
Best Test Accuracy (%)
65.5
65.0
64.5
64.0
63.5
1500
2000
2500
3000
3500
4000
4500
5000
Training Time(s)
Figure 8: Accuracy vs time tradeoff for ResNet20 MuPPET runs on CIFAR100 Final Test Accuracy
14

---

Under review as a conference paper at ICLR 2020
Metric (p) and Threshold (y) over Training Epochs for AlexNet
Metric (p)
Threshold
2.0
8-bit
12-bit
14-bit
1.5
Value of p
16-bit
FP32
1.0
0.5
0.0
0
20
40
60
80
100
Epochs
Figure 9: AlexNet CIFAR10 Demonstration of the generalisability of p over networks, datasets and epochs.
15

---

Under review as a conference paper at ICLR 2020
Metric (p) and Threshold (y) over Training Epochs for GoogLeNet
Metric (p)
3.0
Threshold
8-bit
12-bit
2.5
14-bit
16-bit
2.0
FP32
Value of p
1.5
1.0
0.5
0.0
0
20
40
60
80
100
Epochs
Figure 10: GoogLeNet CIFARIO Demonstration of the generalisability of over networks, datasets and epochs.
16

---

Under review as a conference paper at ICLR 2020
Metric (p) and Threshold (y) over Training Epochs for ResNet20
Metric (p)
Threshold
2.0
8-bit
12-bit
14-bit
16-bit
1.5
FP32
Value of p
1.0
0.5
0.0
0
20
40
60
80
100
Epochs
Figure 11: ResNet20 CIFARIO Demonstration of the generalisability of over networks, datasets and epochs.
17

---

Under review as a conference paper at ICLR 2020
Metric (p) and Threshold (y) over Training Epochs for AlexNet
Metric (p)
Threshold
2.0
8-bit
12-bit
14-bit
16-bit
1.5
FP32
Value of
1.0
0.5
0.0
0
20
40
60
80
100
Epochs
AlexNet ImageNet Demonstration of the generalisability of over networks, datasets and epochs.
Figure 12:
18

---

Under review as a conference paper at ICLR 2020
Metric (p) and Threshold (y) over Training Epochs for GoogLeNet
Metric (p)
Threshold
2.0
8-bit
12-bit
14-bit
16-bit
1.5
FP32
Value of p
1.0
0.5
0.0
20
40
60
80
100
120
Epochs
Figure 13:
GoogLeNet ImageNet Demonstration of the generalisability of over networks, datasets and epochs.
19

---

Under review as a conference paper at ICLR 2020
Metric (p) and Threshold (y) over Training Epochs for ResNet18
Metric (p)
Threshold
2.0
8-bit
12-bit
14-bit
16-bit
1.5
FP32
Value of
1.0
0.5
0.0
20
40
60
80
100
120
Epochs
Figure 14: ResNet18 ImageNet- Demonstration of the generalisability of over networks, datasets and epochs.
20


Reviews:
[{"review_id": "r1xFvatycr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall an interesting paper, though I wished a more detailed presentation of the reasoning behind the algorithm would have been provided. As it stands it feels a bit heuristic. \n\nIn particular I don't understand the motivation between the switching mechanism. Basically it says if the gradients are co-aligned between epochs it means there is not much to learn anymore!? Why? Intuitively if the gradients would go to 0 or become very small maybe you would want to increase precision. Or if you have high variance you could argue that the expected gradient would be 0 and hence you are not really making progress, i.e. you are just moving left-right. But if all gradients agree on a moving direction, why is that a bad thing? I know the heuristic is borrowed from a different work, but since it feels as such an integral part of MuPPET I think you should explain it better. \n\nI guess a few details about the algorithm as well. When you say you look at the diversity of the gradients over the epochs, is this the batch gradient !? \n\nThere are some small typos (e.g. FP23 instead FP32). \n\nI find the justification for AlexNet to be adhoc (it switched at the wrong time, but that allowed to take more advantage of computation in the low precision hence it was faster). The switching mechanism should only care of when the gradients are not informative anymore, not how much compute you are wasting .", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "r1gJesMatS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The article presents an approach to reduce the precision of weights, activations and gradients to speed up the training of deep neural networks. The precision of these values is increased according to a dynamic schedule such that the original classification accuracy is reached after training.\n\nThe manuscript is in most parts well written and the addressed topic is of general interest for the research community represented at ICRL. Still, I recommend a weak reject, since the core idea of the manuscript, i.e. the dynamic switching between precision levels, is not shown to be a necessary condition for good classification results.\n\n\nMajor points:\n•\tThe introduction does not give a clear statement about the novel contribution of the paper. Only the very last paragraph is specific about the paper.\n•\tYour results support that step-wise increasing the resolution speeds up training without significant losses in accuracy. However, the impact of the gradient diversity, choice of p and threshold parameters on the performance of the trained networks are unclear. What is the isolated impact of every of these choices? According to Figure 2, pre-defined switching points between precision levels may also generalize between networks and datasets.\n•\tThe description of the quantization scheme is not clear enough in order to reproduce the results:\no\tPlease give details about every step from FP32 to FPx values or cite appropriate literature.\no\tEquation 4 and 5: How are the scaling factors SC determined?\no\tPlease clarify the difference/relation between n and WL.\n\n\nMinor points:\n•\tEquation 3: What does “represent. range(q^i)” mean?\n•\tText in Figure 1 and 2 is far too small and barely readable\n•\tStep 5 in Algorithm in Section 3.3: What does “p violates y more than gamma times” mean? What is y?\n•\tPlease clarify “distribution approach”. Distribution of what?\n•\tTable 1: For the baseline experiments, the precision is switched from 8 to 32 bits, for MuPPET from 8 to 12 bits (see main text). What is the motivation behind these different choices?\n•\tDo you use any type of data augmentation?\n•\tTable 3: Please clarify “theoretical limit”. Does this limit include 12 and 14 bit quantisation. What do you mean by “optimized quantization implementation” in main text?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeQTHBTtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a training strategy called Multi-Precision Policy Enforced Training(MUPPET). This strategy aims to reduce training time by low-precision data representation and computations during the training stage. According to the gradient diversity, the authors introduce a precision-switching mechanism which chooses the best epoch to increase the precision. The validation accuracy and training time across several networks and datasets are shown in the experiments. However, the results are not superior enough compared with the state-of-the-art.\n\nMy detailed comments are as follows.\n\n\nPositive points: \n\n1. This paper proposes a new reduced-precision training scheme to speed up training by progressively increasing the precision of computations from 8-bit fixed-point to 32-bit floating-point. This scheme moves to reduced-precision fixed-point computations while updating an FP32 model in order to push the boundaries of reduced-precision training. \n\n2. The authors propose a metric to decide when to switch the precision inspired by gradient diversity introduced by [1]. In this paper, the gradient diversity is enhanced by considering gradients across epochs instead of mini-batches. The proposed metric can be seen as a proxy for the amount of new information gained in each training step. Therefore, the metric can decide the most appropriate epoch at run time to increase the precision.\n \n3. The proposed low-precision CNN training scheme is orthogonal and complementary to existing low-precision training techniques.\n\n\n\n\nNegative points:\n\n1. The proposed approach does not match the description in this paper. The authors describe “This approach enables the design of a policy that can decide at run time the most appropriate quantization level for the training process”. In fact, this approach just decides which epoch to increase the quantization level while the levels of quantized precisions are fixed, rather than deciding the most appropriate quantization level. \n\n2. The setting of quantized precision levels (8-, 12-, 14- and 16-bit precisions) is confusing. Please illustrate how to choose the number of quantized bit and the number of quantized precision levels.\n\n3. The presentation of the precision switching policy is confusing and the notations are unclear. For example, in section 3.3, the ratio “p” needs more description because it is a key value in the policy, but lacks an explanation in this section. So please explain more about the motivation of ratio “p” in this section. \tIn section 3.3, in step 5 of the proposed precision switching policy, the authors do not explain the meaning of “y”.\n\n4. In figure 2, the precision switch is not triggered even though the value of p violates the threshold more than 2 times, which mismatches the description in section 3.3.\n\n5. The proposed strategy has no obvious advantages. There are some scenes that the proposed strategy does not perform well. For example, the Top-1 validation accuracy on ImageNet of AlexNet and ResNet with MuPPET strategy is much lower than FP32 baseline. Compared with [2], the proposed method is more complex but not superior enough.\n\n6. The authors do not show the training and validation curves. However, the training and validation curves are common used to show more details of the training process, such as in [2] and [3]. Please show and analyze the training and validation curves of the proposed scheme and the baseline.\n\n\nMinor issues:\nSome spelling and grammar mistakes.\n\n\nReference：\n[1] Dong Yin, Ashwin Pananjady, Max Lam, Dimitris Papailiopoulos, Kannan Ramchandran, and Peter Bartlett. Gradient Diversity: a Key Ingredient for Scalable Distributed Learning. In 21st International Conference on Artificial Intelligence and StatiZZstics (AISTATS), pp. 1998–2007, 2018.\n[2] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. In International Conference on Learning Representations (ICLR), 2018.\n[3]  Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep Learning with Limited Numerical Precision. In 32nd International Conference on Machine Learning (ICML), pp. 1737–1746, 2015.                                                                                                                                                                                                                                                                                                        \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission presents an approach to speed up network training time by using lower precision representations and computation to begin with and then dynamically increasing the precision from 8 to 32 bits over the course of training. The results show that the same accuracy can be obtained while achieving a moderate speed up. 

The reviewers were agreed that the paper did not offer a signficant advantage or novelty, and that the method was somewhat ad hoc and unclear. Unfortunately, the authors' rebuttal did not clarify all of these points, and the recommendation after discussion is for rejection. 
Decision: Reject