DNN Quantization with Attention
Year: 2021

Abstract:
Low-bit quantization of network weights and activations can drastically reduce the memory footprint, complexity, energy consumption and latency of Deep Neural Networks (DNNs). Many different quantization methods like min-max quantization, Statistics-Aware Weight Binning (SAWB) or Binary Weight Network (BWN) have been proposed in the past. However, they still cause a considerable accuracy drop, in particular when applied to complex learning tasks or lightweight DNN architectures. In this paper, we propose a novel training procedure that can be used to improve the performance of existing quantization methods. We call this procedure DNN Quantization with Attention (DQA). It relaxes the training problem, using a learnable linear combination of high, medium and low-bit quantization at the beginning, while converging to a single low-bit quantization at the end of the training. We show empirically that this relaxation effectively smooths the loss function and therefore helps convergence. Moreover, we conduct experiments and show that our procedure improves the performance of many state-of-the-art quantization methods on various object recognition tasks. In particular, we apply DQA with min-max, SAWB and BWN to train 2bit quantized DNNs on the CIFAR10, CIFAR100 and ImageNet ILSVRC 2012 datasets, achieving a very good accuracy comparing to other conterparts.

heading: 1 Introduction and Related Work
text: In the last decade, Deep Neural Networks (DNNs) in general and Convolutional Neural Networks (CNNs) in particular became state-of-the-art in many computer vision tasks, such as image classification or segmentation, object detection and face recognition (LeCun et al. (1998); Iandola et al. (2016); Simonyan & Zisserman (2014); Graham (2014); Szegedy et al. (2015)). However, to be the state-of-the-art, DNNs often contain a large number of trainable parameters and require considerable computational power. Therefore, due to their large power and memory consumption, implementing DNNs on embedded systems with limited resources can be a real challenge. To alleviate this problem, a large number of different network compression methods that reduce the resource requirements of DNNs has been proposed in the past. Among them are for example pruning, distillation or quantization methods. Pruning methods have been first introduced by LeCun et al. (1990). They identify and remove the most insignificant DNN parameters and yield networks with a reduced memory footprint and a smaller computational complexity. As reported by Yamamoto & Maeno (2018); Ramakrishnan et al. (2020); He et al. (2020), pruning methods can either remove single DNN parameters, intermediate inputs or even whole network layers that are, according to a specific criteria, irrelevant for a good network performance. Another line of work is distillation. As introduced by Hinton et al. (2015), it aims at training a small student DNN to reproduce the output of a bigger teacher network. While distillation methods initially only matched the final outputs of the teacher and student networks, methods evolved to take into account intermediate representations (Romero et al. (2014); Koratana et al. (2018)). Our work is about DNN quantization, where weights and activations are represented with a smaller number of bits n ăă 32. Quantization reduces the memory footprint of DNNs
because the number of bits that are required to store their parameters is reduced. However, it also reduces their computational complexity, because low bit operations can be implemented more efficiently on dedicated low precision hardware (Merolla et al. (2014); Farabet et al. (2011); Cowan et al. (2020); Han et al. (2020); Hacene et al. (2018)). Following Gupta et al. (2015), many works have experimentally demonstrated that neural networks do not lose a lot of performance when their parameters are restricted to a small set of possible values. For instance, Choi et al. (2018) introduced PArameterized Clipping acTivation (PACT) in combination with Statistics-Aware Weight Binning (SAWB) –a method that aims at uniformly quantizing both weights and activations to n bit. Learned Step Size Quantization (LSQ) presented by Esser et al. (2019) is a quantization method that learns the quantization steps during training. Unlike other methods, it scales the gradient during backpropagation to speed up the training. Gradient scaling is important especially at transition points. Nikolić et al. (2020) proposed Bit-Pruning, a quantization method to learn the number of bits that each layer requires to represent its parameters and activations. In the same vein, Differentiable Quantization of Deep Neural Networks proposed by Uhlich et al. (2019) (DQDNN) tries to combine the features of both LSQ and Bit-Pruning. They propose a quantization technique where both the number of bits and the quantization steps are learned simultaneous. Other more aggressive quantization methods proposed to use low-bit precision down to binarization (resp. ternarization) with only two (resp. three) possible values and one (resp. two) bit storage for each parameter and/or activation (Hubara et al. (2016); Courbariaux et al. (2015); Li et al. (2016b); Zhu et al. (2016); Li et al. (2016a)). Zhou et al. (2017) observed that training quantized networks to low precision benefits from incremental training. Rather than quantizing all the weights at once, they are quantizing them incrementally in groups, with some training iterations between each quantization step. In practice, 50% of the weights are quantized in the first step, then 75%, 87.5% and finally 100%. Another method for incremental quantization is Binary-Relax (BR) (Yin et al. (2018)). Rather than splitting the parameters into groups, it uses a weighted linear combination of quantized and full-precision parameters and adapts a strategy to push the weights towards the quantized state, by gradually increasing the scaling factor corresponding to the quantized parameters. However, their strategy how to move from full-precision to quantized parameters is handcrafted and may not be optimal. In comparison to most of the previously mentioned works, we do not propose an improved quantization method, but a way how to train a quantized DNN with any existing quantizer. We rely on the fact that the DNN performance can be increased if the network has the ability to learn other features in addition to its own parameters. In the context of DNN quantization, this has already been discussed by Elsken et al. (2019); Ramakrishnan et al. (2020); Hacene et al. (2019); Uhlich et al. (2019); Nikolić et al. (2020). In this contribution, we introduce DNN Quantization with Attention (DQA), an attention mechanism-based learnable approach for DNN quantization (Vaswani et al. (2017)). As shown in Figure 1,
rather than quantizing with just a single bitwidth DQA performs quantization with multiple different bitwidths in parallel. At each stage of the training, DQA can select the quantizer with the optimal bitwidth by putting more attention weight on it. In particular, by starting with a uniform initialization of the attention weights at the beginning of the training, DQA can move smoothly from high precision to low precision quantization. We demonstrate in this paper that this learnable approach results in better DNN accuracy for the exact same complexity and number of bits. The outline of the paper is as follows. In Section 3 we introduce the proposed method. Section 4 presents experiments results and compares our method with other state-of-the-art approaches on challenging computer vision datasets. Finally, we conclude in Section 5.

heading: 2 Background
text: In the following, we explain how differentiable quantization with attention (DQA) can be used to train quantized DNNs. For simplicity reasons, we only discuss how to train DNNs with quantized weights. However, DQA is general and can also be used to train DNNs with quantized weights and activations. In particular, we will also provide experimental results for this case in Section 4. Training DNNs with quantized weights is challenging. Especially when considering low bitwidths n ăă 32, quantization can cause a severe accuracy degradation if compared to the full precision networks. This is mainly caused by the capacity reduction and by additional optimization issues that go hand in hand with the quantization. In particular, quantization yields non-smooth loss functions with gradients that are zero almost everywhere. As discussed by Uhlich et al. (2019), that effectively stops gradient backpropagation and therefore harms training. For these reasons, quantized DNNs are usually not trained with standard gradient based training procedures, but require some tricks that allow for gradient backpropagation and that stabilize the training. The most commonly used trick is to apply straight through gradient estimates (STE) and to ignore the quantization during backpropagation. STE yield non-zero gradients that are suited for DNN training. However, at the same time they introduce a mismatch between the forward (FW) and backward (BW) pass, what often causes training instabilities and oscillations. More specifically, mismatch means that the gradients are calculated at the position of the quantized parameters, assuming the original float32 loss surface. Note, that because of this FW/BW mismatch, training with STE does not necessarily converge to the optimum. This mismatch is for example problematic, if the gradient of the cost function changes signs within one quantization step. For this case, gradient descent with STE would start to oscillate near the quantization thresholds and would not converge to the optimum, even if the cost function is convex. Of course, this problem is most pronounced for low bitwidhts. A training procedure that can alleviate this problem to some degree is Binary-Relax (BR). As proposed by Yin et al. (2018), BR does not only apply STE, but also uses a linear combination of the quantized and the full-precision parameters. This effectively reduces the FW/BW mismatch at the beginning, while still enabling the DNN to use the exact low-bit quantized loss at the end of the training.

heading: 3 Methodology
text: DQA builds on a similar approach. As shown in figure 1, let fpx;Qpw;nqq be the transfer function of a quantized DNN layer, where x P RD, w P RM and Qpw;nq are the layer input, the full-precision weights and the quantized weights, respectively. Similar to the idea of Binary-Relax (BR), DQA relaxes the quantization problem and combines different quantization schemes during training. More specifically, instead of using just one single Qpw;nq, we propose to train a quantized DNN with a set of K different quantization functions that
are combined linearly during training as follows:
y “ fpx;qq (1) q “ QTa (2)
Q “
»
—
—
—
–
Q1pw;n1qT Q2pw;n2qT
... QKpw;nKqT
fi
ffi
ffi
ffi
fl
, (3)
where q is the weighted sum of K quantized weight vectors, Q P RKˆM is a matrix whose row vectors are the quantized weight vectors and a P r0, 1sK is the attention vector on the quantization functions. Note, that each row of Q is calculated, using a different quantization function Qkpw;nkq and bitwidth nk P N. In particular, we assume that the quantization functions in Q are sorted by the bitwidth, i.e., n1 ă n2 ă ... ă nK . In general, DQA is agnostic to the choice of the actual quantization method and can be used with any existing method like min-max, SAWB, binary or ternary quantization. In the following section, we review and define popular quantization methods that we used in our experiments. The attention a is computed from a soft attention vector α P RK , using a softmax function with temperature T P R`, i.e.,
a “ e α T
řK k“1 e αk T
, P RK . (4)
In particular, a reflects the importance of the K quantization methods Qk. During training, the soft attention α is treated as a trainable parameter that is optimized in parallel to the weights w. Note that, increasing αk will also increase the corresponding attention weight ak and therefore the importance of Qkpw;nkq. In this manner, the quantized DNN can learn which bitwidth should be used at which stage, during the training. DQA exponentially cools down the temperature T
T pbq “ T p0qΨb. (5)
Here, b “ 1, 2, ..., B is the batch index for batch-wise training, T p0q P R` is the initial temperature and Ψ P r0, 1r is the decay rate. Because of that schedule, DQA progressively moves from the full mixture of quantization functions at the beginning of the training to just one single quantization function at the end of training. Note, that BR can be seen as a special case of DQA, where q “ rQpw; 2q, Qpw; 32qsT , i.e. for the case that we only use two quantizers with n “ 2bit and n “ 32bit, and for the case that we use a fixed schedule to change the attention vector a. However, DQA has two advantages: 1) The way how we change a and move from high to low precision quantization is learned and data dependent. Hence, DQA can choose the optimal mixture of the quantizers at each training iteration. 2) As shown in Figure 2, DQA gives a smoother transition from high to low-precision parameters. Here, we plot the absolute quantization error for a fixed temperature based schedule for a. In particular, we choose α “ r3{4, 1{4sT and α “ r4{7, 2{4, 1{7sT for BR and DQA with 2, 4, 8bit quantization, respectively. Then, we start with a large temperature T Ñ 8, for which we effectively take the average 1K řK k“1Qkpw;nkq, and move towards 2bit quantization for T Ñ 0. Note, that for the whole interval that we consider for T , DQA results in a lower quantization error, meaning that it also yields a smaller FW/BW mismatch. In general, training quantized DNNs with such a mixture of different weight quantizations and decaying T will not necessarily result in a quantized DNN that uses a low bitwidth. To enforce a low-bit quantized DNN, we therefore add to the loss function a separate regularizer for each layer
rpαq “ λg Tapαq S , (6)
where S is the number of weights in the whole network. Note, that the normalization by S makes the regularizer, and therefore the choice of λ, independent of the actual network size. g “ rg1, g2, ¨ ¨ ¨ , gKsT is a penalty vector, where gk is increasing with growing k. Because we assume, that the quantization functions Qkpw;nkq are sorted by the bitwidth, i.e., n1 ă n2 ă ... ă nK adding gTapαq penalizes if large attention values are assigned to quantizers with a large bitwidth. Hence, it helps the method to converge to the lowest-bit quantization. Algorithm 1 summarizes the DQA training. To quantize a given value x, we use min-max, Statistics-Aware Weight Binning SAWB, Binary Weight Network (BWN) or Ternary Weight Network (TWN) as defined by Nikolić et al. (2020); Choi et al. (2018); Rastegari et al. (2016); Li et al. (2016a) respectively, and detailed in appendix A.1
Algorithm 1 DQA algorithm for a single network layer Inputs: Input vector x, initial softmax temperature T p0q, final softmax temperature T pBq, number of training iterations B, and layer transfer function f Output: Output tensor y
ψ “ e log
ˆ
T pBq T p0q
˙
B ă 1 for each b “ 1, 2, ..., B do T pbq Ð T p0qψb α Ð αstdpαq a Ð softmaxpα{T pbqq q “ QTa (linear combination) y “ fpx,qq Update w and α via backpropagation. end for

heading: 4 Experiments
text: In this section we will first introduce the benchmark protocol that we use to evaluate our method, then we report different results obtained by DQA and compare them with other training procedures.
4.1 Benchmark Protocol
To evaluate our method DNN Quantization with Attention (DQA), we perform experiments on the three object recognition datasets CIFAR10, CIFAR100 and ImageNet ILSVRC 2012. For each dataset, we use DQA to train low-bit quantized versions of the ResNet18 (He et al. (2016)) and MobileNetV2 (Sandler et al. (2018)) network architectures. Low-bit means, that we consider networks that only use n “ 1 or n “ 2bit for quantization.
Under review as a conference paper at ICLR 2022
2-bit 4-bit
8-bit
100
200
300
0
0.5
1
Quantization Epoch
A tt
en tio
n va
lu e a k
(a) Attention values ak
−1 −0.5 0 0.5 1
−1
−0.5
0
0.5
1
Float weight w
A tt
en tio
n qu
an tiz
at io
n ∑ m k=
1 a k Q (w
;n k )
Epoch 1 Epoch 190 Epoch 3001 −0.5 0 0.5 1
−1
−0.5
0
0.5 1
Float weight w
A tt
en tio
n qu
an tiz
at io
n ∑ m k=
1 a k Q (w ;n k )
Epoch 1 Epoch 190 Epoch 300
(b) Resulting quantization function
Figure 3: The evolution of the attention values ak and the resulting quantization function for the first layer of a quantized ResNet18 trained on CIFAR100.
For CIFAR10 and CIFAR100, we start from randomly initialized parameters w and train the quantized networks for 300 epochs. As an optimizer, we use SGD with an initial learning rate γ “ 0.1, which is divided by 10 every 100 epochs. The training batch size is 128. On the ImageNet ILSVRC 2012 dataset, we train the quantized networks for 90 epochs, using a batch size of 256 images. As an initial learning rate, we again use γ “ 0.1 which is divided by 10 every 30 epochs. This way, we again apply two equally spaced learning rate drops over the full 90 epochs. For all our experiments, we either quantize only the weights or both weights and activations using DQA with three different quantization functions tQ1, Q2, Q3u. More specifically, we either consider a mixture of three min-max quantization functions that use n1 “ 2bit, n2 “ 4bit and n3 “ 8bit, respectively or a mixture of BWN, TWN and 8bit min-max quantization. For all experiments, we use an exponential temperature schedule with an initial temperature T p0q “ 100 that is cooled down to the final value of T pBq “ 0.03. The soft attention vector is initialized according to
αk “ řN j“1,j‰k nj řN
j“1 nj . (7)
Note, that since the quantization functions Qkpw;nkq are assumed to be sorted by the bitwidth, i.e. n1 ă n2 ă ¨ ¨ ¨ ă nK , this initialization assigns the highest attention to the quantization function with the lowest bitwidth. The initialization therefore acts as a prior that favours low-bit quantized DNNs and helps us to converge to small bit widths early during training. To further encourage low-bit quantized DNNs, we use the penalty values g “ r1, 4, 16sT that penalize quantization functions with a large bitwidth. Note, that we always compare networks that are quantized to the same bitwidth and thus have the same memory footprint and the same computational complexity.
4.2 Results
In the first experiments, we report the obtained accuracy achieved by DQA and compare it to three different baselines: 1) The full-precision network with float32 parameters. 2) The quantized network that uses 2bit quantized parameters and uses vanilla training without any relaxation scheme. 3) Binary-Relax (BR). To have a fair comparison to BR, we also report some results where we consider BR with the same mixture of quantization functions, i.e.,
q “ ωQ1pw, n1q `Q2pw, n2q `Q3pw, n3q ω ` 2 , (8)
where ω is initialised to 1 and multiplied by 1.02 after each epoch. In other words, we use a fixed schedule to move from 8bit to 2bit quantization.
Table 1 and 2 show the experimental results for the CIFAR10 and CIFAR100 datasets when we quantize only weights and when we quantize both weights and activations, respectively. We report the final validation accuracy of the quantized DNNs for different choices of the quantization functions tQ1, Q2, Q3u. In general, all reported validation accuracies are the result of a single training run. Only for the experiments that use BWN quantization, we report the average validation accuracy computed over 5 runs, because the convergence of BWN quantized networks proved to be noisy, which shadowed the effects of DQA. Our proposed method archives accuracies that are comparable to the full-precision baseline, while outperforming the 2bit quantized baseline and BR. The second experiment aims at studying the behavior of the attention values ak during training. Figure 3 shows the evolution of the attention values ak and the corresponding quantization function. We can observe from 3a that all attention values are equal at the beginning of the training but – due to the penalty term and the temperature schedule – they slowly converge towards a maximum attention value for the 2bit quantization. This evolution can also be seen in 3b where we show how the resulting quantization function evolves during training. Note, that the quantization function is smoothed out at the beginning and converges more and more towards the 2bit quantization curve at the end of the training. This smooth transition is the reason why DQA yields better results than training with just a single fixed quantization. Interestingly, compared to a single low-bit quantization, DQA yields smoother loss surfaces. Figure 4 visualizes the loss surface of a ResNet18 with randomly initialized weights on the CIFAR10 dataset. Here, we apply the method proposed by Li et al. (2017) that samples two random directions in the parameter space of a DNN and visualizes the loss along these directions. Obviously, the loss surface is the smoothest for a float32 network. In comparison, the same ResNet18 with 2bit min-max quantized weights yields a very rough loss surface. For the 2bit case, optimization can get stuck easily in one of the numerous local minima. Moreover, it yields gradients that change quickly in direction and magnitude, causing severe oscillations and effectively making the training unstable. However, if we apply DQA and use a mixture of 2, 4 and 8bit min-max quantization to quantize the network weights, the loss surface is smoothed out. Therefore, compared to DNNs that are trained with only one low-bit quantization scheme, quantized DNNs trained with DQA typically converge faster at the beginning of the training and reach a better final optimum. The third experiment compares DQA with other methods for quantized DNNs trained on the ImageNet ILSVRC 2012 dataset. Table 3 (parameter quantization only) and Table 4 (parameter and activation quantization) show that DQA outperforms the quantized 2bit baseline and BR when considering different quantization approaches. Moreover, DQA causes a significantly smaller drop in accuracy when quantizing MobileNetV2. Thus, it may represent a
promising training procedure that makes existing quantization methods more powerful and, hence, helps us to train lightweight DNN architectures.

heading: 5 Conclusion
text: In this paper, we introduced DQA, a novel learning procedure for training low-bit quantized DNNs, using existing quantization methods. Instead of using only a single quantization precision during training, DQA relaxes the problem and uses a mixture of high, medium and low-bit quantization functions. Our experiments on popular object recognition datasets, such as CIFAR10, CIFAR100 and ImageNet ILSVRC 2012, show that DQA can be used to train highly accurate low-bit quantized DNNs that achieve a good accuracy compared with state-of-the-art counterparts. If we compare to the full-precision networks, DQA yields a significantly lower accuracy drop than other training procedures that only use a single quantization precision and bitwidth during training, This is especially true when quantizing DNN architectures that are already designed to be lightweight and efficient, such as the MobileNetV2. Because such architectures are already small, they are naturally harder to compress. DQA also compares favourably to Binary-Relax (BR), another training procedure for quantized DNNs that applies a mixture of quantized and full-precision weights during training.
.
However, while BR uses a fixed scheme to mix the network weights of different precision, DQA can learn how to mix them in an optimal way and how to gradually move from high precision to low precision. In practice, this helps training and results in quantized DNNs with higher accuracy. DQA is agnostic to and can be used with many different existing quantization methods, such as min-max, PACT-SAWB, Binary-Weight and Ternary-Weight quantization. Therefore, DQA is a very promising extension to existing DNN quantization methods.

heading: A Appendix
text: A.1 Choosing the Quantization Functions
Quantization describes the process of representing a value x P X with a corresponding quantized value q P Q, using a quantization function Q : X Ñ Q. Here, Q “ tq1, q2, ..., q2nu is the set of quantization steps that is much smaller than X , i.e., |Q| ăă |X |. For a given w and Q, the quantization function minimizes the distance between w and q, i.e.,
Qpx;nq “ argmin qPQ }x´ q}, (9)
where } ¨ } is the Euclidean norm. There are different methods how to construct Q that yield different quantization schemes, like uniform or non-uniform quantization. The first method we may consider is the one introduced in Nikolić et al. (2020). For X “ rxmin, xmaxs they define
qi “ xmin ` pi´ 1q xmax ´ xmin
2n ´ 1 , i “ 1, 2, ..., 2 n. (10)
In particular, the values qi are uniformly distributed between the values xmin and xmax, what is known as min-max quantization. The second method we use with our proposed training procedure is Statistics-Aware Weight Binning (SAWB) Choi et al. (2018). The quantization values are again distributed uniformly over a given interval. However, instead of using the limits xmin and xmax, SAWB introduces a limit α, i.e.,
qi “ ´α ` pi´ 1q 2α
2n ´ 1 , i “ 1, 2, ..., 2 n. (11)
The optimal α can be calculated in a calibration step, using data. In particular, we minimize the mean-square quantization error
α˚ “ argmin α Ex„ppxqr}x´Qpx;n, αq}2s (12)
with respect to α. After calibration, we can use Qpx;nq “ Qpx;n, α “ α˚q for quantization. For both min-max and SAWB quantization, the solution of Eq. equation 9 is straight-forward to obtain. It is a uniform quantization function with equally spaced quantization steps that is defined by
Qpw;nq “
$
’ &
’ %
q1 , x ă“ q1 q1 ` q2n´q12n´1 round ´ x 2 n´1
q2n´q1
¯
, others q2n , x ą q2n . (13)
Another quantization function worth to mention was introduced for the Binary Weight Network (BWN) Rastegari et al. (2016). It uses a scaling factor β “ Ep|x|q and constrains
the quantized values to be binary (n “ 1). In particular, with Q “ t´β, βu, the quantization function is defined as
Qpw, 1q “ βw ¨ signpxq “ " β , x ě 0 ´β , others (14)
In the same vein, Ternary Weight Network (TWN) Li et al. (2016a) introduces a third quantization step to improve the accuracy. A TWN uses a bitwidth of n “ 2 and a symmetric Q “ t´β, 0, βu. Similar to the BWN, the range parameter β is calibrated with data. More specifically, we can compute the optimal range β˚ “ Ex ppx||x|ąδqr|x|s, where δ “ 0.7Er|x|s is the symmetric threshold that is used for quantization during calibration. The resulting quantization function is defined as
Qpx; 2q “
$
&
% ´β , x ď ´δ 0 , |x| ď δ β , x ą δ . (15)

Review summaries:
This paper attempt to address a challenging quantization problem, i.e., low-bit quantization. This work utilizes a learnable linear combination of high, medium, and low-bit quantization at the beginning while converging to a single low-bit quantization at the end of the training. In the quantization procedure, multiple quantizers and the corresponding attention matrices are adopted to fuse the quantized weights or activations.

This paper proposes DNN quantization with Attention (DQA), which uses a learnable linear combination of high, medium, and low-bit quantization at the beginning. It gradually converges to a single low-bit quantization at the end of training. Experiments show that DQA outperforms the naive quantization and the Binary-Relax method consistently across three datasets and two networks.

The papers addresses the problem of compression of neural networks. The paper builds upon binary-relax prior work. The precision is adapted during training with a mixture-based quantization method through temperature cooling and a set of an ``attention’’ vector a. The idea of the method, called DQA, is to progressively moves from a mixture of quantization functions (mixing high with low precision, for instance 32 bit with 2 bit training), to a single one (low precision) towards the end of the training. The paper states that the method can be used with several types of quantization methods. The evaluation is carried on computer vision architectures (ResNet18, MobileNet) for image recognition tasks (Cifar-10, Cifar-100, Imagenet ILSVRC 2012).  

This work presents a training method for low-bit network quantization. While training, it employs a multi-bitwidth paradigm in order to alleviate the nonsmooth optimization landscape with lower bitwidth. It uses a temperature parameter and a penalty term to force the network to gradually converge to the target low bit. Experiments are conducted on CIFAR10, CIFAR100, ImageNet Classification with ResNet 18 and MobilenetV2.