SlowGuess/ACL_2020_OCR huggingface dataset:
In Appendix Figure 12, we visualize the models specialized for different hardware mentioned in Table 1. We find that the GPU model is wide but shallow; the Raspberry Pi model is deep but thin. The phenomenon echos with our latency profiling (Figure 2) as GPU latency is insensitive to embedding and hidden dim, but Raspberry Pi is highly sensitive. It guides manual designs: on GPU, we can reduce the layer number and increase dimension to reduce latency and keep high performance.
Ablation Study. HAT achieves higher BLEU with  $1.5 \times$  lower latency and  $1.5 \times$  smaller size compared with the largest SubTransformer (Table 4). This suggests that larger models do not always provide better performance, and demonstrates the effectiveness of HAT. We also compare the evolutionary search with random search (Figure 9). Evolutionary search can find models with lower losses than random search.
7681
WMT'14 En-De	WMT'14 En-Fr
Inherited Val Loss	Inherited BLEU	From- Scratch BLEU	Inherited Val Loss	Inherited BLEU	From- Scratch BLEU
4.71	24.9	25.8	3.92	37.4	38.8
4.40	25.8	27.6	3.71	38.0	40.0
4.07	26.3	28.1	3.48	39.5	41.1
4.02	26.7	28.2	3.46	39.6	41.4
4.01	26.9	28.4	3.45	39.7	41.7
Figure 10: The search cost measured in pounds of  $\mathrm{CO}_{2}$  emission. Our framework for searching HAT reduces the cost by four orders of magnitude than the Evolved Transformer (So et al., 2019).
SubTransformer Performance Proxy. All Sub Transformers inside the SuperTransformer are uniformly sampled and thus equally trained, so the performance order is well-preserved during training. We conduct experiments to show the effectiveness of the SubTransformer performance proxy as in Table 5 and Appendix Figure 11. The BLEUs of Sub Transformers with inherited weights and weights trained from-scratch are very close. More importantly, they also have the same relative performance order. Therefore, we can rely on the proxy to search high-performance model architecture, significantly reducing the search cost.
Low Search Cost. As shown in Table 2 and Figure 10, the search cost of HAT is  $12,041 \times$  lower than the Evolved Transformer. Although both are using Evolutionary Search, the key difference is that Evolved Transformer needs to train all individual models and sort their final performance to pick top ones; on the contrary, HAT trains all models together inside SuperTransformer and sorts their performance proxy to pick top ones. The superior performance of HAT proves that the performance proxy is accurate enough to find good models.
Finetuning Inherited SubTransformers In section 4.1, we trained each searched SubTransformer
Table 5: The performance of SubTransformers with inherited weights are close to those trained from-scratch, and have the same relative performance order.
Task	From-Scratch 40K	Inherit-Finetune 10K
WMT'14	41.5	41.7
En-Fr	40.0	40.2
WMT'14	28.0	28.0
En-De	27.5	27.4
Table 6: The SubTransformer inherited from the SuperTransformer can achieve similar or better performance than the same SubTransformer trained from-scratch. Training steps are saved by  $4 \times$ .
from-scratch in order to conduct fair comparisons with baselines. In practice, we can also directly finetune the SubTransformers with the inherited weights from the SuperTransformer to further reduce the training cost. With 10K finetuning steps (1/4 of from-scratch training), the inherited SubTransformers can achieve similar or better performance than trained from-scratch ones (Table 6). In this way, the training cost for a model under a new hardware constraint can be further reduced by  $4 \times$ , since the SuperTransformer training cost is amortizable among all searched models.
Quantization Friendly. HAT is orthogonal to other model compression techniques such as quantization. We apply K-means quantization to HAT and further reduce the model size. We initialize centroids uniformly in the range of [min, max] of each weight matrix and run at most 300 iterations for each of them. Even without any finetuning, 4-bit quantization can reduce the model size by  $25 \times$  with negligible BLEU loss compared to the Transformer-Big baseline (Table 7). Interestingly, the 8-bit model even has 0.1 higher BLEU than the full precision model, indicating the robustness of searched HAT. Compared with the Transformer-Base 4-bit quantization baseline, which has 24MB model size and 38.9 BLEU score, HAT has 2.2 higher BLEU with similar model size.
Knowledge Distillation Friendly. HAT is also orthogonal to knowledge distillation (KD) because HAT focuses on searching for an efficient architecture while KD focuses on better training a given architecture. We combine KD with HAT by distilling token-level knowledge (top-5 soft labels) from a high-performance SubTransformer to a low-performance SubTransformer on WMT'14 En-Det task. The teacher model has a BLEU of 28.5 and 49M parameters; the student model has 30M parameters. KD can improve the BLEU of the student model from 25.8 to 26.1.
7682
BLEU	Model Size	Reduction
Transformer Float32	41.2	705MB	-
HAT Float32	41.8	227MB	3×
HAT 8 bits	41.9	57MB	12×
HAT 4 bits	41.1	28MB	25×
Table 7: K-means quantization of HAT models on WMT'14 En-Fr. 4-bit quantization reduces the model size by  $25\times$  with only 0.1 BLEU loss compared with the transformer baseline. 8-bit quantization even has 0.1 higher BLEU than its full precision version.
Transformer. Transformer (Vaswani et al., 2017) has prevailed in sequence modeling (Ng et al., 2019; Junczys-Dowmunt, 2018). By stacking identical blocks, the model obtains a large capacity but incurs high latency. Recently, a research trend is to modify the Transformer to improve the performance (Chen et al., 2018; Wu et al., 2019b; Sukhbaatar et al., 2019; Wang et al., 2019). Among them, Wu et al. (2019b) introduced a convolution-based module to replace the attention; Wang et al. (2019) proposed to train deep Transformers by propagating multiple layers together in the encoder. Zhang et al. (2018) and Kim et al. (2019) also proposed AAN and SSRU to replace the attention mechanism. HAT is orthogonal to them and can be combined to search for efficient architecture with those new modules. Another trend is to apply non- or partially-autoregressive models to cut down the iteration number for decoding (Gu et al., 2019; Akoury et al., 2019; Wei et al., 2019; Gu et al., 2018). Although reducing latency, they sometimes suffer from low performance. Bapna et al. (2018) explored using learned linear combinations of encoder outputs as decoder inputs, while HAT concatenates the outputs without linear combinations, thus better preserving the low-level information. Wu et al. (2020) investigated mobile settings for NLP tasks and proposed a multi-branch Lite Transformer. However, it relied on FLOPs for efficient model design, which is an inaccurate proxy for hardware latency (Figure 2). There are also works (Kim and Rush, 2016; Junczys-Dowmunt et al., 2018; Kim et al., 2019; Yan et al., 2020) using Knowledge Distillation (KD) to obtain small student models. Our method is orthogonal to KD and can be combined with it to improve the efficiency further. There are also hardware accelerators (Ham et al., 2020; Zhang et al., 2020) for attention and fully-connected layers in the Transformer to achieve efficient processing.
Neural Architecture Search. In the computer vision community, there has been an increasing interest in automating efficient model design with Neural Architecture Search (NAS) (Zoph and Le, 2017; Zoph et al., 2018; Pham et al., 2018; He et al., 2018). Some applied black-box optimization such as evolutionary search (Wang et al., 2020b) and reinforcement learning (Cai et al., 2019b; He et al., 2018; Wang et al., 2018, 2020a; Mao et al., 2019); Some leveraged backpropagation with differentiable architecture search (Liu et al., 2019). Some also involved hardware constraints into optimizations such as MNasNet (Tan et al., 2019), ProxylessNAS (Cai et al., 2019b), FBNet (Wu et al., 2019a) and APQ (Wang et al., 2020b). To reduce the NAS cost, supernet based methods (Pham et al., 2018; Bender et al., 2018; Guo et al., 2019) apply a proxy for sub-network performance and adopt search algorithms to find good sub-networks. For NLP tasks, the benefits of the architecture search have not been fully investigated. Recently, So et al. (2019) proposed the Evolved Transformer to search for architectures under model size constraints and surpassed the original Transformer baselines. However, it suffered from very high search costs (250 GPU years), making it unaffordable to search specialized models for various hardware and tasks. In addition, hardware latency feedback was not taken into account for better case-by-case specializations. Since different hardware has distinct architecture and features (Cong et al., 2018), feedback from hardware is critical for efficient NLP.
We propose Hardware-Aware Transformers (HAT) framework to solve the challenge of efficient deployments of Transformer models on various hardware platforms. We conduct hardware-aware neural architecture search in an ample design space with an efficient weight-shared SuperTransformer, consuming four orders of magnitude less cost than the prior Evolved Transformer, and discover high-performance low-latency models. We hope HAT can open up an avenue towards efficient Transformer deployments for real-world applications.
We thank NSF Career Award #1943349, MIT-IBM Watson AI Lab, Semi-conductor Research Corporation (SRC), Intel, and Facebook for supporting this research.
7683
Nader Akoury, Kalpesh Krishna, and Mohit Iyyer. 2019. Syntactically supervised transformers for faster neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1269-1281, Florence, Italy. Association for Computational Linguistics.
Ankur Bapna, Mia Chen, Orhan First, Yuan Cao, and Yonghui Wu. 2018. Training deeper neural machine translation models with transparent attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3028-3033, Brussels, Belgium. Association for Computational Linguistics.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. 2018. Understanding and simplifying one-shot architecture search. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 550-559, Stockholmssan, Stockholm Sweden. PMLR.
Han Cai, Chuang Gan, and Song Han. 2019a. Once for all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791.
Han Cai, Ligeng Zhu, and Song Han. 2019b. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations.
Mia Xu Chen, Orhan First, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 76-86, Melbourne, Australia. Association for Computational Linguistics.
Jason Cong, Zhenman Fang, Michael Lo, Hanrui Wang, Jingxian Xu, and Shaochong Zhang. 2018. Understanding performance differences of fpgas and gpus. In 2018 IEEE 26th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), pages 93-96. IEEE.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1243-1252, International Convention Centre, Sydney, Australia. PMLR.
Édouard Grave, Armand Joulin, Moustapha Cisse, David Grangier, and Hervé Jégou. 2017. Efficient softmax approximation for GPUs. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine
Learning Research, pages 1302-1310, International Convention Centre, Sydney, Australia. PMLR.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In International Conference on Learning Representations.
Jiatao Gu, Changhan Wang, and Jake Zhao. 2019. Levenshtein Transformer. arXiv.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. 2019. Single path one-shot neural architecture search with uniform sampling.
Tae Jun Ham, Sung Jun Jung, Seonghak Kim, Young H Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park, Sanghee Lee, Kyoung Park, Jae W Lee, et al. 2020. A^3: Accelerating attention mechanisms in neural networks with approximation. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 328-341. IEEE.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. 2018. Amc: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pages 784-800.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications.
Marcin Junczys-Dowmunt. 2018. Microsoft's submission to the wmt2018 news translation task: How i learned to stop worrying and love the data. arXiv preprint arXiv:1809.00196.
Marcin Junczys-Dowmunt. 2019. Microsoft translator at wmt 2019: Towards large-scale document-level neural machine translation. arXiv preprint arXiv:1907.06170.
Marcin Junczys-Dowmunt, Kenneth Heafield, Hieu Hoang, Roman Grundkiewicz, and Anthony Aue. 2018. Marian: Cost-effective high-quality neural machine translation in  $\mathbf{C} + +$ . In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 129-135, Melbourne, Australia. Association for Computational Linguistics.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882.
Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947.
Young Jin Kim, Marcin Junczys-Dowmunt, Hany Hassan, Alham Fikri Aji, Kenneth Heafield, Roman Grundkiewicz, and Nikolay Bogoychev. 2019. From research to production and back: Ludriciously fast
neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 280-288, Hong Kong. Association for Computational Linguistics.
Diederik Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2019. DARTS: Differentiable architecture search. In International Conference on Learning Representations.
Ilya Loshchilov and Frank Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations.
Hongzi Mao, Parimarjan Negi, Akshay Narayan, Hanrui Wang, Jiacheng Yang, Haonan Wang, Ryan Marcus, Mehrdad Khani Shirkoohi, Songtao He, Vikram Nathan, et al. 2019. Park: An open platform for learning-augmented computer systems. In Advances in Neural Information Processing Systems, pages 2490-2502.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 314-319, Florence, Italy. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. 2018. Efficient neural architecture search via parameters sharing. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4095-4104, Stockholmssan, Stockholm Sweden. PMLR.
Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771.
David So, Quoc Le, and Chen Liang. 2019. The evolved transformer. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5877-5886, Long Beach, California, USA. PMLR.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th
Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 331-335, Florence, Italy. Association for Computational Linguistics.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. 2019. Mnasnet: Platform-aware neural architecture search for mobile. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Conference on Neural Information Processing Systems.
Elena Voita, David Talbot, Fedor Moiseev, Rico Senrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, Florence, Italy. Association for Computational Linguistics.
Hanrui Wang, Kuan Wang, Jiacheng Yang, Linxiao Shen, Nan Sun, Hae-Seung Lee, and Song Han. 2020a. Gcn-rl circuit designer: Transferable transistor sizing with graph neural networks and reinforcement learning. In ACM/IEEE 57th Design Automation Conference (DAC).
Hanrui Wang, Jiacheng Yang, Hae-Seung Lee, and Song Han. 2018. Learning to design circuits. In NeurIPS 2018 Machine Learning for Systems Workshop.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1810-1822, Florence, Italy. Association for Computational Linguistics.
Tianzhe Wang, Kuan Wang, Han Cai, Ji Lin, Zhijian Liu, Hanrui Wang, Yujun Lin, and Song Han. 2020b. Apq: Joint search for network architecture, pruning and quantization policy. In Conference on Computer Vision and Pattern Recognition.
Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. 2019. Imitation learning for non-autoregressive neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1304-1312, Florence, Italy. Association for Computational Linguistics.
7685
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. 2019a. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2019b. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations.
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. 2020. Lite transformer with long-short range attention. In International Conference on Learning Representations.
Zhongxia Yan, Hanrui Wang, Demi Guo, and Song Han. 2020. Micronet for efficient language modeling. Journal of Machine Learning Research.
Biao Zhang, Deyi Xiong, and Jinsong Su. 2018. Accelerating neural transformer via an average attention network. arXiv preprint arXiv:1805.00631.
Zhekai Zhang, Hanrui Wang, Song Han, and William J Dally. 2020. Sparch: Efficient architecture for sparse matrix multiplication. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 261-274. IEEE.
Barret Zoph and Quoc V Le. 2017. Neural Architecture Search with Reinforcement Learning. In International Conference on Learning Representations.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. 2018. Learning Transferable Architectures for Scalable Image Recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Figure 11: The validation loss of SubTransformers is a good performance proxy for BLEU of from-scratch trained SubTransformers. The larger the validation loss, the lower the BLEU score.
A Appendix for "HAT: Hardware-Aware Transformers for Efficient Natural Language Processing"
A.1 SubTransformer Performance Proxy
In Figure 11, we show the relationship between the validation loss of SubTransformers directly inherited from the SuperTransformer, and the BLEU score of the SubTransformers trained from-scratch. We can observe that the larger the validation loss, the lower the BLEU score. Therefore the validation loss can be a good performance proxy.
A.2 Visualizations of Searched Models on WMT'14 En-De Task
We show the HAT models searched for Raspberry Pi ARM Cortex-A72 CPU and Nvidia TITAN Xp GPU in Figure 12. The searched model for Raspberry Pi is deep and thin, while that for GPU is shallow and wide. The BLEU scores of the two models are similar: 28.10 for Raspberry Pi CPU, and 28.15 for Nvidia GPU.
SubTransformer Optimized for Raspberry Pi ARM Cortex-A72 CPU
Figure 12: SubTransformers optimized for Raspberry Pi ARM CPU and Nvidia GPU on WMT'14 En-De task are different. The CPU model has BLEU 28.10, and GPU model has BLEU 28.15.
SubTransformer Optimized for Nvidia TITAN Xp GPU
A.3 Latency, BLEU and SacreBLEU of searched HAT models.
In Table 8, we show the specific latency numbers, BLEU and SacreBLEU (Post, 2018) scores for searched HAT models in Figure 7 and Figure 8.
Task	Hardware	Latency	BLEU	SacREBLEU
WMT'14 En-De	Raspberry Pi ARM Cortex-A72 CPU	3.5s	25.8	25.6
4.0s	26.9	26.6
4.5s	27.6	27.1
5.0s	27.8	27.2
6.0s	28.2	27.6
6.9s	28.4	27.8
Intel Xeon E5-2640 CPU	137.9ms	25.8	25.6
204.2ms	27.6	27.1
278.7ms	27.9	27.3
340.2ms	28.1	27.5
369.6ms	28.2	27.6
450.9ms	28.5	27.9
Nvidia TITAN Xp GPU	57.1ms	25.8	25.6
91.2ms	27.6	27.1
126.0ms	27.9	27.3
146.7ms	28.1	27.5
208.1ms	28.5	27.8
WMT'14 En-Fr	Raspberry Pi ARM Cortex-A72 CPU	4.3s	38.8	36.0
5.3s	40.1	37.3
5.8s	40.6	37.8
6.9s	41.1	38.3
7.8s	41.4	38.5
9.1s	41.8	38.9
Intel Xeon E5-2640 CPU	154.7ms	39.1	36.3
208.8ms	40.0	37.2
329.4ms	41.1	38.2
394.5ms	41.4	38.5
442.0ms	41.7	38.8
Nvidia TITAN Xp GPU	69.3ms	39.1	36.3
94.9ms	40.0	37.2
132.9ms	40.7	37.8
168.3ms	41.1	38.3
208.3ms	41.7	38.8
IWSLT'14 De-En	Nvidia TITAN Xp GPU	45.6ms	33.4	32.5
74.5ms	34.2	33.3
109.0ms	34.5	33.6
137.8ms	34.7	33.8
168.8ms	34.8	33.9
WMT'19 En-De	Nvidia TITAN Xp GPU	55.7ms	42.4	41.9
93.2ms	44.4	43.9
134.5ms	45.4	44.7
176.1ms	46.2	45.6
204.5ms	46.5	45.7
237.8ms	46.7	46.0
Table 8: Specific latency numbers, BLEU and Sacre-BLEU scores for searched HAT models in Figure 7 and Figure 8.
7688