The batch size can affect inference results
Authors: Yunkyung Park
Kyungsoo Kim
Deok-kyu Jang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Matrix operation
Floating-point
Batch size
GEMM

Abstract: When performing matrix multiplication using GPUs, the cuBLAS library is commonly used for computational efficiency. Because of the cuBLAS’ heuristics, a vast, deep neural network model with GPUs may produce different test results owing to the batch sizes in both the training and inference stages. In this paper, we show that the batch size affects the inference results of deep neural network models. Our test models were the well-known bidirectional encoder representations from transformers (BERT) and generative pre-trained transformer (GPT) natural language processing  (NLP) models, and the super-resolution generative adversarial network (SRGAN) image generation model in FP32 and TF32. In the TF32 setting, the evaluation loss in BERT using the general language understanding evaluation (GLUE) data sometimes varied for different batch sizes. The GPT generated sentences depending on batch size, and we show the logit's mean square error by increasing the token length. The SRGAN  model produced different images from batch to batch. However, these phenomena were not observed under the FP32 setting. Therefore, the batch size must be carefully managed in large-sized deep neural networks under the TF32 setting.

Under review as a conference paper at ICLR 2023
THE BATCH SIZE CAN AFFECT INFERENCE RESULTS
Anonymous authors Paper under double-blind review
ABSTRACT
When performing matrix multiplication using GPUs, the cuBLAS library is com- monly used for computational efficiency. Because of the cuBLAS' heuristies, vast, deep neural network model with GPUs may produce different test results owing to the batch sizes in both the training and inference stages. In this paper, we show that the batch size affects the inference results of deep neural network models. Our test models were the well-known bidirectional encoder represen- tations from transformers (BERT) and generative pre-trained transformer (GPT) natural language processing (NLP) models, and the respouuono generative adversarial network (SRGAN) image generation model in FP32 and TF32. In the TF32 setting, the evaluation loss in BERT using the general language understand- ing evaluation (GLUE) data sometimes varied for different batch sizes. The GPT generated sentences depending on batch size, and we show the logit's mean square error by increasing the token length. The SRGAN model produced different im- ages from batch to batch. However, these phenomena were not observed under the FP32 setting. Therefore, the batch size must be carefully managed in large-sized deep neural networks under the TF32 setting.
INTRODUCTION
Several numerical models include matrix multiplication as a fundamental component. For com- putational efficiency, tiling methods are employed for matrix multiplication on computers, but this leads to the accumulation of rounding errors. Considerable research has been undertaken develop accurate and efficient matrix multiplication. Algorithm-bass based fault tolerance (ABFT) Kuang- Huang & Abraham (1984) and autonomous-Al sentative studies. Furthermore, several algorithms have been compared in Févre al. (2021). The most widely used library for performing basic linear algebra operations, such as vector addition, scalar multiplication, dot products, linear combinations, and matrix multiplication, is termed basic linear algebra subprograms (BLAS).
Braun al.) (2014) are two examples of repre-
The BLAS is specification for set of low-level routines execute standard linear algebra oper- ations, such as vector addition, scalar multiplication, dot products, linear combinations, and matrix multiplication. They are the standard low-level routines for linear algebra libraries, including C ("CBLAS interface") and Fortran bindings ("BLAS interface"). Although the BLAS standard is broad, BLAS implementations are frequently optimized for speed on single platform: therefore, employing them can result in significant performance gains. Most libraries that provide linear alge- bra functions adhere to the BLAS interface, allowing library users to create programs not connected to the BLAS library being used. With the development of GPGPU, BLAS implementations are being increasingly used, for instance, cuBLAS and rocBLAS.
Additionally, general matrix multiplication (GEMM) is the matrix multiplication method contained in BLAS. Numerous studies have been conducted to develop GEMM. For instance, the study for efficient non-square and sparse matrix computations ods that fully utilize hardware resources were developed in Fatahalian et al. (2004), and research in Kelefouras et al. (2016) enhanced the optimization and effectiveness of performing GEMM on GPUS.
performed in Qin al. (2020), the meth-
A deep learning model based matrix operations is used in many diverse tasks. To train and infer- ence a deep neural network model, it is important to compute matrices of different sizes efficiently, hence, acceleration research is being conducted to include them in various accelerators extending

---

Under review as a conference paper at ICLR 2023
from tensor cores to TPUs. A hyperscale model such as this requires huge amount of computing resources, because of the numerous matrix computations that may reduce accuracy. Therefore, batch learning is required to overcome memory limitations, and research has been undertaken on adaptive batch sizes for successful batch learning Devarakonda et al. (2017) McCandlish et al. (2018).
However, because the batch size changes the size of the matrix to be computed, the GEMM operation method's tiling size is varied by the batch size. Large batches are often utilized for rapid training, whereas relatively small batches are used for service. In addition, there is problem caused by the difference between the training and inference stages, and at this time, the error is not synced, thus degrading the model's performance.
Floating-poti point arithmetic is method to effectively represent and calculate decimal points in com- puter science. It is expressed an approximation through a floating point method at the expense of decimal-poin precision and is known as a standard, such as IEEE 754. Floating-pots point precision for scientific operations generally uses 64-bit more double-precision arithmetic. Because precision in deep learning has different computational costs and different precision requirements, instead of sac- rificing precision, tensor operation accelerators that significantly increase the computational cost and speed are used. One representative example is NVIDIA's Tensor Core. Even when using low pre- cision in deep learning, in certain cases, the bit configuration differs from the general floating-pont standard. In general, it is composed by changing the combination of the sign bits, exponent bits, and mantissa bits that constitute the floating point. BF16 Kalamkar et al.] (2019), ). which increases the dy- FP16 and reduces significant figures, and TF32, which enables the
namic range of single-precisios dynamic range of 32 bits using the 19-bit representation supported after NVIDIA's Ampere archi- tecture, etc. In general, deep learning algorithms may encounter the problem of gradient explosion by abandoning the dynamic range. Increasing the dynamic range by abandoning significant digits at the same precision is effective.
In this study, we used the bidirectional encoder representations from transformers (BERT) Devlin (et al. (2018) and the generative pre-trained transformer (GPT) Brown et al. (2020) in natural lan- guage processing (NLP) models and the super-resolution generative adversarial network (SRGAN) Ledig et al. (2016) in an image generation model to estimate the phenomenon investigate whether the batch size can affect the inference results under two types of floating-poin arithmetic systems, TF32 and FP32. In BERT, the evaluation loss was compared using the benchmark GLUE data set Wang et al. (2018) by varying the training batch and the inference batch. In GPT, generated sen- tences were compared for batch size with the logit's 's mean square error when increasing the token length. In SRGAN, generated images were compared for inference from 256 data elements by batch to the data trained with 200,000 CelebFaces Attributes Dataset (CelebA) images Liu et al. 2015).
2 EXPERIMENTS
We experimentally tested the three models BERT, GPT, and SRGAN. The GPUs specifications for this experiment were NVIDIA RTX 3090 (ampere) in TF32 and NVIDIA 2080 TI (Turing) in FP32.
2.1 BERT
Developed by Google in 2018 BERT is transformer- based language model. We performed fine tuning according to the batch size using the GLUE dataset based on BERT's pre-training data and performed text classification. A batch size of 32 was used for training and evaluation loss was measured using 1, 2, 4, 8, 16, 32, 64, and 128 batches for inference. Subsequently, significant differences were found in three datasets (CoLA, RTE, SST-2) out of nine GLUE datasets (Figure This phenomenon did not occur in FP32 (Figure
2.2 GPT
GPT is an unsupervised transformer language model created by OpenAI, which translates text, an- swer questions, summarizes passages, and generates text output. In GPT, the inference of generating texts was conducted based on the official trained model. The input data of all batch sizes is same. We choose that the batch size of the model is 1,2.4 and 8, and the FP32 and TF32 settings of NVIDIA activated to enable computation acceleration of the tensor core. Figure consists of the same
2

---

Under review as a conference paper at ICLR 2023
044
330
131
0.0
320
676
1707
6.64
C2LA
611
Figure 1: Evaluation loss according to batches in TF32 and FP32 settings
input as the different batch used for the inference. And the mean square crror(MSE) of the output token logit was shown. From these experimental results, it can be said that the logit value, that is, the probability value of the next token generated by iterative calculation the GPT model, has un- stable numerical calculation consistency precision in TF32 compared FP32. Among them, the forward calculation with different batch sizes while using TF32 precision has a relatively large error. In fact, most deep learning algorithms lear and evaluate by placing data through tensors. However, when inferring the model, most use different batch size than the training time. This effect appears larger as the model larger and the number of calculations increases, and can be easily confirmed in model with a large amount of iterative computation such as GPT.
and batch
2 and 4 batch
and batch
RP32
10a
RP32
11:32
1F32
191
202
30.
Mon My M Manry
20-2
10
30
10
Logit mean square error
Mis TP12
10
20-1
12.
and batch
1 and is batch
and batch
FF32
204
26
36'
10
20
FP92 19332
RP32
10-1
203
1002
400
3000
1000
Number generated max token length
Figure 2: Mean square error between inference logit values of different batches when increasing token length
2.3 SRGAN
SRGAN is generative adversarial network (GAN) for image super-resolution (SR). For SRGAN training, around 200,000 CelebA images were trained in 16 batches, and 4X high-resolution image output, which is the basic size of SRGAN was used. Therefore, the size of 2 becomes SR at 872*712. The inferences stages for the trained model were 1, 2, 4, 8, and 16 batches. In FP32, all generated images were identical regardless of the batch size. In contrast, in TF32, the images not identical when using and 2 batches, 4 and batches, and 16 batch while the images with batch and batch and those with 4 batch and 8 batch were identical, respectively (Table
3

---

Under review as a conference paper at ICLR 2023
Table 1: Mean number of non-identical pixels in 256 generated images with different batch size in TF32
Batch size
2
4
8
16
0
0
121.9
121.9
123.0
0
121.9
121.9
123.0
0
0
22.9
0
22.9
16
0
DISCUSSION
Our experiments show that the results of GEMM and foating-pooi operations in deep learning problems may compute different results from the training and inference batches, and this is due to swapping, which accumulates errors owing to matrix operations and floating-point numbers. This problem may appear in various domains, such as the text and image fields, but it is not recognized as a problem because the known models are not large enough and the numerical precision used in the training is not well disclosed. In addition, in general generative model, it is difficult to construct metric measure the quality of the generative model, and there are almost no studies on performance being reduced by the batch size at the time of inference, which is not part of learning. In the case of text, owing to the presence of a random part of the generative model, the part where the corresponding effect is not well recognized also has considerable influence. In image generation models, such as SRGAN, image quality may be affected continuous images are processed or the batching varied. Therefore, when performing GEMM operations for matrix operation GPUs, the batch sizes should be carefully managed in large-sized deep neural networks.
4 CONCLUSION
We have experimentally shown that errors occur as the batch sizes for training and inference vary in deep learning model. This error affect the model's performance(not performance but stale?). This problem is caused as the GEMM operation on the GPU changes heuristically according to the size of the input matrix. Owing the characteristics of the GPU hardware, this variable calculation by GEMM is essential for optimizing the operation speed; thus, if it forcibly controlled, the speed will inevitably decrease.
REFERENCES
Claus Braun, Sebastian Halder, and Hans Joachim Wunderlich. A-ABFT: Autonomous Algorithm- Based Fault Tolerance for Matrix Multiplications on Graphics Processing Units. 2014 44th An- nual IEEF/FIP International Conference on Dependable Systems and Networks, 6 2014.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, htt...//22..111....
00s/2005.14165, 2020. URL bttpt:://a//
Aditya Devarakonda, Maxim Naumov, and Michael Garland. Adabatch: Adaptive batch sizes for training deep neural networks. CoRR, abs/1712.02029, 2017. URI. htttp://axii....
abs/ /1717 02029
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/18 110.0480,,4 2018. URL
http:://rxii wwwpariwi..ogoaa.ooo
www./ar///a/ag./ 04805
4

---

Under review as a conference paper at ICLR 2023
K. Fatahalian, J. Sugerman, and P. Hanrahan. Understanding the efficiency of GPU algorithms for matrix-matrix multiplication. Proceedings of the ACM SICCRAAPH/EERGGRRPICCCCCCC confererr Graphics hardware HWWS '04, 2004.
Valentin Le Févre, Thomas Herault, Julien Langou, and Yves Robert. A Comparison of Several Fault-Tolerance Methods for the Detection and Correction of Floating-Point Errors in Matrix- Matrix Multiplication. Euro-Par 2020: Parallel Processing Workshops, pp. 303-315, 2021.
Dhiraj D. Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang. Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisck Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A study of BFLOATI6 for deep learning training. CoRR, abs/1905, 12322, 2019. URL http: org. /bs/
1905. 2322
Vasilios Kelefouras, Angeliki Kritikakou, Iosif Mporas, and Vasileios Kolonias. A high perfor- mance matrix- matrix multiplication methodology for epu and gpu architectures. The Journal of Supercomputing, 72, 03 2016. doi: 10.10077s11227-015--1
Kuang-Hua Huang and Jacob A. Abraham. Algorithm-Based Fault Tolerance for Matrix Operations. IEEE Transactions Computers, (-13(6):518-528, 6 1984.
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew P. Aitken, Alykhan Tejani, Johanmes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using generative adversarial network. CoRR, obs/0600, 2016. [Rtt:::///axxiv.or
abs/ Abs/1609. 04802
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. CoRR, abs/1812.061622 2018. URL http:://axxw..rg/////////
66]62
Eric Qin, Ananda Samajdar, Hyoukjun Kwon, Vineet Nadella, Sudarshan Srinivasan, Dipankar Das, Bharat Kaul, and Tushar Krishna. Sigma: A sparse and irregular gemm accelerator with flexible interconnects for dnn training. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 58-70, 2020. doi:
20009//PPA444544222000000000
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/ 10188804477... 2018. URL http: / /rrxiv.. org/ abs 1804. 07461
5


Reviews:
[{"review_id": "bE8rCLP-kU", "reviewer": "Reviewer_YAmr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, authors show that the batch size affects the inference results of deep neural network models. \n\nIn the empirical study, authors studied bidirectional encoder representations from transformers (BERT) and generative pre-trained transformer (GPT) natural language processing (NLP) models, and the super-resolution generative adversarial network (SRGAN) image generation model in FP32 and TF32. \n\nThe study shows batch size will influence TF32 setting while not observed under the FP32 setting.", "summary_of_the_review": "Overall I am afraid the depth in this paper is not sufficient for a long paper in ICLR. It may better fit toas a short workshop paper.", "strength_and_weaknesses": "Overall the topic on the impact of batch size on the performance is interesting. However, this paper lack of sufficient deep understanding on how it happens and how to prevent it. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZXhLUgajhgs", "reviewer": "Reviewer_ReKx", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper sheds light on discrepancies in results when using tf32: depending on the batch size used for inference (and training?), the same instance will give different results (as measured as logits, pixel values or loss depending on the setup).", "summary_of_the_review": "This submission is extremely incomplete and does not meet the standard for a publication venue such as ICLR.\nIt seems like the paper is describing the very first step of a project: verifying and exposing the problem. Beyond that, the contributions are non-existent.", "strength_and_weaknesses": "Strengths:\n- It seems like an interesting problem and I don’t think people are aware enough of this behavior.\n\nWeaknesses:\n- The paper is mostly about shedding light on the discrepancy during inference (could not tell whether the training case was studied). For this study to be interesting, I would have expected some deeper analyses of why such discrepancy is happening, if inference batch sizes need to match training batch sizes, if not to what discrepancy is acceptable, formulate recommendations, etc.\n- The paper is very hard to follow, and I had a hard time putting things together. For instance, I am still unclear on most of the experiment steups.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Pu-S87goqrM", "reviewer": "Reviewer_7wfT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper experimentally shows that errors occur as the batch sizes for training and inference vary in\na deep learning model by caused as the GEMM operation on the GPU.", "summary_of_the_review": "The work has some interesting findings through experiments. But it is not ready to be published at its current form without any deeper analysis.", "strength_and_weaknesses": "An interesting finding! It experimentally finds that the different batch sizes during training and inference will affect the model performance due to the matrix operation on GPU. Therefor the authors suggest to carefully choose the batch size.\n\nHowever,\n1. The paper is not well-written in presentation. The format of the reference is not correct. Table 1 lacks the annotation of the lines and columns. In Figure 2, it is also not clear what the '1 and 2 batch' means. The expression of \"batch/batches\" is also confusing. Do the authors mean batch size? 4 batch and 4 batch size refer to different meanings.\n2. The work is not ready in its current form to be published at the ICLR conference with only very limited experimental results without more deep analysis.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject