Causes and Effects of Unanticipated Numerical Deviations in Neural Network Inference Frameworks
Authors: Alexander Schlögl
Nora Hofer
Rainer Böhme
Venue: NeurIPS 2023 poster (2023)
Keywords: machine learning
security
reproducibility
forensics
TL;DR: We measure how different hardware affects the results of neural network inference while keeping the trained model and the input constant.

Abstract: Hardware-specific optimizations in machine learning (ML) frameworks can cause numerical deviations of inference results. Quite surprisingly, despite using a fixed trained model and fixed input data, inference results are not consistent across platforms, and sometimes not even deterministic on the same platform. We study the causes of these numerical deviations for convolutional neural networks (CNN) on realistic end-to-end inference pipelines and in isolated experiments. Results from 75 distinct platforms suggest that the main causes of deviations on CPUs are differences in SIMD use, and the selection of convolution algorithms at runtime on GPUs. We link the causes and propagation effects to properties of the ML model and evaluate potential mitigations. We make our research code publicly available.

Causes and Effects of Unanticipated Numerical Deviations in Neural Network Inference Frameworks
Alexander Schlögl
Nora Hofer
Rainer Böhme
Department of Computer Science
Universität Innsbruck, 6020 Innsbruck, Austria
[alexander scc.annddrrschooglllll.
nora. hofer rainer. boehme) bubk. ac
Abstract
Hardware-specific optimizations in machine learning (ML) frameworks can cause numerical deviations of inference results. Quite surprisingly, despite using fixed trained model and fixed input data, inference results are not consistent across plat- forms, and sometimes causes of these numerical deviations for convolutional neural networks (CNN) on realistic end-to-end inference pipelines and in isolated experiments. Results from 75 distinet platforms suggest that the main causes of deviations CPUs are dif- ferences in SIMD use, and the selection of convolution algorithms runtime on GPUs. We link the causes and propagation effects to properties of the MI. model and evaluate potential mitigations. We make our research code publicly available.
even deterministic on the same platform. We study the
Introduction
The "reproducibility crisis" machine learning facing, and potentially fueling 18), has drawn at-
tention efforts that improve reproducibility. They include checklists [27] and guidelines (5) 33
benchmarks designed with reproducibility in mind [10 4 22). reproducibility contests (32). and repositories like PapersWitte WithCode (30]. However, we find that
with fully defined environment
and without stochastic processes, runtime optimizations of ML frameworks can cause deviations in
inference results. These are outside of researchers' control, and cannot be fully avoided at present.
Uncontrolled numerical deviations are detrimental to many aspects ML. If deviations occur sys- tematically, key assumptions in federated learning 7 25 heterogeneous MI. 36), and proof-of- learning 111 may not hold. The fact that platforms leave fingerprints in the inference results opens new possibilities for forensies (34]. Finally, numerical deviations have implications ML security: specifically crafted "boundary samples" may trigger label flips depending on the hardware [35]
To study the causes and effects of these deviations, we instrument the popular TensorFlow frame- work (version 2.5.0) on various layers. All our experiments are containerized and automatically deployed and executed on 75 distinct hardware configurations hosted on the Google Cloud Platform (GCP), Amazon Web Services (AWS), and on our premises. (Details of our setup are Section of the supplementary material.) For fixed trained model and input, the 75 platforms produced up to 26 different softmax vectors in the last layer. As label flips remain rare, the existence of deviations apparent in typical performance metries, such test accuracy.
We make three contributions:
We offer the so-far most comprehensive evaluation causes and effects of (known) nu- merical deviations in CNN inference, spanning wide range of heterogeneous platforms.
2. We are the first to associate causes of deviations with properties under control of the MI. engineer, such floating-point precision, layer type, activation function.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

3. We make the code of our infrastructurd and experiments publicly available, allowing follow-up researchers to measure deviations between runs and platforms and inspect them layer by layer. The set of supported platforms can be adjusted with limited effort.
The body of this paper is structured as follows. Section provvidsss background, establishes terminol- ogy, and reviews related work. Section present the experimental results. Starting from end-to-end inference pipelines, we walk through individual causes for CPUs (Section D.I). GPUs (Section 3.2
link them properties of the ML model (Section 3.3), and study potential mitigations (Section 3.4
always supported with experiments. Section discusses the findings and Section [ concludes.
2
Background
The main causes for deviations inference results are different aggregation orders at finite precision, and the use of different approximate convolution algorithms.
Aggregation order The order in which arithmetic operations are executed can affect the result for limited precision. Consider an example in toy decimal floating point representation with only one significant digit. In this representation, integers 0 & |a| S 10 can be represented exactly. For values 10 < |y| I 100, y must be rounded to the next multiple of 10, and the least significant digit is lost. We will denote rounding to the nearest representable value with [a:].
(a + b) + [[ + 4| 5) = [11 - 5] = (b + c) 7 + [1 5]] =
=
(2)
The above example shows how the aggregation order can change the result. In Equation ( [+4] =
[11] is rounded 1E1 10. This effect is known swamping [13]. Hence, optimizations that change the aggregation order depending on the hardware can cause deviations between platforms.
Convolution algorithms Recall the formula for 21) convolution,
h
RE,
I-m.j-n F.m.j-nFin,n. Fmn..
7L=00
for 2-dimensional filter of size h x w with input I and filter F. To reduce the size of the convolution result, sometimes stride is applied to the convolution, which is multiplied with input indices i,
The naive implementation of convolution is nested loop expressing the nested sums. This approach is often inefficient as spatial proximity in the input does not imply locality in memory, in particular for higher dimensions. Modern hardware uses multiple layers of caching, which depend on locality and coalesced access for maximum performance. Due to the prevalence of convolution, especially in ML, many optimized implementations are available
All optimizations reduce convolution to generalized matrix multiplication (GEMM) 6 GEMM convolution, the simplest optimization, extracts the relevant parts of the array into Toeplitz ma- trix, replicates the filter as needed, and multiplies the two matrices. GEMM convolution exists in precomputed, implicit, and explicit variants (3. By contrast, Winograd short convolution g trans- forms both inputs and filters to achieve a lower number of multiplications. There exist fused and non-fused variants [40]. As the transformations used for Winograd convolution are inherently lossy, and rounding occurs at multiple stages, results may deviate. Convolutions using fast Fourier trans- formation (FFT) exploit the fact that spatial convolution is equivalent to point-wise multiplication in the frequency domain. The transformation to and from the frequency domain is inherently lossy for finite numerical precision [24] and introduces deviations from other methods. In addition, modern GPUs include special TensorCores to compute convolution directly (28).
A more detailed description of convolution approaches can be found in Section of the supplemen- tary material, Performance characteristies of the different approaches are excellently explained in Blahut et al. 6 Specific performance characteristics for NVidia GPUs are discussed in (29).
https::/gittub..cmmt/ httpa///w...aoboeeleeeeeeenenenn

---

the remaining Cumulative BQCs
AMD
Intel
NVidia
26
Core count:
4
16
2 32
Aifar10-AA R18
Deapkenda Sarpllledda-BS0000 R50v2
10
Cifar10- 10-alll1
10
20
30
60
Platform index
Figure 1: Main result: cumulative number of different softmax vectors in the last layer over all platforms. Platforms are sorted by vendor, architecture, and core count. Core count for Intel CPUs.
Conventions We use architecture as shorthand for microarchitecture. Architecture, core count, and memory size together form platform. We call inference deterministic if the same input, the same trained model, and the same platform always produce the same output. We say it is consis- tenf if the same input and model always produce the same output across all platforms. We denote our models as Dataset-Architecee 10 (2]].
e.g, Cifar10-R18 is a ResNet18 [1] trained on Cifar-
To measure the effect of numerical deviations, we use the concept of equivalence classes (EQCs): platforms that produce identical outputs form EQC. The number of EQCs is measure of di- versity. Ideally, all platforms fall into a single EQC, meaning that inference deterministic and consistent (as expected in theory). To quantify the magnitude of deviations, precision. This metric counts the number of identical mantissa bits before the first deviation be- tween more (intermediate) results. For single precision floating-point numbers, it is in the range (0, 23). The metric generalizes to tensors by taking the minimum the remaining precision of all elements.
Related work Much work has been done to improve reproducibility in ML B3 5 10:
These works make important contributions to the organizational aspects of reproducibility, but do not address the problem of numerical deviations. The literature has examined the influence of variance in algorithm and implementation to significant differences in model performance and training time. Zhuang et al. [41] extend the experiments and find differences in training performance across different GPUs and TPUs. This end- to-end approach is beneficial for the community and informs practitioners of the potential impact of variances in algorithm and implementation on the final model. Our work differentiates itself by focusing on inference and by drilling down to the root causes of the observed deviations.
model training. Pham al. [31 find that variance
lead
Deviations in ML inference have been reported in the signal processing community by Schlögl al. (34, however with a focus on forensics. Our earlier work offers existential evidence from a few CPU platforms, but does not investigate causes or mitigation strategies. GPUs also not considered.
The computer arithmetic community well aware the non-associativity floating point compu- tations 26 19,, but aims to increase precision and efficiency rather than enforcing associativity.
3 Influences on model outputs
In total, 64 of our 75 platforms were CPU-based. Depending on the model, the softmax vector of the last layer produced between 3 and 16 different EQCs. This means all tested models failed to produce consistent outputs across the CPU platforms. We did not observe indeterminism on CPUs. The remaining 11 of our 75 platforms supported inference GPUs. The softmax outputs produced between 8 and 10 different EQCs. Every EQC had cardinality one, meaning the GPU could be uniquely identified by the numerical deviations in the softmax vector. 39 out of 99 inference outputs on GPUs were indeterministic. Figure [ shows the cumulative number EQCs over all platforms in this scenario (see Tables SUP-1 and SUP-2 for all hardware, model, and input details).
While the main results above are end-to-end measurements for realistic inference pipelines, it instructive to study individual causes with specifically designed experiment on reduced set of platforms. All following results are supported with isolated experiments. We discuss the influences affecting model outputs on CPUs and GPUs in turn.

---

Table Different CPUs produce deviations based on their SIMD capabilities.
Flag cluster
EQC
0
1
3 4
5
6
7
8
10
11
12
13
14
Intel Sandy Bridge
Intel Ivy Bridge
X
X
X
Intel Haswell
2
X*
X
Intel Broadwell
X
X
X
X
X
Intel Skylake
x
X
Intel Cascade Lake
X
X
X
X
X
X
X
X
Intel Ice Lake
4
X
X
X
AMD Rome
5
X
X5
X
X
AMD Milan
X
contains 256-bit SIMD flags; i contains some 512-bit SIMD flags: $ contains more 512-bit SIMD
flags; & contains sse4a (128-bit SIMD) and misaliggsss flags; contains the avx512vnni flag
3.1 Influences when inferring on CPUs
Model outputs computed on CPUs may deviate because of differences data and task parallelism. Both affect the aggregation order of convolutions.
Data parallelism Modern architectures feature variety of SIMD instructions, which affect both the floating-poit accuracy and the aggregation order. While CPUs traditionally compute floating- point operations at very high precision in the FPU, the introduction of SIMD instructions, such as SSE and AVX on x86, enables data parallelism at the cost of reduced precision. Newer CPUs have larger SIMD registers, which allow more flexibility when adjusting the SIMD width (i.e., data parallelism) and desired precision. Some CPUs support fused multiply-and-ade in SIMD [I6]
To measure the effect of data parallelism CPU, perform inference with the Cifar10-R18 model on all dual-core x86 systems available on GCP, and collect all CPUID flags indicating hard- support of SIMD instructions. Since the number of flags (169) exceeds the number of platfoms, we cluster flags that always co-occur. Table [ shows the relation between EQCs and flag clusters.
Observe that subset of the flag clusters perfectly aligns with the EQCs. Flag clusters 0-3 are all related to the support for different SIMD capabilities. Flag cluster 0 contains the avx2 flag, denoting 256-bit SIM) support. Flag cluster 1 contains the avx5 12f flag, indicating SIMD width of 512 bit. Flag cluster contains four SIMD-related flags: avx512vbmt (vector bit manipulation instruc- tions), avx512ifma (fused multiply-add for integers), vpclmulqdq (carry-less multiplication), and avx512vpopentdq (population count). While the co-occurrence prevents us from attributing the deviation single flag, we consider this evidence to show that some SIMD-related feature is sponsible for the EQC split. Flag cluster contains the sse4a flag not present the other CPUs, as well as the misalignsse flag to indicate support for misaligned memory access when using legacy SIMD instructions. For this EQC, however, the effect of different SIMD support may be superseded by general architectural differences of AMD processors. Interestingly, on the support for avx512vnm vector neural-network instructions, indicating that the framework does not use this hardware feature yet.
EQC emerges based
Task parallelism hivid-andd-ooquee algorithms can increase performance and reduce memory overhead by distributing work to multiple cores [38). The use of task parallelism involves additional aggregation steps, the order of which can vary depending on the implementation, e. g., atomic versus sum reduce. As work gets distributed on more cores, the individual workload decreases. When the workload per core becomes too small, data parallelism cannot be used anymore. Figure 2 visual- izes this effect. When increasing core count from 2 to 4, the additional 512-bit SIMD capabilities of the Intel Ice Lake platform become unusable, producing the same outputs as Intel Sky-ICascade Lake. Going from 8 to 16 cores makes the per-core workload too small for 256-bit SIMD instruc- tions, resulting in single EQC for all Intel platforms newer than Ivy/Sandy Bridge. We did not observe additional EQC collapses for core counts larger than 16. Both the Intel Sandy/Ivy Bridge

---

Number of EQCs
AMD Milan
AMD Rome
Intel Ivy Bridge
EQC
Intel Sandy Bridge
Intel Broadwell
2
Intel Haswell
Intel Cascade Lake
Intel Skylake
Intel Ice Lake
Number of cores
Figure 2: Deviations disappear as the number of CPU cores increases.
and the AMD Rome/Milan architectures have sufficiently different SIMD capabilities (cf. Table to produce deviations even for larger core counts.
3.2 Influences when inferring on GPUs
The choice of convolution algorithm is cause of deviations GPUs. Due to different performance characteristics, there is no universally best convolution algorithm ( Modern GPUs implement a large number of convolution algorithms, and select the fastest algorithm for the convolution param- eters (number of filters, stride, etc.) at runtime based on microbenchmarks Therefore, the final algorithm choice may not only vary with the GPU and model architecture, but also with the remain- ing hardware and even uncontrollable conditions, like bus contention and parallel load. As each convolution is benchmarked separately, different layers of model can use different algorithms.
Deviations between GPUs We perform inference with the Cifar10-BB model on all GPUs available GCP and locally. Using the TensorFlow profiler, log all function calls and extract the convolution algorithms. Internally, the TensorFlow profiler uses the NVidia profiler to record calls the CUDA API and some CUDA-internal function calls. A list of all observed convolution algorithms can be found in Table SUP-4 in the supplementary material.
For the GPUs with deterministic outputs, we can explain the deviations by the use of different ap- proximate convolution algorithms. Figure 3a shows exemplary traces from the GTX 1650 and RTX 2070 GPUs
local machines. While 16 different convolution functions were called in our experiments, we group them by their approach for case of reading. The lines in Figure 3a plot the selected algorithm for each convolutional layer of the model. In this case, both GPUs choose the same algorithm for each of the first 15 convolutions. Convolution 16 is computed with different al- gorithms, but they happen to produce identical results. This is because the only difference between explicit and implicit GEMM the fact that explicit stores the Toeplitz matrix in memory, whereas implicit GEMM computes it on the fly. The deviations where the RTX 2070 continues to use implicit GEMM while the GTX 1650 switches back to Wino- grad. The separation of EQCs is indicated by the dashed vertical line, which we verify by comparing the intermediate results. We confirm from all other traces that this pattern is typical for deterministic deviations between GPUs.
caused by the final three convolutions,
Deviations between inferences on the same GPU Microbenchmarks are run just before the first inference. The framework uses random data fill buffer of the problem dimensions and mea- sures the execution time of all supported convolution algorithms. Since these measurements take up valuable execution time, each candidate algorithm is timed only once. This makes the choice of the "winner" susceptible to runtime conditions, g., bus contention and memory latency.
To measure variations between inferences, we perform repeated inference with the Cifar10-PHi -18 model. Each inference happens in a new session. We plot the traces obtained with the same profiler instrumentation. Figure (3b] shows the behavior of an NVidia P100 GPU. For 33 inferences on the same model and data, we observe different paths resulting in 4 EQCs. Observe that up to three
This approach is also called "auto-tuning" in the literature, e.g..
14 31
5

---

GTX 1650
EQC B
loop
RTX 2070
precomputed GEMM
implicit GEMM
explicit GEMM
Winograd fused
Winograd FFT
20
Convolution layers
(a) Different GPUs choose different algorithms in certain layers. All GEMM variants produce identical outputs.
1 EQC
EQCCs
BOCs
direct loop
precomputed GEMM
implicii GEMM
explicit GEMM
Winegrad fused
Winograd non fused
FFT
10
15
20
Convolution layers
(h) The same GPU chooses different algorithms depending on variance in the microhenchmarks.
Figure 3: Choice of convolution algorithm per layer. Function calls are grouped by approach.
different approaches are used to compute the same convolution (layers 2-5). The leftmost dashed line provides evidence that the intermediate results differ that layer and propagate further. Another separation happens at layer 6, whereas the choice of algorithm at layer 17 does not fork out another EQC. We confirm from all other traces that this pattern typical for indeterministic deviations between GPUs.
The host system of the GPU can also influence the race and thus the number of unique algorithm quences. The red trace Figure 3a one of two unique algorithm sequences for an RTX 2070, which both lead to deterministic output. For 33 inferences deterministically observe two EQCs, one for each GPU. We find that the same GPU can produce different algorithm sequences, resulting different number of EQCs on other host systems.
the same model and data,
3.3 Model-specific influences
The architecture, parameters, and input dimensions of an MI. model can influence the number and nature of deviations by amplifying or suppressing influences.
Number of multipications/parameeer
Numerical deviations caused by arithmetic opera- tions, therefore more operations mean potentially more deviations. Both larger inputs and higher number of parameters increase the number of computations during inference. Moreover, the prob- lem dimensions impact the performance of convolution algorithms and affect the algorithm choice.
As deviations arise from rounding errors and different aggregation orders, we hypothesize that more multiplications per convolution layer lead to more deviations. We investigate this with an ablation study for the parameters of standard 2-dimensional convolution: kernel size k. number of filters f. and input dimensions xj X c. Our experiment covers several orders of magnitude of multiplications, ranging from 10 to 10S which is more than the number of multiplications in Cifar 1 model. For each order of magnitude, we fix k, f and c, and adjust and j to reach the desired number of multiplications while aiming for square inputs, i. e., i but other values give the same results. Weights are drawn from a uniform distribution over [0, 1)
the number
Figure 4 shows
3,
Our remaining precision metric (cf. Section 2 measures the magnitude of the deviations. Figure 4a shows that the remaining precision decreases with the number of multiplications and does not vary
This does not affect results, discussed the "Parameter and input distribution" paragraph of Section

---

Remaining precision
k:
20
Number of EQCS
15
10
10a
10
10a
10
10
10
10a
10
10
10
107
10
Multiplications
Multiplications
(a) Remaining precision over # of multiplications.
(b) Number of EQCs for the same platforms.
Figure 4: Effect of different convolution dimensions (mapped to the number of multiplications).
20
Remaining precision
bat-ch_norn_1
Equivalence classes
10
5
0
Segment
Segment 2
Segment 3
Segment 4
Classifier
ResNet layer
Figure 5: Causes and effects by layer: number of EQCs and remaining precision for Cifar10- 1--810
significantly between the platforms (excluding the smallest case == 2,
where deviations emerge). This is plausible, as approximately Gaussian deviations lead to lower minimum remain- ing precision if more independent realizations are aggregated. The remaining precision drops to O, which means that at least one deviation is at least as large as factor 2.
function of the number of multiplications. Observe that the relation is not monotonic. Both f = 1 and f = produce more EQCs for 10 to 10" multiplications than for 10 The number of filters seems to have larger influence than the filter size. Note that the maximum number of EQCs in this experiment is four, whereas we observe up to five EQCs for the same set of platforms in main experiment with complete models. We interpret this as evidence that deviations propagate between layers and may produce forks into different EQCs later layers.
Figure 4b shows the number of equivalence classes
Architecture and layer types The model architecture defines the operations and hence determines if cause of deviation is present not. In particular, convolutional and other parallel data processing layers tend to introduce deviations. Layers that aggregate results and reduce information, such as pooling layers, may reduce eliminate deviations from preceding layers. This extends to activation functions. Continuous functions that preserve information (e. g.. sigmoid and softmax) can maintain potentially amplify deviations. Functions that reduce information, like the rectified linear unit (ReLU), can have a diminishing effect on deviations. This means that a sufficiently large model can introduce deviations in earlier layers, and remove them in pooling and activation layers. Skip layers can preserve deviations by bypassing information reduction.
To investigate the influence of different layer types in full inference computation, instrument model to output the intermediate results of all layers in addition to the final class label. We verify that this does not alter the EQCs. Figure 5S plots the number of EQCs as well as the remaining precision over the outputs of the 60 layers of Cifar10-R18. Note that activations count as distinct layers, marked with thick dark-red tick, and not every convolution is immediately followed by an activation. The annotated segments on the x-axis refer to the ResNet convention bundling convolutions of the same size into segments [15].
The remaining precision changes after every layer. This can be explained by the fact that the very first convolution following the input layer already produces deviations that result in multiple EQCs.
7

---

As long as more than one EQC exists (i. e., for all remaining layers), the remaining precision indi- cates how far these EQCs fall apart in the most extreme case. In terms of magnitude. the remaining precision never reaches zero. This corroborates the results of our ablation study (cf. Figure 44 as the maximum number of multiplications per layer is in the order of 10 Although the number of to- tal multiplications increases monotonically, the ReLU activations increase the remaining precision, visible as peaks of the remaining precision. One instance of aggregation very pronounced at the rightmost layer, where the output is projected to small label space.
Turning to the number of EQCs, new ones only emerge as outputs of convolutional and batch nor- malization layers. The number of EQCs tends to increase over the execution of the model. However, we also observe reduction in the number EQCs, which occurs once in convolutional layer (conv2d_ 14) in the third segment of the model. We did not expect to sees this, in particular given that this layer does not perform any quantization or aggregation. This shows that past deviations may be cancelled out. Another unexpected finding is the emergence of the fourth EQC, which occurs in a batch normalization layer (batch, fircch_oom_____ 1), and deviations arise in layers other than convolution.
3.4 Influence of floating point precision
Figure 6hroesssswwwss how casting trained model different mootiing-point precision affects deviations. The figure shows similarities between the EQCs as dendrograms, with remaining precision used as distance metric. Branch lengths within the dendrograms are proportional to the remaining preci- sion between the EQCs, and connections further to the right indicate higher remaining precision. Branches extend past the maximum remaining precision of the format to indicate platforms that produce identical outputs and thus are in the same EQC.
AMD Rome 48
AMD Rome 48
AMD Rome 48
AMD Rome 16
AMD Rome 16
AMD Rome 16
Intel Ivy Bridge
Intel Skylake
Intel Broadwell 4
Intel Sandy Bridge
Intel Cascade Lake
AMD Rome 4
AMD Rome
AMD Rome 2
AMD Rome
AMD Milan
AMD Milan
AMD Milan
Intel Broadwell 2
Intel Haswell
Intel Broadwell 2
Intel Haswell
Intel Broadwell 2
Intel Cascade Lake
Broadwell
Broadwell
Haswell
AMD Rome 4
AMD Rome 8
Intel Ice Lake
Intel Cascade Lake
Intel Icc Lake
Intel Ivy Bridge
Intel Ice Lake
Intel Sandy Bridge
Intel Sandy Bridge
Skylake
Ivy Bridge
Intel Skylake
15
20
40
43
30
Remaining precision
Remaining precisioo
Remaining precision
(a) Half precision: 3 EQCs.
(b) Single precision: 9 EQCs.
(c) Double precision: 4 EQCs.
Figure 6: Influence of casting the Cifar10-R18 model different floating- point precisions. Both half (float16) and double precision (f1oat64) floating-point numbers generate fewer deviations than single precision. The same pattern holds for all models, ef. Section E in the supplementary material.
Rounding to IEEE-754 16-bit half-precision (float16) suppresses deviations and reduces the num- ber of EQCs. However, which EQCs fall together seems rather chaotic our experiments. For example, AMD Rome processors fall into all three EQCs, depending their core count. Reducing precision thus mitigates deviations, but in potentially unpredictable ways. Alternatively, the model can be converted to fixed point integer arithmetic, which reportedly eliminates all deviations (34).
Interestingly, increasing precision also reduces the number of EQCs. While we observe total of four different EQCs when inferring with our model in 64-bit double precision, this is still fewer than for 3i-bit single precision. As we simply cast the model the higher precision without modifying the weights, the lower part of the 64-bit mantissas are zero. This provides more room for shifting and prevents swamping to some extent. Platforms from different EQCs for 32 bit match for 64 bit according to their core count, indicating that deviations due to task parallelism persist. We conjecture that zeroing the last mantissa bits has the same effect, reducing the performance cost of the mitigation measure at the cost of reduced fidelity.
8

---

3.5 Potential influences not observed in experiments
Our research identified number of potential influences that did not surface in isolated experiments.
Compute graph optimization Recent approaches go beyond optimized algorithms and instead transform the operations and their ordering (the compute graph) [23]. For example, the Accelerated Linear Algebra (XLA) project is JIT compiler for ML operations [39]. The resulting compute graph is specific to the hardware [4]. determines and how deviations caused in the hardware come to effect. XLA is not yet used by default [39]. As we did not activate it, we can rule out special
compute- graph optimizations as cause for deviations.
Device placement Most ML frameworks allow running operations on both CPU and GPU. In theory this lets them choose the target device at runtime. Under certain conditions, the framework may decide to execute operations on the CPU even though GPU is available [77.. Since CPU and GPU implementations very likely differ in algorithm, floating-pole -aiin precision, and aggregation order, the device placement can cause deviations. We log the device placement in our experiments and verify that EQCs do not change with our instrumentation.
execution can lead to different aggregation orders depending on the implementation. If all intermediate results are stored in an arry and then summed up (e. g., with sum- sum-reduce), the execution time of individual tasks will not change the aggregation order. However, the results are summed up immediately, with some sort of locking mechanism, the execution time of individual tasks determines the aggregation order. In our experiments, all model outputs CPUs and GPUs are consistent and deterministic when controlling for the influences we discovered, ruling out race conditions as causes of deviations.
Race conditions
Parameter and input distributions The distributions of model parameters and input values can both affect the shape and magnitude of deviations. To measure their influence, we first extract the conv2d_ 11 layer from our Cifar10-R18 model and capture its intermediate input during a forward pass. Then, we perform inference on twelve combinations of input and parameter distributions on all dual-core GCP instances. We use the original weights, weights drawn from the Glorot uniform distribution [12], and weighs drawn from Gaussian with μ -0.019, trained weights. For the inputs we use the captured values, two random permutations of these values (preserving the marginal distribution), and random inputs drawn uniformly from 1). for all cases are identical. This suggest that the emergence of FQCs is independent of the parameter
0.226, fitted to the
1). The EQCs
and input distribution. Incidentally, the microbenchmarks for algorithm choice GPUs make similar assumption. Their inputs are filled with random data.
4 Discussion
Impact The deviations we observe might impair ML reproducibility and security. Concerning reproducibility, as different platforms can produce different outputs from the exact same model and input data, it unlikely that high-precision numerical results can be reproduced exactly. Assertions in automated software tests are prone to fail on specific hardware. Such limitations to reproducibility are not always visible in headline indicators because classifier performance is typically measured on the level of labels. The deviations we study rarely cause label flips for natural inputs. However, we have demonstrated (35 that label flips can be provoked by searching for synthetic inputs that map to the space between decision boundaries of different platforms. Moreover, Casacuberta et al. [8] point out that numerical deviations can undermine security guarantces of ML implementations with differential privacy. Differences between symbolic math and actual floating-por -ponnt computations at limited precision have been used to fool ML verifiers [7]. Varying algorithm choices break these assumptions even further, they fundamentally change the type and order of arithmetic operations. These examples highlight the security impact of our observation. Our study of causes breaks ground for principled approach to assess and mitigate these risks.
Federated learning distributes ML training tasks over many machines under the assumption of com- patible gradients. The systematic deviations observed here call for robustness checks when com- bining gradients from heterogeneous hardware 77] 25). The fact that deviations leave traces of the executing hardware enables forensicability and attribution of ML outputs [34]. This could be used,

---

g., in the combat against harmful generated content: but can also be misused, e. g., for unautho- rized surveillance. This adds a new aspect the debate on societal and ethical aspects of ML.
Mitigation strategies As shown in Figure 6 quantization can suppress deviations and reduce the number of EQCs. However, these measures do not mitigate deviations due to algorithm choice, and thus are not applicable to GPUs. While researchers can instruct the MI. framework to use deter- ministic implementations of operators the fact that different GPUs support different algorithms can lead deviations even when such options are set. Similarly, TensorCores (28) may be used available, which can also lead to deviations. While they can be disabled, their use cannot be en- forced. Reaching full reproducibility requires giving researchers access to low-level functionalities and providing them with fine-graiin fine-grained control over optimizations. Another approach is to sidestep platform-speccfic optimizations at runtime by transforming the model into a flat compute graph. TFLite, for example, does this by default; yet for the purpose of smaller memory footprint rather than consistency.
While all these strategies are heuristic, developers of future MI. frameworks and accelerator libraries should consider supporting an option for fully deterministic and consistent computation. This comes almost unavoidably at the cost of lower performance. Cryptographic libraries, for example, went similar path when constant-time options were added in response to the discovery of side-channel attacks (20). In both domains, successful mitigation depends on exact knowledge of the entire stack down to the hardware.
Limitations While our work covers a lot of ground concerning numerical deviations, there are still some areas for further exploration. Our experiments have intentionally kept some factors con- stant: the versions of TensorFlow (2.5.0,, all dependent libraries and drivers, the compiler version and options used to build TensorFlow, as well as the concurrent load the system besides our experiments. Additional layer types and other frameworks can also be subject to follow-up work.
5 Conclusion
Performing inference the same trained model and input data is not always consistent across plat- forms, and sometimes not even deterministic on the same platform. This paper explores the causes for the numerical deviations, in particular in convolution operations. They include the choice of the specific convolution algorithm, the floating-pon point precision, and the order of aggregation. The order of aggregation is defined by SIMD capabilities and the number of cores of the executing CPU. As the number of CPU cores grows, task parallelism can supersede the deviations caused by SIMD capabilities. GPUs use microbenchmarks select the fastet supported convolution algorithm at run- time. We find that this can produce seemingly indeterministic behavior when different algorithms chosen in different sessions. We validate our findings on 75 platforms, including CPUs and GPUs, hosted locally and on two large commercial cloud platforms. Our measurement infrastruc- ture, which is made available on GitHub can facilitate future comparative studies. Our analysis offers number of mitigation strategies, but certain deviations appear unavoidable at present. These findings imply that authors interested improving reproducibility should meticulously document specifics of the computation hardware, in addition to the amount of compute [27], and provide high- precision intermediate results as reference points for replication attempts.
Acknowledgments and Disclosure of Funding
We thank Matthias Kramm for insightful comments earlier versions of this paper. We also thank Michael Fröwis, Jakob Hollenstein, Martin Beneš, and Patrik Keller for fruitful discussions during this research. Parts of this work were funded by the European Union's Horizon 2020 research and innovation programme under grant agreement No. 101021687, and a PhD Fellowship from NEC Laboratories Europe (2021-2022).
$ 'For example. by setting the enable beaalle_o_deeee deeeermminnnnn flag. TensorFlow chooses the first available convo- lution algorithm rather than the fastest one.
httpp:: : thub. www/wiokmmoooeeeeeee.........e.e iNeference
10

---

References
[1] AMD. 128 bit and 256 bit VPX instructions. pdf 2009.
amd. com//Techhoocs/4444999
[2] Andrew Anderson and David Gregg- Optimal DNN primitive selection with partitioned boolean quadratic programming. In International Symposium Code Generation and Op- timization (GCO), pages 340-351. ACM, 2018.
[3] Andrew Anderson, Aravind Vasudevan, Cormac Keane, and David Gregg. High-performance low-memory lowering: GEMM-based algorithms for DNN convolution. In Interantional Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), pages 99-106. IEEE, 2020.
[4] Sebastian Pineda Arango, Hadi Samer Jomaa, Martin Wistuba, and Josif Grabocka. HPO- B: A large-scale reproducible benchmark for black-box HPO based on OpenML. In Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2021.
[5] Daniel Arp, Erwin Quiring, Feargus Pendlebury, Alexander Warnecke, Fabio Pierazzi, Chris- tian Wressnegger, Lorenzo Cavallaro, and Konrad Rieck. Dos and don't of machine learning in computer security. In USENIX Security Symposium (USENIX Security), pages 3971-3988. USENIX Association, 2022.
[6] Richard E. Blahut. Fast Algorithms for Signal Processing. Cambridge University Press, 2010.
[7] Keith Bonawitz al. Towards federated learning scale: System design. In Machine Learning and Systems (MLSys). mlsys.org, 2019.
[8] Sílvia Casacuberta, Michael Shoemate, Salil P. Vadhan, and Connor Wagaman. Widespread underestimation of sensitivity in differentially private libraries and how to fix it. Conference on Computer and Communications Security (CCS), pages 471-484 ACM, 2022.
[9] Don Coppersmith and Shmuel Winograd. Matrix multiplication via arithmetic progressions. Journal of Symbolic Compulation, 93):251-280, 1990.
[10] Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, Rene Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. HPOBeneh: A collection of repro- ducible multi- fultityidelity benchmark problems for hpo. Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2021.
[11] Congyu Fang, Hengrui Jia, Anvith Thudi, Mohammad Yaghini, Christopher Choquette-Choo, Natalie Dullerud, Varun Chandrasekaran, and Nicolas Papernot. Proof-of-learning is currently more broken than you think. In European Symposium Security and Privacy (EuroS&P) pages 797-816 16. IEEE, 2023.
[12] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics (AIS- TATS), pages 249-256. JMLR, 2010.
[13] David Goldberg. What every computer scientist should know about floating-poin arithmetic. ACM Computing Surveys (CSUR), 23(1):5-48, 1991.
[14] Scott Grauer-Gray, Lifan Xu, Robert Searles, Sudhee Ayalasomayajula, and John Cavazos. Auto-tuning a high-level language targeted to GPU codes. In Innovative Parallel Computing (InPar), 2012.
[15] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference Computer Vision and Pattern Recognition (CVPR), pages 770- 778. IEFF, 2016.
[16] Intel. Intel@ C++ content/ haataa//aaa/eeeeooe//eeeee/////r comp:/lllppppccpppppppcpppcpclee clapsccccasccccc 2021.
developer guide and http:eenccchttt:://////:: http:///wwww.ll.l...mmom.. intel.nell....... cante.comm

---

[17] Kai Jia and Martin Rinard. Exploiting verified neural networks via floating point numerical error. In International Symposium on Static Analysis (SAS), pages 191-205. Springer. 2021.
[18] Sayash Kapoor and Arvind Narayanan. Leakage and the reproducibility crisis in ML-based science. arXiv, 2022. arxiv Computing Research Repository (CoRR), abs/2207.07048
[19] Nachiket Kapre and Andre DeHon. Optimistic parallelization of floating-point accumulation.
In Symposium on Computer Arithmetic (ARITH), pages 205-216. IEEE, 2007.
[20] Paul C. Kocher. Timing attacks implementations of Liffie-Hellmann RSA, DSS, and other systems. In Advances in Cryptology (CRYPTO), pages 104-113. Springer, 1996.
[21] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 dataset. Master's thesis, University of Toronto, 2014.
[22] Yuhang Li, Mingzhu Shen, Jian Ma. Yan Ren, Mingxin Zhao, Qi Zhang. Ruihao Gong, Feng- wei Yu, and Junjie Yan. Mqbench: Towards reproducible and deployable model quantization benchmark. In Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2021.
[23] Yizhi Liu, Yao Wang. Ruofei Yu, Mu Li, Vin Sharma, and Yida Wang. Optimizing CNN model inference on CPUs. In USENIX Annual Technical Conference (USENIX ATC), pages 1025-1040. USENIX Association, 2019.
[24] Raimund Meyer. Error analysis and comparison of FFT implementation structures. In Inter- national Conference Acoustics, Speech, and Signal Processing (ICASSP), pages 888-891. IEEE, 1989.
[25] Fan Mo, Hamed Haddadi, Kleomenis Katevas, Eduard Marin, Diego Perino, and Nicolas Kourtellis. PPFL: Privacy-preserving federated learning with trusted execution environments. In International Conference on Mobile Systems, Applications, and Services (MobiSys), pages 94-108. ACM, 2021.
[26] Jean-Michel Muller, a- (x ..) or (z) xx? In Symposium on Computer Arithmetic (ARITH), pages 17-24. IEEE, 2021.
[27] NeurIPS. NeurlPS 2021 paper checklist. httes://aarrreeccc///otrrrrerrss
Rager//nnfommtttoo/Ppeerrcclllll 2021.
[28]
NVidia Corporation. NVidia TensorCores.
data- cetta-centeer/tnnnor-crreee/ 2022.
[29]
NVidia Corporation. ing.
Optimizing convolutional
layers in deep learn-
betps://dooe nvidia. http///eepeeaannnararrooooo Opg.anziiinccmoouuuttttoaa/tteee
2023.
[30]
PapersWithCode. Papers with code. httpps:/ppaperssiiticoooe. com/ 2018.
[31] Hung Viet Pham, Shangshu Qian, Jiannan Wang, Thibaud Lutellier, Jonathan Rosenthal, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan. Problems and opportunities in training deep learning software systems: An analysis of variance. In IEEE/ACM International Conference on Automated Software Engineering, pages 771-783. IEEE, 2020.
[32] Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelz- imer, Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research (a report from the NeurlPS 2019 reproducibility program). Journal
of Machine Learning Research, 22(1):7459-744 -478, 2021.
[33] Edward Raff. A step toward quantifying independently reproducible machine learning search. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
[34] Alexander Schlögl, Tobias Kupek, and Rainer Böhme. Forensicability of deep neural network inference pipelines. In International Conference on Acousties, Speech, and Signal Processing (ICASSP), pages 2515-2519, IEFF, 2021.
12

---

[35] Alexander Schlögl, Tobias Kupek, and Rainer Böhme. iNNformant: Boundary samples as tell- tale watermarks. In Workshop on Information Hiding and Multimedia Security (IH&MMSec), pages 81-86. ACM, 2021.
[36] Surat Teerapitaaanon and Bradley McDanel. Branchynet: Fast inference via early exiting from deep neural networks. In International Conference on Pattern Recognition (ICPR), pages 1464-2469. IEEE, 2016.
[37] TensorFlow. TensorFlow GPU guide. https://ww..eessorr.ooo org/gogggiio. httg/guid//gp) 2021.
[38] Aravind Vasudevan, Andrew Anderson, and David Gregg. Parallel multi channel convolu- tion using general matrix multiplication. In International Conference on Application-specific Systems, Architectures and Processors (ASAP), pages 19-24. IEEE, 2017.
[39] XLA Developer Team. XLA: TensorFow,, compiled, https://wwwt www.trennn...oooo oow.ll....oo/../. org/xla# xla# enable, xla_ for tensorflov soo.lllwsooollsssolloeol 2022. Accessed on 2022-07-04.
[40] Da Yan, Wei Wang, and Xiaowen Chu. Optimizing batched winograd convolution gpus. In SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 32 44. ACM, 2020.
[41] Donglin Zhuang, Xingyao Zhang, Shuaiwen Song, and Sara Hooker. Randomness in neural network training: Characterizing the impact of tooling. Proceedings of Machine Learning and Systems, 4:316-336, 2022.
[42] Dániel Zombori, Balázs Bánhelyi, Tibor Csendes, István Megyeri, and Márk Jelasity. Fooling a complete neural network verifier. In International Conference on Learning Representations (ICLR). OpenReview.net, 2021.
13


Reviews:
[{"review_id": "pH9F4BDka1", "reviewer": "Reviewer_djPp", "summary": "The paper presents an extensive empirical study on the numerical instabilities for convolutions across different platforms. The findings of why these instabilities occur are interesting and informative. They further cluster them into equivalence classes for ease of explanation. ", "questions": "I find the paper very interesting to read. It is not a research solutions paper, but an extensive study on numerical stabilities. Even though, the research contribution on novel techniques do not exist, I find the community would benefit from these findings the paper reports through extensive experimentation. \n\nPlease answer the following questions in the rebuttal.\n\n* Why is Toeplitz based convolution lossy? \n* How significant are the loss of precision? Does it affect results in inference? Can this be trained away by fine tuning?\n* How do you use the EQCs to better mitigate any inaccuracies in inference?\n* Algorithmic choice in GPUs are anyway supposed to get different results, since they are different approximations of the convolution. I find this expected and obvious.\n", "limitations": "Authors mention the limitations explicitly under discussion.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Extensive evaluation on a large number of platforms across both CPUs and GPUs.", "weaknesses": "The results and findings are very interesting, but the utility values of the EQCs and the errors they find is not demonstrated. Essentially, how do you use this information to make inference more robust for example?\n\nNo results to corroborate line 285. Will these imperfections lead to different model outputs? Some of these errors are tolerable specially in classification setting. Authors themselves mention this is rare. Some evidence or experiment on flips would be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for carefully reviewing our paper.\n\n> \"The results and findings are very interesting, but the utility values of the EQCs and the errors they find is not demonstrated. Essentially, how do you use this information to make inference more robust for example?\"\n\nThe concrete utility of this work lies in identifying the root causes of deviations as a necessary first step towards reproducibility.  Our findings give researchers and ML framework developers concrete starting points for improving robustness. Thank you for asking this question and bringing this lack of framing to our attention. We will clarify our contributions in the introduction.\n\n> \"No results to corroborate line 285. Will these imperfections lead to different model outputs? Some of these errors are tolerable specially in classification setting. Authors themselves mention this is rare. Some evidence or experiment on flips would be useful.\"\n\nWe agree that our formulation in line 285 is too general, and will revise it to state that bit-level reproducibility of inference results is a necessary requirement for reproducibility. While our study focuses on inference for comprehensibility, the causes we identify may also explain phenomena observed (but not explained) during training, as shown for instance by [Zhuang et al. \"Randomness in neural network training: Characterizing the impact of tooling\" (2022) and Pham et al. \"Problems and opportunities in training deep learning software systems: An analysis of variance.\" (2020)].\nExperiments on forcing label flips have been described in related work which we cite in line 288.\n\n> \"Why is Toeplitz based convolution lossy?\"\n\nOur presentation of convolution approaches was imprecise. In fact, convolution with Toeplitz matrices is not lossy. The \"lossy\" transformations in lines 60 and 63 refer to Winograd and FFT-based convolution, respectively. Thank you for pointing this out, we will revise accordingly.\n\n > \"How significant are the loss of precision? Does it affect results in inference? Can this be trained away by fine tuning?\"\n\nAs stated in line 287, reductions in common model performance characteristics are unlikely.\nWhile incorporating deviations into the loss function to reduce deviations by fine-tuning is conceivable in principle, we deem it impractical with current frameworks. The deviations happen at such a low level that the ML framework does not capture them without the kind of instrumentation we built for the purpose of this research. We thank you for this question and will keep it in mind for future work on this topic.\n\n> \"How do you use the EQCs to better mitigate any inaccuracies in inference?\"\n\nWe introduce EQCs - a metric linked to the effect numerical deviations - as a tool to understand their causes. Engineers striving for deterministic and consistent results should aim for a single EQC, by restricting the inference platforms to those within the same EQC or by applying mitigation measures that reduce the number of EQCs. Researchers can use EQCs to test new mitigation measures. Thank you for this question, we will elaborate more on the practical use of EQCs in the paper.\n\n> \"Algorithmic choice in GPUs are anyway supposed to get different results, since they are different approximations of the convolution. I find this expected and obvious.\"\n\nIt is known that algorithm choice can vary on GPUs, and that different algorithms yield different results. However, to the best of our knowledge, we are the first to report the variations in chosen algorithms, given the same models, inputs, and hardware.\nAs mentioned in line 300, even with \"enforced\" determinism (through setting determinism-flags in TensorFlow and PyTorch), different algorithms can be chosen based on the algorithms supported by the hardware.\n\nThank you again for your thorough review and your valuable remarks.\nIf you have any further suggestions or questions, please do not hesitate to let us know.\n"}, {"review_id": "A7goQdS1wc", "reviewer": "Reviewer_u1sW", "summary": "The paper explores the causes and effects of the same model and data for inference on different platforms that result in different variations of numerical values. Furthermore, many different hardware specifications within the cloud which are mostly CPU platforms are chosen to evaluate different factors that cause this discrepancy. For the CPU environments, the emphasis is on parallelism and certain Instructions while for the GPU, the focus is on the convolution algorithm that is selected by the GPU at a given time with respect to multiple factors. Finally, this paper offers some mitigation strategies to solve these issues.", "questions": "C1. It would be better to list the contribution of the work in the introduction. Also since this paper is an analysis of cause and effect, it would be better to evaluate some of the mitigation strategies to show effectiveness.\n\nC2. Although the title suggests a general analysis of neural networks, most of the analysis is about layers of CNN or ResNet and ways of calculating convolution. Therefore it would be better to either analyze other neural networks as well or indicate this in the introductory sections of the paper.\n\nC3. It would be better to have a list of CPU flags and corresponding clusters as an appendix. Also, More description on how often certain SIMID flags within a cluster are activated would help the readers to understand the importance of SIMID instructions for differences in inference results.\n\nC4. Figure 6 might need more explanation regarding the importance of floating point precision. Also Maybe other Neural Networks could be tested to see if they follow the same pattern for single-point precision.\n\nC5. More description about how different the convolution algorithms do the calculations might extract new insights into how they affect EQCs. Also, some results might confuse the reader, for example, how different explicit and implicit GEMMs are and the reasoning behind a GPU choosing one over the other.", "limitations": "See Questions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "++ Good effort to separate each factor of deviation for evaluation\n\n++ High diversity of CPUs used for the experiments\n\n++ Most of the graphs are informative and hold useful pieces of information\n\n++ EQC is a good metric to measure diversity", "weaknesses": "– The abstract and introduction section could be more organized and well written\n\n– Minor Writing issues(transitions, cohesion, word usage)\n\n– More description of related work could be utilized to differentiate in detail between the paper and other works in the literature\n\n– Real environments might take multiple factors all at once to generate different inference results. This real-world scenario could be explored in the experiments instead of evaluating each factor separately", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your remarks regarding the organization of sections, writing issues, and description of related work. We will update the revised version of the paper to ensure the contributions of our research are well-articulated.\n\n> \"Real environments might take multiple factors all at once to generate different inference results. This real-world scenario could be explored in the experiments instead of evaluating each factor separately.\"\n\nIn addition to our study of isolated influences, our main results in Figure 1 of the paper, as well as Tables 1 and 2 in the supplemental material, in fact present the results of a large-scale validation study using real models in real CPU and GPU environments. We will revise line 92 to explicitly state that these measurements capture the interplay of all influences in real-world scenarios.\n\n> C1. It would be better to list the contribution of the work in the introduction. Also since this paper is an analysis of cause and effect, it would be better to evaluate some of the mitigation strategies to show effectiveness.\n\nWe will include a clear and concise description of the main findings in the introduction. We will also highlight that our results of Section 3.4 do evaluate a mitigation strategy (varying the floating point precision) and analyze its effectiveness.\n\n> C2. Although the title suggests a general analysis of neural networks, most of the analysis is about layers of CNN or ResNet and ways of calculating convolution. Therefore it would be better to either analyze other neural networks as well or indicate this in the introductory sections of the paper.\n\nThis assessment is fair. We will clarify the scope of our work in the introduction. By publishing the measurement and orchestration code we make it easy for researchers to extend our research to other layer types and models. As an indication that our findings generalize, our experiments for Figure 5 show that batch normalization can introduce deviations.\n\n> C3. It would be better to have a list of CPU flags and corresponding clusters as an appendix.\n\nWe have generated a full table including all CPU flags and added it to the supplemental material. (Unfortunately, we cannot provide it here as it would exceed the page limit for the rebuttal document.)\n\n> C3. (cont.) Also, More description on how often certain SIMID flags within a cluster are activated would help the readers to understand the importance of SIMID instructions for differences in inference results.\n\nWe fully agree with your comment and have in fact made efforts to measure the usage of SIMD instructions in TensorFlow. However, despite numerous attempts at dynamic instrumentation (gdb, rr, valgrind/cachegrind, and TensorFlow's own profiler) and code analysis (of TensorFlow and its underlying Eigen library), we were unable to obtain reliable results due to the complexity of TensorFlow's monolithic architecture. While certainly desirable, we believe that the value of our work does not hinge on this description.\n\n> C4. Figure 6 might need more explanation regarding the importance of floating point precision. Also Maybe other Neural Networks could be tested to see if they follow the same pattern for single-point precision.\n\nWe recognize the need for more explanation of Figure 6 and will provide a more detailed description in the final version of the paper. To address your comment, we extended the experiment to all three models used in the paper. We find that the pattern we report holds across all tested models and samples. Due to the 1-page limit of the rebuttal document we cannot include dendrograms, but can confirm that the EQCs follow the same patterns as shown in Figure 6. These results will be included in the supplemental material.\n\n> C5. More description about how different the convolution algorithms do the calculations might extract new insights into how they affect EQCs.\n\nDue to the page limit we omitted a detailed description of calculations of different convolution algorithms. We do agree that they might provide insights into their effect on EQCs and have therefore included them in the supplementary material along with references to the relevant literature and vendor documentation.\n\n> C5. (cont.) Also, some results might confuse the reader, for example, how different explicit and implicit GEMMs are and the reasoning behind a GPU choosing one over the other.\n\nIt seems that our description of the difference between explicit and implicit GEMM in line 153 has been imprecise (specifically \"presumably\"). Thank you for the comment, we will refer to the literature and clarify that the only difference is that explicit GEMM stores the Toeplitz matrix in memory, whereas implicit GEMM computes it on the fly.\n\nThank you for bringing these points to our attention.\nIf you have any further suggestions or questions, please do not hesitate to let us know."}, {"review_id": "B1r64tLNEb", "reviewer": "Reviewer_3gga", "summary": "This paper explores why the same code & data can result in different results from a trained neural network on multiple different, or even the same, architectures. Considering CPU, GPU, and algorithmic implementation, the paper isolates several key factors that cause variance in the calculated results, which hardware group together, which can cause variation on the same hardware, and eliminate other factors as confounders. \n\nThis really is a great paper. I could go into more detail spitting the nuances back at the author, but this is excellent and valuable needed within the ML/DL community. ", "questions": "1. Was any experiment performed to alter the depth/size of the network under test, to see if that would impact the probability of a divergence occurring as depth increased? ", "limitations": "There are no limitations worth noting. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Tackles a core issue and question of reproducibility within the research community\n2. Performs an extensive analysis requiring immense work to obtain many different hardware platforms to perform the tests. \n3. Identified surprising results on the impact of low-precision in mitigating the issue but in unstable ways. \n", "weaknesses": "1. There is some missing related work that should be incorporated for scholastic completeness. In particular the history of numerical differences in ML was documented in \"A Siren Song of Open Source Reproducibility\" - which covers some older history of related work, but also has two recent works that must also be included: \"Problems and Opportunities in Training Deep Learning Software Systems: An Analysis of Variance\" and \"Randomness In Neural Network Training: Characterizing The Impact of Tooling\". Neither work was as thorough in platforms and nailing down what exactly the problem is as this work, so I see this as no barrier to accepting the paper. But we should give completeness to the work by citing the ones who have documented this space previously. \n2. The work doesn't fully prepare the reader for their journey in the introduction. It would help the reader to summarize the results in the introduction, and then the last paragraph can explain where/how these results will be reached. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for pointing us to additional related work, which we unfortunately overlooked when doing our literature search.\n\nZhuang et al.’s work on training variance gives valuable insights on the training process as a whole, whereas we focus on pinpoint observations concerning inference. Combining their methodology with our instrumentation is promising future work, and we will indicate it as such.\n\nPham et al. review the influence of implementation-level variance during training, including the same observations that led to our study of microbenchmarks, on the final performance of the trained models. They also survey how often such variances are mentioned and accounted for in the literature, as well as how prevalent knowledge of such variances is among ML researchers. We will cite their paper in our coverage of microbenchmarks.\n\nRaff et al.’s discussion of the influence of code on reproducibility and replicability focuses on the process of ML research. We share the sentiment that current practices of publishing research code are often insufficient. We will include these points in the discussion and cite the reference.\nWe will expand our discussion section by the points raised in all three papers to highlight the problem of lacking reproducibility, and in fact the lack of a common definition of this term in the first place.\n\n> \"The work doesn't fully prepare the reader for their journey in the introduction. It would help the reader to summarize the results in the introduction, and then the last paragraph can explain where/how these results will be reached.\"\n\nThank you for pointing this out. We will restructure the introduction to make it perfectly clear what results will be presented, and in what order.\n\n> \"Was any experiment performed to alter the depth/size of the network under test, to see if that would impact the probability of a divergence occurring as depth increased?\"\n\nThere is no experiment that explicitly cuts, shrinks or enlarges middle layers to investigate the effects. However, the results in Figure 5 were obtained by outputting intermediate layer results, and the results in Figure 4 can be interpreted as varying the size of a single convolutional layer. Both figures show clear trends on how the size affects the number of EQCs as well as the remaining precision.\nOur analysis of TensorFlow and its underlying Eigen library tells us that actually cutting the layers would result in the same remaining precision and EQCs as shown in Figure 5 because implementation choice on CPUs depends only on the hardware and not on the model.\nOn GPUs, cutting layers will affect these metrics, as early layers will take up memory on the GPU and affect the microbenchmarks of subsequent layers.\n\nUnfortunately, the page limit of NeurIPS forced us to aggressively select which questions we answer in the paper. We hope to convince you that our focus on breadth (for a comprehensive coverage of causes) while compromising on depth (concerning potential interactions between causes) is the right choice at this stage of the research.\n\nIf you have any further suggestions or questions, please do not hesitate to let us know."}, {"review_id": "0Wdr0XGeYM", "reviewer": "Reviewer_s6bp", "summary": "The authors a wide range of hardware platforms including both CPUs and GPUs and found inference result (bitwise) of ML model is not consistent across the platforms and even non-deterministic on the same platform. The authors identified the causesof these numerical deviations and attibuted them to accumulation/aggregation rounding errors due to finite precision and different convolution algorithms.", "questions": "The remaining precisions remain relatively stable (don't diminish to 0) throught the entire model (Figure 5) and the authors attribute it to activaiton layers like ReLU/Pooling reduce information and the author also indicated that EQCs are not related to weight distributions(line 273).\n\nTheoretically, if weights are properly distributed (e.g., like Xavier/He initialization that scales to sqrt(accumulation size)) both the activations and their deviations should follow the same distribution (with similar mean/var) throughout the model, regardless of what activation functions are used. So one question is whether the authors have checked if the weight distributions indeed have such property. And to verify the stable remaining precision is indeed due to ReLU/Pooling that reduce information, can the authors try Xavier initialization + sigmoid activation (instead of He + ReLU in typical Resnet) or replace MaxPool to MeanPool to see if this behavior still holds. This should tell which is the cause of non-diminishing remaining precision.", "limitations": "The authors have talked about a few in line 315, additionally, as discussed by last point in Weakness, the study is limited to single device and single precision, and these may not be a significant source of deviations compared to distributed systems and quantizations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The authors have studied a wide range (75) of hardware platforms, including both CPUs and GPUs, which represents the majorities of ML inference platforms and quatified the deviations through EQCs and Remaining Precisions\n\n* The causes of numerical deviations on different platforms are clearly presented, e.g., CPUs mainly due to precision issues of various parallelisms (SIMD/Multi-cores), and GPUs additionally due to convolution algorithms", "weaknesses": "* The numerical deviations author disclosed are well known to the industry, e.g., arithmetic precisions due to accumulation/aggregation order/times and various convolution algorithms. What's new to me is the non-determinism of convs is actually due to runtime variance of microbenchmarks, however which can be surpressed by forcing Frameworks to enable determinism like in TensorFlow (as suggested by authors at line 300) or Pytorch, which is a common practice when serving models. So I don't think authors found significant new sources of numerical deviations.\n\n* Though authors provided a metric \"Remaining Precisions\" to quantify numerical deviations, it is unclear how it translates to the final inference accuracy/capacity of the model. In the end, unlike traditional software development or high performance computing, ML applications are more robust to numerical deviations in the neurons, and people are even considering \"mortal computing\" through analog devices. So while \"Remaining Precisions\" might be insightful to computer arithmetic community, it doesn't directly reflect the capacity of a hardware/software system for ML applications. Studies to bridge the connection between \"Remaining Precisions\" and accuracies might be desired.\n\n* The takeaway/value of this work is not clear for broader audience of NeurIPS readers. While it reveals the causes of numercial deviations across platforms, and might be valuable to certain domains like security (line 293-296), it doesn't provide a methodology for general audience to benchmark/quantify the numerial precisions of new hardware/softwares, e.g., transformer engines of H100. For example, users could have collected average numerical results of a model from a few mature platforms and run it on a new hardware and compare the remaining precisions, which might refelect how the reliability of a new hardware.\n\n* If my understand is correct, all cases studied are single CPU/GPU and mainly on single floating point precisions, however, in practice, a lot of ML workloads are run in quantized version (bf16/fp16/int8, etc) and in a distributed fashion, which also heavily rely on the ML framework/versions. These might present a first order deviations, compared to the SIMDs/Conv Algos studied by authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your careful analysis of our paper!\n\n> \"The numerical deviations author disclosed are well known to the industry [...] So I don't think authors found significant new sources of numerical deviations.\"\n\nClearly, the sources of numerical deviations are known. This is why we do not claim to have identified new sources of numerical deviations. Instead, we offer the so-far most comprehensive evaluation of causes and effects of (known) numerical deviations in CNN inference, spanning a wide range of relevant platforms. The novelty lies, among other things, in associating them to properties under control of the machine learning engineer, such as layer type or activation function.\n\n> \"[…] What's new to me is the non-determinism of convs is actually due to runtime variance of microbenchmarks, however which can be surpressed by forcing Frameworks to enable determinism […]\"\n\nIndeed, some frameworks allow users to enable deterministic operations. This disables microbenchmarks and selects the first supported algorithm instead of the fastest one. The selected algorithm, however, can again vary between GPUs, especially of different generations. Thank you for pointing this out. We will clarify this in the final version of the paper.\n\n> \"[…] it is unclear how it [\"Remaining Precisions\"] translates to the final inference accuracy/capacity of the model. […]  while \"Remaining Precisions\" might be insightful to computer arithmetic community, it doesn't directly reflect the capacity of a hardware/software system for ML applications. […] \"\n\nOur remaining precision metric is not meant for measuring system-level accuracy. The metric is intended to facilitate analysis and discussion of deviations. It is a refinement of the simple (but crude) EQC metric and is much easier to reason about than, e.g., difference norms and cosine distances.\nMoreover, while users are rarely affected *at this point in time*, they might be in the future when ML results are used other applications that require deterministic and reproducible inputs.\n\n> \"[…] might be valuable to certain domains like security (line 293-296), it doesn't provide a methodology for general audience to benchmark/quantify the numerial precisions of new hardware/softwares, e.g., transformer engines of H100. […]\"\n\nOur understanding is that a broad audience is indeed concerned about security and replicability. By providing our infrastructure on GitHub, we enable other researchers to use our metrics and codebase to trace deviations through the model and produce visualizations like Fig. 5.\nNew hardware, such as H100 (released just before the submission deadline, and only very recently available on AWS), can be compared to existing hardware for concrete models. As demonstrated in the paper, deviations have multiple causes at various level of the stack. A single metric that can predict the reliability of new hardware in general may not exist.\n\n> \"[…] all cases studied are single CPU/GPU and mainly on single floating point precisions, however, in practice, a lot of ML workloads are run in quantized version (bf16/fp16/int8, etc) and in a distributed fashion […].\",\n\"[…] the study is limited to single device and single precision, and these may not be a significant source of deviations compared to distributed systems and quantizations.\"\n\nThank you for pointing this out. The primary goal of this paper is to answer the question if inference is deterministic and to identify causes of non-determinism. For this reason, we chose the approach of analyzing single CPU/GPU machines. Our measurement and orchestration code enables researchers to analyze multi CPU/GPU machines without much effort.\n\nPlease recall that quantization to fp16 is covered in Figure 6 of the paper. As bfloat16 and int8 are not directly compatible with our method of converting the models, we were not able to generate results in time for this rebuttal but will include them in the camera-ready version of the paper.\n\n> \"[…] if weights are properly distributed (e.g., like Xavier/He initialization that scales to sqrt(accumulation size)) both the activations and their deviations should follow the same distribution […] So one question is whether the authors have checked if the weight distributions indeed have such property.  \n> […] to verify the stable remaining precision is indeed due to ReLU/Pooling that reduce information, can the authors try Xavier initialization + sigmoid activation (instead of He + ReLU in typical Resnet) or replace MaxPool to MeanPool to see if this behavior still holds.\"\n\nThank you for this observation. We ran new experiments to confirm that choosing Sigmoid/MeanPool (Xavier initialization) over ReLU/MaxPool (He initialization) does not change the pattern of remaining precision throughout the model.  The results of this experiment are shown in Figures R1 and R2 of the attached PDF. The first and last Sigmoid activation layers decrease the remaining precision, whereas the remaining layers increase or do not affect it. This confirms our reasoning that activation functions simply propagate deviations. The single pooling layer after the first convolution increases the remaining precision for both MaxPool and MeanPool.\nWe also measure the effect for a small model with a second pooling layer and find that Sigmoid activation reduces the remaining precision in three out of four cases. MaxPool increases the remaining precision for both layers, whereas MeanPool increases it for one and leaves it unaffected for the other layer.\n\nMoreover, Table R1 shows the influence of activations on the remaining precision and Table R2 compares the number of deviations before and after activations for each layer.\n\nAgain, thank you for your valuable input. It helped us to improve the quality and clarity of our work!\n\nIf you have any further suggestions or questions, please do not hesitate to let us know.\n"}]

Author rebuttal:
We thank all four reviewers for their insightful feedback and thoughtful suggestions. We are very glad that the suggestions by different reviewers are compatible (and partly overlapping). This enables us to incorporate all of them into the final version of the paper.

Responses to specific questions and remarks are posted under the respective reviews.
The attached PDF document contains the figures and tables of the experiments we ran to address specific questions in the reviews.

Meta-review: Reviewers found this paper exciting and unanimously recommended accept.
Decision: Accept (poster)