The Grand Illusion: The Myth of Software Portability and Implications for ML Progress.
Authors: Fraser Mince
Dzung Dinh
Jonas Kgomo
Neil Thompson
Sara Hooker
Venue: NeurIPS 2023 poster (2023)
Keywords: hardware
software
meta study
portability

Abstract: Pushing the boundaries of machine learning often requires exploring different hardware and software combinations. However, this ability to experiment with different systems can be at odds with the drive for efficiency, which has produced increasingly specialized AI hardware and incentivized consolidation around a narrow set of ML frameworks. Exploratory research can be further restricted if software and hardware are co-evolving, making it even harder to stray away from a given tooling stack. While this friction increasingly impacts the rate of innovation in machine learning, to our knowledge the lack of portability in tooling has not been quantified. In this work we ask: How portable are popular ML software frameworks? We conduct a large scale study of the portability of mainstream ML frameworks across different hardware types. Our findings paint an uncomfortable picture -- frameworks can lose more than 40% of their key functions when ported to other hardware. Worse, even when functions are portable, the slowdown in their performance can be extreme. Collectively, our results reveal how costly straying from a narrow set of hardware-software combinations can be - and thus how specialization incurs an exploration cost that can impede innovation in machine learning research.

The Grand Illusion: The Myth of Software Portability and Implications for ML Progress.
Fraser Mince
Dzung Dinh Cohere for Al Community dinhd@dickinsee edu
Cohere for AI Community fraserminceeggail. com
Jonas Kgomo
Neil Thompson MIT Beil.tmit..o edu
Sara Hooker Cohere for AI
Cohere for AI Community
jonaskgmoo@gmail.cm com
sarahookerrCcherre.com
Abstract
Pushing the boundaries of machine learning often requires exploring different hardware and software combinations. However, the freedom experiment across different tooling stacks can be at odds with the drive for efficiency, which has produced increasingly specialized AI hardware and incentivized consolidation around narrow set of ML frameworks. Exploratory research can be restricted if software and hardware are co-evolving, making it even harder to stray away from mainstream ideas that work well with popular tooling stacks. While this friction increasingly impacts the rate of innovation machine learning, to our knowledge the lack of portability in tooling has not been quantified. In this work, we ask: How portable are popular MI. software frameworks? We conduct a large-scale study of the portability of mainstream MI. frameworks across different hardware types. Our findings paint an uncomfortable picture frameworks can lose more than 40% of their key functions when ported other hardware. Worse, even when functions are portable, the slowdown in their performance can be extreme and render performance untenable. Collectively, our results reveal how costly straying from a narrow set of hardware- combinations can be and suggest that specialization of hardware impedes innovation in machine learning research.
Introduction
The field of machine learning (ML) has made significant strides in recent years, thanks in large part to advances hardware and software [Chowdhery et al. 2022 Zhang et al.] 2022 Kaplan et al. 2020]. However, the pursuit of efficiency has led to the creation of increasingly specialized Al hardware and the consolidation of ML frameworks around a narrow set of tools [Hooker 2021]
This specialization has limited the ability of researchers to experiment with different hardware and software combinations, hindering the rate of innovation in the field.
The portability challenge has been amplified by the ever more heterogeneous landscape of hardware and software [Reddi et al. 2020). In particular, differences in hardware create a vexing problem for software: how to allow portability while maximizing performance [Hooker] 202 Lee al. 2011 Barham and Isard 2019). Many commercial hardware suppliers purport to support a variety of popular ML libraries, however qualitative evidence from machine learning researchers suggests that this is often far from a straightforward process that requires significant changes to the code before
* "These authors contributed equally to this work.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

can be transferred successfully (Johansen et al. 2014). In this work, we ask how has the increasingly fragmented and specialized hardware and software landscape impacted the portability of research?
To our knowledge, there has been no prior work that has sought to quantify the ease of portability between hardware types. In this work, we seek address this gap, by explicitly quantifying the portability of popular mainstream ML libraries, TensorFlow [Abadi et al. 2013), PyTorch (Paszke) al. 2019), and JAX (Bradbury et al. 2018). that are used by millions of developers across different hardware types. We embark on extensive data collection and annotation to hand-curate representative tests for each library and subsequently benchmark transferability and latency across different hardware types.
Our results reveal highly uneven portability, suggesting that there will be increasingly uneven gains from progress in computer science research. Exploration in MI. research appears to be hindered by failing functions and dismal performance. While some operations benefit from portability across devices, there are large gaps in coverage for widely used software frameworks. We find that there are frustrating differences in the subset of software operations supported on different types of hardware which prevent the portability of algorithms across hardware types. Even where there is portability, significant gaps exist in performance between each framework. Software kernels are often overly optimized for a specific type of hardware which causes huge lags in efficiency when used with a different type of hardware (Hennessy and Patterson 2019b]. Our main contributions can be enumerated as follows:
We gather a human-curated and annotated collection of functions from popular ML libraries that can be benchmarked across hardware types. We open source this dataset for use in future benchmarking at the provided repo: htttps://githui com/ f for- ai ai/portability
We find that PyTorch and TensorFlow, in particular, have portability issues. On GPUs, 22% of the TensorFlow benchmark functions fail partially or completely. On TPUs, a remarkable 44% of PyTorch benchmark functions partially or completely fail.
Even where functions are portable, we see significant gaps in performance - with both unexpected speedups and slowdowns moving functions between the GPU and the TPU. For example, 81.4% of functions in PyTorch exhibit more than a 10x slowdown when transferring functions from GPU to TPU.
We illustrate that certain software libraries are locked into a particular tooling stack. JAX co-designed with TPUs in mind, and this is reflected in its performance. In all, 91.8% of our JAX function set is faster on the TPU.
We compare how software portability has evolved over time by comparing different versions of GPUs and TPUs Specifically, we run experiments on both GPUs T4 and A100 and observe that the portability remains the same for PyTorch while it differs by only up to 1% for TensorFlow and JAX. Moreover, we observe that 28.07% and 9.09% of PyTorch functions achieve a 1.5X speed improvement when operating newer GPU and TPU versions, respectively. Hence, although newer generations of hardware have not improved software portability, they have yielded modest speed enhancements for certain frameworks.
Importance of this work: This paper presents an evaluation framework at the beginning of a time when hardware and software specialization is growing, and thus where comparative evaluations will become ever more important. The economics of chip specialization have dramatically changed over the last decade so [Thomsoon and Spanuth| 2021]. leading Hennessy and Patterson term this new golden age for computer architecture in their Turing lecture (Hennessy and Patterson 2019a]. Specialization carries with it radical changes in performance, and disparities will only increase, as will the importance of co-designing implementations to those chips. Thus, we should expect that the type of quantitative portability analyses that we do in this paper will only become more important in the coming years to aid the design of tooling that is both efficient and portable.
2 Methodology
We are interested in quantifying the portability of mainstream Python libraries used for machine learning workloads. We define portability as the ease with which a machine learning workload (code, data, and models) can transfer between different hardware types. We consider several types of failure:
2

---

Comparison of TPU and GPU Failure and Success Rates
GPUs
TPUs
Success Pass
Failure
Success Pass
Failure
Partial
Complete
Partial
Complete
TensorFlow
78%
8%
14%
71%
15%
14%
PyTorch
92%
3%
5%
57%
27% 0%
17%
JAX
98%
0%
2%
97%
3%
Table Comparison portability success and failure rates of random stratified sample of Tensor- Flow, PyTorch, and JAX functions across TPUs and GPUs.
Faster TPU
Faster GPU
GPU TPU
109
10
TPU time (nanosecond]
10a
10
10
10?
10
10c
10
10a
10a
10a
10a
10a
10
10a
10"
10"
10a
10* 10"
10" 10
10 10 10 10' 10a 10 GPU time (nanosecond)
GPU time (nanosecond)
GPU time (nanosecond)
(a) TensorFlow
(b) PyTorch
(c) JAX
Figure 1: Comparison of average execution time on Log scale for TensorFlow, PyTorch, and JAX functions GPU versus TPU. In total, there are 51 functions in TensorFlow, 43 functions in Py Torch, and functions in JAX. The number of data points is lower than the overall count of functions because we have excluded all subtests that failed on either device. This exclusion was to ensure valid comparison.
1. Complete failure to run: If the function does not the device at all.
2. Partial failure to run: Some but not all the benchmark tests for given function fail to
3. Intolerable latency: High latencies may be prohibitively inefficient, which may impair usability even if the function technically is able to run on multiple hardware types.
Our goal to benchmark the portability of libraries that claim to be portable across hardware types, and which are widely adopted. Hence, evaluate the portability of JAX version 0.4.8, Py PyTrch version 1.12.0, and TensorFlow version 2.1..0.
2.1 Data collection
Function sampling procedure: To obtain a full list of functions, we iterate through the module structure of Torch, TensorFlow, and JAX to enumerate all functions and classes contained in the library. This process results in 2718 TensorFlow functions, 2898 PyTorch functions, and 1091 JAX functions.
Sampling procedure: To have a representative view each libraries performance, we stratified sampling. including 1) the top 20 functions as ranked by frequency of use, and 2) 5 random functions from each decile of all functions ranked by frequency of use for each library (JAX, PyTorc, TensorFlow). The random sample allows us to capture variety of different engincering

---

Device
Device
Devise
TPU
TPU
GPU
GPU
Densty
0.6
Density
0.8
Density
04
0.
10
10"
0.0
10
10"
10"
10"
Time (ns)
Time (ns)
Time (ns)
(a) TensorFlow
(b) PyTorch
(c)
Figure 2: TensorFlow, PyTorch, and JAX time densities.
use cases and not overfit to scripts that may only rely on small subset of the library. Benchmarking the top 20 functions measures how the frequency of use of given function impacts portability our expectation at the outset was that more frequently used functions would be prioritized for support across hardware types.
To identify the top 20 functions and the decile samples, we measure the frequency of how often these PyTorch, TensorFlow, and JAX functions appear in scripts in the CodeParootcclan datase CodeParrot-clean a deduplicated version of the CodeParrot datase which is a collection of approximately 22 million Python files used originally build code generation model part of the O' O'Reilly Transformers book [Tunstall et al. 2022]. This was created by collecting Python files from the Github Google BigQuery dataset We filtered to restrict files that string matched import statements from either library. In the Appendix Section procedure used.
12
we provide more details about the filtering
Thus, for each framework, we sample approximately 70 functions, 50 random decile functions, and the 20 most-used The total number of samples per framework balanced the need for coverage with the time-intensive process need to human annotate and procure the minimal test for each function, to add in code track latency to each script, and modify and verify that each test is only on the hardware in question. We intend to release this as dataset that can be used by others for benchmarking portability.
Human annotation and curation tests: Our goal is to benchmark as conservatively as possible rely where possible on the test included in
the expected behavior of the function. To do the library for a given function. We manually match each function to its associated tests in the TensorFlow, PyTorch, and JAX libraries. Given these widely used and well-maintained libraries, our expectation is that the tests within the formal library reasonably represent the minimal code needed validate expected function behavior under different conditions. For all tests that failed, ensured that all failed tests were due to errors in the function being tested. Once tests are identified, manually modify the test files to ensure 1) only the relevant tests and the initialization code needed for running were preserved, 2) the code was being run on the device we were evaluating, and 3) the code needed record the latency in consistent way for each script added.
Top 20 test exclusion: In the top 20 functions, there were occasions when it was not possible to test function:
1. Overlapping functions: Due to the inherited randomness of our sampling and the static nature of our top 20 there are some overlaps between deciles and the overall top 20: 4, O, and
an--iigquery-anaae aeee--lll-hhee all-the-opeer tace-opee--suucee source-
code
'This number is approximate due to needing to exclude some functions from the list. More details are in the next paragraph. In all, we include 63 unique functions from PyTorch, 65 unique functions from TensorFlow, and 63 functions for JAX.

---

100
100
80
80
60
80
40
40
0
TensorFlow
PyTorch
JAX
TensorFlow
PyTorch
JAX
(a) % of Functions Faster on v3-8 TPU
(b) of all Functions 10X Faster A100 GPU
Figure 3: Percentage of functions faster A100 GPU vs v3-8 TPU.
overlapping functions in PyTorch, TensorFlow, and JAX, respectively. These we excluded from testing since could replace the top 20.
2. Functions without a relevant test: For some functions in the top 20 there was either no relevant test or situations where testing them would be somewhat nonsensical such as int, which shows up in PyTorch but as is a type. is not quite clear what we should test in this case, and thus we decided to exclude them.
Replacement criteria: After completing the sampling procedure, we found that a subset of functions was not viable for benchmarking. In total, we found 14 functions in TensorFlow, 13 functions in PyTorch, and 13 functions in JAX that needed replacement by resampling at random from the same decile. Our criteria for justifying replacing a function are detailed below:
1. No present in the respective suites. For example, arctan within PyTorrc was not tested in the PyTorch open-sourced test suite. Respectively, there were 12, 12, and 13 functions for P PoTch,, TensorFlow, and JAX that were not tested the open-sourced test suite.
2. The tests designed to validate the error handling of the functions; therefore, the timing doesn't t work in this case. For example, when testing the batch norm function in Py Torch, the test focuses solely on checking if the function correctly raises errors for unreasonable inputs. So, the core functionality of the method is not tested, just error throwing. Only one function in Py PyTorch fell into this case.
3. The functions operate on an older version of the framework. For instance, the test for TensorFlow's raw pnnn is only compatible with TensorFlow version 1, yet we are conducting tests in TensorFlow version 2. There were two functions TensorFlow that fit this case.
For functions that needed be resampled, we sampled new function random from the same frequency decile as the original function.
Results and discussion
3.1 Portability of functions across hardware types
Overall failure and partial failure rates: We observe rates of failure for all libraries we benchmark across hardware types. However, the rate of failure differs between frameworks. As seen in Table ! on GPUs TensorFlow had the highest failure rate with total of 21.54% complete and partial failures. On TPUs PyTorc has the highest failure rate with a remarkable total of 44.44% complete and partial failures. Across both platforms, we observe the lowest rate of failure for JAX with 1.59% complete failure on the GPU and 3.17% complete failure on the TPU. In particular, PyTorch's 's TPU failure rates stand out, as double the failure rate of TensorFlow and the highest failure rate overall.
Failure rate aeross different percentiles: One of the questions we wanted to explore was whether portability was impacted by the frequency of use of functions. Our expectation was that the more heavily used a function was, the more portable would be given the incentive to support the top use cases. However, as shown in Figure I there is fairly consistent failure and partial failure rate across

---

Table Comparison of the latency in milliseconds for the two functions with the greatest and least increase in latency in TensorFlow, PyTorch, and JAX on GPU and TPU. The table is ordered by the ratio GPU/TPU in descending order, and the top two biggest ratio functions are highlighted. Note that values are rounded to 3 decimal places.
Function
GPU
TPU
TPU/GPU
Tensorflow
tf lilaallggg.. sinalg.svd svd w.m.ath. redutthhrrdueeeeocuuueeeee
0.931
112.843
121.206
13.028
474.586
36.428
tf estimator LoggingTennsoriook tf. compat. Senmpat.v..Sesii... ww.Session...... run
0.042
0.038
0.905
5.722
3.804
0.665
PyTorch
torch. argsort farch.opting Adamax
0.157
948.210
6039.554
0,069
392.712
5691.478
torch. cuda torch. Conv2d
0.041
0.061
1.488
46.053
67.081
1.457
JAX
jax. nax.name_ccall jax numpy. array
0.007
0.012
1.714
0,435
0.638
1.467
jax.numpp numpy cos jax.
172.002
26.102
0.152
98.118
13.860
0.141
(a) TensorFlow
(b) Py'orrc
(c) JAX
100
100
100
Complete Feilure
functions
80
Partial Falluure
80
GPU
60
40
20
Percentile
0
10
20
30
40 50 50
Percentile
70 80 90 100
10
20
30
40 50 60 70 80 90 100 Percentile
100
100
of functions
80
TPU
60
60
40
20
20
20
10
20
30
40
60 Percentile
50
70
90
90
100
10
60 Percentile
70
50 60
100
Percentile
Figure 4: Complete and partial failure rates by percentile bucket for TensorFlow, PyTorch, and JAX functions on GPU and TPU. Note that charts include functions across deciles.
deciles. This holds for all libraries, which suggests that frequency of use has not had a significant effect on the prioritization of support across hardware types.
To further explore whether the frequency of use has influenced portability, directly compare rates of portability for the top 20 functions other deciles. In Table we observe that some libraries like JAX have 0% failure rates in the top-20 and low overall failure rates across all functions. However, surprisingly failure rates in the top 20 functions than across all functions (46% vs 44%).. We also observe that on GPUs, TensorFlow also presents considerably higher rate of failure in the top-20 functions (33% vs 22%). Across the board, we observe the rates of error between the deciles and the top 20 are quite similar showing even the most used functions do not benefit from greatly increased portability.
Failure rate of top-20 functions:
TPUs, PyTorch actually presents slightly higher
Comparing GPUs generations: When analyzing the portability success and failure rates of Tensor- Flow, Porch, and JAX functions across T4 and A100 GPUs, we observe surprisingly similar trends

---

Complete Fallure
JAX
Partial Failure PASS
JAX
Torch
PyTorch
TensorFlow
TensorFlow
0
20
40
60
80
100
0
20
40
60
80
100
% functions
of functions
(a) GPU
(b) TPU
Figure 5: Complete and partial failure rates for the top 20 functions in TensorFlow, PyTorch, and JAX functions on GPU and TPU.
Comparison of GPU A100 and T4 Failure and Success Rates
T4
A100
Success Pass
Failure
Success Pass
Failure
Partial
Complete
Partial
Complete
TensorFlow
78%
8%
14%
79%
9%
12%
Py Torch 92%
3%
5%
92%
3%
5%
JAX
98%
0%
2%
97%
0%
3%
Table 3: Comparison of portability success and failure rates of random stratified sample of Tensor- Flow, PyTorch, and JAX functions across T4s and A100s.
between the two hardware generations, differing by only up to 1% for TensorFlow and JAX as shown in Table B Success rates remain consistently high for all frameworks both GPUs, indicating robust compatibility. The percentages of Partial and Complete Failures also exhibit comparability across the GPUs and frameworks. This concerning it indicates that the advancements in A 1100 architecture have minimal influence on the overall portability.
First class citizen effect different hardware: One noted effect we see in our results could be described as a first class citizen effect. Or simply frameworks built for a device or compilation target perform much better in that environment. The most striking example of this is in JAX on TPUs. As seen in Table [ in JAX, we see much lower rate of eтrors on TPUs when compared to other frameworks with only 3% of functions failing. This is likely due to JAX being built with XLA as a target in mind. We see a similar but less pronounced effect with TensorFlow when compared to PyTorch. TensorFlow them with 29% of functions failing when compared to Py Torch which has 44% of functions failing. The first-class citizen effect is less pronounced in TensorFlow. likely due the age of the framework and the newer JAX giving the teams at Google a chance to rethink what XLA as a compilation target looks like. Compare both of these PyTorch, and you see a significant difference. PyTorch is framework where XLA support was tacked on in a separate library, and it very much shows.
of the original targets for XIAA and thus performed decently well
We observe rates of failure for all libraries benchmark across hardware types. To understand better what impacts hardware portability, we annotate failures into categories. We briefly describe each below:
Reason for failure:
Type failure: Some types are not implemented in given device. For instance, in PyTorch, while a TPU tensor might claim to double, it will always represented by a Float instead. Another example of this is in the case of the PyTorch function SmoothLlLoss, which on GPUs we attempt to call the bFloat16 type. However, this not implemented and fails.

---

100
100
Percenlage
Percentage
60
60
40
40
20
20
0
Type Failure
Timeout
Memory Issue
Float Precision Emrr
Type Failure
Timeout
Memory Issue
Float Precision Error
Implemented
Implemented
(a) TensorFlow GPU Error Categories
(b) TensorFlow TPU Error Categories
100
100
Percentage
Percentage
60
60
40
40
20
20
Type Fallure
Memory Isaue
Type Failure
Timeout
Memory Issue
Implemented
Predision
Implemented
Precision Error
Error
(c) Py GPU Error Categories
(d) PyTorch TPU Error Categories
100
100
Percentage
Percentage
60
60
40
40
20
20
Type Fallure
Timeout
Memory Issue
Type Failure
Timeout
Memory Issue
Float Precision Error
Implemented
Precision
Implemented
Error
(e) JAX GPU Error Categories
(I) JAX TPU Error Categories
Figure Percentage of failure categories per framework device pair.
Not implemented: Some operations do not have kernels implemented on TPUs GPUs at all or for specific categories of inputs. For example, on the TensorFlow numpy function, the kernel PyFuncStateless is not implemented.
Timeout:
We set a threshold of 15 minutes for each test. If test goes above that, we timeout failure. Given the minimal test code and the
automatically kill and record size of the test inputs (designed to run quickly), we believe 15 minutes was conservatively long.
Memory issue: Captures all cases where memory attempted to be allocated accessed as part of the operation and failed. For example, PyTorch Dataset attempted to use pin-memory. but this does not work on TPU.
Float precision error: TPUs have special float class called bFloat16, which has fewer mantissa bits and more bits for the exponent. This allows for much smaller and larger values but at the cost of floating point precision. This can break assertions in the tests.
As shown in Figure 6 the most common reason for failure across all frameworks is the Not Implemented error, which pronounced in TensorFlow and JAX, accounting for over 60% of failures. Moreover, Py Torch has distinctive rise in Type Failures, contributing to more than 30% of its failures, rate noticeably higher than the almost negligible or at most nearly 10% in other frameworks. Both TensorFlow and PyTorch exhibit relatively low failure rate due to Memory Issues and Type Failures. As expected, the Float Precision error is unique to the TPU, representing around 20% of the failures for both TensorFlow and PyTorch.

---

100
100
60
60
20
20
TensorFlow
PyTorch
JAX
TensorFlow
PyTorch
JAX
(a) % of all Functions 1.5X Faster on A100 GPU Compared to T4 GPU
(b) % of all Functions 1.5X Faster on v3-8 Com- pared to v2-8 TPU.
Figure 7: Percentage functions faster on new GPU/TPU compared with old ones.
3.2 Efficiency cost to switching tooling
As depicted in Figure and Figure B 96% and 100% TensorFlow and Py functions experi- ence significant deceleration when migrating from GPU to TPU, respectively. Specifically, within TensorFlow and Py Torch, a clear latency gap emerges when functions previously operating on the GPU transferred the TPU. As seen in Table 2 the lowest observed latency gap is 0.665 times. This gap tends to be more pronounced for slower operations on the GPU, reaching maximum latency prominent, with speed reductions of up to 6039 times when migrating from GPU to TPU. The latency densities also follow this trend, as shown in Figure 2
gap of 121.206 times. In PyTorch, the latency gap is
In contrast, most functions perform faster on TPU in JAX. When comparing the two devices, there is minimal latency gap in JAX for both quick and slow operations. The ratio of performance in the migration from GPU to TPU in JAX remains minor, ranging from 0,141 to 1.714 14 times. In all, we see slowdowns 100% functions for Py PyTochh, 96% of functions for TensorFlow, and 8% functions JAX while moving to the TPU.
There are unique circumstances here that might make this different from using these frameworks in real-life situations (specifically in your standard training situation, you have long-running processes, and in our case, we are running simple functions that finish quickly), but this is clear that the benefits of switching specialized hardware can be uneven and variable.
Related work
Deep learning frameworks: The rapid adoption and commercial success of Deep learning has spurred the development of software frameworks tailored deep neural network workloads. Many of the most widely used libraries for machine learning workloads are Python libraries like TensorFlow Abadi et al. 20155, Theano [Team et al.) 2016). Chainer (Tokui et al. 2201)) MXNet (Chen et al.) 2015). Py Torch Paszke et al. 2019] and JAX [Bradbury et al. 2018). Despite the variety in frameworks, there has been study
knowledge of the difficulty of porting these frameworks
between different types of hardware.
Narrowing of AI research: The specialization of hardware create efficiencies for machine learning workloads has created concerns about narrowing research contributions. Recent work (Hooee 202T Barham and Isard 2019] suggests that inflexible nigh-performance kernels and limited programming abstractions are hindering innovative machine learning research. (Hooker, 2021 argues that the availability of accelerator hardware determines the success of ML algorithms potentially more than their intrinsic merits that the success of ideas hinges on alignment with hardware on software. [Klinger et al. 2022 analyzes arXiv papers and finds that AI research has stagnated in recent years and that AI research involving the private sector tends to be less diverse and more influential than research in academia. Several works [Ahmed and Wahed 2020 point to the growing compute divide, which impacts accessibility to research and ideas.
Portability of software frameworks: Different designs for technology are possible, and some designs are more desirable from an innovation standpoint than others (David al. 2009). However, circumstances such as chance events, shortsightedness, and lack of coordination can lead to

---

Table 4: Comparison of failure rates between functions the top and the overall failure rate across all deciles.
TensorFlow
Py PyTorch
JAX
GPU
Top 20 All Functions
33%
0%
0%
22%
10%
2%
TPU
Top 20 All Functions
27%
46%
0%
30%
44%
4%
situation where an inferior design becomes dominant and difficult to transition away from, even after its limitations become evident [Arthur 1994 David 1985] In the face of uncertainty regarding the advantages and risks associated with different technologies and the desire to avoid getting stuck with an inferior design prematurely, it might be sensible implement policies that maintain diversity in the technological landscape (David et al. 2009]. A third and final reason to preserve technological mobility and reduce the cost of exploration: innovation involves the creative recombination of ideas, and unusual mixes are often an important source of radical and transformative innovations (Arthur
2011
5
Limitations
While our work does great deal to quantify existing gaps in portability, it has some important limitations. Firstly we recorded latency calculations and failure categories on two types of GPUs (A100s and T4s) and two types of TPUs (v2-8 and v3-8). We believe the similar error rates between types of GPUs show that at least for failure rates there is a good deal consistency between types of GPUs. Worthwhile extensions of this work would include adding more device types to get more robust view overall portability and its trend.
Secondly, this paper does not explore in depth why these portability gaps exist. We provide some broad hypotheses on why there might be differences in Section 3.2 but we leave it to future work to pinpoint why these differences exist. One reason for our limitation is due to the lack of access to CUDA internals as it is not completely open source. Understanding the differences kernels between devices and framework implementations is daunting task and outside of the scope of this work.
Conclusion
We benchmark three widely used and adopted machine learning libraries to evaluate the ease of portability across different hardware types. We find large differences in the subset of software operations supported on different types of hardware. We find that PyTorch and TensorFlow, in particular, have pronounced portability issues. On GPUs, 22% of the TensorFlow benchmark functions fail partially completely. On TPU, remarkable 44% of PyTorch benchmark functions partially or completely fail. Even where there is portability, significant gaps exist in performance between each framework. We observe that when transferring functions from GPU to TPU, 81.4% of functions in PyTorc exhibit more than 10x slowdown.
Significant work remains ensure portability and performance between device types. Currently, jor slowdowns and broken operations are the norms, and widely used frameworks have overpromised when it comes to portability. This lack of portability has costs to innovation: incentivizing researchers to stick with the tooling they have, which often makes it harder to stray off the beaten path of research ideas. Innovation often occurs when it is cheaper explore, but our tooling stacks have clearly quantifiable friction that deters exploration. Valuable future work includes developing standardized approaches machine learning tooling that enable greater portability between different hardware types, and benchmarking additional hardware types. We hope by releasing our benchmark dataset, we can spur greater visibility into what frameworks need more support.
10

---

References
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L.. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster,
J. Shlens, B. Steiner, Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas,
O. Vinyals. P. Warden, M. Wattenberg, M. Wicke, Yu, and X. Zheng. TensorFlow: Large-Scale URL htttp::/wwww.tessstt
Machine Learning on Heterogeneous Systems, Jan. 2015.
org/ Software available from tensorflow.org.
N. Ahmed and M. Wahed. The de- deedemmocratization of AI: Deep learning and the compute divide in artificial intelligence research, 2020. URL htt::http:/aaat /ttpvv/aaxvvvvvvm org/ abaaaba////a/ 15581
W. B. Arthur. Increasing Returns and Path Dependence in the Economy. University of Michigan Press, 1994. ISBN 978-0-472-09496-77 doi: 10.3998/mpub.1002/ 10029. URL https::/www/ jstor org/stable/10. 1999/mmpb... 10029
W. B. Arthur. The Nature of Technoloy:: What It Is and How It Evolves. Free Press, reprint edition edition, 2011 1. ISBN 978- 978-1-4165-4406-7. 10
Barham and M. Isard. Machine learning systems are stuck in rut. Proceedings of the Workshop on Hot Topics in Operating Systems, HotOS '19, page 177-183, New York, NY, USA, 2019, Association for Computing Machinery. ISBN 9781450367271.
doi: 10.1145/311555033314444
URL https: //doi.:://do..oo org 10. 0114/33315555 3321441
P. Barham, A. Chowdhery, J. Dean, S. Ghemawat, S. Hand, D. Hurt, M. Isard. H. Lim, R. Pang, S. Roy, B. Sacta, P. Schuh, R. Sepassi, L. E. Shafey, C. A. Thekkath, and Y. Wu. Pathways:
Asynchronous distributed dataflow for ml, 2022. 14
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke,
J. VanderPlas, S. Wanderman-Milne, and Q Zhang. JAX: composable transformations of Python+NummP programs, 2018. URL URLLhtpp:// http:/ http::/githuu. www com/google/jar
2
T. Chen, M. Li, Li, M. Lin, N. Wang, M. Wang. Xiao, B. Xu, C. Zhang, and Z. Zhang. Mxnet:
A flexible and efficient machine learning library for heterogeneous distributed systems. CoRR, abs/1512.01274, 2015. URL http:///aaaa .rt. 1512.01274
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, Bradbury, J. Austin, M. Isard, G. Gur-Ari. P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski.
X. Garcia, V. Misra, K. Robinson, I.. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, F. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellsterm, D. Eck, Dean, S. Petrov, and N. Fiedel. Palm: Scaling language modeling with pathways, 2022. URL htttt::/t/taatttt
Baarg//baa/eee 2204 02311
David, D. Foray, and Aghion. Science, technology and innovation for economic growth: Linking policy research and practice in "stig systems". Research Policy, 38:681 1-63,, 05 2009. doi: (0.2139//smm.128555
10
A. David, Clio and the economics of QWERTY. The American Economic Review, 75(2):332- 337, 1985. ISSN 0002-8282, URL httpp://wwwi http://wwwwww.t...o//o wttp://ot//tttttbgeeeeeeea o American Economic Association. IC
Publisher:
K. M. Eisenhardt and B. N. Tabrizi. Accelerating adaptive processes: Product innovation in the global computer industry. Administrative Science Quarterly, 40(1):84-110, 1995. ISSN 00018392. URL http://www..Boooorrgggaaaeeee//
jstor wttp:stable//33300...
J. L. Hennessy and D. A. Patterson. A new golden age for computer architecture. Commun. ACM, 62(2):48 60, jan 2019 1a. ISSN 0001-0782, doi: 10. 10.1145/3282307. URL. https //doi //t//oi.oggg//////// org/10 1145/3282307

---

J. L. Hennessy and D. A. Patterson. A new golden age for computer architecture. Communicatiins of the ACM, 62(2):48 60, 2019b. ISSN 0001-0782. //doi 7/doi.org/10 org/10. 10. 1145/3282307
doi: 10. 10.1145/3282307.
URL https
Hooker. The hardware lottery. Commun. ACM, (4()))411111))))) 65, nov 2021. ISSN 0001-0782. doi: 10.1145/3467017. URL
9
H. Johansen, L. C. McInnes, D. E. Bernholdt, Carver, M. Heroux, R. Hormung. P. Jones, B. Lucas, and A. Siegel. Workshop software productivity for extreme-scate science, january 13-14, 2014, hilton hotel, rockville, MD, 2014. Uttptss::://ee http::/wwwwwwttt.ggoooo htttl/ooooooo/////111..
2
N. P. Jouppi, C. Young. N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers, R. Boyle, P.-. Cantin, C. Chao, C. Clark, J. Coriell, M. Daley, M. Dau, J. Dean,
B. Gelb, T. Ghaemmaghami, R. Gottipati, W. Gulland, R. Hagmann, C. R. Ho, D. Hogberg, J. Hu, R. Hundt, D. Hurt, J. Ibarz, A. Jaffey, A. Jaworski, A. Kaplan, H. Khaitan, D. Killebrew, A. Koch, N. Kumar, S. Lacy, J. Laudon, Law, D. Le, C. Leary, Z. Liu, K. Lucke, A. Lundin, G. MacKean, A. Maggiore, M. Mahony, K. Miller, R. Nagarajan, R. Narayanaswami, R. Ni, K. Nix,
T. Norrie, M. Omernick, N. Penukonda, A. Phelps, Ross, M. Ross, A. Salek, E. Samadiani, C. Severn, G. Sizikov, M. Snelham, J. Souter, D. Steinberg, A. Swing, M. Tan, G. Thorson, B. Tian, H. Toma, Tuttle, V. Vasudevan, R. Walter, W. Wang. E. Wilcox, and D. Yoon. In-Datacenter Performance Analysis of Tensor Processing Unit. SIGARCH Comput. Archir. News, 45(2):1 June 2017. ISSN 0164-5964. doi: 45/54///345430000 //doi. 114// /0111///14000 3106659 3080246 15
URL https
Kaplan, S. McCandlish, Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. CoRR, abs/2001.0836 2020. URL
08361
J. Klinger, J. Maaatos--Grriia, and K. Stathoulopoulos. A narrowing of Al research?, 2022. URL http: http:://axxiv..rr/abbbe. wwwgg/bb///00/0 10385
H. Lee, K. Brown, A. Sujeeth, H. Chafi, T. Rompf, M. Odersky, and K. Olukotun. Implementing domain-specific languages for heterogeneous parallel computing. IEEE Micro, 31(5):42-53, 2011. ISSN 0272-1732. doi: 10.1109/MM.201..0 URL http://jeeeepllore.e ieee org/
document/5963629/
N. Parikh and Boyd. Proximal algorithms. Foundations and Trends Optimization, 1:123-231, 01 2014. 14
A. Paszke, S. Gross, Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035, Curran Associates, Inc., 2019. URL http::/papees. www.eeewww.ccpppeeeeeeeee 9015 2015-pytorch--o an- hrtprpacceeettttteeeeerprrorroee pdf
V. J. Reddi, C. Cheng. D. Kanter, P. Mattson, G. Schmuelling, C.-J. Wu, B. Anderson, M. Breughe, M. Charlebois, W. Chou, R. Chukka, C. Coleman, S. Davis, Deng, G. Diamos, J. Duke, D. Fick, S. Gardner, Hubara, S. Idgunji, B. Jablin, J. Jiao, T. S. John, P. Kanwar, D. Lee, J. Liao,
A. Lokhmotov, F. Massa, P. Meng, P. Micikevicius, C. Osborne, G. Pekhimenko, A. T. R. Rajan, D. Sequeira, A. Sirasao, F. Sun, H. Tang, M. Thomson, F. Wei, E. Wu, L. Xu, K. Yamada, B. Yu, G. Yuan, A. Zhong, P. Zhang, and Y. Zhou. MLPerf inference benchmark, 2020. URL
http://arxizs www.aiiwgg/aaa..... www.//gg////eeo..ee...ooeeeeeeee 1atbb//9991..... 02549
C. Richter and H. Wehrheim. Tssb-3m: Mining single statement bugs at massive scale, In 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR), pages 418 422. Los Alamitos, CA, USA, may 2022. IEEE Computer Society. doi: 10. 145/3528842.35288066 URL htttt::///aatt hew.beoeeeeecottttesscceeeeee httpa//cc.caatacooa.ooooooooooaa
11tp://.111111/2242221433 3528505
T6
S. Sabour, N. Frosst, and G. E. Hinton. Dynamic routing between capsules, 2017.
I4
12

---

T. T. D. Team, R. AI-Rfou, G. Alain, A. Almahairi, C. Angermueller, D. Bahdanau, N. Ballas,
F. Bastien, Bayer, A. Belikov, A. Belopolsky, Y. Bengio, A. Bergeron, J. Bergstra, V. Bisson, J. B. Snyder, N. Bouchard, N. Boulanger-- Lewaddowsi, X. Bouthillier, A. de Brébisson, O. Breuleux, P.- L. Carrier, K. Cho, Chorowski, P. Christiano, T. Cooijmans, M.-A. Côté, M. Côté, A. Courville, Y. N. Dauphin, O. Delalleau, J. Demouth, G. Desjardins, S. Dieleman, L. Dinh, M. Ducoffe, E. Kahou, D. Erhan, Z. Fan, O. Firat, M. Germain, X. Glorot, Goodfellow, M. Graham, C. Gulcehre, P. Hamel, I. Harlouchet, J.-P. Heng, B. Hidasi, S. Honari, A. Jain, S. Jean,
V. Dumoulin,
K. Jia, M. Korobov, Kulkarni, A. Lamb, P. Lamblin, E. Larsen, C. Laurent, S. Lee, S. Lefrancois, S. Lemieux, N. Léonard, Z. Lin, J. A. Livezey, C. Lorenz, J. Lowin, Q. Ma, P.-A. Manzagol, O. Mastropietro, R. T. McGibbon, R. Memisevic, B. van Merriënboer, V. Michalski, M. Mirza,
A. Orlandi, C. Pal, R. Pascanu, M. Pezeshki, C. Raffel, D. Renshaw, M. Rocklin, A. Romero, M. Roth, P. Sadowski, J. Salvatier, F. Savard, J. Schlüter, J. Schulman, G. Schwartz, 1. V. Serban, D. Serdyuk, S. Shabanian, Simon, S. Spieckermann, S. R. Subramanyam, Sygnowski, J. Tanguay, G. van Tulder, J. Turian, S. Urban, P. Vincent, F. Visin, H. de Vries, D. Warde-Farley, J. Webb, M. Willson, K. Xu, L. Xue, L. Yao, S. Zhang, and Y. Zhang. Theano: A python framework for fast computation of mathematical expressions, 2016. URL URL http::ttp////iie httpaaawiiv....gaaaaaa aba/ www.aa////....0000
S. Tecrapitaaaanon B. McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. CoRR, abs/1709.01666, 2017. URL http://arxiv. www.ooggg//bbbooeeeeeeeeeeeeeeee abs/ 1709
01686 14
N. C. Thompson and S. Spanuth. The decline of computers as a general purpose technology. Commun. ACM, 64(3):64-72, feb 2021. ISSN 0001-0782. doi: 10.1145/3430936 URL https //doi. odgoi.org/.... 001111//34400
S. Tokui, R. Okuta, T. Akiba, Y. Niitani, T. Ogawa, S. Saito, S. Suzuki, K. Uenishi, B. Vogel, and H. Y. Vincent. Chainer: deep learning framework for accelerating the research cycle, 2019. URL
http://arxiv.cmm org/ /bs///b///// /ates///9855 19089990..0 00203
L. Tunstall, L. von Werra, and T. Wolf. Natural Language Processing with Transformers: Build- ing Language Applications with Hugging Face. O'Reilly Media, Incorporated, 2022. ISBN 1098103246. URL http::t//bbooo
google. wtt.//ooss7..///yyya.a.....
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, S. Koura, A. Sridhar, T. Wang, and L.. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL https
//arxiv. farxiv.orgee www.:o/ggg/..o......11 01068 [
H. Zhu, A. Phanishayee, and G. Pekhimenko. Daydream: Accurately estimating the efficacy of optimizations for DNN training. In 2020 USENIX Annual Technical Conference (USENIX ATC 20), pages 337-352. USENIX Association, July 2020. ISBN 938 URL
14
13


Reviews:
[{"review_id": "4C0gEDBnzt", "reviewer": "Reviewer_FGUA", "summary": "The authors study the portability of a core set of PyTorch, TensorFlow, and JAX functions across hardware platforms, specifically GPUs and TPUs. They find significant fractions of functions fail to run on a given platform fully, partially, or within a tolerable latency. They provide the generated dataset.", "questions": "- What versions of the software were used?\n- Is it possible to add comparisons to CPU (x86 or ARM)?", "limitations": "Yes, the authors have addressed limitations and broader societal impacts.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Timely analysis of portability of widely used ML software frameworks across different hardware devices\n- Significant and surprising results that can go a long way toward shining a light on the issue and improving the current situation\n- Well-written and clear exposition\n- The public dataset release is highly beneficial for the community.", "weaknesses": "- For reproducibility, it would be useful to detail the exact software versions used for the benchmarking explicitly\n- No CPU comparisons are made, which would be quite interesting to add given that many ML applications are run on CPUs (both x86 and ARM)\n- In related work, there could be some discussion of the emerging ideas around FAIR for ML (https://www.rd-alliance.org/groups/fair-machine-learning-fair4ml-ig) and FAIR for research software (https://www.rd-alliance.org/groups/fair-research-software-fair4rs-wg) ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal (FGUA)\n\nWe thank the reviewer for their review and constructive feedback. We appreciate your recognition of the timely importance of our analysis of the portability of ML software frameworks across distinct hardware devices. It is great that you found our paper provides “Significant and surprising results that can go a long way toward shining a light on the issue and improving the current situation.” These results, as you pointed out, expose the existing challenges in the domain. Your acknowledgment of our “Well-written and clear exposition” and “The public dataset release is highly beneficial for the community” motivates us to continue this path.\n\nFrom reviewer FGUA, we are rebutting the following claims:\n\n1. **\"What versions of the software were used\"**: We clarify that the Framework’s version: TensorFlow version 2.11.0, PyTorch version 1.12.0, and (3) JAX version 0.4.8. We will add these details to our paper, and thank the reviewer for pointing out this inadvertent omission.\n2. **\"Is it possible to add comparisons to CPU (x86 or ARM)?\"**: Unfortunately, we were unable to complete these during the time allowed for the rebuttal – we prioritized the additional hardware variants on GPU since it takes time to run all functions and variants. However, we are happy to commit to adding these experiments to the final manuscript and agree that this is an important variant.\n3. **\"In related work, there could be some discussion of the emerging ideas around FAIR for ML (https://www.rd-alliance.org/groups/fair-machine-learning-fair4ml-ig) and FAIR for research software (https://www.rd-alliance.org/groups/fair-research-software-fair4rs-wg)\"**: We thank the reviewer for the suggestions for a wider body of work that would be good to highlight. The two suggestions FAIR for ML (https://www.rd-alliance.org/groups/fair-machine-learning-fair4ml-ig) and FAIR for research software (https://www.rd-alliance.org/groups/fair-research-software-fair4rs-wg) relate to open science initiatives and multi-institutional collaborations to improve findability, accessibility, interoperability, and reuse of research software. These are important motivators of our current work. We will add these to the final manuscript as adjacent directions to understand the role of software in ensuring fairness.\n"}, {"review_id": "MbXJ2YSr1B", "reviewer": "Reviewer_Twor", "summary": "This paper studies the portability of three different ML frameworks (PyTorch, TensorFlow, and JAX) across different hardware types (GPUs and TPUs). The authors sample a variety of functions implemented in these frameworks and test to what extent these functions are interoperable across platforms, and also compare the function latency between the hardware types. The evaluation finds that many functions are either unimplemented, fail to execute, or execute far more slowly on one hardware type versus the other. For example, the authors report that JAX functions are almost always faster on TPUs as opposed to GPUs.", "questions": "- Why are certain functions so much slower on TPUs than GPUs and vice versa? This question may be difficult to answer given some kernel implementations are black box, but for kernels where source code is available it would be helpful to understand the performance behavior in more detail. For example, do certain kernels more effectively use hardware caches on GPUs vs TPUs?\n", "limitations": "The paper does not discuss limitations in much detail, but does discuss broader impacts. I do not foresee any negative societal impacts of this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- In general, portability is a key practical concern and therefore the paper provides a valuable contribution for the ML community. \n- Methodology for selecting functions for evaluation is comprehensive and the benchmarking is carefully executed.\n- Thorough evaluation of GPU vs TPU execution reveals many interesting insights about function performance across hardware types.\n- Paper includes granular analysis of failure cases.\n", "weaknesses": "- While portability is a very relevant and practical concern for ML practitioners, this paper only explores portability in the dimension of GPU vs TPU execution within the same framework. Another related dimension is how well functions (and more broadly models) map from one framework to another; for example, if I have a convolution kernel written in PyTorch executing on a GPU, how comparable is this kernel’s execution to a similar convolution operation in TensorFlow? While frameworks may share underlying cuDNN kernel implementations under the hood, it would be interesting to see to what extent this symmetry exists (it appears that some basic version of this analysis could already be performed using the data in Tables 4, 5, and 6). An extension of this dimension is how well frameworks operate with intermediate representation (IR) libraries like ONNX and OpenXLA. In particular, a helpful experiment might be taking a list of functions in PyTorch, exporting them to an IR and then exporting that IR to TensorFlow, and seeing what percentage of functions fail to execute, produce different outputs, and/or experience slowdowns. \n- The paper only evaluates two specific hardware types: T4 GPUs and v3-8 TPUs. While the evaluation revealed significant disparities even between these platforms, the analysis would be even more compelling if expanded to include additional GPU / TPU types or new hardware types (e.g. AMD and/or Cerebras machines). \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal (Twor)\n\nWe thank reviewer Twor for their feedback and for noting the core strengths of our work, including \"portability is a key practical concern and therefore the paper provides a valuable contribution for the ML community\" and \"evaluation of GPU vs TPU execution reveals many interesting insights about function performance across hardware types.\" We appreciate them mentioning the importance of portability. We believe portability is of extreme importance, and quantifying the lack of portability is a valuable contribution. We also appreciate review 2 highlighting the comprehensiveness of our methodology and benchmarking: \"Methodology for selecting functions for evaluation is comprehensive and the benchmarking is carefully executed.\" Specifically, the reviewer mentioned our comparison of operations on GPU vs TPU and the granular categorization of failure modes.\n\nFrom reviewer Twor, we are addressing the following claims:\n\n1. **\"While portability is a very relevant and practical concern for ML practitioners, this paper only explores portability in the dimension of GPU vs TPU execution within the same framework. Another related dimension is how well functions (and more broadly models) map from one framework to another; for example, if I have a convolution kernel written in PyTorch executing on a GPU, how comparable is this kernel's execution to a similar convolution operation in TensorFlow?\"**: We choose to focus our effort on hardware portability instead of framework portability because we believe it is the larger, more relevant problem. Software, and specifically frameworks, have no physical limitations on usage. While it may take the developer time to switch from framework to framework, this is a question of bits. Hardware is in the physical world. And as such, you are often limited by the devices you own or the amount of budget you have to access. While it would be fascinating to dive into how frameworks map to the underlying operations in CUDA and other compilation targets, we believe the first step was our process to find latencies and failure rates within a framework. Between the experimental cost of time and the compelling narrative these results gave, we believe this was a sufficient step to release this paper. We believe this is a promising direction for future research and will add it to our limitations and future work section in the manuscript.\n2. **\"An extension of this dimension is how well frameworks operate with intermediate representation (IR) libraries like ONNX and OpenXLA. In particular, a helpful experiment might be taking a list of functions in PyTorch, exporting them to an IR and then exporting that IR to TensorFlow, and seeing what percentage of functions fail to execute, produce different outputs, and/or experience slowdowns.\"**: While we agree exploring compilation targets and intermediate representations is a useful direction for this research, we believe it is a rich direction of future work. Here, our primary goal is to provide a primary rigorous contribution to evaluate the portability as a preliminary step to motivate such future work. Our experimental setup is already costly, requiring hand coding verifying, and running in a total of 1,146 experiments.\n3. **\"The paper only evaluates two specific hardware types: T4 GPUs and v3-8 TPUs. While the evaluation revealed significant disparities even between these platforms, the analysis would be even more compelling if expanded to include additional GPU / TPU types or new hardware types (e.g. AMD and/or Cerebras machines).\"**: Agreed that more device types make for a stronger narrative when it comes to our results. We have taken the rebuttal period to incorporate the feedback of the reviewer and run additional experiments. In the attached pdf, we include a table showing failure rates for both the A100s and the T4s. Please see Table 1 in the attached file for the evaluation of A100s.\n4. **\"Why are certain functions so much slower on TPUs than GPUs and vice versa?\"**: We thank the reviewer for the interesting question. To deeply understand why specific functions perform differently on TPUs and GPUs, we would need a granular profile of the operations down to the hardware level. However, in some cases, we do not have access due to the closed-source nature of CUDA and XLA code. Given that and the time cost of doing our current experiments, we believe that understanding why certain latency differences exist is beyond the scope of this paper. That said, even if the exact reasons for these lags are not fully investigated due to proprietary software barriers, recognizing their existence is crucial for the ML community.\n"}, {"review_id": "kuoRn3hTUk", "reviewer": "Reviewer_fKz5", "summary": "The paper explores the combination between hardware and software for machine learning, and ask the question:  How portable are popular ML software frameworks?  The authors conduct a quantity study of the portability of mainstream ML frameworks across different hardware types. With the experiment result, they claim that machine-learning frameworks can lose more than 40% of their key functions when ported to other hardware, and significantly slowdown if portable.  Collectively, the results reveal how costly straying from a narrow set of hardware-software combinations can be and suggest that specialization of hardware impedes innovation in machine learning research.", "questions": "1. Paper significance: How the lack of portability of software frameworks is significant in ML? It is not easy to understand since there are no realistic evidence. What does the quantity result suggest comparing to the intuitive result? We would like to know more about why quantity is important here.\n2. Methods: Why the paper uses the framework operations instead of NN workloads for evaluation? The codes need to be manually check to port on different hardware.", "limitations": "The authors have not discussed their limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The paper raise a concern for lack of machine learning portable quantity, which is not payed enough attention to in the current research.\n2. This paper reports large-scale experimental results to quantity the hardware-software combination, which can be reference for the industry developers.", "weaknesses": "1. The lack of portability with different hardware/software structures is widely known knowledge. For example, APPs work on Arm/Android systems is difficult to work on the X86/Windows system. Under mature commercial circumstances, the paper lacks proving why portable is important and necessary in ML.\n2. The paper concludes, “Specialization of hardware impedes innovation in machine learning research”, which is not been proved in this paper. According to the No Free Lunch theory in computer architecture studies, domain-specific hardware needs to find a tradeoff between generality and efficiency on specific workloads. It is not realistic to design hardware supporting every workload as well as other hardware frameworks, while still keeping its own advantage in some domains.\n3. The paper needs to focus more on the quantity standard for ML portable. The method now only considers the slowdowns of the software framework APIs, but not the actual tasks or NNs (such as Resnet or VGG). The standard is confusing and the authors need more analysis or evidence.\n4. The paper is not easy to follow. It is not clear to understand the significance and originality of the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal (fKz5):\n\nWe thank the reviewer for their feedback, including their positive observation that this work \"raise[d] a concern for lack of machine learning portable quantity, which is not payed enough attention to in the current research\" and the positive view of the experimental rigor and breadth of our work. These experimental results were the core of our research. We believe we are among the first to quantify the lack of portability between machine learning frameworks. Moreover, we will release the dataset to the ML community.\n\nFrom reviewer fKz5, we are rebutting the following claims:\n\n1. **\"The lack of portability with different hardware/software structures is widely known knowledge.\"**: While knowledge of hardware disparities may be known by some, we believe the magnitude of the findings presented in this paper are not known. We believe our findings are striking: PyTorch and TensorFlow have portability issues across GPUs and TPUs, with performance discrepancies over 10x slowdown when moved from GPU to TPU for PyTorch; meanwhile, JAX functions, designed with TPUs in mind, run 91.8% faster on TPUs. Our contribution is to offer a rigorous evaluation and benchmark this issue, which is in keeping with the goals of the Evaluation track.\n2. **\"Under mature commercial circumstances, the paper lacks proving why portable is important and necessary in ML.\"**: Our primary focus is on the portability of ML frameworks for researchers, as stated in Lines 29-30. We are interested in how a lack of portability can increase the barrier of entry for new ideas. A lack of software portability means that researchers are locked into certain hardware/software combinations. This can bias against certain ideas and advantage others. This has been widely discussed in research, including Hooker et al. (2021) and Barham et al. (2019).\n\n   The effect on developers and researchers of significant differences in hardware latency is that ideas overfit certain tooling stacks and fail to transfer to other software stacks. This can turn the act of writing machine learning code into an arcane art of understanding framework sharp edges and hardware complexity.\n\n3. **\"The method now only considers the slowdowns of the software framework APIs, but not the actual tasks or NNs (such as Resnet or VGG)\"**: Our objective was to develop an analysis that would broadly cover a wide variety of workloads. A major concern when overly focusing on popular architectures or tasks is that we might sideline the diverse range of code and ideas that researchers are exploring, some of which might not have reached popularity or high optimization levels. To address our concerns about workload representation, we sampled our functions from a distribution based on function frequency. This was designed not to underestimate workloads, thus ensuring a balanced view of the performance landscape across the range of ideas researchers may want to try.\n\n   Analyzing workloads instead of individual functions also poses additional challenges in terms of the rigor of our analysis. We will add the following reasoning to the final manuscript to make more clear:\n\n   1. Sometimes, it is not possible to analyze. For example, if we have input x, go through three functions F, G, and H in the order F(G(H(x))). If the middle function, which is G in this case, fails because it is not portable, we will not be able to test the function F.\n   2. Different workloads also use different framework versions. Therefore, if we use a deprecated function, we might face (1).\n   3. It is biased. The function X might work on the common task, but it might not work in the niche case. Therefore, the test case itself is much more thorough and thus more suitable for testing.\n   4. Operations are the building blocks of ML workloads. The performance and portability of the operations directly impact the workloads that use them.\n\n4. **\"Domain-specific hardware needs to find a tradeoff between generality and efficiency on specific workloads. It is not realistic to design hardware supporting every workload as well as other hardware frameworks, while still keeping its own advantage in some domains\"**: While it does not make sense to implement ML operations for every feasible piece of hardware, it is not an unrealistic observation to point out that the current state of ML frameworks can increase friction in the adoption and exploration of ideas across hardware because of the severe drops in performance.\n5. **\"How the lack of portability of software frameworks is significant in ML? It is not easy to understand since there are no realistic evidence. What does the quantity result suggest comparing to the intuitive result? We would like to know more about why quantity is important here.\"**: We motivate our work by citing prior works (Hooker et al. (2021) and P. Barham et al. (2019)), which make a strong case for why the lack of portability is significant. As for the quantities themselves, we believe quantifying latency differences over multiple runs tells a compelling story about the portability differences between frameworks.\n6. **\"The authors have not discussed their limitations.\"**: We thank the reviewer for pointing this out! Our research has three limitations. Firstly, we used only one type of GPU and TPU. During the rebuttal, we added an additional variant, which we believe mitigates some of the limitations of the original submission. Secondly, our current focus is on portability for researchers, so a future direction would be to focus on commercial/application-based portability, which would include analysis involving CPUs. Lastly, we couldn't discern the underlying reasons for the observed performance mitigation when transferring functions across devices since we don't have access to the source code (CUDA is not open source). Exploring this aspect would be valuable for the ML community. We will add these limitations in our final manuscript."}]

Author rebuttal:
# Global Response

We thank all reviewers for taking the time to evaluate our manuscript. We appreciate the recognition of the timeliness and relevance of our study (R fKz5 and FGUA), emphasizing the often overlooked issue of ML software framework portability across different hardware types (R fKz5). It is encouraging to see the consensus on the value of our large-scale experimental results, which can serve as a reference for both academia and industry (R fKz5, Twor, FGUA). The thorough and careful evaluation methodologies employed, particularly our comprehensive approach to function selection and benchmarking (R Twor), as well as our granular analysis of failure cases (R Twor), are recognized positively. Furthermore, we are grateful for acknowledging our contribution in revealing significant insights about function performance across GPUs and TPUs (R FGUA, Twor).

As reviewer (Twor) has suggested we have included a pdf with results for another type of GPUs. We now include both T4 and A100 GPU results. We believe this makes our research even stronger and we hope to extend to more types of devices in the future.


Meta-review: I have some reservations to accept this paper but given the overall support from the reviewers, I recommend acceptance for the importance of the topic which is not discussed in ML conferences enough.

My main reservation is that the results are not very surprising. Jax was built on a solid foundation of XLA. Thus it is more portable than other frameworks by design. 99 % of the PyTorch users do not have access to TPUs. Thus it is not surprising that the operations are not supported on TPUs. TensorFlow's lack of portability is a bit disappointing. One can only speculate that this must be a consequence of Google's internal resourcing decisions.

On the methodology, I have the following questions

1. Does it make sense that the top 20 functions are chosen by the frequency of usage rather than their impact to some representative model's runtime? For example, certainly `torch.cuda` is used a lot but I don't expect this function to be a bottleneck.
2. Does it make sense that the functions are sampled independently for each framework? For example, `svd` is included in tensorflow and jax but `torch.svd` is not. I think the authors should benchmark important functions for all frameworks.
3. Does it make sense that the authors benchmark the unit tests? Inputs to unit tests are usually small and the performance on them can be very different from the performance on typical inputs to these functions in a real model.
Decision: Accept (poster)