SlowGuess/NeurIPS_2023_OCR huggingface dataset:
There are unique circumstances here that might make this different from using these frameworks in real-life situations (specifically in your standard training situation, you have long-running processes, and in our case, we are running simple functions that finish quickly), but this is clear that the benefits of switching to specialized hardware can be uneven and variable.
Deep learning frameworks: The rapid adoption and commercial success of Deep learning has spurred the development of software frameworks tailored to deep neural network workloads. Many of the most widely used libraries for machine learning workloads are Python libraries like TensorFlow [Abadi et al., 2015], Theano [Team et al., 2016], Chainer [Tokui et al., 2019], MXNet [Chen et al., 2015], PyTorch [Paszke et al., 2019] and JAX [Bradbury et al., 2018]. Despite the variety in frameworks, there has been no study to our knowledge of the difficulty of porting these frameworks between different types of hardware.
Narrowing of AI research: The specialization of hardware to create efficiencies for machine learning workloads has created concerns about a narrowing in research contributions. Recent work [Hooker, 2021; Barham and Isard, 2019] suggests that inflexible high-performance kernels and limited programming abstractions are hindering innovative machine learning research. [Hooker, 2021] argues that the availability of accelerator hardware determines the success of ML algorithms potentially more than their intrinsic merits – that the success of ideas hinges on alignment with hardware on software. [Klinger et al., 2022] analyzes arXiv papers and finds that AI research has stagnated in recent years and that AI research involving the private sector tends to be less diverse and more influential than research in academia. Several works [Ahmed and Wahed, 2020] point to the growing compute divide, which impacts accessibility to research and ideas.
Portability of software frameworks: Different designs for technology are possible, and some designs are more desirable from an innovation standpoint than others [David et al., 2009]. However, circumstances such as chance events, shortsightedness, and lack of coordination can lead to a
Table 4: Comparison of failure rates between functions in the top 20 and the overall failure rate across all deciles.
TensorFlow	PyTorch	JAX
GPU	Top 20	33%	0%	0%
All Functions	22%	10%	2%
TPU	Top 20	27%	46%	0%
All Functions	30%	44%	4%
situation where an inferior design becomes dominant and difficult to transition away from, even after its limitations become evident [Arthur, 1994, David, 1985]. In the face of uncertainty regarding the advantages and risks associated with different technologies and the desire to avoid getting stuck with an inferior design prematurely, it might be sensible to implement policies that maintain diversity in the technological landscape [David et al., 2009]. A third and final reason to preserve technological mobility and reduce the cost of exploration: innovation involves the creative recombination of ideas, and unusual mixes are often an important source of radical and transformative innovations [Arthur, 2011].
While our work does a great deal to quantify existing gaps in portability, it has some important limitations. Firstly we recorded latency calculations and failure categories on two types of GPUs (A100s and T4s) and two types of TPUs (v2-8 and v3-8). We believe the similar error rates between types of GPUs show that at least for failure rates there is a good deal of consistency between types of GPUs. Worthwhile extensions of this work would include adding more device types to get a more robust view of overall portability and its trend.
Secondly, this paper does not explore in depth why these portability gaps exist. We provide some broad hypotheses on why there might be differences in Section 3.2 but we leave it to future work to pinpoint why these differences exist. One reason for our limitation is due to the lack of access to CUDA internals as it is not completely open source. Understanding the differences in kernels between devices and framework implementations is a daunting task and outside of the scope of this work.
We benchmark three widely used and adopted machine learning libraries to evaluate the ease of portability across different hardware types. We find large differences in the subset of software operations supported on different types of hardware. We find that PyTorch and TensorFlow, in particular, have pronounced portability issues. On GPUs,  $22\%$  of the TensorFlow benchmark functions fail partially or completely. On TPU, a remarkable  $44\%$  of PyTorch benchmark functions partially or completely fail. Even where there is portability, significant gaps exist in performance between each framework. We observe that when transferring functions from GPU to TPU,  $81.4\%$  of functions in PyTorch exhibit more than 10x slowdown.
Significant work remains to ensure portability and performance between device types. Currently, major slowdowns and broken operations are the norms, and widely used frameworks have overpromised when it comes to portability. This lack of portability has costs to innovation: incentivizing researchers to stick with the tooling they have, which often makes it harder to stray off the beaten path of research ideas. Innovation often occurs when it is cheaper to explore, but our tooling stacks have clearly quantifiable friction that deters exploration. Valuable future work includes developing standardized approaches to machine learning tooling that enable greater portability between different hardware types, and benchmarking additional hardware types. We hope by releasing our benchmark dataset, we can spur greater visibility into what frameworks need more support.
T. T. D. Team, R. Al-Rfou, G. Alain, A. Almahairi, C. Angermueller, D. Bahdanau, N. Ballas, F. Bastien, J. Bayer, A. Belikov, A. Belopolsky, Y. Bengio, A. Bergeron, J. Bergstra, V. Bisson, J. B. Snyder, N. Bouchard, N. Boulanger-Lewandowski, X. Bouthillier, A. de Brébisson, O. Breuleux, P.-L. Carrier, K. Cho, J. Chorowski, P. Christiano, T. Cooijmans, M.-A. Côté, M. Côté, A. Courville, Y. N. Dauphin, O. Delalleau, J. Demouth, G. Desjardins, S. Dieleman, L. Dinh, M. Ducoffe, V. Dumoulin, S. E. Kahou, D. Erhan, Z. Fan, O. Firat, M. Germain, X. Glorot, I. Goodfellow, M. Graham, C. Gulcehre, P. Hamel, I. Harlouchet, J.-P. Heng, B. Hidasi, S. Honari, A. Jain, S. Jean, K. Jia, M. Korobov, V. Kulkarni, A. Lamb, P. Lamblin, E. Larsen, C. Laurent, S. Lee, S. Lefrancois, S. Lemieux, N. Léonard, Z. Lin, J. A. Livezey, C. Lorenz, J. Lowin, Q. Ma, P.-A. Manzagol, O. Mastropietro, R. T. McGibbon, R. Memisevic, B. van Merrienboer, V. Michalski, M. Mirza, A. Orlandi, C. Pal, R. Pascanu, M. Pezeshki, C. Raffel, D. Renshaw, M. Rocklin, A. Romero, M. Roth, P. Sadowski, J. Salvatier, F. Savard, J. Schlüter, J. Schulman, G. Schwartz, I. V. Serban, D. Serdyuk, S. Shabanian, Simon, S. Spieckermann, S. R. Subramanyam, J. Sygnowski, J. Tanguay, G. van Tulder, J. Turian, S. Urban, P. Vincent, F. Visin, H. de Vries, D. Warde-Farley, D. J. Webb, M. Willson, K. Xu, L. Xue, L. Yao, S. Zhang and Y. Zhang: Theano: A python framework for fast computation of mathematical expressions, 2016. URL http://arxiv.org/abs/1605.02688
S. Teerapittayanon, B. McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. CoRR, abs/1709.01686, 2017. URL http://arxiv.org/abs/1709.01686 [14]
N. C. Thompson and S. Spanuth. The decline of computers as a general purpose technology. Commun. ACM, 64(3):64-72, feb 2021. ISSN 0001-0782. doi: 10.1145/3430936. URL https://doi.org/10.1145/3430936.
S. Tokui, R. Okuta, T. Akiba, Y. Niitani, T. Ogawa, S. Saito, S. Suzuki, K. Uenishi, B. Vogel, and H. Y. Vincent. Chainer: A deep learning framework for accelerating the research cycle, 2019. URL http://arxiv.org/abs/1908.00213.
L. Tunstall, L. von Werra, and T. Wolf. Natural Language Processing with Transformers: Building Language Applications with Hugging Face. O'Reilly Media, Incorporated, 2022. ISBN 1098103246. URL https://books.google.ch/books?id=7hhyzgEACAAJ.4
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL https://arxiv.org/abs/2205.01068 [1]
H. Zhu, A. Phanishayee, and G. Pekhimenko. Daydream: Accurately estimating the efficacy of optimizations for DNN training. In 2020 USENIX Annual Technical Conference (USENIX ATC 20), pages 337-352. USENIX Association, July 2020. ISBN 978-1-939133-14-4. URL https://www.usenix.org/conference/atc20/presentation/zhu-hongyu. [14]