###### Abstract  
Due to the recent growing data size of deep learning workloads, loading data from storage is increasingly becoming a performance bottleneck for neural network systems. In this article, we analyze the data reference characteristics of neural network workloads and observe that they are significantly different from conventional desktop workloads. In particular, during the training phase of deep learning, data blocks are referenced in a fully random manner, which significantly degrades the performance of a buffer cache. To handle this situation, this article suggests a new data shuffling scheme that aims to accelerate data loading in deep neural networks. Unlike the default shuffling method used in PyTorch that randomly shuffles full dataset in every epoch, the proposed scheme defines a shuffling unit called bundle, and enhances the locality of data references to improve buffer cache performances. Specifically, the proposed scheme performs data shuffling by the unit of a bundle, and the bundles used in each epoch are arranged alternately, thereby improving the locality of references at the viewpoint of the buffer cache. Based on simulation and measurement studies, we show that the hit rate of the buffer cache is improved by 37.2%, and the data loading time is also shortened by 11.4% without degrading the model's training efficiency. buffer cache; deep learning; data reference; data shuffling; neural network; performance +
Footnote †: journal: Applied Sciences  
for neural network workloads where temporal locality-based algorithms such as LRU do not perform well [9]. In this article, we extract data reference traces of neural network workloads and observe some special reference characteristics while training is performed. In particular, all data in the dataset are equally referenced in a random manner at each epoch. We observe that these reference characteristics degrade the buffer cache performance significantly when traditional algorithms based on temporal locality are used. To cope with this situation, we present a new data shuffling scheme for improving buffer cache performance in neural network systems. Unlike traditional shuffling used in neural network training, which randomly shuffles all data in every epoch, the proposed scheme defines shuffling units that we call bundles, to improve the locality of data references without degrading the model's training efficiency. Our scheme performs data shuffling by the unit of bundle, and the bundles used in each epoch are arranged alternately, thereby improving the locality of data references at the viewpoint of the buffer cache. Figure 1 shows a brief overview of the proposed shuffling scheme in comparison with the default shuffling used in PyTorch. By utilizing the data shuffling scheme presented in this article, we show that the hit rate of the buffer cache can be improved by 37.2% on average, and the data loading time can also be accelerated by 11.4% while training is being performed. Figure 1: An overview of the proposed shuffling scheme in comparison to the default shuffling in PyTorch. The main contributions of this article can be summarized as follows. * First, we analyze the data reference characteristics of deep learning workloads from a buffer cache's perspective and find out that block references generated during deep learning's training significantly degrade buffer cache performance. * Second, we observe that data blocks in deep learning workloads are referenced in a completely random manner in each epoch, which causes performance impairments in current buffer cache systems that focus on temporal locality of references. * Third, we propose a new data shuffling scheme that groups a portion of data blocks into bundles and performs bundle-based shuffling to improve the locality of data block references. In addition, we alternately select bundles for each epoch to maximize temporal locality. The remainder of this article is organized as follows. Section 2 discusses the data reference characteristics of neural network workloads based on trace analysis.