## 3 Data Shuffling in Neural Networks Considering Buffer Cache  
In this section, we suggest a new data shuffling scheme for neural network workloads to improve the buffer cache performance. As observed in the previous section, data block references in neural network workloads are totally random, which degrades the effectiveness of the buffer cache. To handle this situation, we present a new data shuffling scheme in bundle units, which improves the locality of data block references. Our scheme behaves different from conventional neural network training that randomly shuffles a full dataset in every epoch. That is, we define a shuffling unit, called a bundle, and shuffles data within a bundle randomly such as conventional training, but bundles are selected sequentially in each epoch instead of randomly shuffled. To this end, we inject our code in the components of PyTorch, consisting of Dataset, DataLoader, and Sampler. The Dataset makes use of the __geitem__ method to retrieve a data sample from the corresponding index. The DataLoader loads data samples from a dataset in mini-batches by making use of a worker process. The Sampler extracts a subset of dataset samples during this process [22]. When the shuffle parameter is set to true, the DataLoader invokes RandomSampler to shuffle the entire dataset randomly, and retrieves the dataset based on this sequence. In this article, we add the BundleRandomSampler to shuffle a dataset in bundle units. The BundleRandomSampler takes the bundle size as an input parameter and obtains the offset index of each bundle in the entire dataset. Then, each bundle is shuffled based on the offsets determined. This is different from RandomSampler that shuffles all indices of the entire dataset and SubsetRandomSampler that shuffles the samples for given indices. Figure 2: Data references while training of neural networks. To set the bundle size of BundleRandomSampler, we define a parameter called bundle_ratio that is used as the input parameter to invoke DataLoader. The bundle_ratio can be set between 0 and 1, representing the ratio of the bundle size relative to the entire dataset. For example, if the bundle_ratio is 0.2, each bundle contains 20% of the entire dataset, such that there are 5 bundles in the dataset. Figure 3 depicts the data block references while training the same dataset shown in Figure 2 with the proposed shuffling scheme. As we see from this figure, intra-bundle references are randomly shuffled, whereas inter-bundle references exhibit sequential patterns from the second epoch of the training. Along with the bundle-based shuffling scheme, we suggest another simple but efficient scheme that can improve the effectiveness of buffer caching. Specifically, in each epoch, the referencing orders of bundles are reversed from the previous epoch, thereby generating the alternating sequence of bundle references. Suppose that there are 9 bundles, \(B_{1}\), \(B_{2}\), \(\ldots\), \(B_{9}\), in the full dataset and the bundles are referenced in increasing order of \(B_{1}\), \(B_{2}\), \(B_{3}\), \(\ldots\), \(B_{9}\) in the \(n\)-th epoch. Then, the reference order of the bundles in the \((n+1)\)-th epoch in our scheme is \(B_{9}\), \(B_{8}\), \(B_{7}\), \(\ldots\), \(B_{1}\). Such an alternating sequence of references is intended to improve the temporal locality effect from the perspective of the buffer cache so that data blocks that were last referenced in the previous epoch are referenced first in the next epoch. In particular, if the dataset is larger than the size of the buffer cache, data blocks cached in the previous epoch are certain to be discarded from the buffer cache by the eviction algorithm before they are referenced in the current epoch. This significantly degrades the buffer cache performance. However, our bundle alternating scheme resolves this problem by taking advantage of temporal locality to a certain level.