# Memory-Efficient Batching for Time Series Transformer Training: A Systematic Evaluation
## 6 Discussion  
In this section, we will go through each of the experimental results and analyze the behaviors we observed. Our investigation into batch memory optimizations via view-based sliding windows for GPU-resident multivariate time series data, has revealed compelling insights. The \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{5}{c}{**Model**} \\ \hline  & **PatchTST** & **Informer** & **Autoformer** & **DLinear** & **Transformer** \\ \hline \multirow{2}{*}{**Methods**} & **Optimized** & 6.97\% & 21.86\% & 19.20\% & 15.40\% & 21.99\% \\  & **Optimized in memory** & 8.48\% & 38.01\% & 30.51\% & 95.36\% & 31.50\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Average percentage reduction in end-to-end epoch wall-clock time achieved by the proposed batching methods compared to the PyTorch baseline. Figure 15: Model runtime comparison. experimental results on both memory and runtime benchmarks demonstrate the superiority of our proposed optimizations over the conventional PyTorch tensor-stacking baseline. The first set of experiments, examining memory consumption with varying feature counts, batch sizes, and sequence lengths (Figures 8-10), reveal the advantages of our two optimization variants. The \"preload\" variant, suitable when the entire dataset fits within GPU memory, exhibits a stable memory footprint regardless of any batch size or sequence length. This behavior is expected due to the one-time upfront transfer of data. On the other hand, the \"per-batch\" optimization, designed for larger datasets, demonstrates a linear relationship between memory usage and the three varying dataset parameters, aligning with the batch-wise data-loading mechanism. Even in this constrained setting, our approach consistently outperforms the baseline, particularly when handling high-dimensional time series data, showcasing its effectiveness in mitigating memory pressure. The subsequent runtime analyses (Figures 11-13) across varying numbers of features, batch sizes, and sequence lengths further solidify the efficacy of our optimizations. In all scenarios, both our optimization variants consistently surpass the PyTorch baseline in terms of runtime performance. The preloading variant, as anticipated, delivers the most substantial speedup due to the elimination of repetitive CPU-to-GPU data transfers. This result strongly advocates for preloading data onto the GPU whenever feasible. Notably, even the per-batch optimization variant, despite the unavoidable data transfers, maintains a consistent speed advantage over the baseline, emphasizing the efficiency of view-based operations in reducing computational overhead. Applying our optimization techniques on five distinct model architectures (PatchTST, autoformer, informer, transformer, and DLinear) operating in a distributed environment for the task of time series forecasting further demonstrates the generalizability and scalability of our approach. The consistent runtime improvements observed across all models, regardless of their complexity, highlight the broad applicability of our memory optimizations. However, the application of our optimizations to the PatchTST architecture revealed a unique behavior, while both variants still improved runtime compared to the baseline, the magnitude of the reduction was notably smaller than that observed for other transformer-based models (Figure 15). This can be attributed to PatchTST's internal default mechanism, which involves a form of padding and patching of input tensors. This process requires input data materialization of views. Since we have not modified any model architecture to directly take advantage of the view-based tensors, this could be a future improvement that would significantly enhance the impact of our optimizations. Despite the limitation, our approach still offers a measurable performance boost even for PatchTST, suggesting its broad applicability. A particularly distinct runtime reduction was observed for the DLinear model.