# Memory-Efficient Batching for Time Series Transformer Training: A Systematic Evaluation
## 1 Introduction  
The popularity of transformer-based models is evident and has been gaining more traction in several industries in recent years. These large-scale pre-trained models have the ability to learn general representations from vast amounts of data and generalize across a wide range of downstream tasks. Applications of these models in the natural language processing (NLP) domain are emerging at an unprecedented rate, ranging from conversational agents [1, 2, 3] and machine translation [4, 5, 6] to content generation [7, 8, 9] and sentiment analysis [10, 11], proving the usability of these models in real-world settings. In addition to the natural language processing domain, transformer architectures havebeen adopted by closely related fields such as computer vision [12; 13; 14; 15], code [16; 17; 18], and increasingly time series [19; 20; 21]. In time series data analysis, transformer-based models have emerged as a powerful tool for extracting meaningful patterns and insights from sequential data. These models exhibit remarkable capabilities in tasks such as forecasting [20; 22], anomaly detection [21; 23], and classification [19; 24]. However, as the size of datasets grow, training time series transformer models becomes increasingly memory intensive, particularly when using sliding-window approaches to expose temporal dependencies. Although these methods are commonly used because of their effectiveness in capturing sequential patterns, they introduce substantial memory and computational overhead during batch preparation. Conventional training pipelines generate overlapped sliding windows on the CPU and then transfer stacked tensors to the GPU, resulting in redundant memory copies and inefficient data transfer. Although asynchronous data loaders can partially mask transfer costs, they do not eliminate data duplication. As batch sizes, sequence lengths, and feature dimensions increase, these inefficiencies become significant bottlenecks for scalability, especially in multivariate forecasting tasks [25; 26; 27; 28]. To address this limitation, this paper revisits the often-overlooked issue of batching overhead and proposes an effective solution, view-based sliding-window batching. By replacing default tensor stacking operations with zero-copy tensor views, we eliminate redundant data materialization without changing model architectures or affecting downstream accuracy, while view-based operations have long existed in frameworks like PyTorch [29], their integration into large-scale training pipelines for time series models has not been systematically explored. We develop a data preparation pipeline ([[https://github.com/psinthong/ME_Batch_Formation](https://github.com/psinthong/ME_Batch_Formation)]([https://github.com/psinthong/ME_Batch_Formation](https://github.com/psinthong/ME_Batch_Formation)) (accessed on 3 May 2025)). that applies view-based sliding-window optimizations to GPU-resident data for training time series transformer models, while our focus is on transformer architectures, we also validate the method's broader applicability using a simple linear forecasting model. Although existing memory optimization methods, such as quantization, tensor rematerialization, and asynchronous CPU-GPU data loading, have shown significant progress, they primarily target model-level optimization or computational overlap techniques. Our work optimizes the batching pipeline itself by performing a GPU-based, zero-copy sliding-window formation using tensor views. By eliminating redundant data materialization, our approach reduces memory usage and runtime without sacrificing model performance. Through extensive empirical analysis, we demonstrate that view-based batching yields non-trivial runtime and memory savings. We evaluate its impact across varying batch sizes, sequence lengths, feature dimensions, and model architectures. This evaluation offers a comprehensive view of its practical effectiveness. Our contributions are the following.