# Memory-Efficient Batching for Time Series Transformer Training: A Systematic Evaluation
## 3 Related Works  
In terms of memory optimizations for transformer models, the innovations in this field fall into three main categories: quantization, layer optimization, and tensor rematerialization.  
**Quantization**: Quantization reduces the precision of weights and activations within a model, typically from 32-bit floating-point numbers (FP32) to lower precision formats like 16-bit (FP16) or even 8-bit integers (INT8). This compression significantly reduces memory footprint without sacrificing significant accuracy. Post-training quantization techniques (PTQ) [33, 34] achieve this by analyzing the distribution of weights and activations and identifying suitable lower-precision representations. Alternatively, quantization-aware training (QAT) [35, 36] integrates quantization into the training process itself, allowing the model to adapt to the lower precision format.  
**Layer Optimization**: Certain layers in transformer models are inherently more memory-intensive than others. For instance, self-attention layers, with their quadratic complexity in sequence length, can consume a substantial portion of memory. Layer optimization techniques focus on reducing the memory footprint of these specific layers. One approach involves using techniques like sparse attention [37, 38], where attention weights are concentrated on a smaller subset of relevant tokens, leading to sparser activation tensors. Another strategy involves exploiting low-rank approximations for attention matrices [39, 40], effectively capturing the essential information while reducing the number of parameters to be trained.  
**Tensor Swapping and Rematerialization:** Transformer models often produce large intermediate tensors during the forward and backward passes. Tensor swapping and materialization techniques aim to optimize the management of these tensors in memory. Tensor swapping [41] leverages techniques like checkpointing [42] to temporarily store intermediate activations on secondary storage (e.g., disks) when they are not actively needed. This technique frees up precious GPU memory for computations involving other tensors. Rematerialization [43, 44], on the other hand, focuses on reducing memory overheads associated with creating these intermediate tensors altogether. Recomputing intermediate values rather than retaining them in memory is often accompanied by techniques like operation fusion which combines multiple operations into a single kernel, minimizing the number of temporary tensors generated during the computation.  
Existing batching techniques for time series transformer models typically rely on CPU-based preprocessing followed by GPU data transfers, resulting in redundant tensor replication, memory bottlenecks, and limited scalability. Even asynchronous loading methods, which overlap computation and data transfer, fail to fully eliminate data duplication. In contrast, our approach optimizes the batching pipeline itself rather than modifying model architectures or numerical precision by employing GPU-resident, view-based sliding windows through a custom data-loading pipeline, thereby entirely preventing redundant tensor materialization, seamlessly complementing other memory optimizations, and enabling integration into distributed training workflows.