## 3 Methodology and Materials
### Applying with Attention Mechanisms
#### 3.2.2 Applying Self-Attention with Novel Cycling Layer  
However, while self-attention mechanisms offer significant advantages, they still rely on positional encodings to maintain the order of the elements in a sequence. These positional encodings are designed to capture the linear order of data points but do not inherently account for cyclical patterns, such as daily, weekly, or seasonal cycles, which are crucial in many real-world time series datasets. Moreover, the quadratic complexity of self-attention can make it computationally expensive for very long sequences, limiting its efficiency in practical applications. In light of this, we advocate for adopting cycling encoding as a substitute for positional self-attention. This method refines the attention's focus based on the temporal characteristics of the target data, ensuring a more nuanced and contextually relevant representation. Based on our analysis of weather data with seasonal features incorporated into the positional layer Figure 1, we observed inconsistencies in the step length of the positional layer. While a straightforward numerical step relationship may be effective for dynamic features, such as language, it is less suitable for capturing the temporal characteristics of time series data. To address this issue, we have developed a cycling layer. We anticipate that our cycling layer, with its consistent step length, will more accurately capture the seasonal features inherent to time series data. The advent of the cycling layer self-attention signifies a pivotal advancement in unsupervised machine learning, particularly in preprocessing time series data. This innovative architectural layer is motivated by the intrinsic need to capture the cyclical nature of temporal data more effectively. Traditional layers, such as positional encodings, handle sequential information but often need more capability to grasp cyclical patterns critical in datasets, such as weather patterns, where features fluctuate periodically. The cycling layer is specifically designed to address this gap. Unlike the one-size-fits-all approach of traditional positional encoding, it provides a framework to isolate and understand the seasonal and daily cycles inherent in time series datasets. The flexibility of the cycling layer allows for the manual adjustment of periodic time features, enabling a more tailored and precise capture of cyclical trends. This ability to adjust and focus on specific cycle lengths with a reduced risk of overfitting irrelevant patterns represents a significant improvement over the traditional method. Furthermore, the simpler construction of the cycling layer compared to more complex positional encoding mechanisms can potentially reduce computational overhead and streamline the preprocessing phase, leading to cost efficiencies in both time and resources. Suppose that the input representation \(x_{i}\in\mathbb{R}^{n\times d}\), each value \(x\) in \(X\) is a timestamp \(t\). Here, the cycling period represents the total time for one full cycle, such as 24 h for daily cycles. The result \(t\) / will be a fraction between 0 and 1 representing the position of the timestamp within the cycle. Then, convert the fraction of the cycle \(t\) / to radians, representing the position within the cycle in radians, covering a full circle from 0 to 2\(\pi\). This equation (Equation (5)) transforms the timestamp \(t\) into two values, \(sin_{t}\) (Equation (6)) and \(cos_{t}\) (Equation (7)), which together represent the cyclic nature of the timestamp. \[\theta=\left(\left(\frac{t}{cycling\ period\ time}\right)mod\ 1\right)\times 2\pi \tag{5}\]  
\[sin_{t}=sin(\theta) \tag{6}\]  
\[cos_{t}=\cos(\theta) \tag{7}\]  
The cycling layer is a novel construct in the architecture of machine learning models designed for processing time series data. Algorithm 3 begins with the input tensor \(x\), which is assumed to have a shape of (batch_size, sequence_length, embed_dim), where each value in \(x\) represents a timestamp.