## 3 Proposed Framework: PE-DOCC
### Periodicformer Encoder for Unsupervised Representation Learning
#### 3.1.2 Criss-Cross Periodic Attention  
As shown in Figure 3, we propose the criss-cross periodic attention mechanism to mine richer periodic scale characteristics. Row autocorrelation and column autocorrelation explore periodic dependencies by calculating autocorrelation. * Period-based dependencies: Inspired by the autocorrelation mechanism [29], for an input sequence \(X_{t}\) with length \(L\), we can obtain the autocorrelation \(R_{XX}(\tau)\) by the following Equation (1): \[R_{XX}(\tau)=\lim_{L\rightarrow\infty}\frac{1}{L}\sum_{t=1}^{L}X_{t}X_{t-\tau},\] (1)  
\(R_{XX}(\tau)\) reflects the time-delay similarity between \(\{X_{t}\}\) and its \(\tau\) lag series \(\{X_{t-\tau}\}\). Taking the row autocorrelation in Figure 3 as an example, for single head \(h\) and input sequence \(F^{\prime}_{i-1}\in\mathbb{R}^{L_{i}\times d}\) where \(L_{i}=\frac{L_{i-1}}{p_{i}}\), after the linear projector, we get query \(Q_{h}\), key \(K_{h}\), and value \(V_{h}\). At this point, we consider \(Q_{h}\) as sequences along the row direction, i.e.,\(Q_{h}=[row_{1},...,row_{d_{k}}]\) where \(d_{k}\) is the embedding dimension of \(K_{h}\), with \(K_{h}\) treated in the same manner. Then, we can obtain the row autocorrelation \(\hat{R}_{Row}(Q_{h},K_{h})\) using the following Equation (2): \[\hat{R}_{Row}(Q_{h},K_{h})=[R_{Q_{h},K_{h}}(1),...,R_{Q_{h},K_{h}}(d_{k})],\] (2) where \(R_{Q_{h},K_{h}}\) is the autocorrelation between \(Q_{h}\) and \(K_{h}\). The calculation of the column autocorrelation is similar to that of row autocorrelation, with the only difference being that \(Q_{h}\) and \(K_{h}\) are treated as sequences along the column direction. The formula for calculating column autocorrelation is provided in Equation (3). \[\hat{R}_{Column}(Q_{h},K_{h})=[R_{Q_{h}^{T},K_{h}^{T}}(1),...,R_{Q_{h}^{T},K_{ h}^{T}}(L_{i})], \tag{3}\]  
Figure 2: The overall architecture of the proposed Periodicformer encoder. where \(R_{Q_{h}^{T},K_{h}^{T}}\) is the autocorrelation between the transpose of \(Q_{h}\) and the transpose of \(K_{h}\);  
* Criss-cross periodic attention: To comprehensively capture the different periodic features extracted by row and column autocorrelation, we use the scaled dot product attention, with the product of row and column autocorrelation serving as the new attention weights. For a single head \(h\), the criss-cross periodic attention (CCPA) mechanism is as shown in the following Equation (4): \[\text{CCPA}(Q_{h},K_{h},V_{h})=softmax(\frac{\hat{R}_{Row}(Q_{h},K_{h})\hat{R }_{Column}^{T}(Q_{h},K_{h})}{\sqrt{d_{k}}})V_{h},\] (4) For the multi-head version, with \(H\) heads, the process is as shown in Equation (5): \[\text{Multi-head}(Q,K,V) =concat(head_{1},...,head_{H})W_{o},\] (5) where \(head_{i}\) = Criss-cross periodic attention(\(Q_{i},K_{i},V_{i}\)),
* Efficient computation: The Wigner-Khinchin theorem establishes that the Fourier transform of the autocorrelation function of a signal equals its power spectral density. For the autocorrelation computation (Equation (1)), the given input sequence \(X_{t}\), \(R_{XX}(\tau)\) can be calculated by the discrete Fourier transforms (DFT) as Equation (6): \[\mathcal{F}(X_{t}) =\sum_{t=1}^{L}X_{t}e^{-2\pi i\frac{\mathrm{tr}}{\mathrm{tr}}},\] (6) \[S_{XX}(k) =\mathcal{F}(X_{t})\mathcal{F}^{*}(X_{t}),\] \[R_{XX}(\tau) =\mathcal{F}^{-1}(S_{XX}(k))=\frac{1}{L}\sum_{k=1}^{L}S_{XX}(k)e^ {2\pi i\frac{\mathrm{tr}}{L}},\]  
Figure 3: Criss-cross periodic attention (**left**) and the corresponding row and column autocorrelation for a single head \(h\) (**right**). where \(\tau\in\{1,...,L\}\), \(\mathcal{F}\) denotes DFT and \(\mathcal{F}^{-1}\) is its inverse transformation. \(*\) denotes the conjugate operation, and \(S_{XX}(k)\) is the power spectral density. In practice, we use fast Fourier transforms (FFT) to reduce the time complexity from \(O(L^{2})\) to \(O(L\log L)\). The FFT algorithm process can be seen in Algorithm 1.