# CQFormer: Cyclic Quasi-Dynamic Transformers for Hourly Origin-Destination Estimation
## 4 Methodology
Since different \(r_{k}\) contains not exactly the same sequence of states \([A_{i}]\), different distributions of \(r_{k}\) lead to various spatiotemporal distributions of traffic flows. Under the quasi-dynamic assumption, this is reflected in the generation of various \(F_{T}\). Meanwhile, the spatiotemporal trajectories \(r_{k}\) connect different origins and destinations, and the distribution of \(r_{k}\) is also associated with the distribution of traffic demand simultaneously. Self-attention mechanism allows models to establish connections between multiple positions within a sequence of data and capture their contextual relationships, and is applied here to extract the association weights between the spatial feature vectors in \(n_{t}\) subperiods, as a way to implicitly represent the probability distribution of \(r_{k}\), and thus map different spatiotemporal flow distributions to different types of traffic demand structures. Because the \(n_{t}\) feature vectors do not explicitly contain sequential information, the position embedding is operated first as in [49]. \[Attention(Q,K,V)=softmax\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\bigg{)}V \tag{9}\]  
Figure 1: **Architecture of CQDPormer. ** CQDPormer adopts a bidirectional encoder–decoder architecture. It uses self-attention and cyclic attention mechanisms to capture the diverse and nonlinear features of traffic demand and assignment. Feature vectors from both forward and backward networks are fed into the cyclic attention module. \[FFN(x)=ReLU(xW_{1}+b_{1})W_{2}+b_{2} \tag{10}\]  
\[AN(x)=LayerNorm(x+Sublayer(x)) \tag{11}\]  
Equation (9) is the core formula of self-attention, where \(Q,K,V\) are query, key, and value matrices, which are generated from the embedded feature vectors through three different linear transformations. The scaled dot product of \(Q\) and \(K\) scores the correlation degree between feature vectors in \(n_{t}\) subperiods, where the scaling factor \(1/\sqrt{d_{k}}\) prevents the influence of vector dimensions. The activation function \(softmax\) normalizes the dot product of \(Q\) and \(K\) to indicate the importance weights of value vectors in \(V\). In multi-head attention, each attention head performs attention operation upon a corresponding group of \(Q,K,V\) to capture multiple association patterns. The Feedforward Layer deploys two fully connected layers to enhance the nonlinearity of the attention outcome as expressed in Equation (10). The Add&Norm establishes residual connections between the front and back of each sublayer (i.e., Attention layer, Feed Forward layer) and avails layer normalization to improve the performance as in Equation (11). Encoded by the self-attention module, the feature vectors are fed into two separate decoders. One of them extracts the distributional features of the OD matrices directly from the training data and produces a tensor \(\hat{D}_{T}^{\prime}\) sharing the same dimension with the OD matrix through two fully connected layers. The other produces the weight vector \(W_{D}\) of candidate OD matrices including the OD matrix from the last one decoder \(\hat{D}_{T}^{\prime}\) and \(n_{d}\) prior OD matrices \(D_{1},D_{2},\cdots,D_{n_{d}}\), then the final estimation matrix is expressed as  
\[F_{T}[\alpha_{i},\beta_{i}]=N(A_{i})=\sum_{r_{k}}N(r_{k})\delta_{A_{i}\in r_{ k}} \tag{5}\]0  
Since \(\hat{D}_{T}^{\prime}\) generated mainly from the training of pairwise simulation data is susceptible to distributional shift, the weight tensor \(W_{D}\) receives the gradient backpropagation from the loss of cyclic consistency training with the realistic data and adaptively adjusts the weights to capture the structure of the OD matrix in the real environment. The inputs of the backward network are a batch of flattened OD matrices. The feature encoder first maps the OD matrix to a \(n_{t}\cdot n_{f}\)-dimensional feature space, where \(n_{f}\) is the feature space dimension of the forward process and \(n_{t}\) is the number of sub-periods.