# HTN: Hybrid Transformer Network for Curvature of Cervical Spine Estimation
## 2 Materials and Methods
### Integration of Transformer  
Given the input image \(X\in R^{H_{0}*W_{0}*3}\) with a spatial resolution of \(H_{0}*W_{0}\) and three-color channels, our designed down-sampling path generates the high-level feature representation \(X\in R^{H*W*C}\). Here we use \(C=512\) and \(H,W=H_{0}/32,W_{0}/32\). The feature representation \(F\) is fed to the main decoding path and intermediate branch. Before decoding, a transformer is integrated. To perform tokenization, we uniformly slip the feature representation F into a sequence of flattened 2D patches \(x_{F}\Big{\{}x_{F}^{i}\in R^{N*(P^{2}*C)}\Big{|}i=1,\ldots,N\Big{\}}\), where the size of each patch is P\({}^{2}\) and N = H * W/P\({}^{2}\) is the resulting number of patches. The transformer expects a 1D sequence of token embeddings as input; hence, we flatten the 2D patches into 1D and map it into a D-dimensional embedding space using a trainable linear projection. Moreover, we add a learnable 1D position embedding to the patch embeddings to preserve the positional information. The obtained embedding vector sequence is utilized as the input of the encoder, as shown in Equation (1):  
\[Y_{0}=\Big{[}x_{F}^{1}E;\ x_{F}^{2}E;\ \ldots;x_{F}^{N}E;\Big{]}+E_{pos} \tag{1}\]  
where \(E\in R^{P^{2}*C*D}\) is the linear projection matrix, \(E_{pos}\in R^{N*D}\) is the position embedding, and \(Y_{0}\) is the final feature embedding. The output of this projection is regarded as the patch embedding. In general, a transformer includes an encoder and a decoder. In its encoding path, there are \(L\) alternating layers, each of which consists of multi-head self-attention (_MHSA_) and multilayer perceptron (_MLP_) blocks. A residual connection is added to the input and output of each module, and layer normalization is applied before every block. In addition, each _MLP_ block contains two layers with GELU nonlinearity. Therefore, the output of the \(l\)-th layer can be expressed as Equations (2) and (3):  
\[\hat{Y}_{l}=MHSA[LN(Y_{l-1})]+Y_{l-1} \tag{2}\]  
\[Y_{l}=MLP\big{[}LN(\hat{Y}_{l})\big{]}+\hat{Y}_{l} \tag{3}\]  
where \(LN(*)\) refers to the layer normalization operator, the output of the \(l\)-th layer is \(Y_{l}\), and \(l\in[1,2,\ldots,L]\). In our experiments, \(L=4\). The multi-head self-attention block is the most important part in the transformer. Multiple self-attention modules are processed, connected, and projected in parallel. The inputs \(x_{F}\) are transformed into three parts: the \(d_{k}\)-dimension queries \(Q\), \(d_{k}\)-dimension keys  
Figure 2: The overall framework of our proposed method. Our model is mainly composed of an encoder–decoder structure and a prediction module. The dimensions of residual modules with the same ID are the same. A transformer is integrated into the middle of the encoder and decoder. \(K\), and \(d_{v}\)-dimension values \(V\). All of them have the same sequence length as the inputs \(x_{F}\). The scaled dot-product attention is applied as follows:  
\[Attn(Q,K,V)=softmax\Big{(}\frac{QK^{T}}{\sqrt{d_{k}}}\Big{)}*V \tag{4}\]  
The decoder of the transformer also consists of \(L\) identical layers, but it inserts an additional multi-head layer that pays more attention to the output of the encoder stack. A residual connection is added to its input and output, similar to the operation outside each module, and layer normalization is applied. To predict landmarks of the spine, we use a 2D CNN to restore the spatial order and to map the serialized data back to the feature map. Specifically, the \(D\)-dimensional output sequence of transformer \(Y_{L}\in R^{N*L}\) is projected back to the \(C\)-dimension through a linear projector. We obtain the feature map \(Y\in R^{H*W*C}\) with the same size of the feature representation \(F\). After feature mapping, we stack the up-sampling layer and convolutional layer to decode the hidden feature for landmark prediction. Although they have similar structures, the two branches have their own functions.