# Twin-FPN++: Rwin Transformer with Feature Pyramid Network for Dense Scene Text Spotting
## 3 Proposed Method
### Text Detection
#### 3.2.1 Rwin Transformer  
In recent years, Transformer has been adapted for computer vision and achieved state-of-the-art performances in image classification [12, 13] and object detection [13]. Inspired by the success of Swin Transformer [13] on object detection tasks, we propose a method called Rwin-FPN++, which incorporates the long-range dependency merit of Rwin Transformer into the feature pyramid networks (FPN) to effectively enhance the functionality and generalization of FPN. Swin Transformer proposes a self-attention mechanism based on the shifted windows. First, the self attention of the window is calculated, and then the self attention after the shifted window is carried out. In this way, communication between windows is realized, thus achieving the effect of global modeling and better translation invariance. For the task of scene text detection, because there are a large number of rotated and distorted texts, we modified the Swin Transformer by adding a rotating window self-attention mechanism. Thus our Rwin Transformer can enhance the attention to rotated and distorted scene text.  
The main idea of the Rwin Transformer is to compute the multi-head self-attention (MSA) on regular and rotated window partitions consecutively. Following the design of [13], the Rwin Transformer block is composed of the windows-based MSA (W-MSA) module and the rotated windows-based MSA (RW-MSA) module. Both W-MSA and RW-MSA are composed of four layers of windows-based MSA and two layers of Multi-Layer Perceptron (MLP) with GELU operation. Layernorm (LN) is calculated between each MSA module and each residual connection. The Rwin Transformer block can be computed as follows:  
\[\hat{z_{1}}=W\_MSA(LN(z_{0}))+z_{0} \tag{1}\]  
\[z_{1}=MLP(LN(\hat{z_{1}}))+\hat{z_{1}} \tag{2}\]  
\[\hat{z_{2}}=RW\_MSA(LN(z_{1}))+z_{1} \tag{3}\]  
\[z_{2}=MLP(LN(\hat{z_{2}}))+\hat{z_{2}} \tag{4}\]  
where \(z_{0}\), \(\hat{z_{1}}\), \(z_{1}\), \(\hat{z_{2}}\) and \(z_{2}\) represent the output of the encoder, W_MSA, MLP, RW_MSA, and MLP, respectively.  
The MSA block is computed as follows:  
\[MultiHead(Q,K,V)=Concat(head_{1},\dots,head_{n}) \tag{5}\]  
\[head_{i}=SoftMax(\frac{QK^{T}}{\sqrt{d}}+B)V \tag{6}\]  
where \(K\), \(Q\), and \(V\) represent the key, query, and value matrices; \(d\) is the dimension of query and key, and \(B\) is the relative position bias; Concat denotes the concatenation operation.  
Figure 3: The overall pipeline of detection. (**a**) is the input image with dense scene text. (**b**) At each layer of the feature pyramid network, the Rwin Transformer is adopted to model the rotational invariance of the self-attention mechanism. (**c**) The four layers of the feature pyramid network (FPN) are combined into a new Module (FFPM). (**d**) The module FFPM produces the text score matrix and keypoints matrix. (**e**) We merge the values in the score matrix and keypoints matrix and obtain the final segmentation result.  
Following [13], we partition each output layer produced by Resnet into multiple windows of size 4 \(\times\) 4. The windows are then reshaped into a sequence of 1D patches, which are treated as the token of the Transformer. These embedded patches are then mapped into three matrices (\(Q\), \(K\), \(V\) in Equation (6)) through a linear operation. Then one attention module can be computed using Equation (6), and the multi-head self-attention (MSA) block can be obtained using Equation (5). The residual output \(\hat{z_{2}}\) can thus be calculated using Equation (3). Finally, the global self-attention output \(z_{2}\) is obtained using Equation (4).