# CommuSpotter: Scene Text Spotting with Multi-Task Communication
## 3 Text Spotter with Communication
### Text Recognition Communication  
As mentioned above, a segmentation-based text recognition module adopts text or character instance segmentation. Similar to the text detection module, the shared features are first extended by the reverse Extensive Representation (ER) to embed global information, as shown in Figure 3b. Specifically, the filtered pyramid features, \(P\), consisting of multiple levels, from \(P_{2}\) to \(P_{5}\), are presented for the module as follows:  
\[P^{**}_{i+1}=Gcm(Dr(P_{i})\cdot P_{i+1}),i=2,3,4, \tag{5}\]  
where \(Dr(\cdot)\) is the downsampling operation.  
The instance segmentation-based recognition methods always suffer from the problem of missing characters in texts, especially when some characters inside a text are artistically designed with different textures, shapes, fonts, etc. From this view, the purpose of the Conversation Mechanism (CM) is to integrate (1) the renewed expertise from text detection module \(F^{\prime\prime}\) as global guidance and (2) text semantic segmentation expertise \(S^{\prime}\), which provides local details. Specifically, the extensive features, \(P^{**}\), and filtered proposals, \(R^{\prime}\), from RPN are fed into the RoIAlign with text semantic segmentation expertise \(S^{\prime}\), to generate the aligned RoI feature, \(F^{\prime}_{RoI}\), of the shape \(N^{\prime}\times C\times H^{\prime}_{RoI}\times W^{\prime}_{RoI}\), where \(N^{\prime}\) is the number of instances, and \(H^{\prime}_{RoI}\) and \(W^{\prime}_{RoI}\) are the resolutions of 16 and 64, respectively.  
\[F^{\prime}_{RoI}=RoIAlign(P^{**},R^{\prime})+S^{\prime}, \tag{6}\] \[F^{\prime\prime}=Lr(Lr(F^{\prime}_{RoI})),\] (7) \[F^{**}_{RoI}=F^{\prime\prime}+F^{\prime}_{RoI}. \tag{8}\]  
Then, the aligned RoI feature, \(F^{\prime}_{RoI}\), is transferred by two consequent linear convolutional layers \(Lr(\cdot)\). Finally, we concatenate text detection expertise \(F^{\prime\prime}\) with the RoI features \(F^{\prime}_{RoI}\).  
Another problem of segmentation-based recognition is that character details are ignored in text spotters. The pixel-level expertise of characters can provide information to differentiate characters of similar shapes. For example, \"hot\" and \"hat\" only differ in one character's details. Due to the lack of pixel-level annotations for scene text datasets, text semantic segmentation is not applied in existing scene text spotters. Instead of designing complex modules to tighten the character regions with extra annotations, we introduce text semantic segmentation [35] expertise \(S\) and expertise \(S^{\prime}\) with individually pre-trained  
Figure 3: The communication networks of (**a**) the text detection module and (**b**) text recognition module.  
weights. The initial semantic segmentation results, \(S_{e}\), are refined by the attention mechanism [36] to obtain a better text semantic segmentation \(S_{re}\), as shown in Figure 2.  
\[S_{f}=ResNet(I), \tag{9}\] \[S_{e}=Softmax(S_{f}),\] (10) \[S_{att}=Att(S_{e},S_{f}),\] (11) \[S_{re}=Convolution(S_{e},S_{att}), \tag{12}\]  
where \(S_{f}\) is the fusion of pyramid features \(P_{2}\) to \(P_{3}\), \(Softmax\) is the softmax operation, \(Att\) is an attention layer of the dot-product operation, and \(Convolution\) is a series of convolution layers of kernel size 5 \(\times\) 5 and 1 \(\times\) 1, to fuse the attention with the feature maps. To communicate this expertise in the text spotter, we take pooling operations on text semantic segmentation \(S_{re}\) with different resolutions. Then, text semantic segmentation expertise \(S\) and expertise \(S^{\prime}\) are generated.  
\[S=\mathcal{P}(S_{re}), \tag{13}\] \[S^{\prime}=\mathcal{P}^{\prime}(S_{re}), \tag{14}\]  
where \(\mathcal{P}\) and \(\mathcal{P}^{\prime}\) are the pooling layers with different resolutions. The shape of expertise \(S\) is \(N\times C\times H_{RoI}\times W_{RoI}\), and \(S^{\prime}\) is \(N^{\prime}\times C\times H^{\prime}_{RoI}\times W^{\prime}_{RoI}\).