# TMCrack-Net: A U-Shaped Network with a Feature Pyramid and Transformer for Mural Crack Segmentation
## 3 Proposed Methods
### AG-BiFPN Module  
UCTransNet demonstrates that not all jump-joins are beneficial for segmentation, and simple replication of features at the encoder stage can be detrimental to feature fusion, so more suitable feature fusion methods are needed to join the encoder and decoder.  
Figure 4: Feature network design. The * stands for multiplication (**a**) ConvNeXt block’s structure; (**b**) downsample structure, where k represents the convolution kernel size, s represents the step size, and p represents the padding.  
Figure 3: Overall model framework of TMCrack-Net.  
In order to better utilize the features extracted from the backbone network, we invoke a combination of BiFPN and CCT modules instead of the original hop-join in the U-net. By fusing feature information at different scales through the BiFPN network, the input image retains a richer level of detail contained in the shallow features, better utilizes local information, and focuses on extracting crack details.  
BiFPN was proposed by Google Research, based on the bi-directional scale join. It removes the nodes with only one input edge, which do not perform feature fusion and contribute little to the network, and add the residual structure to join the original junction with the output node, which can fuse more features. The BiFPN infrastructure is shown in Figure 2d.  
The single-use feature fusion network with only a convolutional structure is biased toward local interaction and brings some confounding effect, which interferes with recognizing cracks. By adding an attention mechanism after BiFPN, as shown in Figure 5, the CCT module can better emphasize the global information and alleviate the confounding effect at each layer of the pyramid.  
CCT computes the weights of different feature layers through a multi-headed self-attentive mechanism and sums the weights of each layer with the corresponding feature layer to establish global relationships and suppress invalid features. The implementation of CCT can be expressed as follows:  
1. Five features \(\text{E}_{i\_\text{out}}\,,(i=1,2,3,4,5)\) are obtained by feature fusion. Firstly, the features are divided into a series of patches, and the number of feature channels is kept constant. The patches are flattened and given location information to obtain tokens of different scales as \(T_{i}(1,2,3,4,5)\), and the four tokens are combined to obtain \(T_{\text{C}}=Concat(T_{1},T_{2},T_{3}T_{4},T_{5})\).
2. Tokens are fed into the multi-headed self-attentive mechanism, with five \(T_{i}\) as the query and \(T_{\text{C}}\) as the key and value: \[Q_{\text{i}}=T_{i}M_{Q_{i}},K=T_{c}M_{K},V=T_{c}M_{V}\] (1) where \(M_{Q}\), \(M_{K}\), and \(M_{V}\) are the weights of the different input features, and the weights \(W_{vi}\  
where \(\sigma\) represents the instance normalization [21], \(C_{\Sigma}=\sum_{i=1}^{N}C_{i}\), \((i=1,2,3,4,5)\) denotes the sum of the number of channels of the input features. In our implementation, \(C_{1}=96\), \(C_{2}=192\), \(C_{3}=384\), \(C_{4}=768\), and \(C_{5}=768\), values weighted by \(W_{vi}\):  
\[At_{i}=W_{vi}V^{T} \tag{3}\]  
In the case of N head attention, the output after multi-head cross-attention is calculated as follows:  
\[MAt_{i}=\left(At_{i}^{1}+At_{i}^{2}+At_{i}^{3}+\ldots,At_{i}^{N}\right)/N \tag{4}\]  
where \(N\) is the number of heads. A series of experiments on layers 1, 2, 3, 4, and 5 of CCT in the implementation found that the best performance can be achieved on the tomb mural dataset with one layer and six heads of CCT. The following output is obtained by using MLP and residual join:  
\[O_{i}=MAt_{i}+MLP(Q_{i}+MAt_{i}) \tag{5}\]  
Finally, the five outputs \(O_{1}\), \(O_{2}\), \(O_{3}\), \(O_{4}\), \(O_{5}\) are reconstructed by up-sampling through convolution, and the corresponding decoder features are concatenated.