# STDecoder-CD: How to Decode the Hierarchical Transformer in Change Detection Tasks
## 2 Materials and Methods
### Design of the Decoder Module
#### 2.4.3 Type III STDecoder-CD  
Intuitively, it is also better to use the transformer decoder to decode the proposed backbone, which is a combined Siamese transformer-CNN architecture.  
As displayed in Figure 4a, we first use the \"upsampling + concatenation + convolution\" operation to fuse the four difference detectors (Difference Detector II). The output of this step is a context-rich feature representation (denoted by \(M\)) with a dimension of 240 and with the same size as the first difference detector--\(1/4\) of the original image size. Then, \(M\) is split into \(L\) visual words, which contain compact high-level semantic information that effectively reveals the change of interest, and each corresponds to one token vector of 240 elements. The relevant calculation process is shown in Figure 4c. The input of the transformer decoder is \(M\), the semantic tokens derived from it are denoted by \(T\), and its output is \(M^{\prime}\), which is a representation with the same size and dimension as \(M\). Apparently, the purpose of introducing the transformer decoder is to further refine the image features from \(M\) to \(M^{\prime}\). Finally, \(M^{\prime}\) is sent to the classifier FCN to generate the predicted change map(s).  
Traditionally, the transformer encoder and decoder usually come in pairs [27]. However, we only use the transformer decoder here. Actually, the TransCNN module itself can be considered a hierarchical transformer encoder, so it is unnecessary to add an extra one. As displayed in Figure 4b, the decoder in use consists of \(N_{D}\) layers of multi-head cross attention (MHCA) and MLP blocks [19] instead of MHSA.  
Formally, at each layer \(l\), MHCA is defined as:  
\[MHCA\Big{(}D^{l-1},\ T\Big{)}=Concat(head_{1},\ \ldots,head_{h})W^{O} \tag{5}\]  
and  
\[head_{j}=Att(D^{l-1}W_{j}^{q},TW_{j}^{k},TW_{j}^{v}) \tag{6}\]  
where  
\[Att(Q,\ K,V)=\sigma(\frac{QK^{T}}{\sqrt{d}})V \tag{7}\]In the above equations, \(W_{j}^{d},\ W_{j}^{k},\ W_{j}^{v}\in\mathbb{R}^{C\times d}\) (C is the channel dimension of the feature map \(M\)) are the learnable parameters of three linear projection layers generating Q, K, and V, and \(d\) is the channel dimension of the triplet. \(W^{O}\in\mathbb{R}^{hd\times C}\) is a linear projection matrix, and \(h\) is the number of attention heads. \(\sigma(\cdot)\) denotes the softmax function running on the channel dimension.  
Formally, at each layer \(l\), MLP is defined as:  
\[MLP\Big{(}T^{l-1}\Big{)}=GELU\Big{(}T^{l-1}W_{1}\Big{)}W_{2} \tag{8}\]  
where \(W_{1}\in R^{C\times 2C}\) and \(W_{2}\in R^{2C\times C}\) are linear projection matrices, and GELU is the activation function.  
In MHSA, the query, key, and value are derived from the same token embeddings, and its output is also a 1D sequence of tokens. In order to exploit the relation between each pixel in \(M\) and the token set \(T\) and generate the refined 2D feature \(M\)', here, we consider the MHCA-based decoder. It treats pixels in \(M\) as queries and tokens in \(T\) as keys and values. In [19], before the decoding operation, the token embeddings derived from the feature map generated by FCN are first sent to a transformer encoder to obtain compact high-level semantic tokens, but here, we assume that the token embeddings (\(T\)) derived from the global context-rich \(M\) are directly available to MHCA.  
Figure 4: Flowchart of the Type III STDecoder-CD architecture. Note: (**b,c**) are modified after Ref. [19]. The legends in this figure are consistent with those in Figures 1–3 if not specified.  
Note that in Figure 4b, \(N_{D}\) is set to 1, and the number of heads of MHCA = 8. In Figure 4c, the token length \(L\) = 16, and the token dimension \(C\) = 240. Other hyperparameter settings are consistent with those in [19].